# HG changeset patch # User Mychaela Falconia # Date 1703044206 0 # Node ID 9e364c18e0e8c97bdc96151618174fe5b1a718f0 beginning of architectural design spec diff -r 000000000000 -r 9e364c18e0e8 doc/Arch-design --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Arch-design Wed Dec 20 03:50:06 2023 +0000 @@ -0,0 +1,334 @@ +Themyscira Wireless SMSC implementation +Architectural design specification + +1. Purpose and scope of the software + +The purpose of the present software project is to facilitate store-and-forward +SMS exchange among the following parties: + +* Locally owned mobile telephone numbers (LOMTNs) that belong to Themyscira + Wireless, with Short Message Service accessed either via the local GSM network + (Osmocom-based) or via direct command line access to the SMSC; + +* The outside world: the total set of all SMS-capable E.164 telephone numbers + in the world, with whom our users must be able to freely exchange SMS just + like users of any other cellular phone carrier in USA; + +* USA-specific 5-digit and 6-digit short codes: these services aren't accessible + from anywhere in the world, only from USA (each country has its own services + of this type), but because we are located in USA, we must provide the same + access to public services as any other cellular phone carrier; + +* Any downstream parties who enter into an interconnection agreement with ThemWi + for the purpose of sharing our SMS uplink to the outside world. + +1.1. NANP specifics + +The design of our SMSC makes the following assumptions that are specific to +North American Numbering Plan: + +* All LOMTNs and all downstream peer MTNs are expected to be NANP numbers; + any/all SMS source or destination numbers in country codes other than +1 are + treated as belonging in the Outside World, accessible only via the SMPP + "uplink" connection to our upstream SMS connectivity provider. + +* The set of SMS destination numbers that can be sent to the upstream includes + not only non-NANP and not-locally-known NANP E.164 numbers, but also any/all + SMS short codes in USA-specific NXXXX or NXXXXX format. + +* In the case of Mobile-Originated SMS from the local GSM network, if the + user-entered destination number is not explicitly international (TON=1) and + does not fit the format of a USA SMS short code, other USA-customary dialing + formats are supported, as in 10-digit NPANXXXXXX or 11-digit 1NPANXXXXXX + without '+' prefix. + +themwi-nanp software package is a strict dependency for themwi-smsc: themwi-nanp +utilities must be used to manage the database of locally owned NANP numbers, +and the present software uses themwi-nanp libraries to access that database. + +1.2. Hierarchical arrangement of upstream and downstream peers + +The telecom landscape in USA is such that anyone can obtain 10-digit telephone +numbers (TNs) very easily and very cheaply, but making them SMS-capable (able +to function as Mobile Telephone Numbers or MTNs) is much more difficult. +Suitably equipped providers such as Bandwidth.com are generally unwilling to +provide service directly to small customers, and we (Themyscira Wireless team) +were able to find only one company (Sopranica Telecom) who buys P2P SMS +interconnection service from Bandwidth and was willing to resell to us. + +Suppose that many different ultra-small parties wish to set up their own indie +GSM networks in different parts of USA. Each of these tiny fiefdoms can serve +as its own administration and get its own TNs from a provider such as BulkVS. +How would all of these tiny fiefdoms then add SMS capability? The feedback we +got from Sopranica is that asking them to set up a sub-account on their +Bandwidth service for each microfiefdom would be too much work - hence San Diego +2G Association (the primary instance of Themyscira Wireless) will need to serve +as a third-level reseller, getting Bandwidth SMS interconnection service from +Sopranica and then further subletting it to other microfiefdoms. + +Vertical hierarchy support in ThemWi-SMSC is designed to support the just- +described use case. Each SMSC instance has a set of locally owned mobile TNs +(LOMTNs, owned by the local fiefdom operating this SMSC instance), a single +upstream SMPP link pointing up the hierarchy tree (toward the Outside World) +and any number of downstream SMPP links to downstream peers. The total set of +phone numbers known to each SMSC instance is its own local set (themwi-nanp +database of locally owned TNs) plus the set of numbers assigned to downstream +peers - all other E.164 numbers everywhere in the world (plus all non-E.164 USA +SMS short codes) belong in the Outside World and are sent to the "uplink" +connection. Messages are then routed as follows: + +* Any SM originating from a local GSM subscriber can go to another GSM + subscriber, to a known downstream peer or to the Outside World. + +* Any SM that are injected directly into the SMSC from local shell access are + treated the same way as Mobile-Originated SMS from local GSM users - hence + this mechanism can be used to send SMS to the local GSM network or to the + Outside World. + +* Any SM coming from the uplink connection can be addressing a local GSM + subscriber or a downstream peer - but either way it must be a number known + to this SMSC, otherwise something is badly misconfigured somewhere. + +* Any SM coming from a downlink connection can go to a local GSM subscriber, to + a different downstream peer or to the Outside World. + +1.2.1. Direction of SMPP connections + +Despite the name "Short Message Peer to Peer", SMPP is an asymmetric client- +server protocol, not symmetric peer-to-peer. Our primary, above-all-else +requirement when it comes to SMPP is to connect to the "big daddy" SMSC of +Bandwidth.com, the one that allows us to receive SMS from and send SMS to +anywhere in the Outside World. BW requires that we connect to their SMSC server +in the role of an SMPP client and bind as a bidirectional transceiver - both +message directions then flow over this single long-lived TCP connection from our +client to their server. + +This externally imposed requirement dictates the entire architectural design of +ThemWi-SMSC with respect to SMPP. Each instance of ThemWi-SMSC can have a +single upstream peer to whom we connect in the role of an SMPP client, and it +can optionally act as an SMPP server accepting TCP connections from downstream +peers. The master instance of ThemWi-SMSC at smsc.sandiego2g.org will point +its "upstream" link at Bandwidth.com SMPP server, using credentials given to us +by Sopranica, whereas other small fiefdoms who wish to join our service resale +tree will point the "upstream" link of their ThemWi-SMSC instances to +smsc.sandiego2g.org, and we (SD2G) will assign them authentication credentials +and manage their downstream number pools. + +1.3. Possible use outside of originally intended North American use case + +If your situation and/or interests do not match the very specific use case for +which the present software is designed (if you are located outside of North +America, and/or you have no interest in attaining SMS interconnection with the +national mobile telephony environment of whichever country you call home), you +can still play with the present implementation of GSM-oriented SMSC: the uplink +connection to the Outside World can be omitted, and if you don't have real TNs +(telephone numbers) in North American Numbering Plan (either because you are +outside of North America or because you are in NA but not interested in official +phone network interconnection), you can operate ThemWi-SMSC (plus the attached +Osmocom GSM network) with fake NANP numbers instead. + +To be clear, this support for modes of usage outside of the primary design goals +of ThemWi-SMSC is intended only to facilitate "play" and evaluation (getting a +feel for what may be the first SMSC implementation connecting to Osmocom CNI +via GSUP), not for serious long-term usage. If your actual desired use case is +an isolated GSM network with a totally ad hoc or "free" numbering plan (the +default which one gets with a "vanilla" installation of Osmocom CNI), or a GSM +network that is interconnected with the national mobile telephony environment +of some country other than USA, you need a different SMSC design that is +tailored for your numbering plan (free-form or non-USA national) that will be +different from NANP, and for local telecom environment quirks that will almost +certainly be different from those in USA. + +If you like the general idea and overall design of ThemWi-SMSC, but require an +adaptation to a different numbering plan or a different telecom environment +(isolated or a national interconnect in some other country), you should be able +to take the present code base and modify just the numbering plan aspects, +producing a derivative-work SMSC for your different needs. + +2. ThemWi-SMSC software architecture + +2.1. Modularity of components + +A complete deployment of ThemWi-SMSC, as in our own use case at Themyscira +Wireless, includes a local GSM network (Osmocom-based) and a connection to the +hierarchical SMPP tree that eventually leads to the Outside World SMS +connectivity provider at the top. However, our software implementation will be +modular, divided into separate software components for: + +* The internal core of the SMSC (one daemon process and some command line + utilities); + +* A pair of daemon processes devoted to the task of connecting the SMSC to the + local Osmocom-based GSM network, to be omitted if you don't have one; + +* A dedicated daemon process serving the SMPP link to the upstream peer, to be + omitted if you have no upstream link; + +* Another dedicated sw component serving downstream peer SMPP connections, one + process instance per downstream peer, or none if you have no such peers. + +This modularity allows the software to be used and (hopefully) appreciated +outside of its primary intended use case. At one extreme, someone could have +an isolated Osmocom GSM network, modify it slightly to use MSISDNs that look +like (fake) NANP numbers, hook up ThemWi-SMSC and use this SMSC as a replacement +for the Osmocom-default one, paving the way for factoring the SMSC function out +of OsmoMSC. At the other extreme, if someone is located in USA and wishes to +interconnect to the world of SMS through the chain of 3 resellers (Bandwidth +followed by Sopranica followed by San Diego 2G Association), they can run an +instance of ThemWi-SMSC without any GSM network at all. (You will still need +Osmocom libraries, but no Osmocom processes and no hardware.) In such a +deployment, all incoming SMS to your number(s) will be written into the +persistent store which you can read, and you can send outgoing SMS with a +command line utility. + +2.2. Persistent message store + +Every SM that passes through ThemWi-SMSC gets written into an append-only +persistent message store (PMS). Because this store is append-only, no messages +are ever deleted - however, each message in PMS can be in one of two states: +active or historical. An active SM is one for which the SMSC still needs to +make delivery attempts, either attempts at GSM MT delivery or attempts at +delivery to the appropriate upstream or downstream SMPP peer. A historical SM +is one for which no further action will be taken by any component of our SMSC. +An SM can enter "historical" state in several ways: + +* For some LOMTNs the act of writing incoming messages into PMS constitutes + final delivery in itself, and no other delivery actions are needed. In this + case a newly entered SM is directly written into PMS in the "historical" + state, without ever going through "active". + +* For messages that need to be delivered to a GSM MS or to an SMPP peer, once + that delivery has been made successfully, the message transitions from active + to historical. + +* In the case of failed deliveries (permament error, or expiration time reached + after repeated temporary failures), the failed message also transitions from + active to historical. + +The persistent message store is a simple binary file (/var/sms/pms.bin) +consisting of directly abutted 'struct sm_record' records. Each message record +is exactly 256 bytes (see struct definition - we were able to fit everything we +needed under the 256 byte mark, and then padded the struct to perfect round +size), and this perfect power-of-2 record size makes it very easy to perform +operations such as binary search via mmap or stripping initial megabytes of +historical records - see subsequent sections for more detailed description. + +PMS is append-only as already stated, but already-written records do not become +fully immutable until they become historical. For as long as a given SM is in +the active state, themwi-smsc-core daemon can and will update that record in +pms.bin: + +* For messages addressed to local GSM subscribers, dest_imsi will be filled + when the MSISDN-to-IMSI lookup operation on the destination number succeeds; + +* Upon discharge (successful delivery, permanent error or validity period + expiration after temporary failures), themwi-smsc-core will transition the + sm_record into historical state by filling disposition and time_disch struct + members; + +* Additional info may be written into dest_extra_info upon discharge, depending + on the destination type and thus the mode of final delivery. + +Once an sm_record transitions into historical state, it is then immutable for +archival purposes; archives of historical messages can be kept for years or even +decades, depending on local administration policy. + +2.2.1. Historical megabyte count + +Given the simple binary structure of the main PMS file, each megabyte (2**20 +bytes) holds exactly 4096 messages. It is envisioned that as a busy SMSC runs +for a long time, a significant number of historical messages will accumulate, +and the content of PMS may become many megabytes of historical messages followed +by some active SMs at the end. When themwi-smsc-core daemon restarts, it has +to read the entire PMS in order to collect all still-active SMs. Having to +read through many megabytes of historical SMs to get to active ones at the end +becomes unacceptable at large archive sizes, hence a mechanism is needed for +marking where the historical-only portion ends and the possibly-active portion +begins. + +There will be an auxiliary file named historical-mb, containing a single ASCII +line giving the number of historical megabytes in pms.bin. If this file reads +1, the first 4096 SM records are historical, if the auxiliary file reads 2, the +first 8192 SM records are historical, and so forth. This auxiliary file will be +used as follows: + +* Upon startup, themwi-smsc-core will read this historical-mb file and skip that + many initial megabytes of pms.bin; + +* At run time, themwi-smsc-core will track the index of the oldest still-active + SM in PMS. Whenever this index crosses a megabyte boundary, historical-mb + will be updated. + +2.2.2. Offline storage + +Even with the historical-mb mechanism of the previous section, the fact remains +that disk space on live servers is not infinite. If the archive of historical +messages grows so big that it needs to be removed from the SMSC server to free +up disk space, one can carry out the following procedure: + +* Temporarily stop themwi-smsc-core daemon at the level of runit or systemctl + or whatever you are using - this operation will bring down the entire SMSC, + so do it during a scheduled maintenance window; + +* Use dd to split pms.bin into historical and active portions: + + dd if=pms.bin of=pms-hist.bin bs=1048576 count=N + dd if=pms.bin of=pms-new.bin bs=1048576 skip=N + +* Move pms-hist.bin to offline storage; + +* Replace the long file with the shortened one: + + mv pms-new.bin pms.bin + echo 0 > historical-mb + +* Re-enable themwi-smsc-core and restart all other SMSC daemons. + +2.2.3. themwi-smsc-dump reading tool + +The program named themwi-smsc-dump will be a standalone command line utility +(fully static in its operation, not talking to any daemons or services) for +reading and parsing (decoding) pms.bin. It will open pms.bin with O_RDONLY, do +a read-only mmap on it, and then access this PMS as a memory-mapped file. +Several different modes of operation will be provided: + +* It will be possible to dump and decode the entire PMS, as needed during early + debugging. + +* It will be possible to specify a starting date/time at which the dump should + begin. As records are added in strict forward chronological order, it is + possible to find a record nearest (by time_entry timestamp) to a given time + point by binary search, very efficient on a memory-mapped file. + +* Once the dump has a starting point (beginning of the file or a time point + found by binary search), the tool can be told to dump till the end, display + some count of messages, or run until a certain ending date/time is crossed. + +* The tool can dump all message records in the selected range, or only those + matching specific filters such as a particular source or destination type, or + a specific phone number. + +The complexity described above is needed for the following reasons: + +* One radical idea is to grant limited access (by way of a very strict wrapper) + to themwi-smsc-dump to unprivileged users of the network served by the SMSC, + i.e., to end users. The idea is that each individual user should be able to + give their ssh public key to the administrator of the community network, and + then ssh into a special restricted service on the SMSC that does not grant + any system shell access, but allows them to access services under their own + phone number. Such an empowered end user should be able to submit SMS from + their own phone number using the power of a full-size computer (as opposed to + very painful text entry on the numeric keypad of a traditional GSM phone), + and to see a full log of all messages received by or sent from their own + phone number. + +* By the nature of her job, the administrator of the SMSC (and of the community + GSM network to which this SMSC belongs) necessarily has access to every + message that passes through the system, all metadata and actual content. + While this access is technically necessary, an administrator who is worthy of + her trusted position must not abuse this trust, and must do everything + possible to avoid looking at users' private message content when it is not + necessary to do so for technical troubleshooting reasons. Toward this + objective, themwi-smsc-dump must make it easy to look at only technically + necessary information, without throwing unnecessary private info into the + operator's eyeballs. diff -r 000000000000 -r 9e364c18e0e8 include/sm_record.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/include/sm_record.h Wed Dec 20 03:50:06 2023 +0000 @@ -0,0 +1,55 @@ +/* + * This header file defines the message structure as stored in ThemWi-SMSC + * persistent message store (PMS) binary files. + */ + +#pragma once + +#include + +struct sm_address { + uint8_t ndigits; + uint8_t ton_npi; + uint8_t bcd[10]; +}; + +/* values for src_class and dest_class bytes */ +#define SME_CLASS_LOCAL 0 +#define SME_CLASS_GSM 1 +#define SME_CLASS_UPSTREAM 2 +#define SME_CLASS_DOWNSTREAM 3 + +/* values for disposition byte */ +#define SM_DISP_ACTIVE 0 +#define SM_DISP_DELIVERED 1 +#define SM_DISP_EXPIRED 2 +#define SM_DISP_IMSI_FAIL 3 +#define SM_DISP_SMPP_REJECT 4 + +struct sm_record { + uint64_t time_entry; /* time entered in our SMSC */ + uint64_t time_expiry; /* end of validity period */ + uint64_t time_disch; /* discharge time */ + struct sm_address addr_from; + struct sm_address addr_to_orig; + struct sm_address addr_to_final; + /* + * Each of the two IMSI fields is repurposed as system_id + * in the case of downstream SMPP peers. The maximum length + * is conveniently the same. + */ + char src_imsi[16]; + char dest_imsi[16]; + uint32_t src_extra_info; + uint32_t dest_extra_info; + uint8_t src_class; + uint8_t dest_class; + uint8_t disposition; + uint8_t gsm_src_mr; + uint8_t pid; + uint8_t dcs; + uint8_t udhi; + uint8_t udl; + uint8_t ud[140]; + uint8_t reserved[8]; +};