--- a/doc/Sysmocom-SIM-notes	Mon Mar 22 21:28:32 2021 +0000
+++ b/doc/Sysmocom-SIM-notes	Mon Mar 22 21:30:42 2021 +0000
@@ -1,73 +1,194 @@
-The present suite of tools (fc-simtool and fc-uicc-tool) is NOT a good fit for
-programming sysmoUSIM-SJS1 and sysmoISIM-SJA2 cards made by Sysmocom and sold
-in their webshop, because of the following combination of factors:
+The current programmable SIM card model sold by Sysmocom in their webshop
+(sysmoISIM-SJA2) is probably good for people who run their own cellular networks
+of the LTE/5G kind, but it is NOT a good choice for those of us who are only
+interested in GSM/2G, to the exclusion of all later G's:
+
+* The triple-cut physical form factor is inferior (compared to solid-piece 2FF
+  without 3FF or 4FF cuts) for use in classic GSM/2G phones with 2FF SIM
+  sockets.
+
+* The presence of unwanted USIM and ISIM applications with their associated
+  ADF.USIM and ADF.ISIM file systems is very unpleasant: it forces us to either
+  study up on completely unwanted-to-us USIM and ISIM specs and program all
+  those files to something sensible (and just what would be sensible programming
+  of USIM and ISIM files for a 2G-only network that exists solely to provide
+  service to classic GSM/2G phones?), plus expend oodles of time and effort to
+  develop the necessary programming tools that can write all those files under
+  ADF.USIM and ADF.ISIM, or leave all those files unprogrammed, and take a
+  gamble if someone sticks the partially-programmed card (classic SIM
+  programmed, USIM and ISIM left unprogrammed) into a phone that knows about
+  USIM and/or ISIM.
 
-1) These cards are primarily USIM/ISIM, with classic GSM 11.11 SIM support
-   regarded as "backward compatibility" - thus they have a lot of important
-   files under ADF.USIM and ADF.ISIM which are not accessible via the classic
-   GSM 11.11 SIM protocol.
+* Some of the advertising which Sysmocom prints on their current webshop cards,
+  plus the very name sysmoISIM (emphasizing and glorifying ISIM rather than
+  plain SIM) is offensive at least to me (Mother Mychaela), and should be
+  offensive to any truly devoted lover of classic GSM/2G technology.
 
-2) Our main feature-rich tool is fc-simtool, but this tool speaks only the
-   classic GSM 11.11 SIM protocol, hence it cannot access any of the USIM/ISIM
-   files.
+Because of the above considerations, we (FreeCalypso) are currently in the
+process of getting our own community SIMs made, to serve as an alternative to
+Sysmocom webshop product.  Our FreeCalypso community SIMs are currently as of
+this writing (2021-03) being made for us by Grcard in China, they are a GSM-only
+SIM card model (GrcardSIM2) without USIM/ISIM (they don't speak UICC protocol
+at all, yay!), and we are having them made in a 2FF-only cut, meaning that the
+2FF piece is fully solid.
 
-3) We have fc-uicc-tool which speaks the UICC protocol that is native to these
-   Sysmocom cards, but it is only a low-level debug tool, not a feature match
-   to fc-simtool.
+However, despite our general dislike of Sysmocom's current USIM/ISIM-centric
+product and our ongoing effort to produce a GSM/2G-centric alternative, we do
+have some support in FC SIM tools for Sysmocom's current sysmoISIM-SJA2 card
+and for their previous sysmoUSIM-SJS1 model.  This limited support exists
+because these webshop cards are very readily and inexpensively available, and
+because of natural human curiosity - we've been playing with these readily
+available Sysmocom webshop cards while enduring the long delays involved in our
+Grcard-based quest for a better alternative.
+
+Sysmocom webshop card database
+==============================
 
-The proper long-term solution for our 2G-centric GSM community is to get our own
-SIMs made, either by paying big bucks to Sysmocom to produce a run of custom
-cards (presumably based on their current SJA2 platform) with USIM and ISIM
-removed, leaving only the file system tree under MF that can be fully
-manipulated via the classic SIM protocol, or preferably by resurrecting the
-older Grcard SIM-only platform if possible - it may take a long time to find out
-if the latter option is possible or not.  But in the meantime, if someone needs
-to program a SIM right now, when Sysmocom webshop cards are the only available
-option, we do have limited support for programming these SIMs:
+Whenever you buy a 10-pack of sysmoUSIM-SJS1 or sysmoISIM-SJA2 cards from
+Sysmocom webshop, they send you an email with per-card identities and keys.
+The information in that email is essential for doing any kind of admin writes
+to the cards (the necessary ADM1 key is randomly assigned per card), and also
+for any CHV2 operations: the randomly assigned PIN1 and PUK1 are printed on the
+plastic, but not PIN2 or PUK2, which are also randomly assigned.
+
+To reduce the need for manual lookups in email data, we have implemented a tool
+that converts Sysmocom webshop emails into our own database format, and we have
+integrated support for this database into fc-simtool.  (Replicating the same
+functionality in fc-uicc-tool, as would be appropriate for these UICC-native
+cards, is on the to-do list.)
+
+Sysmocom webshop emails with USIM/ISIM card key material feature a MIME
+multipart/alternative structure with text/plain and text/html parts, with each
+part further encoded in base64.  To extract the bits of interest and convert
+them into our sws-card-db format, follow these steps:
+
+1) Extract the text/plain portion from the MIME structure and decode it from
+   base64.
 
-* It is possible to authenticate with the ADM1 key from within fc-simtool on
-  both sysmoUSIM-SJS1 and sysmoISIM-SJA2, as explained below.
+2) Open the extracted and decoded text/plain email portion in your favourite
+   text editor and find the heading block of 19 lines, beginning with a line
+   that reads "IMSI" and ending with a line that reads "KIK3".  (If you bought
+   the cheaper option without ADM and OTA keys, there will only be 9 lines here,
+   starting with IMSI and ending with OPC.)  Then there should be a blank line,
+   followed by 19 lines of data per card (or 9 lines for sans-ADM/OTA variant),
+   with blank lines separating each card data block from the next.  Extract the
+   portion beginning with the heading block and ending with the last card data
+   block in the batch.
+
+3) Feed the data extract from the previous step to our sws-email2db utility.
+
+sms-email2db sends its output to stdout, thus you should run it like this
 
-* Once you have authenticated with ADM1, you can use fc-simtool admin write
-  commands (write-imsi, SDN phonebook write operations, manual update-bin-imm
-  on various small transparent EFs) just as if you were working with a Grcard
-  SIM.
+sws-email2db email_extract.txt >> /opt/freecalypso/sim-data/sws-card-db
+
+If you have bought multiple card batches from Sysmocom over the years, you will
+need to collect those old emails and repeat the extraction procedure for each of
+them, using the '>>' form of output redirection to gather all data in one
+sws-card-db file.  Edit the finished database file with vi if necessary.
+
+Using fc-simtool to program Sysmocom webshop cards
+==================================================
+
+Even though it is a UICC-native card that clearly prefers being admin-programmed
+via the UICC protocol, sysmoISIM-SJA2 allows its ADM1 PIN to be entered in a
+GSM 11.11 SIM protocol session with a VERIFY CHV command with P2=0x0A.
+Therefore, the command to enter sysmoISIM-SJA2 ADM1 manually in fc-simtool is:
 
-* You can also use fc-uicc-tool to access and program every file on Sysmocom
-  cards, including files under ADF.USIM and ADF.ISIM - but in this case you will
-  have to do everything manually in raw hex, with a hex data file for every
-  update-bin and update-rec command.
+verify-ext 10 xxxxxxxx
+
+Unlike the situation with sysmoUSIM-SJS1 (see below), there are no restrictions
+as to when this command may be given in an fc-simtool session.
+
+The above is the manual command, requiring the operator to manually look up the
+correct ADM1 key for the card being programmed.  However, if you have your
+sws-card-db file initialized with data from email per above instructions, you
+can authenticate with ADM1 as simply as:
+
+sws-auth-adm1
 
-Authenticating with ADM1
-========================
+This command reads the ICCID record from the card (totally immutable on SJA2
+cards, and always readable without depending on CHV1 status), looks up this
+ICCID in sws-card-db, and sends a VERIFY CHV P2=0x0A command to the card with
+ADM1 extracted from the card db record.
+
+The following additional commands are available that work in a similar manner:
+
+sws-auth-pin1		-- send VERIFY CHV1 with PIN1 from sws-card-db
+sws-auth-pin2		-- send VERIFY CHV2 with PIN2 from sws-card-db
+sws-pin1-disable	-- send DISABLE CHV with PIN1 from sws-card-db
+sws-pin1-enable		-- send ENABLE CHV  with PIN1 from sws-card-db
 
-The method for sending your ADM1 key to the card varies depending on whether
-you are in an fc-simtool or fc-uicc-tool session, and whether your card is
-sysmoUSIM-SJS1 or sysmoISIM-SJA2.  There are 3 possibilities:
+sysmoUSIM-SJS1 difference
+=========================
 
-* If you are in an fc-uicc-tool session with either type of card, the command
-  to authenticate with ADM1 is as follows:
+Both sysmoUSIM-SJS1 and sysmoISIM-SJA2 are UICC-native cards, and both really
+prefer to be admin-programmed via the UICC protocol, rather than GSM 11.11 SIM
+protocol.  Both cards do allow ADM1 authentication to be performed in a GSM
+11.11 SIM protocol session, but sysmoUSIM-SJS1 is less "happy" about it, and
+imposes a more burdensome restriction.  sysmoISIM-SJA2 allows its ADM1 key to
+be submitted via a VERIFY CHV (CLA=A0, P2=0A) APDU in a GSM 11.11 SIM session,
+but sysmoUSIM-SJS1 does not allow the same.  sysmoUSIM-SJS1 accepts its ADM1 key
+only via UICC-style (CLA=00) VERIFY PIN APDUs, thus at first it appears that
+these cards cannot be admin-programmed via the classic GSM 11.11 SIM protocol.
+They do have one open loophole, however: if the UICC-style VERIFY PIN command
+for ADM1 is sent as the very first command in a card session, it can be followed
+by other UICC protocol commands (making a regular UICC session), or it can be
+followed by GSM 11.11 SIM protocol commands with CLA=A0, thus allowing one
+special exception to the general rule which prohibits mixing these two protocols
+in the same card session.
 
-  verify-pin 10 xxxxxxxx
+Our fc-simtool command for sending SJS1 ADM1 keys in the manner this card model
+requires is as follows:
+
+verify-sjs1-adm1 xxxxxxxx
 
-  where xxxxxxxx are the 8 digits of the ADM1 secret code.  There are no
-  restrictions as to when this command may be given in an fc-uicc-tool session.
+The really big restriction is that this command must be issued at the very
+beginning of your fc-simtool session, before any other commands.  If you issue
+this command later, after some GSM 11.11 SIM APDUs have already been exchanged,
+it won't work.  For this reason, our sws-auth-adm1 "macro" command cannot be
+used in fc-simtool with SJS1 cards: in order to use sws-card-db, one has to read
+the ICCID record to identify the specific card out of the pool, and once some
+APDUs have been exchanged to make that ICCID read, the special exception to the
+protocol mixing prohibition is no longer available.  One could develop a more
+complicated system where you read the ICCID, then reset the card and have a new
+card session beginning with ADM1 authentication - but because this
+sysmoUSIM-SJS1 card model is no longer sold by Sysmocom, there is no
+justification for expending the effort.
 
-* If you are in an fc-simtool session with sysmoISIM-SJA2, the command becomes:
-
-  verify-ext 10 xxxxxxxx
+Using fc-uicc-tool with Sysmocom webshop cards
+==============================================
 
-  There are no restrictions as to when this command may be given in an
-  fc-simtool session.
-
-* If you are in an fc-simtool session with sysmoUSIM-SJS1, the command becomes:
+The UICC protocol is native to both sysmoUSIM-SJS1 and sysmoISIM-SJA2, thus
+fc-uicc-tool works like a charm with both card models.  The problem, however,
+is that fc-uicc-tool is only a low-level debug and manual tinkering tool: it
+can do "everything", but only 100% manually in raw hex.  Most of the high-level
+functions of fc-simtool are not replicated in fc-uicc-tool, and furthermore, an
+approach of mindlessly translating fc-simtool high-level functions to use the
+UICC protocol for card file access won't work either: the USIM spec definition
+of many important files is quite different from the original DF_GSM and
+DF_TELECOM definitions for classic SIM.
 
-  verify-sjs1-adm1 xxxxxxxx
+The issue is ultimately one of project purpose and direction: FreeCalypso
+focuses on GSM/2G to the exclusion of later G's, our preferred SIM cards are
+our own FCSIM1, our primary SIM card manipulation tool is fc-simtool, and
+fc-uicc-tool exists only as a bounded-effort side utility.  For people who
+prefer to work with USIM/ISIM cards natively, programming all of their new
+files for later-G functionality, other software tool projects like pysim-shell
+would be more appropriate.
+
+ADM1 and other PIN authentication in fc-uicc-tool
+=================================================
 
-  Unlike the other two cases, this command must be issued at the very beginning
-  of your fc-simtool session, before any other commands.  If you issue this
-  command later, after some GSM 11.11 SIM APDUs have already been exchanged, it
-  won't work.
+If you are in an fc-uicc-tool session with either sysmoUSIM-SJS1 or
+sysmoISIM-SJA2, the command to authenticate with ADM1 is as follows:
+
+verify-pin 10 xxxxxxxx
+
+where xxxxxxxx are the 8 digits of the ADM1 secret code.  There are no
+restrictions as to when this command may be given in an fc-uicc-tool session.
+
+sws-auth-* commands have not been ported over fc-uicc-tool yet, but this
+omission will be easy to fill.
 
 Changing the ADM1 PIN
 =====================
@@ -89,40 +210,13 @@
 remain forever proprietary to Sysmocom, especially given the lack of any
 practical need for such downstream changing of PUK1/PUK2.
 
-Thoughts on card (re)formatting
-===============================
-
-ETSI and 3GPP specs give many more degrees of freedom to SIM card issuers than
-just the content of various EFs: the card issuer gets to decide which DFs and
-EFs will be present vs. which ones won't be present at all, and for many EFs
-the size (allocated space) is variable per the specs and up to the card issuer.
-In the case of record-based EFs, both the record size and the number of records
-are often left up to card issuers to tune as desired.
-
-In the Mother's opinion, a truly programmable SIM would be one where every
-downstream owner of each card (not just the initial factory or the party putting
-up big bucks for a large custom production run) can do a full reformat: erase
-the file system and then create whatever tree of DFs and EFs she desires, with
-full control over each file's allocated size, structure and access conditions.
-
-In the case of Sysmocom webshop SIMs, we (FreeCalypso) are not aware of any
-publicly available documents describing how to perform such a reformat - it
-appears that Sysmocom keeps this knowledge proprietary.  In contrast, the older
-Grcard-based SIMs had some publicly documented commands for erasing the card
-and creating new directories and files:
-
-https://osmocom.org/projects/cellular-infrastructure/wiki/GrcardSIM
-
-It remains to be seen whether we (FreeCalypso) can get new SIMs from Grcard
-which are also freely formattable.
-
 MSISDN misprogramming on early sysmoUSIM-SJS1 cards
 ===================================================
 
-Referring to the previous section regarding formatting degrees of freedom,
-Sysmocom webshop cards have their EF_MSISDN file allocated as 6 records of 34
-bytes each.  Record length of 34 bytes translates into 20 bytes of alpha tag
-plus the required 14-byte structure at the end of each record.
+Sysmocom webshop cards (both sysmoUSIM-SJS1 and sysmoISIM-SJA2) have their
+EF_MSISDN file allocated as 6 records of 34 bytes each.  Record length of 34
+bytes translates into 20 bytes of alpha tag plus the required 14-byte structure
+at the end of each record.
 
 When Sysmocom made their early sysmoUSIM-SJS1 cards, they intended to program
 the first record of EF_MSISDN as +882110xxxxx, where xxxxx are equal to the last