--- a/doc/TIFFS-Overview	Sun Nov 04 01:55:35 2018 +0000
+++ b/doc/TIFFS-Overview	Sun Nov 04 02:00:03 2018 +0000
@@ -1,4 +1,14 @@
-All TI GSM firmwares known to this author (FreeCalypso developer Space Falcon)
+[Historical note: this document was originally written in 2014 when the vision
+ of FreeCalypso was very different from what it is today.  Since then we have
+ transitioned from making aftermarket hacks on pre-existing Calypso phones and
+ modems to producing and supporting our own FreeCalypso hardware products, and
+ our firmware work has changed from a nebulous dream to stable production code.
+ The ways in which we approach various technical aspects have changed
+ accordingly, but much of our documentation has been slow to catch up.  The
+ documentation is now being updated, but there may still be some passages where
+ the old world view shines through.]
+
+All TI GSM firmwares known to this author (Mother Mychaela of FreeCalypso)
 implement some kind of flash file system, or FFS.  Several different FFS code
 implementations, and correspondingly several different on-flash data formats,
 have been used throughout the history of TI's involvement in the wireless
@@ -17,17 +27,21 @@
   building our own full source, multi-target GSM fw contains a turnkey-working
   implementation of this very FFS, using the on-flash format in question and
   providing run-time APIs expected by the rest of the GSM fw suite.  Following
-  the principle of ``if it ain't broke, don't fix it'', we can use this FFS not
+  the principle of ``if it ain't broke, don't fix it'', we use this FFS not
   only on the gtamodem target, but also on other targets, including those where
-  we would be starting from a blank state and thus have the freedom to use
-  whatever FFS we like.
+  we are starting from a blank state and thus have the freedom to use whatever
+  FFS we like.
 
 * The original proprietary fw on the Pirelli DP-L10 phone also happens to use
-  an FFS in the same format.  Pirelli's FFS does *not* contain the IMEI or any
-  of the RF calibration values though, and trying to reuse it directly for our
-  own FC GSM fw seems to be more trouble than benefit - so we'll probably have
-  our fw start with a blank TIFFS instead - but there is still insight to be
-  gained from in-vitro examination of captured Pirelli FFS images.
+  an FFS in the same format, although Pirelli's FFS does *not* contain the IMEI
+  or any of the RF calibration values.  This Pirelli phone was originally a
+  target of high interest for FreeCalypso, as we had high hopes of turning it
+  into a libre phone by way of our aftermarket firmware - but this plan has
+  since been abandoned when it became clear that Pirelli's hw is unsuitable for
+  aftermarket fw development because of the multitude of extra peripheral chips
+  for non-GSM functions which get in the way.  In the earlier years of
+  FreeCalypso a lot of effort had been invested into studying all aspects of
+  the Pirelli DP-L10 phone and its original firmware, including the FFS.
 
 Naming
 ======
@@ -60,8 +74,8 @@
 and TIFFS even supports symlinks, although that support is a little under-
 developed, and apparently no FFS symlinks were ever used in any production GSM
 device.  Thus the FFS implemented in TI-based GSM devices (modems and
-"dumbphones") is really no different from, for example, JFFS2 in embedded Linux
-systems.
+"dumbphone" handsets) is really no different from, for example, JFFS2 in
+embedded Linux systems.
 
 (The only traditional UNIX file system features which are missing in TIFFS are
  the creation/modification/access timestamps and the ownership/permission
@@ -86,10 +100,10 @@
 One would naively think that non-volatile storage of data in flash outside of
 factory programming would be needed only in a device with its own UI, and that
 a modem subservient to external AT commands would be completely stateless
-across reboot/power cycles; but that is not the case in actuality.  TI's GSM
-firmwares, including the Openmoko ones (the "standard" mokoN), are designed to
-always "mount" their FFS with read/write access; TI's FFS implementation in the
-firmware has no concept of a "read-only mount".
+across reboot/power cycles; but that is not the case in actuality.  TI-baseed
+GSM firmwares, including Openmoko's mokoN and our own FreeCalypso, are designed
+to always "mount" their FFS with read/write access; TI's FFS implementation in
+the firmware has no concept of a "read-only mount".
 
 I am still investigating just what kinds of data are routinely written into the
 non-volatile FFS by the firmware in normal operation on devices like the GTA0x
@@ -110,11 +124,11 @@
 flash chip itself) into units called "sectors" or more descriptively, erase
 blocks.  The typical NOR flash sector size (in Calypso GSM devices) ranges from
 64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the
-S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total).
-The key physical property is that any bit may be changed from a '1' to a '0' at
-any time, in any combination, but resetting of '0' bits back to ones can be
-done only on the granularity of these largish sectors, in an operation called
-"sector erase".
+S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 and in our own FreeCalypso
+hardware designs (16 MiB of flash total).  The key physical property is that
+any bit may be changed from a '1' to a '0' at any time, in any combination, but
+resetting of '0' bits back to ones can be done only on the granularity of these
+largish sectors, in an operation called "sector erase".
 
 The location of TIFFS within the flash memory of a given GSM device is defined
 by the firmware design of that device, but is always some integral number of
@@ -132,6 +146,12 @@
   C11x/123 use smaller FFS configurations, whereas C155/156 seem to have
   switched to some other FFS format, different from our familiar TIFFS.
 
+* On our own FreeCalypso hardware family we have put our FFS in the first 8
+  sectors (of 256 KiB each) in the 2nd flash chip select bank, which appears at
+  0x01800000 in the ARM7 address space instead of Pirelli's 0x02000000 because
+  we have wired the 2nd flash chip select to nCS2 on the Calypso instead of
+  Pirelli's nCS3.
+
 * The smallest real FFS configuration called for by the table in dev.c in TI's
   original Leonardo fw source is 3 sectors of 64 KiB each; the same table also
   sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in
@@ -160,8 +180,8 @@
 expected to recover on the next boot cycle.  I won't be able to document here
 all gory details of exactly how this goal is achieved, partly because I haven't
 studied the code to the requisite level of depth myself yet, but all of the
-responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source
-tree; feel free to study it.
+responsible code lives under src/cs/drivers/drv_app/ffs in our fc-magnetite and
+fc-selenite source trees; feel free to study it.
 
 In its "normal" or "clean" state (i.e., when not in the middle of a write
 operation or recovery from an ungracefully interrupted one), a TIFFS instance
@@ -220,9 +240,8 @@
 =============================
 
 (This section is only relevant to you if you plan on physically producing your
- own GSM phones or modems on your own factory production line, like this author
- fancies doing in the not-too-distant future, or if you simply enjoy knowing
- how it is done.)
+ own GSM phones or modems on your own factory production line, like we currently
+ do at our family company, or if you simply enjoy knowing how it is done.)
 
 To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the
 embedded Linux world, which would produce a TIFFS image complete with some
@@ -231,7 +250,10 @@
 "in vivo" by TI's firmware running on the device itself during the "production
 test" phase.
 
-The process seems to go like this:
+We never got a copy of the original factory production line software that was
+used by Openmoko, but we have successfully replicated the process using our own
+Unix/Linux-based FreeCalypso host tools, the very same tools that are contained
+in the present source package you are looking at.  The process goes like this:
 
 * When the printed circuit board is physically populated with components such
   as the Calypso chip and the flash chip, the latter can be blank - if the
@@ -271,46 +293,58 @@
 
 The IMEI is assigned and written into FFS in this process, but it is not the
 only data item that will be unique for each individual device made.  Much more
-important are the RF calibration values: I have yet to learn exactly what is
-being (or needs to be) measured, how these measurements are performed (under
-what conditions; what external test equipment is needed), and how these measured
-and recorded RF calibration values affect GSM device operation, but this TI
-presentation gives some clues:
+important are the RF calibration values; the factory calibration procedure does
+the following for each individual unit:
+
+* Measures the frequency offset produced by the VCXO as a function of the AFC
+  DAC control value and constructs the afcparams table based on these
+  measurements;
+
+* Characterizes the dBm output of the Tx chain as a function of the APC DAC
+  control value and comes up with a set of these DAC values which produce Tx
+  power levels prescribed by the GSM 05.05 spec;
 
-ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf
+* Determines the correction values which need to be applied in order to set the
+  correct Rx path gains and to determine the true Rx signal level from the dBfs
+  power measurements made in the DSP from Rx I&Q samples.
 
-All of these calibration values are stored in a bunch of files under the
-/gsm/rf subtree, and these files seem to be "owned" by the L1 code.  The latter
-has RAM data structures which correspond to these files; upon normal boot the
+These calibration procedures are performed by connecting a suitable RF test
+instrument (R&S CMU200 is the industry gold standard) to the GSM device's
+antenna connector or RF test port and running special calibration programs
+which talk both to the CMU200 or other test instrument and to the L1TM (Layer 1
+test modes) component in the DUT (device under test).  In FreeCalypso hardware
+manufacturing we use a CMU200 instrument which is itself maintained in good
+calibration standing, and for the calibration software we use our own
+fc-rfcal-tools which talk to the DUT via rvinterf.
+
+All of the resulting calibration values are stored in a bunch of files under the
+/gsm/rf subtree, and these files are "owned" by the L1 code.  The latter has
+RAM data structures which correspond to these files; upon normal boot the
 initialization code looks in FFS, and if it finds any of the RF calibration
 files, it reads each present file into the corresponding RAM data structure,
-overwriting the compiled-in defaults.  It appears (slightly uncertain because I
-have not yet reintegrated the code in question into our own gsm-fw) that the RF
-calibration files in FFS come into being as follows:
-
-* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the
-  measurements and stores results in its RAM data structures as commanded by
-  the production test station through the "test mode" interface;
+overwriting the compiled-in defaults.  With TI's standard production calibration
+procedure which we have replicated in our FreeCalypso hw manufacturing setup,
+these RF calibration files in FFS come into being as follows:
 
-* A final test mode command directs the above L1 code to write its RAM data
-  structures into FFS.
+* The Test Mode support code in L1 (i.e., part of the main GSM fw) performs the
+  measurements and stores results in its RAM data structures as commanded by
+  the production test station through the Test Mode interface;
 
-Once I actually learn this RF calibration process properly in connection with
-building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what
-it would take to recreate these RF calibration values if they are lost.  But
-until then the only advice I can give is to make a backup copy of your modem
-FFS with fc-loadtool, and to save it securely.
+* Certain special test mode commands encoded via the MISC_ENABLE opcode direct
+  the above L1TM code to write its RAM data structures into FFS.
+
+See the RF_tables article for more information.
 
 Compal and Pirelli differences
 ==============================
 
 The above description refers to TI's vanilla reference version, and it seems
-like Openmoko (FIC) was the only phone/modem manufacturer who followed it
-without major deviations.  In contrast, both Compal (Mot C1xx) and Foxconn
-(Pirelli DP-L10) moved the vital per-unit factory data (IMEI and RF calibration)
-out of the FFS into their own ad hoc flash data structures (which are very
-difficult to reverse-engineer and make use of, unfortunately), leaving their FFS
-only for less critical data.
+like Openmoko (FIC) was the only phone/modem manufacturer (prior to us!) who
+followed it without major deviations.  In contrast, both Compal (Mot C1xx) and
+Foxconn (Pirelli DP-L10) moved their vital per-unit factory data (IMEI and RF
+calibration) out of the FFS into their own ad hoc flash data structures (which
+are very difficult to reverse-engineer and make use of, unfortunately), leaving
+their FFS only for less critical data.
 
 In Compal's case (at least on the C139 model with which I have extensive
 personal experience) the FFS stores only users' personal information and nothing
@@ -329,67 +363,114 @@
 must have uploaded these files to each phone via the TMFFS2 protocol, and our
 FreeCalypso suite now features a tool that can replicate this feat: fc-fsio.
 
-FreeCalypso support for TIFFS
-=============================
-
-Aside from implementing and using it in our own gsm-fw, FreeCalypso offers
-the following support for TIFFS:
+Aftermarket FFS for FreeCalypso on Compal & Pirelli targets
+===========================================================
 
-1. We have a utility for "in vitro" examination of FFS images read out of GSM
-   devices with fc-loadtool.  This tiffs utility (along with mokoffs and pirffs
-   wrappers) lives in the ffstools top-level directory of the freecalypso-sw
-   source tree.  This TIFFS "in vitro analyzer" utility supplants the earlier
-   mpffs-* tools, and adds some additional examination functionality.  It is
-   strictly a "read only" tool, however - it is not designed for "in vitro"
-   editing of TIFFS images.
+When we run our own FreeCalypso fw on "alien" (not native to us) Mot C1xx and
+Pirelli DP-L10 hardware, we don't use the FFS from their respectively original
+firmwares: those original FFS instances don't contain any bits of interest to
+us, trying to make our fw use the same FFS as Mot/Compal's or Pirelli's original
+fw would be more trouble than benefit, and on one of the target devices in this
+family (Mot C155/156) the original FFS is in some different format.  Instead we
+create our own aftermarket FFS for our FreeCalypso fw on these alien hw targets,
+using a different flash location from the original so that the original fw's FFS
+cannot be mistaken for our own.
+
+On the Pirelli DP-L10 we put our aftermarket FFS in an area of the flash which
+the official fw family uses as a staging area for over-the-air firmware updates,
+thus as long as you are not doing official fw updates over WLAN (i.e., if you
+only run one fixed official fw version or flash different official fw versions
+with fc-loadtool without going through "fw update" protocols), our aftermarket
+FFS used by our run-from-RAM FreeCalypso firmwares should remain undisturbed
+while the phone is in the official fw mode.
 
-2. A number of FC tools may be strung together into a kit for editing the FFS
-   content of a GSM device, e.g., for changing the IMEI.  The following pieces
-   will be involved:
+The situation is different on Mot C1xx phones.  The lower-end C1xx models
+including the C139 (our primary hw target in this family) have too little RAM
+to run our FreeCalypso fw entirely out of RAM without flashing; the C155/156
+subfamily does have enough RAM to allow a complete FC GSM fw image to be loaded
+and run via fc-xram under some conditions (we previously supported such usage
+in our now-retired Citrine fw and may bring it back in the gcc-built config of
+FC Selenite), but there is no place in the flash where we can put our
+aftermarket FFS without overwriting some part of the original fw or its data -
+thus our general procedure for running FreeCalypso on any C1xx model is to
+convert the victim phone to FC on a long-term basis by flashing our modified
+bootloader, flashing one of our fw builds and establishing a FreeCalypso
+aftermarket FFS in a flash area designated by us.
 
-* What is destined to eventually become our totally free GSM fw (the gsm-fw
-  source subtree at the top of freecalypso-sw) does not contain any of the
-  actual GSM protocol stack (or even L1) functionality yet, but it already
-  contains both the FFS code and those components (ETM and TMFFS[12]) which
-  are needed for interfacing an external "test mode shell" to this FFS
-  implementation through the RVTMUX interface.  And when our gsm-fw does gain
-  the actual GSM functionality, the ability to build a minimal FFS+ETM-only
-  configuration will still be retained.
+It was already mentioned earlier that the factory RF calibration values on these
+alien phones are stored in non-TIFFS flash data structures of Mot/Compal's or
+Foxconn's invention, and our currently supported FreeCalypso firmwares
+(Magnetite and Selenite) do not contain any code for reading these alien data
+structures.  (FC Citrine could read directly from Pirelli's factory data block,
+but none of our fw offerings ever parsed Compal's weird flash records.)  Instead
+our current approach is to have an external tool extract the bits of interest
+from the alien factory records, convert them to our TI-standard format if
+necessary, and upload them into our FreeCalypso aftermarket FFS.  The specifics
+are as follows:
 
-* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs
-  entirely from RAM via fc-xram, no flashing), and run on a physical device
-  such as the GTA0x GSM modem via the fc-xram host utility;
+* The Pirelli DP-L10 is a breeze: simply run our special pirelli-magnetite-init
+  command in fc-fsio while connected to a Magnetite or Selenite fw instance
+  running with our aftermarket FFS, and the tool will copy both the IMEI and
+  the RF calibration records from Pirelli's factory data block into our
+  aftermarket FFS.
 
-* After loading the ramImage, fc-xram will immediately exec our rvinterf host
-  utility (see rvinterf/README);
+* Mot C1xx phones present a lot more hassle: our current official procedure is
+  to make a dump of the flash prior to the xenotransplantation procedure (also
+  serves as a backup), extract the RF calibration values with our c1xx-calextr
+  tool, and then later in the procedure when you initialize your aftermarket FFS
+  with fc-fsio, upload these extracted and format-converted RF calibration files
+  as one of the several steps involved.  You will need to enter your IMEI
+  manually: we never figured out where Mot/Compal have it stored and how it is
+  obfuscated.
 
-* Once the GSM device is running what is effectively an FFS editing agent out
-  of RAM, accessed via rvinterf over the serial channel, the user can run
-  fc-tmsh or fc-fsio, and this "test mode shell" provides commands for writing
-  things to FFS exactly like one would do in the factory production line
-  environment for which TI taylored their tools.
+FreeCalypso host tools for TIFFS
+================================
+
+Our FC host tools package supports TIFFS in two ways:
+
+1) Our primary tool for working with GSM device file systems is fc-fsio.  When
+run against a compatible firmware version (primarily our own, but Pirelli's
+proprietary fw is also compatible), fc-fsio allows various read and write
+operations to be performed on the target device FFS.
+
+2) We have a TIFFS In Vitro Analyzer (IVA) tool for "in vitro" examination of
+FFS images that have been read out of raw flash with fc-loadtool.  See the
+TIFFS-IVA-usage article for more information.
 
-The "in vivo" method of editing the FFS content of a GSM device described above
-will probably sound very convoluted, and you may find yourself asking for a way
-to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit
-that image "in vitro" with some utility on your PC, and then use fc-loadtool
-again to program it back into your device.  But consider that an "in vitro" FFS
-modification would involve erasing and rewriting all sectors of your FFS,
-whereas an "in vivo" modification of some small file like the IMEI would be
-just a short flash write operation without any erasures at all, i.e., kinder
-on the flash.
+In addition to the above, back in the days of Openmoko (back when the Openmoko
+community was still active and we considered ourselves a part of it) we had
+produced a kit for editing the modem FFS on Openmoko GTA01/02 devices, giving
+users an easy way to change their /pcm/IMEI file.  Changing IMEIs for no good
+reason is completely pointless and is actually detrimental rather than helpful
+for privacy, but whoever came up with the edicts that "the IMEI MUST be
+immutable" had obviously failed Human Psychology 101: declaring something to be
+forbidden causes people to want it simply because it is forbidden and for no
+other reason - hence the popular demand for IMEI changing tools.
+
+Our Openmoko FFS editing kit from early 2014 consisted of a very early version
+of what much later became the present FC host tools package (more specifically,
+it was before fc-fsio, and the set-imeisv command had been hacked into fc-tmsh
+instead) plus a pair of "in vivo" FFS editing agent target binaries.  Our
+current FC host tools fully supplant the ancient version in that 2014 kit, but
+we do not currently have a non-deprecated replacement for the old "in vivo" FFS
+editing agent: those FFS editing agent binaries were built from our old gsm-fw
+framework which was deemed to be a dead end in 2016 and fully retired in 2018.
 
-In any case, the "in vivo" method is already available now because all of the
-components involved therein are also needed for other development uses in the
-FreeCalypso project, whereas developing a fully-functional "in vitro"
-alternative (one that can create an FFS image "de novo" from a tree of files
-and directories a la mkfs.jffs2, or add new files to an existing TIFFS image
-etc) would be a good amount of extra work which we otherwise don't need - hence
-the latter is not very likely to be written any time soon.
+We have not produced a new and actively maintained replacement for the "in vivo"
+FFS editing function which was previously provided as a side product of the old
+gsm-fw framework because we have no real need for it: the *only* use case we
+have for it involves Openmoko devices, and even then only two special use cases:
+
+1) If someone wishes to change their IMEI from within the FreeRunner (without
+using an external serial cable) *or* while keeping an old firmware version
+(without updating to moko13 or later), or
 
-However, if the "in vitro" modification you seek is something trivial like
-changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without
-changing its length, you can use the existing "in vitro, read-only" tiffs host
-utility to find the exact byte location of the file data within the TIFFS image,
-and then use your favourite hex editor to whack whatever new byte content you
-like at that offset.
+2) If someone has updated the modem fw on their FR to one of our current
+releases, wishes to fix the /gsm/com/rfcap file to reflect the true tri900 or
+tri850 band configuration of their device, but wishes to do it from within the
+FR, without using an external serial cable.
+
+If anyone does need to exercise one of the two special use cases listed above,
+they can use the "in vivo" FFS editing agent binaries from the ancient
+ffs-edit-kit-r1.tar.bz2 package - they are ancient, but will do the job just
+fine for both use cases in question.