--- a/doc/TIFFS	Mon Jan 13 10:15:59 2014 +0000
+++ b/doc/TIFFS	Tue Jan 14 02:31:09 2014 +0000
@@ -48,3 +48,321 @@
 link between this FFS format and the Openmoko+Pirelli duo, other than the
 happenstance of me having first encountered this FFS on these two GSM device
 brands, and the name TIFFS is more neutrally-descriptive.
+
+What it is
+==========
+
+In a rare departure from TI's norm (most of TI's GSM firmware and associated
+development tools suffer from heavy Windows poisoning), what I call TIFFS is
+very Unixy.  It is a file system with a hierarchical directory tree structure
+and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics
+of "what is a file" and "what is a directory" are exactly the same as in UNIX;
+and TIFFS even supports symlinks, although that support is a little under-
+developed, and apparently no FFS symlinks were ever used in any production GSM
+device.  Thus the FFS implemented in TI-based GSM devices (modems and
+"dumbphones") is really no different from, for example, JFFS2 in embedded Linux
+systems.
+
+(The only traditional UNIX file system features which are missing in TIFFS are
+ the creation/modification/access timestamps and the ownership/permission
+ fields.)
+
+The FFS in a GSM device typically stores two kinds of content:
+
+* Factory data: IMEI, RF calibration values, device make/model/revision
+  ID strings etc.  These files are expected to be programmed on the factory
+  production line and not changed afterward.
+
+* Dynamic data written into the FFS in normal device operation: when you use a
+  "dumbphone" running TI-based firmware, every time you store something "on the
+  phone" or in "non-volatile memory", that item is actually stored in the FFS.
+  (Where else, if you think of it?)  That includes contacts and received SMS
+  stored "on the phone" instead of the SIM, any selections you make in the
+  settings/preferences menus which persist across reboots (power cycles), call
+  history etc.
+
+It needs to be noted that the "dynamic data" aspect of FFS usage applies not
+only to complete phones, but also to modems like the one used in the GTA01/02.
+One would naively think that non-volatile storage of data in flash outside of
+factory programming would be needed only in a device with its own UI, and that
+a modem subservient to external AT commands would be completely stateless
+across reboot/power cycles; but that is not the case in actuality.  TI's GSM
+firmwares, including the Openmoko ones (the "standard" mokoN), are designed to
+always "mount" their FFS with read/write access; TI's FFS implementation in the
+firmware has no concept of a "read-only mount".
+
+I am still investigating just what kinds of data are routinely written into the
+non-volatile FFS by the firmware in normal operation on devices like the GTA0x
+modem, but there most definitely are some.
+
+There is no hard separation between "static" and "dynamic" data in the file
+system structure; TIFFS is thus akin to an embedded Linux system with just a
+single root file system containing both "static" files like userland binaries
+and "dynamic" ones like configuration files under /etc which the user is
+expected to edit with vi after logging into the box, or log and similar files
+created by the system itself under /var, for example.
+
+Where it lives
+==============
+
+The type of flash memory used in Calypso GSM modems and "dumbphones" is called
+NOR flash.  This NOR flash memory is physically divided (by the design of the
+flash chip itself) into units called "sectors" or more descriptively, erase
+blocks.  The typical NOR flash sector size (in Calypso GSM devices) ranges from
+64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the
+S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total).
+The key physical property is that any bit may be changed from a '1' to a '0' at
+any time, in any combination, but resetting of '0' bits back to ones can be
+done only on the granularity of these largish sectors, in an operation called
+"sector erase".
+
+The location of TIFFS within the flash memory of a given GSM device is defined
+by the firmware design of that device, but is always some integral number of
+contiguous flash sectors.  Some examples:
+
+* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at
+  flash offset 0x380000.
+
+* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies
+  18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning
+  of the 2nd flash chip select (0x02000000 in the ARM7 address space).
+
+* The smallest real FFS configuration called for by the table in dev.c in TI's
+  original Leonardo fw source is 3 sectors of 64 KiB each; the same table also
+  sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in
+  RAM without real flash).
+
+* The largest FFS configuration that has been envisioned by the original
+  designers seems to be somewhere around 128 sectors.
+
+Each flash sector used for TIFFS begins with this 6-byte signature:
+
+46 66 73 23 10 02
+
+The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the
+format version number of 0x0210 in little-endian byte order.  The following two
+bytes give a count of how many times that sector has been erased and rewritten
+(FF FF in "fresh" or "virgin" FFS images), and the following byte indicates
+that block's role and status in the FFS life cycle.
+
+How it works
+============
+
+Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to
+recover gracefully from any possible power failure or crash: one can yank the
+battery from the GSM device (or induce a firmware crash) at the most mis-
+opportune moment in the middle of an FFS write operation, and the FFS is
+expected to recover on the next boot cycle.  I won't be able to document here
+all gory details of exactly how this goal is achieved, partly because I haven't
+studied the code to the requisite level of depth myself yet, but all of the
+responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source
+tree; feel free to study it.
+
+In its "normal" or "clean" state (i.e., when not in the middle of a write
+operation or recovery from an ungracefully interrupted one), a TIFFS instance
+consists of the following 3 types of blocks:
+
+* One block containing inode records, indicated by AB in its type/flags/status
+  byte in the block header;
+* N-2 blocks (where N is the total number of flash sectors allocated for the
+  FFS) containing (or waiting to be filled with) data chunks - indicated by BD
+  in the type/flags/status byte;
+* One "free" block, indicated by BF - destined to become a new AB or a new BD
+  at some point.
+
+Each object written into the FFS (file, directory or symlink) consists of a
+16-byte inode record written into the AB block and a data chunk written into
+one of the BD blocks.  The data chunk includes the name of the object, hence
+one is required even for directories.  Data chunks are contiguous, uncompressed,
+and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS
+configuration.  Files larger than this limit are stored in a "segmented" form,
+giving rise to a 4th inode or object type (after file, directory and symlink):
+segment.  Each segment of a segmented file consists of not only a data chunk,
+but also an inode record for the segment, which gives the location of the data
+chunk and ties the segment object into the overall FFS structure, making it
+accessible.
+
+Because aside from complete sector erasure, flash memory bits can only
+transition from '1' to '0' but not the other way around, overwriting an existing
+file with some new content (an operation which any reasonable file system must
+implement in some way) cannot be done in place.  Instead like most flash file
+systems, TIFFS implements this common operation by writing the new version of
+the file to a new location (previously blank flash) and then invalidating the
+old version - and doing all that while keeping in mind the possibility of an
+ungraceful crash or powerdown at any moment, and the requirement of recovering
+gracefully from any such event.
+
+Of course as an FFS receives more write activity, even if one keeps overwriting
+some existing files with new content of the same size, without adding to the
+visible total content size (think du(1) command), eventually all remaining blank
+flash space will fill up.  At that point (or at some earlier point, depending
+on the FFS design and/or configuration) the FFS has to invoke a compaction or
+reclamation or garbage collection procedure: any "mixed" blocks containing both
+valid and stale data are transitioned into a "stale-only" state by having the
+active data moved to a new block, and then the "all stale" blocks are subjected
+to sector erasure, becoming new blank sectors.  The logic responsible for these
+operations once again needs to be resilient to the possibility of a crash or
+powerdown occurring at the most mis-opportune moment, and it also needs to
+implement flash wear leveling: there is a physical limit to how many times a
+given flash sector can be erased and rewritten before it goes bad.
+
+All of the above are common and well-known principles, successfully implemented
+in well-known flash file systems such as JFFS2 in Linux.  TIFFS is absolutely
+no different in this regard; for the implementation details, read the source
+code.
+
+How this FFS comes into being
+=============================
+
+(This section is only relevant to you if you plan on physically producing your
+ own GSM phones or modems on your own factory production line, like this author
+ fancies doing in the not-too-distant future, or if you simply enjoy knowing
+ how it is done.)
+
+To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the
+embedded Linux world, which would produce a TIFFS image complete with some
+initial directory and file content "in vitro".  Instead it appears that the FFS
+instances found in shipped products such as Openmoko phones have been created
+"in vivo" by TI's firmware running on the device itself during the "production
+test" phase.
+
+The process seems to go like this:
+
+* When the printed circuit board is physically populated with components such
+  as the Calypso chip and the flash chip, the latter can be blank - if the
+  board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM
+  (Openmoko and Pirelli both good on this one, but shame on Compal!), there is
+  no need to preprogram the flash chip with anything prior to populating it on
+  the board, and the device remains fully unbrickable at all times afterward.
+
+* When the assembled board is powered up for the first time, with completely
+  blank flash, the Calypso boot ROM will sit there and patiently wait for a
+  code download on either of its two UARTs.
+
+* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's
+  fc-loadtool free replacement, the factory production station loads the main
+  firmware image into the flash.  Note, it is just the firmware image at this
+  step, and the FFS sectors remain blank.
+
+* The board is commanded to reboot (or power-cycled), and the firmware image
+  boots for the first time.
+
+* TI's FFS implementation code in their standard firmware reacts to all blank
+  flash in the FFS sectors as follows: it performs what they call the preformat
+  operation, writing the TIFFS signature and a BF state byte into every FFS
+  sector, but the main "format" operation, which sets up the AB/BD block roles,
+  creates the root inode and makes the FFS ready to accept the creation of its
+  first directories and files, is not done automatically.
+
+In order to perform the FFS format operation and then fill the new FFS with
+whatever directories and files are deemed needed to be present in "fresh"
+shipping products, the factory production station connects to the just-booted
+firmware running on the target via the RVT/ETM protocol (see the RVTMUX
+write-up), and sends "test mode" commands to this running firmware.  These
+"FFS test mode" (or TMFFS) commands include the format operation, an mkdir
+operation to create directories, and a "file write" operation akin to doing
+'cat > /dir/whatever/file', creating files in FFS and storing any desired data
+in them.
+
+The IMEI is assigned and written into FFS in this process, but it is not the
+only data item that will be unique for each individual device made.  Much more
+important are the RF calibration values: I have yet to learn exactly what is
+being (or needs to be) measured, how these measurements are performed (under
+what conditions; what external test equipment is needed), and how these measured
+and recorded RF calibration values affect GSM device operation, but this TI
+presentation gives some clues:
+
+ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf
+
+All of these calibration values are stored in a bunch of files under the
+/gsm/rf subtree, and these files seem to be "owned" by the L1 code.  The latter
+has RAM data structures which correspond to these files; upon normal boot the
+initialization code looks in FFS, and if it finds any of the RF calibration
+files, it reads each present file into the corresponding RAM data structure,
+overwriting the compiled-in defaults.  It appears (slightly uncertain because I
+have not yet reintegrated the code in question into our own gsm-fw) that the RF
+calibration files in FFS come into being as follows:
+
+* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the
+  measurements and stores results in its RAM data structures as commanded by
+  the production test station through the "test mode" interface;
+
+* A final test mode command directs the above L1 code to write its RAM data
+  structures into FFS.
+
+Once I actually learn this RF calibration process properly in connection with
+building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what
+it would take to recreate these RF calibration values if they are lost.  But
+until then the only advice I can give is to make a backup copy of your modem
+FFS with fc-loadtool, and to save it securely.
+
+FreeCalypso support for TIFFS
+=============================
+
+Aside from implementing and using it in our own gsm-fw, FreeCalypso will offer
+the following support for TIFFS:
+
+1. A tiffs host utility is being written which will allow a user to list and
+   extract content from saved FFS images (read out of flash with fc-loadtool)
+   "in vitro".  It will be a restructured and (hopefully) improved version of
+   the mpffs-* tools released back in the summer of SE52 (A.D. 2013); the latter
+   already perform the advertised function, but I seek to integrate some other
+   functionality which I developed in an ad hoc side project ("pirollback"),
+   and I'm taking the opportunity to make the MPFFS->TIFFS renaming.
+
+(The mpffs-* tools mentioned above have been written based on reverse eng only,
+ before I found any source code for TI's FFS firmware implementation!  Now that
+ we have the source, some terminological and other inevitable misunderstandings
+ can be corrected.)
+
+2. A number of FC tools may be strung together into a kit for editing the FFS
+   content of a GSM device, e.g., for changing the IMEI.  The following pieces
+   will be involved:
+
+* What is destined to eventually become our totally free GSM fw (the gsm-fw
+  source subtree at the top of freecalypso-sw) does not contain any of the
+  actual GSM protocol stack (or even L1) functionality yet, but it already
+  contains both the FFS code and those components (ETM and TMFFS[12]) which
+  are needed for interfacing an external "test mode shell" to this FFS
+  implementation through the RVTMUX interface.  And when our gsm-fw does gain
+  the actual GSM functionality, the ability to build a minimal FFS+ETM-only
+  configuration will still be retained.
+
+* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs
+  entirely from RAM via fc-xram, no flashing), and run on a physical device
+  such as the GTA0x GSM modem via the fc-xram host utility;
+
+* After loading the ramImage, fc-xram will immediately exec our rvinterf host
+  utility described in the RVTMUX write-up;
+
+* Once the GSM device is running what is effectively an FFS editing agent out
+  of RAM, accessed via rvinterf over the serial channel, the user will be able
+  to run fc-tmsh (or perhaps the FFS operations will be implemented in some
+  other utility, we'll see), and that "test mode shell" will provide commands
+  for writing things to FFS exactly like one would do in the factory production
+  line environment for which TI taylored their tools.
+
+The "in vivo" method of editing the FFS content of a GSM device described above
+will probably sound very convoluted, and you may find yourself asking for a way
+to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit
+that image "in vitro" with some utility on your PC, and then use fc-loadtool
+again to program it back into your device.  But consider that an "in vitro" FFS
+modification would involve erasing and rewriting all sectors of your FFS,
+whereas an "in vivo" modification of some small file like the IMEI would be
+just a short flash write operation without any erasures at all, i.e., kinder
+on the flash.
+
+In any case, the "in vivo" method will definitely be available soon because all
+of the components involved therein are also needed for other development uses
+in the FreeCalypso project, whereas developing a fully-functional "in vitro"
+alternative (one that can create an FFS image "de novo" from a tree of files
+and directories a la mkfs.jffs2, or add new files to an existing TIFFS image
+etc) would be a good amount of extra work which we otherwise don't need - hence
+the latter is not very likely to be written any time soon.
+
+However, if the "in vitro" modification you seek is something trivial like
+changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without
+changing its length, you will be able to use the "in vitro, read-only" tiffs
+host utility to find the exact byte location of the file data within the TIFFS
+image, and use your favourite hex editor to whack whatever new byte content you
+like at that offset.