changeset 250:3d88461d8284

beginning of TIFFS IVA documentation
author Michael Spacefalcon <msokolov@ivan.Harhan.ORG>
date Fri, 31 Jan 2014 07:37:39 +0000
parents 66a6f1652909
children 63cd64625597
files README doc/TIFFS doc/TIFFS-Overview ffstools/README ffstools/Usage
diffstat 5 files changed, 487 insertions(+), 370 deletions(-) [+]
line wrap: on
line diff
--- a/README	Mon Jan 27 08:05:43 2014 +0000
+++ b/README	Fri Jan 31 07:37:39 2014 +0000
@@ -6,8 +6,15 @@
 		modems and basic phones, consisting of Nucleus RTOS, RiViera
 		and GPF frameworks, TI's BSP/driver and L1 code, Condat G23
 		GSM/GPRS protocol stack, AT command interpreter and UI layers.
-		The current code is in the earliest embryonic stages of
-		development, and does not do anything useful yet.
+
+		At the present, none of the actual GSM code (L1 or G23) has
+		been integrated yet, but most of the underlying RTOS environment
+		is present and working: one can exercise RVT, ETM and FFS.
+		L1 integration will begin soon.
+
+ffstools	Here you will find tools for "in vitro" examination of FFS
+		(flash file system) images read out of TI-based GSM devices.
+		See doc/TIFFS-Overview for more information.
 
 loadtools	The suite of tools which run on a development host machine
 		(normally PC/Linux desktop or laptop) and communicate with
--- a/doc/TIFFS	Mon Jan 27 08:05:43 2014 +0000
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,368 +0,0 @@
-All TI GSM firmwares known to this author (FreeCalypso developer Space Falcon)
-implement some kind of flash file system, or FFS.  Several different FFS code
-implementations, and correspondingly several different on-flash data formats,
-have been used throughout the history of TI's involvement in the wireless
-terminal business.  The FFS incarnation of primary interest to the FreeCalypso
-project is the one invented by Mads Meisner-Jensen at TI in the early 2000s
-(at least according to the comments in the sources available to us), and it is
-relevant to us in the following ways:
-
-* When targeting the GSM modem in Openmoko's GTA01/02 smartphones, we need to
-  work with the original FFS from the factory (call it MokoFFS), the same FFS
-  as used by the mokoN firmwares: this FFS contains the IMEI and the RF
-  calibration values from the factory, which we most certainly don't want to go
-  without.
-
-* The Leonardo firmware semi-src which we are using as the reference for
-  building our own full source, multi-target GSM fw contains a turnkey-working
-  implementation of this very FFS, using the on-flash format in question and
-  providing run-time APIs expected by the rest of the GSM fw suite.  Following
-  the principle of ``if it ain't broke, don't fix it'', we can use this FFS not
-  only on the gtamodem target, but also on other targets, including those where
-  we would be starting from a blank state and thus have the freedom to use
-  whatever FFS we like.
-
-* The original proprietary fw on the Pirelli DP-L10 phone also happens to use
-  an FFS in the same format.  Pirelli's FFS does *not* contain the IMEI or any
-  of the RF calibration values though, and trying to reuse it directly for our
-  own FC GSM fw seems to be more trouble than benefit - so we'll probably have
-  our fw start with a blank TIFFS instead - but there is still insight to be
-  gained from in-vitro examination of captured Pirelli FFS images.
-
-Naming
-======
-
-I have previously referred to the FFS format in question as Mokopir-FFS or
-MPFFS, from "Moko" and "Pirelli".  I was originally hesitant to call it TIFFS,
-as lacking the source code, I had no way of knowing whether the FFS format and
-implementation were of TI's own invention, or something that TI licensed as a
-black box from one of their many proprietary software partners.  (I was unable
-to identify it as any well-known, industry-standard FFS format, but absence of
-evidence is not evidence of absence.)  But now that we have TI's original source
-code which implements this FFS (first the MV100-0.1.rar source, then the full
-Leonardo one), complete with comments and a HISTORY file, we know that our FFS
-was invented and implemented by someone named Mads Meisner-Jensen at TI - I'm
-guessing in the SSA group in Nice, France.
-
-I am now making a naming transition from MPFFS to TIFFS: there is really no
-link between this FFS format and the Openmoko+Pirelli duo, other than the
-happenstance of me having first encountered this FFS on these two GSM device
-brands, and the name TIFFS is more neutrally-descriptive.
-
-What it is
-==========
-
-In a rare departure from TI's norm (most of TI's GSM firmware and associated
-development tools suffer from heavy Windows poisoning), what I call TIFFS is
-very Unixy.  It is a file system with a hierarchical directory tree structure
-and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics
-of "what is a file" and "what is a directory" are exactly the same as in UNIX;
-and TIFFS even supports symlinks, although that support is a little under-
-developed, and apparently no FFS symlinks were ever used in any production GSM
-device.  Thus the FFS implemented in TI-based GSM devices (modems and
-"dumbphones") is really no different from, for example, JFFS2 in embedded Linux
-systems.
-
-(The only traditional UNIX file system features which are missing in TIFFS are
- the creation/modification/access timestamps and the ownership/permission
- fields.)
-
-The FFS in a GSM device typically stores two kinds of content:
-
-* Factory data: IMEI, RF calibration values, device make/model/revision
-  ID strings etc.  These files are expected to be programmed on the factory
-  production line and not changed afterward.
-
-* Dynamic data written into the FFS in normal device operation: when you use a
-  "dumbphone" running TI-based firmware, every time you store something "on the
-  phone" or in "non-volatile memory", that item is actually stored in the FFS.
-  (Where else, if you think of it?)  That includes contacts and received SMS
-  stored "on the phone" instead of the SIM, any selections you make in the
-  settings/preferences menus which persist across reboots (power cycles), call
-  history etc.
-
-It needs to be noted that the "dynamic data" aspect of FFS usage applies not
-only to complete phones, but also to modems like the one used in the GTA01/02.
-One would naively think that non-volatile storage of data in flash outside of
-factory programming would be needed only in a device with its own UI, and that
-a modem subservient to external AT commands would be completely stateless
-across reboot/power cycles; but that is not the case in actuality.  TI's GSM
-firmwares, including the Openmoko ones (the "standard" mokoN), are designed to
-always "mount" their FFS with read/write access; TI's FFS implementation in the
-firmware has no concept of a "read-only mount".
-
-I am still investigating just what kinds of data are routinely written into the
-non-volatile FFS by the firmware in normal operation on devices like the GTA0x
-modem, but there most definitely are some.
-
-There is no hard separation between "static" and "dynamic" data in the file
-system structure; TIFFS is thus akin to an embedded Linux system with just a
-single root file system containing both "static" files like userland binaries
-and "dynamic" ones like configuration files under /etc which the user is
-expected to edit with vi after logging into the box, or log and similar files
-created by the system itself under /var, for example.
-
-Where it lives
-==============
-
-The type of flash memory used in Calypso GSM modems and "dumbphones" is called
-NOR flash.  This NOR flash memory is physically divided (by the design of the
-flash chip itself) into units called "sectors" or more descriptively, erase
-blocks.  The typical NOR flash sector size (in Calypso GSM devices) ranges from
-64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the
-S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total).
-The key physical property is that any bit may be changed from a '1' to a '0' at
-any time, in any combination, but resetting of '0' bits back to ones can be
-done only on the granularity of these largish sectors, in an operation called
-"sector erase".
-
-The location of TIFFS within the flash memory of a given GSM device is defined
-by the firmware design of that device, but is always some integral number of
-contiguous flash sectors.  Some examples:
-
-* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at
-  flash offset 0x380000.
-
-* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies
-  18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning
-  of the 2nd flash chip select (0x02000000 in the ARM7 address space).
-
-* The smallest real FFS configuration called for by the table in dev.c in TI's
-  original Leonardo fw source is 3 sectors of 64 KiB each; the same table also
-  sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in
-  RAM without real flash).
-
-* The largest FFS configuration that has been envisioned by the original
-  designers seems to be somewhere around 128 sectors.
-
-Each flash sector used for TIFFS begins with this 6-byte signature:
-
-46 66 73 23 10 02
-
-The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the
-format version number of 0x0210 in little-endian byte order.  The following two
-bytes give a count of how many times that sector has been erased and rewritten
-(FF FF in "fresh" or "virgin" FFS images), and the following byte indicates
-that block's role and status in the FFS life cycle.
-
-How it works
-============
-
-Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to
-recover gracefully from any possible power failure or crash: one can yank the
-battery from the GSM device (or induce a firmware crash) at the most mis-
-opportune moment in the middle of an FFS write operation, and the FFS is
-expected to recover on the next boot cycle.  I won't be able to document here
-all gory details of exactly how this goal is achieved, partly because I haven't
-studied the code to the requisite level of depth myself yet, but all of the
-responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source
-tree; feel free to study it.
-
-In its "normal" or "clean" state (i.e., when not in the middle of a write
-operation or recovery from an ungracefully interrupted one), a TIFFS instance
-consists of the following 3 types of blocks:
-
-* One block containing inode records, indicated by AB in its type/flags/status
-  byte in the block header;
-* N-2 blocks (where N is the total number of flash sectors allocated for the
-  FFS) containing (or waiting to be filled with) data chunks - indicated by BD
-  in the type/flags/status byte;
-* One "free" block, indicated by BF - destined to become a new AB or a new BD
-  at some point.
-
-Each object written into the FFS (file, directory or symlink) consists of a
-16-byte inode record written into the AB block and a data chunk written into
-one of the BD blocks.  The data chunk includes the name of the object, hence
-one is required even for directories.  Data chunks are contiguous, uncompressed,
-and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS
-configuration.  Files larger than this limit are stored in a "segmented" form,
-giving rise to a 4th inode or object type (after file, directory and symlink):
-segment.  Each segment of a segmented file consists of not only a data chunk,
-but also an inode record for the segment, which gives the location of the data
-chunk and ties the segment object into the overall FFS structure, making it
-accessible.
-
-Because aside from complete sector erasure, flash memory bits can only
-transition from '1' to '0' but not the other way around, overwriting an existing
-file with some new content (an operation which any reasonable file system must
-implement in some way) cannot be done in place.  Instead like most flash file
-systems, TIFFS implements this common operation by writing the new version of
-the file to a new location (previously blank flash) and then invalidating the
-old version - and doing all that while keeping in mind the possibility of an
-ungraceful crash or powerdown at any moment, and the requirement of recovering
-gracefully from any such event.
-
-Of course as an FFS receives more write activity, even if one keeps overwriting
-some existing files with new content of the same size, without adding to the
-visible total content size (think du(1) command), eventually all remaining blank
-flash space will fill up.  At that point (or at some earlier point, depending
-on the FFS design and/or configuration) the FFS has to invoke a compaction or
-reclamation or garbage collection procedure: any "mixed" blocks containing both
-valid and stale data are transitioned into a "stale-only" state by having the
-active data moved to a new block, and then the "all stale" blocks are subjected
-to sector erasure, becoming new blank sectors.  The logic responsible for these
-operations once again needs to be resilient to the possibility of a crash or
-powerdown occurring at the most mis-opportune moment, and it also needs to
-implement flash wear leveling: there is a physical limit to how many times a
-given flash sector can be erased and rewritten before it goes bad.
-
-All of the above are common and well-known principles, successfully implemented
-in well-known flash file systems such as JFFS2 in Linux.  TIFFS is absolutely
-no different in this regard; for the implementation details, read the source
-code.
-
-How this FFS comes into being
-=============================
-
-(This section is only relevant to you if you plan on physically producing your
- own GSM phones or modems on your own factory production line, like this author
- fancies doing in the not-too-distant future, or if you simply enjoy knowing
- how it is done.)
-
-To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the
-embedded Linux world, which would produce a TIFFS image complete with some
-initial directory and file content "in vitro".  Instead it appears that the FFS
-instances found in shipped products such as Openmoko phones have been created
-"in vivo" by TI's firmware running on the device itself during the "production
-test" phase.
-
-The process seems to go like this:
-
-* When the printed circuit board is physically populated with components such
-  as the Calypso chip and the flash chip, the latter can be blank - if the
-  board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM
-  (Openmoko and Pirelli both good on this one, but shame on Compal!), there is
-  no need to preprogram the flash chip with anything prior to populating it on
-  the board, and the device remains fully unbrickable at all times afterward.
-
-* When the assembled board is powered up for the first time, with completely
-  blank flash, the Calypso boot ROM will sit there and patiently wait for a
-  code download on either of its two UARTs.
-
-* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's
-  fc-loadtool free replacement, the factory production station loads the main
-  firmware image into the flash.  Note, it is just the firmware image at this
-  step, and the FFS sectors remain blank.
-
-* The board is commanded to reboot (or power-cycled), and the firmware image
-  boots for the first time.
-
-* TI's FFS implementation code in their standard firmware reacts to all blank
-  flash in the FFS sectors as follows: it performs what they call the preformat
-  operation, writing the TIFFS signature and a BF state byte into every FFS
-  sector, but the main "format" operation, which sets up the AB/BD block roles,
-  creates the root inode and makes the FFS ready to accept the creation of its
-  first directories and files, is not done automatically.
-
-In order to perform the FFS format operation and then fill the new FFS with
-whatever directories and files are deemed needed to be present in "fresh"
-shipping products, the factory production station connects to the just-booted
-firmware running on the target via the RVT/ETM protocol (see the RVTMUX
-write-up), and sends "test mode" commands to this running firmware.  These
-"FFS test mode" (or TMFFS) commands include the format operation, an mkdir
-operation to create directories, and a "file write" operation akin to doing
-'cat > /dir/whatever/file', creating files in FFS and storing any desired data
-in them.
-
-The IMEI is assigned and written into FFS in this process, but it is not the
-only data item that will be unique for each individual device made.  Much more
-important are the RF calibration values: I have yet to learn exactly what is
-being (or needs to be) measured, how these measurements are performed (under
-what conditions; what external test equipment is needed), and how these measured
-and recorded RF calibration values affect GSM device operation, but this TI
-presentation gives some clues:
-
-ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf
-
-All of these calibration values are stored in a bunch of files under the
-/gsm/rf subtree, and these files seem to be "owned" by the L1 code.  The latter
-has RAM data structures which correspond to these files; upon normal boot the
-initialization code looks in FFS, and if it finds any of the RF calibration
-files, it reads each present file into the corresponding RAM data structure,
-overwriting the compiled-in defaults.  It appears (slightly uncertain because I
-have not yet reintegrated the code in question into our own gsm-fw) that the RF
-calibration files in FFS come into being as follows:
-
-* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the
-  measurements and stores results in its RAM data structures as commanded by
-  the production test station through the "test mode" interface;
-
-* A final test mode command directs the above L1 code to write its RAM data
-  structures into FFS.
-
-Once I actually learn this RF calibration process properly in connection with
-building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what
-it would take to recreate these RF calibration values if they are lost.  But
-until then the only advice I can give is to make a backup copy of your modem
-FFS with fc-loadtool, and to save it securely.
-
-FreeCalypso support for TIFFS
-=============================
-
-Aside from implementing and using it in our own gsm-fw, FreeCalypso will offer
-the following support for TIFFS:
-
-1. A tiffs host utility is being written which will allow a user to list and
-   extract content from saved FFS images (read out of flash with fc-loadtool)
-   "in vitro".  It will be a restructured and (hopefully) improved version of
-   the mpffs-* tools released back in the summer of SE52 (A.D. 2013); the latter
-   already perform the advertised function, but I seek to integrate some other
-   functionality which I developed in an ad hoc side project ("pirollback"),
-   and I'm taking the opportunity to make the MPFFS->TIFFS renaming.
-
-(The mpffs-* tools mentioned above have been written based on reverse eng only,
- before I found any source code for TI's FFS firmware implementation!  Now that
- we have the source, some terminological and other inevitable misunderstandings
- can be corrected.)
-
-2. A number of FC tools may be strung together into a kit for editing the FFS
-   content of a GSM device, e.g., for changing the IMEI.  The following pieces
-   will be involved:
-
-* What is destined to eventually become our totally free GSM fw (the gsm-fw
-  source subtree at the top of freecalypso-sw) does not contain any of the
-  actual GSM protocol stack (or even L1) functionality yet, but it already
-  contains both the FFS code and those components (ETM and TMFFS[12]) which
-  are needed for interfacing an external "test mode shell" to this FFS
-  implementation through the RVTMUX interface.  And when our gsm-fw does gain
-  the actual GSM functionality, the ability to build a minimal FFS+ETM-only
-  configuration will still be retained.
-
-* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs
-  entirely from RAM via fc-xram, no flashing), and run on a physical device
-  such as the GTA0x GSM modem via the fc-xram host utility;
-
-* After loading the ramImage, fc-xram will immediately exec our rvinterf host
-  utility described in the RVTMUX write-up;
-
-* Once the GSM device is running what is effectively an FFS editing agent out
-  of RAM, accessed via rvinterf over the serial channel, the user will be able
-  to run fc-tmsh (or perhaps the FFS operations will be implemented in some
-  other utility, we'll see), and that "test mode shell" will provide commands
-  for writing things to FFS exactly like one would do in the factory production
-  line environment for which TI taylored their tools.
-
-The "in vivo" method of editing the FFS content of a GSM device described above
-will probably sound very convoluted, and you may find yourself asking for a way
-to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit
-that image "in vitro" with some utility on your PC, and then use fc-loadtool
-again to program it back into your device.  But consider that an "in vitro" FFS
-modification would involve erasing and rewriting all sectors of your FFS,
-whereas an "in vivo" modification of some small file like the IMEI would be
-just a short flash write operation without any erasures at all, i.e., kinder
-on the flash.
-
-In any case, the "in vivo" method will definitely be available soon because all
-of the components involved therein are also needed for other development uses
-in the FreeCalypso project, whereas developing a fully-functional "in vitro"
-alternative (one that can create an FFS image "de novo" from a tree of files
-and directories a la mkfs.jffs2, or add new files to an existing TIFFS image
-etc) would be a good amount of extra work which we otherwise don't need - hence
-the latter is not very likely to be written any time soon.
-
-However, if the "in vitro" modification you seek is something trivial like
-changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without
-changing its length, you will be able to use the "in vitro, read-only" tiffs
-host utility to find the exact byte location of the file data within the TIFFS
-image, and use your favourite hex editor to whack whatever new byte content you
-like at that offset.
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/TIFFS-Overview	Fri Jan 31 07:37:39 2014 +0000
@@ -0,0 +1,363 @@
+All TI GSM firmwares known to this author (FreeCalypso developer Space Falcon)
+implement some kind of flash file system, or FFS.  Several different FFS code
+implementations, and correspondingly several different on-flash data formats,
+have been used throughout the history of TI's involvement in the wireless
+terminal business.  The FFS incarnation of primary interest to the FreeCalypso
+project is the one invented by Mads Meisner-Jensen at TI in the early 2000s
+(at least according to the comments in the sources available to us), and it is
+relevant to us in the following ways:
+
+* When targeting the GSM modem in Openmoko's GTA01/02 smartphones, we need to
+  work with the original FFS from the factory (call it MokoFFS), the same FFS
+  as used by the mokoN firmwares: this FFS contains the IMEI and the RF
+  calibration values from the factory, which we most certainly don't want to go
+  without.
+
+* The Leonardo firmware semi-src which we are using as the reference for
+  building our own full source, multi-target GSM fw contains a turnkey-working
+  implementation of this very FFS, using the on-flash format in question and
+  providing run-time APIs expected by the rest of the GSM fw suite.  Following
+  the principle of ``if it ain't broke, don't fix it'', we can use this FFS not
+  only on the gtamodem target, but also on other targets, including those where
+  we would be starting from a blank state and thus have the freedom to use
+  whatever FFS we like.
+
+* The original proprietary fw on the Pirelli DP-L10 phone also happens to use
+  an FFS in the same format.  Pirelli's FFS does *not* contain the IMEI or any
+  of the RF calibration values though, and trying to reuse it directly for our
+  own FC GSM fw seems to be more trouble than benefit - so we'll probably have
+  our fw start with a blank TIFFS instead - but there is still insight to be
+  gained from in-vitro examination of captured Pirelli FFS images.
+
+Naming
+======
+
+I have previously referred to the FFS format in question as Mokopir-FFS or
+MPFFS, from "Moko" and "Pirelli".  I was originally hesitant to call it TIFFS,
+as lacking the source code, I had no way of knowing whether the FFS format and
+implementation were of TI's own invention, or something that TI licensed as a
+black box from one of their many proprietary software partners.  (I was unable
+to identify it as any well-known, industry-standard FFS format, but absence of
+evidence is not evidence of absence.)  But now that we have TI's original source
+code which implements this FFS (first the MV100-0.1.rar source, then the full
+Leonardo one), complete with comments and a HISTORY file, we know that our FFS
+was invented and implemented by someone named Mads Meisner-Jensen at TI - I'm
+guessing in the SSA group in Nice, France.
+
+I am now making a naming transition from MPFFS to TIFFS: there is really no
+link between this FFS format and the Openmoko+Pirelli duo, other than the
+happenstance of me having first encountered this FFS on these two GSM device
+brands, and the name TIFFS is more neutrally-descriptive.
+
+What it is
+==========
+
+In a rare departure from TI's norm (most of TI's GSM firmware and associated
+development tools suffer from heavy Windows poisoning), what I call TIFFS is
+very Unixy.  It is a file system with a hierarchical directory tree structure
+and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics
+of "what is a file" and "what is a directory" are exactly the same as in UNIX;
+and TIFFS even supports symlinks, although that support is a little under-
+developed, and apparently no FFS symlinks were ever used in any production GSM
+device.  Thus the FFS implemented in TI-based GSM devices (modems and
+"dumbphones") is really no different from, for example, JFFS2 in embedded Linux
+systems.
+
+(The only traditional UNIX file system features which are missing in TIFFS are
+ the creation/modification/access timestamps and the ownership/permission
+ fields.)
+
+The FFS in a GSM device typically stores two kinds of content:
+
+* Factory data: IMEI, RF calibration values, device make/model/revision
+  ID strings etc.  These files are expected to be programmed on the factory
+  production line and not changed afterward.
+
+* Dynamic data written into the FFS in normal device operation: when you use a
+  "dumbphone" running TI-based firmware, every time you store something "on the
+  phone" or in "non-volatile memory", that item is actually stored in the FFS.
+  (Where else, if you think of it?)  That includes contacts and received SMS
+  stored "on the phone" instead of the SIM, any selections you make in the
+  settings/preferences menus which persist across reboots (power cycles), call
+  history etc.
+
+It needs to be noted that the "dynamic data" aspect of FFS usage applies not
+only to complete phones, but also to modems like the one used in the GTA01/02.
+One would naively think that non-volatile storage of data in flash outside of
+factory programming would be needed only in a device with its own UI, and that
+a modem subservient to external AT commands would be completely stateless
+across reboot/power cycles; but that is not the case in actuality.  TI's GSM
+firmwares, including the Openmoko ones (the "standard" mokoN), are designed to
+always "mount" their FFS with read/write access; TI's FFS implementation in the
+firmware has no concept of a "read-only mount".
+
+I am still investigating just what kinds of data are routinely written into the
+non-volatile FFS by the firmware in normal operation on devices like the GTA0x
+modem, but there most definitely are some.
+
+There is no hard separation between "static" and "dynamic" data in the file
+system structure; TIFFS is thus akin to an embedded Linux system with just a
+single root file system containing both "static" files like userland binaries
+and "dynamic" ones like configuration files under /etc which the user is
+expected to edit with vi after logging into the box, or log and similar files
+created by the system itself under /var, for example.
+
+Where it lives
+==============
+
+The type of flash memory used in Calypso GSM modems and "dumbphones" is called
+NOR flash.  This NOR flash memory is physically divided (by the design of the
+flash chip itself) into units called "sectors" or more descriptively, erase
+blocks.  The typical NOR flash sector size (in Calypso GSM devices) ranges from
+64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the
+S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total).
+The key physical property is that any bit may be changed from a '1' to a '0' at
+any time, in any combination, but resetting of '0' bits back to ones can be
+done only on the granularity of these largish sectors, in an operation called
+"sector erase".
+
+The location of TIFFS within the flash memory of a given GSM device is defined
+by the firmware design of that device, but is always some integral number of
+contiguous flash sectors.  Some examples:
+
+* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at
+  flash offset 0x380000.
+
+* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies
+  18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning
+  of the 2nd flash chip select (0x02000000 in the ARM7 address space).
+
+* The smallest real FFS configuration called for by the table in dev.c in TI's
+  original Leonardo fw source is 3 sectors of 64 KiB each; the same table also
+  sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in
+  RAM without real flash).
+
+* The largest FFS configuration that has been envisioned by the original
+  designers seems to be somewhere around 128 sectors.
+
+Each flash sector used for TIFFS begins with this 6-byte signature:
+
+46 66 73 23 10 02
+
+The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the
+format version number of 0x0210 in little-endian byte order.  The following two
+bytes give a count of how many times that sector has been erased and rewritten
+(FF FF in "fresh" or "virgin" FFS images), and the following byte indicates
+that block's role and status in the FFS life cycle.
+
+How it works
+============
+
+Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to
+recover gracefully from any possible power failure or crash: one can yank the
+battery from the GSM device (or induce a firmware crash) at the most mis-
+opportune moment in the middle of an FFS write operation, and the FFS is
+expected to recover on the next boot cycle.  I won't be able to document here
+all gory details of exactly how this goal is achieved, partly because I haven't
+studied the code to the requisite level of depth myself yet, but all of the
+responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source
+tree; feel free to study it.
+
+In its "normal" or "clean" state (i.e., when not in the middle of a write
+operation or recovery from an ungracefully interrupted one), a TIFFS instance
+consists of the following 3 types of blocks:
+
+* One block containing inode records, indicated by AB in its type/flags/status
+  byte in the block header;
+* N-2 blocks (where N is the total number of flash sectors allocated for the
+  FFS) containing (or waiting to be filled with) data chunks - indicated by BD
+  in the type/flags/status byte;
+* One "free" block, indicated by BF - destined to become a new AB or a new BD
+  at some point.
+
+Each object written into the FFS (file, directory or symlink) consists of a
+16-byte inode record written into the AB block and a data chunk written into
+one of the BD blocks.  The data chunk includes the name of the object, hence
+one is required even for directories.  Data chunks are contiguous, uncompressed,
+and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS
+configuration.  Files larger than this limit are stored in a "segmented" form,
+giving rise to a 4th inode or object type (after file, directory and symlink):
+segment.  Each segment of a segmented file consists of not only a data chunk,
+but also an inode record for the segment, which gives the location of the data
+chunk and ties the segment object into the overall FFS structure, making it
+accessible.
+
+Because aside from complete sector erasure, flash memory bits can only
+transition from '1' to '0' but not the other way around, overwriting an existing
+file with some new content (an operation which any reasonable file system must
+implement in some way) cannot be done in place.  Instead like most flash file
+systems, TIFFS implements this common operation by writing the new version of
+the file to a new location (previously blank flash) and then invalidating the
+old version - and doing all that while keeping in mind the possibility of an
+ungraceful crash or powerdown at any moment, and the requirement of recovering
+gracefully from any such event.
+
+Of course as an FFS receives more write activity, even if one keeps overwriting
+some existing files with new content of the same size, without adding to the
+visible total content size (think du(1) command), eventually all remaining blank
+flash space will fill up.  At that point (or at some earlier point, depending
+on the FFS design and/or configuration) the FFS has to invoke a compaction or
+reclamation or garbage collection procedure: any "mixed" blocks containing both
+valid and stale data are transitioned into a "stale-only" state by having the
+active data moved to a new block, and then the "all stale" blocks are subjected
+to sector erasure, becoming new blank sectors.  The logic responsible for these
+operations once again needs to be resilient to the possibility of a crash or
+powerdown occurring at the most mis-opportune moment, and it also needs to
+implement flash wear leveling: there is a physical limit to how many times a
+given flash sector can be erased and rewritten before it goes bad.
+
+All of the above are common and well-known principles, successfully implemented
+in well-known flash file systems such as JFFS2 in Linux.  TIFFS is absolutely
+no different in this regard; for the implementation details, read the source
+code.
+
+How this FFS comes into being
+=============================
+
+(This section is only relevant to you if you plan on physically producing your
+ own GSM phones or modems on your own factory production line, like this author
+ fancies doing in the not-too-distant future, or if you simply enjoy knowing
+ how it is done.)
+
+To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the
+embedded Linux world, which would produce a TIFFS image complete with some
+initial directory and file content "in vitro".  Instead it appears that the FFS
+instances found in shipped products such as Openmoko phones have been created
+"in vivo" by TI's firmware running on the device itself during the "production
+test" phase.
+
+The process seems to go like this:
+
+* When the printed circuit board is physically populated with components such
+  as the Calypso chip and the flash chip, the latter can be blank - if the
+  board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM
+  (Openmoko and Pirelli both good on this one, but shame on Compal!), there is
+  no need to preprogram the flash chip with anything prior to populating it on
+  the board, and the device remains fully unbrickable at all times afterward.
+
+* When the assembled board is powered up for the first time, with completely
+  blank flash, the Calypso boot ROM will sit there and patiently wait for a
+  code download on either of its two UARTs.
+
+* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's
+  fc-loadtool free replacement, the factory production station loads the main
+  firmware image into the flash.  Note, it is just the firmware image at this
+  step, and the FFS sectors remain blank.
+
+* The board is commanded to reboot (or power-cycled), and the firmware image
+  boots for the first time.
+
+* TI's FFS implementation code in their standard firmware reacts to all blank
+  flash in the FFS sectors as follows: it performs what they call the preformat
+  operation, writing the TIFFS signature and a BF state byte into every FFS
+  sector, but the main "format" operation, which sets up the AB/BD block roles,
+  creates the root inode and makes the FFS ready to accept the creation of its
+  first directories and files, is not done automatically.
+
+In order to perform the FFS format operation and then fill the new FFS with
+whatever directories and files are deemed needed to be present in "fresh"
+shipping products, the factory production station connects to the just-booted
+firmware running on the target via the RVT/ETM protocol (see the RVTMUX
+write-up), and sends "test mode" commands to this running firmware.  These
+"FFS test mode" (or TMFFS) commands include the format operation, an mkdir
+operation to create directories, and a "file write" operation akin to doing
+'cat > /dir/whatever/file', creating files in FFS and storing any desired data
+in them.
+
+The IMEI is assigned and written into FFS in this process, but it is not the
+only data item that will be unique for each individual device made.  Much more
+important are the RF calibration values: I have yet to learn exactly what is
+being (or needs to be) measured, how these measurements are performed (under
+what conditions; what external test equipment is needed), and how these measured
+and recorded RF calibration values affect GSM device operation, but this TI
+presentation gives some clues:
+
+ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf
+
+All of these calibration values are stored in a bunch of files under the
+/gsm/rf subtree, and these files seem to be "owned" by the L1 code.  The latter
+has RAM data structures which correspond to these files; upon normal boot the
+initialization code looks in FFS, and if it finds any of the RF calibration
+files, it reads each present file into the corresponding RAM data structure,
+overwriting the compiled-in defaults.  It appears (slightly uncertain because I
+have not yet reintegrated the code in question into our own gsm-fw) that the RF
+calibration files in FFS come into being as follows:
+
+* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the
+  measurements and stores results in its RAM data structures as commanded by
+  the production test station through the "test mode" interface;
+
+* A final test mode command directs the above L1 code to write its RAM data
+  structures into FFS.
+
+Once I actually learn this RF calibration process properly in connection with
+building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what
+it would take to recreate these RF calibration values if they are lost.  But
+until then the only advice I can give is to make a backup copy of your modem
+FFS with fc-loadtool, and to save it securely.
+
+FreeCalypso support for TIFFS
+=============================
+
+Aside from implementing and using it in our own gsm-fw, FreeCalypso offers
+the following support for TIFFS:
+
+1. We have a utility for "in vitro" examination of FFS images read out of GSM
+   devices with fc-loadtool.  This tiffs utility (along with mokoffs and pirffs
+   wrappers) lives in the ffstools top-level directory of the freecalypso-sw
+   source tree.  This TIFFS "in vitro analyzer" utility supplants the earlier
+   mpffs-* tools, and adds some additional examination functionality.  It is
+   strictly a "read only" tool, however - it is not designed for "in vitro"
+   editing" of TIFFS images.
+
+2. A number of FC tools may be strung together into a kit for editing the FFS
+   content of a GSM device, e.g., for changing the IMEI.  The following pieces
+   will be involved:
+
+* What is destined to eventually become our totally free GSM fw (the gsm-fw
+  source subtree at the top of freecalypso-sw) does not contain any of the
+  actual GSM protocol stack (or even L1) functionality yet, but it already
+  contains both the FFS code and those components (ETM and TMFFS[12]) which
+  are needed for interfacing an external "test mode shell" to this FFS
+  implementation through the RVTMUX interface.  And when our gsm-fw does gain
+  the actual GSM functionality, the ability to build a minimal FFS+ETM-only
+  configuration will still be retained.
+
+* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs
+  entirely from RAM via fc-xram, no flashing), and run on a physical device
+  such as the GTA0x GSM modem via the fc-xram host utility;
+
+* After loading the ramImage, fc-xram will immediately exec our rvinterf host
+  utility described in the RVTMUX write-up;
+
+* Once the GSM device is running what is effectively an FFS editing agent out
+  of RAM, accessed via rvinterf over the serial channel, the user will be able
+  to run fc-tmsh (or perhaps the FFS operations will be implemented in some
+  other utility, we'll see), and that "test mode shell" will provide commands
+  for writing things to FFS exactly like one would do in the factory production
+  line environment for which TI taylored their tools.
+
+The "in vivo" method of editing the FFS content of a GSM device described above
+will probably sound very convoluted, and you may find yourself asking for a way
+to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit
+that image "in vitro" with some utility on your PC, and then use fc-loadtool
+again to program it back into your device.  But consider that an "in vitro" FFS
+modification would involve erasing and rewriting all sectors of your FFS,
+whereas an "in vivo" modification of some small file like the IMEI would be
+just a short flash write operation without any erasures at all, i.e., kinder
+on the flash.
+
+In any case, the "in vivo" method will definitely be available soon because all
+of the components involved therein are also needed for other development uses
+in the FreeCalypso project, whereas developing a fully-functional "in vitro"
+alternative (one that can create an FFS image "de novo" from a tree of files
+and directories a la mkfs.jffs2, or add new files to an existing TIFFS image
+etc) would be a good amount of extra work which we otherwise don't need - hence
+the latter is not very likely to be written any time soon.
+
+However, if the "in vitro" modification you seek is something trivial like
+changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without
+changing its length, you can use the existing "in vitro, read-only" tiffs host
+utility to find the exact byte location of the file data within the TIFFS image,
+and then use your favourite hex editor to whack whatever new byte content you
+like at that offset.
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/ffstools/README	Fri Jan 31 07:37:39 2014 +0000
@@ -0,0 +1,31 @@
+You are looking at the source for the TIFFS In Vitro Analyzer utility.  You may
+have downloaded it either as a separate package or as part of the larger
+freecalypso-sw suite.
+
+See TIFFS-Overview (in ../doc if you are working with the full freecalypso-sw
+source tree) for a general description of what TIFFS is and why it matters.
+
+The utility contained in the present package runs on a general purpose GNU/Linux
+(or other Unix) host and enables "in vitro" examination of Flash File System
+images read out of TI-based GSM devices.  Using this utility, you can list the
+directory and file content of an FFS image, cat any individual file in the FFS,
+or extract the complete FFS content into your regular Unix file system.  Some
+"forensic" operations are also supported: by listing the inode array, one can
+deduce the order in which the present FFS content got created, and see what
+files have been overwritten or deleted in the span of still-visible history.
+One can then cat the old byte content of those overwritten or deleted files,
+if those data chunks are still in the FFS image (i.e., if the flash sector in
+question has not been reclaimed yet).
+
+Compilation and installation are straightforward: run 'make' to compile the
+source; you should get 3 executable binaries named tiffs, mokoffs and pirffs;
+then run 'make install' as root to install them in /usr/local/bin.  The binary
+named tiffs is the main program; mokoffs and pirffs are wrappers that simplify
+the most common current use cases.
+
+To install somewhere other than /usr/local/bin, edit the INSTBIN= setting in
+the subdirectory Makefiles.  You will also need to edit
+tiffs-wrappers/installpath.c accordingly, as the mokoffs and pirffs wrappers
+are designed to exec tiffs by its absolute installed pathname.
+
+See Usage for the usage instructions.
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/ffstools/Usage	Fri Jan 31 07:37:39 2014 +0000
@@ -0,0 +1,84 @@
+The generic tiffs utility needs to be invoked as follows:
+
+tiffs [global-options] <imgfile> <org> <cmd> [command-args]
+
+The first 3 non-optional arguments are the filename of the TIFFS image under
+examination, the FFS organization being examined, and the operation to be
+performed.  The present utility is designed in the classic Unix manner in that
+each invokation performs a single operation and exits, such that invokations of
+tiffs (or one of the wrappers described below) may be plumbed into pipes and
+the like.
+
+The 2nd argument to tiffs after the FFS image filename describes how the TIFFS
+instance under study is organized in terms of flash sectors.  The syntax of
+this argument is KxN, where K is the flash sector size in KiB and N is the
+number of sectors occupied by the FFS.  For MokoFFS images the correct
+organization argument is 64x7 (7 sectors of 64 KiB each); for Pirelli's FFS
+images it is 256x18 (18 sectors of 256 KiB each).
+
+The following global options may be given before the image filename argument:
+
+-a num
+
+	Use the specified flash block (sector) as the inode array block.
+
+-o offset
+
+	The FFS image begins at the specified offset within the file, rather
+	than at the beginning.  This option is useful when working with complete
+	device flash dumps of which FFS is only a part, starting somewhere
+	other than at 0.
+
+-r ino
+
+	Use the specified inode as the root.  Per Falcon's convention, TIFFS
+	inode numbers are always given in hex, hence this argument is
+	interpreted as hex without needing a 0x prefix.
+
+The invokation syntax for mokoffs and pirffs wrappers is the same as for tiffs,
+except that the FFS organization argument (64x7 or 256x18) is omitted; the
+wrapper fills that argument in before passing the command to the main tiffs
+program.  The only other difference is that instead of the generic -o global
+option, mokoffs takes a -f global option (no argument) which indicates that one
+is working with a complete flash dump image, rather than just the FFS portion;
+mokoffs -f gets translated into tiffs -o0x380000.  (pirffs has no such option
+at all because Pirelli's FFS starts at offset 0 within its respective flash
+chip select.)
+
+The next argument after the FFS organization for tiffs (or after the image
+filename for mokoffs/pirffs) is the command (or operation) to be performed.
+The following tiffs commands are currently available:
+
+Standard listing/extraction commands
+====================================
+
+These commands list or extract the normally-visible content of the FFS, i.e.,
+the content which is visible when the FFS is "mounted" normally, and which the
+FFS promises to preserve - as opposed to deleted or overwritten content.
+
+ls [-v[v]] [pathname...]
+
+Tiffs ls without additional arguments yields a listing of the complete FFS
+directory tree, akin to tar tv.  Example output fragment:
+
+fr    4096 /.journal
+d          /gsm
+d          /gsm/rf
+d          /gsm/rf/tx
+f      512 /gsm/rf/tx/ramps.900
+f      128 /gsm/rf/tx/levels.900
+f      128 /gsm/rf/tx/calchan.900
+
+The first character is 'f' for files or 'd' for directories.  An 'r' following
+immediately afterward means that the object has the read-only attribute set.
+For files the listing includes the content size in bytes, and the last part is
+the pathname of the object within the FFS.
+
+With a single -v option added after ls, the output will include verbose
+information as to the segmentation structure of each file.  With two -v options
+or with -vv, this additional output will also include the byte offset of each
+data chunk, relative to the beginning of the FFS image.
+
+Tiffs ls with a pathname argument yields information about the specified FFS
+object; -v and -vv options act as already described, but are arguably more
+useful when listing single files.