# HG changeset patch # User Michael Spacefalcon # Date 1389666669 0 # Node ID 1852900ce9ea4ac52d4648f7d9ca46897eb3640b # Parent 4d706a4134b03e7b48c67ee8daba2fdb95dd0f4c doc/TIFFS write-up diff -r 4d706a4134b0 -r 1852900ce9ea doc/TIFFS --- a/doc/TIFFS Mon Jan 13 10:15:59 2014 +0000 +++ b/doc/TIFFS Tue Jan 14 02:31:09 2014 +0000 @@ -48,3 +48,321 @@ link between this FFS format and the Openmoko+Pirelli duo, other than the happenstance of me having first encountered this FFS on these two GSM device brands, and the name TIFFS is more neutrally-descriptive. + +What it is +========== + +In a rare departure from TI's norm (most of TI's GSM firmware and associated +development tools suffer from heavy Windows poisoning), what I call TIFFS is +very Unixy. It is a file system with a hierarchical directory tree structure +and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics +of "what is a file" and "what is a directory" are exactly the same as in UNIX; +and TIFFS even supports symlinks, although that support is a little under- +developed, and apparently no FFS symlinks were ever used in any production GSM +device. Thus the FFS implemented in TI-based GSM devices (modems and +"dumbphones") is really no different from, for example, JFFS2 in embedded Linux +systems. + +(The only traditional UNIX file system features which are missing in TIFFS are + the creation/modification/access timestamps and the ownership/permission + fields.) + +The FFS in a GSM device typically stores two kinds of content: + +* Factory data: IMEI, RF calibration values, device make/model/revision + ID strings etc. These files are expected to be programmed on the factory + production line and not changed afterward. + +* Dynamic data written into the FFS in normal device operation: when you use a + "dumbphone" running TI-based firmware, every time you store something "on the + phone" or in "non-volatile memory", that item is actually stored in the FFS. + (Where else, if you think of it?) That includes contacts and received SMS + stored "on the phone" instead of the SIM, any selections you make in the + settings/preferences menus which persist across reboots (power cycles), call + history etc. + +It needs to be noted that the "dynamic data" aspect of FFS usage applies not +only to complete phones, but also to modems like the one used in the GTA01/02. +One would naively think that non-volatile storage of data in flash outside of +factory programming would be needed only in a device with its own UI, and that +a modem subservient to external AT commands would be completely stateless +across reboot/power cycles; but that is not the case in actuality. TI's GSM +firmwares, including the Openmoko ones (the "standard" mokoN), are designed to +always "mount" their FFS with read/write access; TI's FFS implementation in the +firmware has no concept of a "read-only mount". + +I am still investigating just what kinds of data are routinely written into the +non-volatile FFS by the firmware in normal operation on devices like the GTA0x +modem, but there most definitely are some. + +There is no hard separation between "static" and "dynamic" data in the file +system structure; TIFFS is thus akin to an embedded Linux system with just a +single root file system containing both "static" files like userland binaries +and "dynamic" ones like configuration files under /etc which the user is +expected to edit with vi after logging into the box, or log and similar files +created by the system itself under /var, for example. + +Where it lives +============== + +The type of flash memory used in Calypso GSM modems and "dumbphones" is called +NOR flash. This NOR flash memory is physically divided (by the design of the +flash chip itself) into units called "sectors" or more descriptively, erase +blocks. The typical NOR flash sector size (in Calypso GSM devices) ranges from +64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the +S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total). +The key physical property is that any bit may be changed from a '1' to a '0' at +any time, in any combination, but resetting of '0' bits back to ones can be +done only on the granularity of these largish sectors, in an operation called +"sector erase". + +The location of TIFFS within the flash memory of a given GSM device is defined +by the firmware design of that device, but is always some integral number of +contiguous flash sectors. Some examples: + +* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at + flash offset 0x380000. + +* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies + 18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning + of the 2nd flash chip select (0x02000000 in the ARM7 address space). + +* The smallest real FFS configuration called for by the table in dev.c in TI's + original Leonardo fw source is 3 sectors of 64 KiB each; the same table also + sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in + RAM without real flash). + +* The largest FFS configuration that has been envisioned by the original + designers seems to be somewhere around 128 sectors. + +Each flash sector used for TIFFS begins with this 6-byte signature: + +46 66 73 23 10 02 + +The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the +format version number of 0x0210 in little-endian byte order. The following two +bytes give a count of how many times that sector has been erased and rewritten +(FF FF in "fresh" or "virgin" FFS images), and the following byte indicates +that block's role and status in the FFS life cycle. + +How it works +============ + +Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to +recover gracefully from any possible power failure or crash: one can yank the +battery from the GSM device (or induce a firmware crash) at the most mis- +opportune moment in the middle of an FFS write operation, and the FFS is +expected to recover on the next boot cycle. I won't be able to document here +all gory details of exactly how this goal is achieved, partly because I haven't +studied the code to the requisite level of depth myself yet, but all of the +responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source +tree; feel free to study it. + +In its "normal" or "clean" state (i.e., when not in the middle of a write +operation or recovery from an ungracefully interrupted one), a TIFFS instance +consists of the following 3 types of blocks: + +* One block containing inode records, indicated by AB in its type/flags/status + byte in the block header; +* N-2 blocks (where N is the total number of flash sectors allocated for the + FFS) containing (or waiting to be filled with) data chunks - indicated by BD + in the type/flags/status byte; +* One "free" block, indicated by BF - destined to become a new AB or a new BD + at some point. + +Each object written into the FFS (file, directory or symlink) consists of a +16-byte inode record written into the AB block and a data chunk written into +one of the BD blocks. The data chunk includes the name of the object, hence +one is required even for directories. Data chunks are contiguous, uncompressed, +and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS +configuration. Files larger than this limit are stored in a "segmented" form, +giving rise to a 4th inode or object type (after file, directory and symlink): +segment. Each segment of a segmented file consists of not only a data chunk, +but also an inode record for the segment, which gives the location of the data +chunk and ties the segment object into the overall FFS structure, making it +accessible. + +Because aside from complete sector erasure, flash memory bits can only +transition from '1' to '0' but not the other way around, overwriting an existing +file with some new content (an operation which any reasonable file system must +implement in some way) cannot be done in place. Instead like most flash file +systems, TIFFS implements this common operation by writing the new version of +the file to a new location (previously blank flash) and then invalidating the +old version - and doing all that while keeping in mind the possibility of an +ungraceful crash or powerdown at any moment, and the requirement of recovering +gracefully from any such event. + +Of course as an FFS receives more write activity, even if one keeps overwriting +some existing files with new content of the same size, without adding to the +visible total content size (think du(1) command), eventually all remaining blank +flash space will fill up. At that point (or at some earlier point, depending +on the FFS design and/or configuration) the FFS has to invoke a compaction or +reclamation or garbage collection procedure: any "mixed" blocks containing both +valid and stale data are transitioned into a "stale-only" state by having the +active data moved to a new block, and then the "all stale" blocks are subjected +to sector erasure, becoming new blank sectors. The logic responsible for these +operations once again needs to be resilient to the possibility of a crash or +powerdown occurring at the most mis-opportune moment, and it also needs to +implement flash wear leveling: there is a physical limit to how many times a +given flash sector can be erased and rewritten before it goes bad. + +All of the above are common and well-known principles, successfully implemented +in well-known flash file systems such as JFFS2 in Linux. TIFFS is absolutely +no different in this regard; for the implementation details, read the source +code. + +How this FFS comes into being +============================= + +(This section is only relevant to you if you plan on physically producing your + own GSM phones or modems on your own factory production line, like this author + fancies doing in the not-too-distant future, or if you simply enjoy knowing + how it is done.) + +To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the +embedded Linux world, which would produce a TIFFS image complete with some +initial directory and file content "in vitro". Instead it appears that the FFS +instances found in shipped products such as Openmoko phones have been created +"in vivo" by TI's firmware running on the device itself during the "production +test" phase. + +The process seems to go like this: + +* When the printed circuit board is physically populated with components such + as the Calypso chip and the flash chip, the latter can be blank - if the + board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM + (Openmoko and Pirelli both good on this one, but shame on Compal!), there is + no need to preprogram the flash chip with anything prior to populating it on + the board, and the device remains fully unbrickable at all times afterward. + +* When the assembled board is powered up for the first time, with completely + blank flash, the Calypso boot ROM will sit there and patiently wait for a + code download on either of its two UARTs. + +* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's + fc-loadtool free replacement, the factory production station loads the main + firmware image into the flash. Note, it is just the firmware image at this + step, and the FFS sectors remain blank. + +* The board is commanded to reboot (or power-cycled), and the firmware image + boots for the first time. + +* TI's FFS implementation code in their standard firmware reacts to all blank + flash in the FFS sectors as follows: it performs what they call the preformat + operation, writing the TIFFS signature and a BF state byte into every FFS + sector, but the main "format" operation, which sets up the AB/BD block roles, + creates the root inode and makes the FFS ready to accept the creation of its + first directories and files, is not done automatically. + +In order to perform the FFS format operation and then fill the new FFS with +whatever directories and files are deemed needed to be present in "fresh" +shipping products, the factory production station connects to the just-booted +firmware running on the target via the RVT/ETM protocol (see the RVTMUX +write-up), and sends "test mode" commands to this running firmware. These +"FFS test mode" (or TMFFS) commands include the format operation, an mkdir +operation to create directories, and a "file write" operation akin to doing +'cat > /dir/whatever/file', creating files in FFS and storing any desired data +in them. + +The IMEI is assigned and written into FFS in this process, but it is not the +only data item that will be unique for each individual device made. Much more +important are the RF calibration values: I have yet to learn exactly what is +being (or needs to be) measured, how these measurements are performed (under +what conditions; what external test equipment is needed), and how these measured +and recorded RF calibration values affect GSM device operation, but this TI +presentation gives some clues: + +ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf + +All of these calibration values are stored in a bunch of files under the +/gsm/rf subtree, and these files seem to be "owned" by the L1 code. The latter +has RAM data structures which correspond to these files; upon normal boot the +initialization code looks in FFS, and if it finds any of the RF calibration +files, it reads each present file into the corresponding RAM data structure, +overwriting the compiled-in defaults. It appears (slightly uncertain because I +have not yet reintegrated the code in question into our own gsm-fw) that the RF +calibration files in FFS come into being as follows: + +* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the + measurements and stores results in its RAM data structures as commanded by + the production test station through the "test mode" interface; + +* A final test mode command directs the above L1 code to write its RAM data + structures into FFS. + +Once I actually learn this RF calibration process properly in connection with +building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what +it would take to recreate these RF calibration values if they are lost. But +until then the only advice I can give is to make a backup copy of your modem +FFS with fc-loadtool, and to save it securely. + +FreeCalypso support for TIFFS +============================= + +Aside from implementing and using it in our own gsm-fw, FreeCalypso will offer +the following support for TIFFS: + +1. A tiffs host utility is being written which will allow a user to list and + extract content from saved FFS images (read out of flash with fc-loadtool) + "in vitro". It will be a restructured and (hopefully) improved version of + the mpffs-* tools released back in the summer of SE52 (A.D. 2013); the latter + already perform the advertised function, but I seek to integrate some other + functionality which I developed in an ad hoc side project ("pirollback"), + and I'm taking the opportunity to make the MPFFS->TIFFS renaming. + +(The mpffs-* tools mentioned above have been written based on reverse eng only, + before I found any source code for TI's FFS firmware implementation! Now that + we have the source, some terminological and other inevitable misunderstandings + can be corrected.) + +2. A number of FC tools may be strung together into a kit for editing the FFS + content of a GSM device, e.g., for changing the IMEI. The following pieces + will be involved: + +* What is destined to eventually become our totally free GSM fw (the gsm-fw + source subtree at the top of freecalypso-sw) does not contain any of the + actual GSM protocol stack (or even L1) functionality yet, but it already + contains both the FFS code and those components (ETM and TMFFS[12]) which + are needed for interfacing an external "test mode shell" to this FFS + implementation through the RVTMUX interface. And when our gsm-fw does gain + the actual GSM functionality, the ability to build a minimal FFS+ETM-only + configuration will still be retained. + +* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs + entirely from RAM via fc-xram, no flashing), and run on a physical device + such as the GTA0x GSM modem via the fc-xram host utility; + +* After loading the ramImage, fc-xram will immediately exec our rvinterf host + utility described in the RVTMUX write-up; + +* Once the GSM device is running what is effectively an FFS editing agent out + of RAM, accessed via rvinterf over the serial channel, the user will be able + to run fc-tmsh (or perhaps the FFS operations will be implemented in some + other utility, we'll see), and that "test mode shell" will provide commands + for writing things to FFS exactly like one would do in the factory production + line environment for which TI taylored their tools. + +The "in vivo" method of editing the FFS content of a GSM device described above +will probably sound very convoluted, and you may find yourself asking for a way +to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit +that image "in vitro" with some utility on your PC, and then use fc-loadtool +again to program it back into your device. But consider that an "in vitro" FFS +modification would involve erasing and rewriting all sectors of your FFS, +whereas an "in vivo" modification of some small file like the IMEI would be +just a short flash write operation without any erasures at all, i.e., kinder +on the flash. + +In any case, the "in vivo" method will definitely be available soon because all +of the components involved therein are also needed for other development uses +in the FreeCalypso project, whereas developing a fully-functional "in vitro" +alternative (one that can create an FFS image "de novo" from a tree of files +and directories a la mkfs.jffs2, or add new files to an existing TIFFS image +etc) would be a good amount of extra work which we otherwise don't need - hence +the latter is not very likely to be written any time soon. + +However, if the "in vitro" modification you seek is something trivial like +changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without +changing its length, you will be able to use the "in vitro, read-only" tiffs +host utility to find the exact byte location of the file data within the TIFFS +image, and use your favourite hex editor to whack whatever new byte content you +like at that offset.