FreeCalypso > hg > freecalypso-tools
view doc/TIFFS-Overview @ 402:817f3b5f019e
doc/RF_tables: typo fix
author | Mychaela Falconia <falcon@freecalypso.org> |
---|---|
date | Sun, 18 Mar 2018 08:37:23 +0000 |
parents | e7502631a0f9 |
children | 579441d7dcd8 |
line wrap: on
line source
All TI GSM firmwares known to this author (FreeCalypso developer Space Falcon) implement some kind of flash file system, or FFS. Several different FFS code implementations, and correspondingly several different on-flash data formats, have been used throughout the history of TI's involvement in the wireless terminal business. The FFS incarnation of primary interest to the FreeCalypso project is the one invented by Mads Meisner-Jensen at TI in the early 2000s (at least according to the comments in the sources available to us), and it is relevant to us in the following ways: * When targeting the GSM modem in Openmoko's GTA01/02 smartphones, we need to work with the original FFS from the factory (call it MokoFFS), the same FFS as used by the mokoN firmwares: this FFS contains the IMEI and the RF calibration values from the factory, which we most certainly don't want to go without. * The Leonardo firmware semi-src which we are using as the reference for building our own full source, multi-target GSM fw contains a turnkey-working implementation of this very FFS, using the on-flash format in question and providing run-time APIs expected by the rest of the GSM fw suite. Following the principle of ``if it ain't broke, don't fix it'', we can use this FFS not only on the gtamodem target, but also on other targets, including those where we would be starting from a blank state and thus have the freedom to use whatever FFS we like. * The original proprietary fw on the Pirelli DP-L10 phone also happens to use an FFS in the same format. Pirelli's FFS does *not* contain the IMEI or any of the RF calibration values though, and trying to reuse it directly for our own FC GSM fw seems to be more trouble than benefit - so we'll probably have our fw start with a blank TIFFS instead - but there is still insight to be gained from in-vitro examination of captured Pirelli FFS images. Naming ====== I have previously referred to the FFS format in question as Mokopir-FFS or MPFFS, from "Moko" and "Pirelli". I was originally hesitant to call it TIFFS, as lacking the source code, I had no way of knowing whether the FFS format and implementation were of TI's own invention, or something that TI licensed as a black box from one of their many proprietary software partners. (I was unable to identify it as any well-known, industry-standard FFS format, but absence of evidence is not evidence of absence.) But now that we have TI's original source code which implements this FFS (first the MV100-0.1.rar source, then the full Leonardo one), complete with comments and a HISTORY file, we know that our FFS was invented and implemented by someone named Mads Meisner-Jensen at TI - I'm guessing in the SSA group in Nice, France. I am now making a naming transition from MPFFS to TIFFS: there is really no link between this FFS format and the Openmoko+Pirelli duo, other than the happenstance of me having first encountered this FFS on these two GSM device brands, and the name TIFFS is more neutrally-descriptive. What it is ========== In a rare departure from TI's norm (most of TI's GSM firmware and associated development tools suffer from heavy Windows poisoning), what I call TIFFS is very Unixy. It is a file system with a hierarchical directory tree structure and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics of "what is a file" and "what is a directory" are exactly the same as in UNIX; and TIFFS even supports symlinks, although that support is a little under- developed, and apparently no FFS symlinks were ever used in any production GSM device. Thus the FFS implemented in TI-based GSM devices (modems and "dumbphones") is really no different from, for example, JFFS2 in embedded Linux systems. (The only traditional UNIX file system features which are missing in TIFFS are the creation/modification/access timestamps and the ownership/permission fields.) The FFS in a GSM device typically stores two kinds of content: * Factory data: IMEI, RF calibration values, device make/model/revision ID strings etc. These files are expected to be programmed on the factory production line and not changed afterward. * Dynamic data written into the FFS in normal device operation: when you use a "dumbphone" running TI-based firmware, every time you store something "on the phone" or in "non-volatile memory", that item is actually stored in the FFS. (Where else, if you think of it?) That includes contacts and received SMS stored "on the phone" instead of the SIM, any selections you make in the settings/preferences menus which persist across reboots (power cycles), call history etc. It needs to be noted that the "dynamic data" aspect of FFS usage applies not only to complete phones, but also to modems like the one used in the GTA01/02. One would naively think that non-volatile storage of data in flash outside of factory programming would be needed only in a device with its own UI, and that a modem subservient to external AT commands would be completely stateless across reboot/power cycles; but that is not the case in actuality. TI's GSM firmwares, including the Openmoko ones (the "standard" mokoN), are designed to always "mount" their FFS with read/write access; TI's FFS implementation in the firmware has no concept of a "read-only mount". I am still investigating just what kinds of data are routinely written into the non-volatile FFS by the firmware in normal operation on devices like the GTA0x modem, but there most definitely are some. There is no hard separation between "static" and "dynamic" data in the file system structure; TIFFS is thus akin to an embedded Linux system with just a single root file system containing both "static" files like userland binaries and "dynamic" ones like configuration files under /etc which the user is expected to edit with vi after logging into the box, or log and similar files created by the system itself under /var, for example. Where it lives ============== The type of flash memory used in Calypso GSM modems and "dumbphones" is called NOR flash. This NOR flash memory is physically divided (by the design of the flash chip itself) into units called "sectors" or more descriptively, erase blocks. The typical NOR flash sector size (in Calypso GSM devices) ranges from 64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total). The key physical property is that any bit may be changed from a '1' to a '0' at any time, in any combination, but resetting of '0' bits back to ones can be done only on the granularity of these largish sectors, in an operation called "sector erase". The location of TIFFS within the flash memory of a given GSM device is defined by the firmware design of that device, but is always some integral number of contiguous flash sectors. Some examples: * On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at flash offset 0x380000. * On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies 18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning of the 2nd flash chip select (0x02000000 in the ARM7 address space). * On Motorola/Compal C139/140 phones, the FFS used by the original proprietary fw occupies 5 sectors of 64 KiB each (320 KiB in total), starting at 0x370000. C11x/123 use smaller FFS configurations, whereas C155/156 seem to have switched to some other FFS format, different from our familiar TIFFS. * The smallest real FFS configuration called for by the table in dev.c in TI's original Leonardo fw source is 3 sectors of 64 KiB each; the same table also sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in RAM without real flash). * The largest FFS configuration that has been envisioned by the original designers seems to be somewhere around 128 sectors. Each flash sector used for TIFFS begins with this 6-byte signature: 46 66 73 23 10 02 The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the format version number of 0x0210 in little-endian byte order. The following two bytes give a count of how many times that sector has been erased and rewritten (FF FF in "fresh" or "virgin" FFS images), and the following byte indicates that block's role and status in the FFS life cycle. How it works ============ Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to recover gracefully from any possible power failure or crash: one can yank the battery from the GSM device (or induce a firmware crash) at the most mis- opportune moment in the middle of an FFS write operation, and the FFS is expected to recover on the next boot cycle. I won't be able to document here all gory details of exactly how this goal is achieved, partly because I haven't studied the code to the requisite level of depth myself yet, but all of the responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source tree; feel free to study it. In its "normal" or "clean" state (i.e., when not in the middle of a write operation or recovery from an ungracefully interrupted one), a TIFFS instance consists of the following 3 types of blocks: * One block containing inode records, indicated by AB in its type/flags/status byte in the block header; * N-2 blocks (where N is the total number of flash sectors allocated for the FFS) containing (or waiting to be filled with) data chunks - indicated by BD in the type/flags/status byte; * One "free" block, indicated by BF - destined to become a new AB or a new BD at some point. Each object written into the FFS (file, directory or symlink) consists of a 16-byte inode record written into the AB block and a data chunk written into one of the BD blocks. The data chunk includes the name of the object, hence one is required even for directories. Data chunks are contiguous, uncompressed, and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS configuration. Files larger than this limit are stored in a "segmented" form, giving rise to a 4th inode or object type (after file, directory and symlink): segment. Each segment of a segmented file consists of not only a data chunk, but also an inode record for the segment, which gives the location of the data chunk and ties the segment object into the overall FFS structure, making it accessible. Because aside from complete sector erasure, flash memory bits can only transition from '1' to '0' but not the other way around, overwriting an existing file with some new content (an operation which any reasonable file system must implement in some way) cannot be done in place. Instead like most flash file systems, TIFFS implements this common operation by writing the new version of the file to a new location (previously blank flash) and then invalidating the old version - and doing all that while keeping in mind the possibility of an ungraceful crash or powerdown at any moment, and the requirement of recovering gracefully from any such event. Of course as an FFS receives more write activity, even if one keeps overwriting some existing files with new content of the same size, without adding to the visible total content size (think du(1) command), eventually all remaining blank flash space will fill up. At that point (or at some earlier point, depending on the FFS design and/or configuration) the FFS has to invoke a compaction or reclamation or garbage collection procedure: any "mixed" blocks containing both valid and stale data are transitioned into a "stale-only" state by having the active data moved to a new block, and then the "all stale" blocks are subjected to sector erasure, becoming new blank sectors. The logic responsible for these operations once again needs to be resilient to the possibility of a crash or powerdown occurring at the most mis-opportune moment, and it also needs to implement flash wear leveling: there is a physical limit to how many times a given flash sector can be erased and rewritten before it goes bad. All of the above are common and well-known principles, successfully implemented in well-known flash file systems such as JFFS2 in Linux. TIFFS is absolutely no different in this regard; for the implementation details, read the source code. How this FFS comes into being ============================= (This section is only relevant to you if you plan on physically producing your own GSM phones or modems on your own factory production line, like this author fancies doing in the not-too-distant future, or if you simply enjoy knowing how it is done.) To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the embedded Linux world, which would produce a TIFFS image complete with some initial directory and file content "in vitro". Instead it appears that the FFS instances found in shipped products such as Openmoko phones have been created "in vivo" by TI's firmware running on the device itself during the "production test" phase. The process seems to go like this: * When the printed circuit board is physically populated with components such as the Calypso chip and the flash chip, the latter can be blank - if the board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM (Openmoko and Pirelli both good on this one, but shame on Compal!), there is no need to preprogram the flash chip with anything prior to populating it on the board, and the device remains fully unbrickable at all times afterward. * When the assembled board is powered up for the first time, with completely blank flash, the Calypso boot ROM will sit there and patiently wait for a code download on either of its two UARTs. * Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's fc-loadtool free replacement, the factory production station loads the main firmware image into the flash. Note, it is just the firmware image at this step, and the FFS sectors remain blank. * The board is commanded to reboot (or power-cycled), and the firmware image boots for the first time. * TI's FFS implementation code in their standard firmware reacts to all blank flash in the FFS sectors as follows: it performs what they call the preformat operation, writing the TIFFS signature and a BF state byte into every FFS sector, but the main "format" operation, which sets up the AB/BD block roles, creates the root inode and makes the FFS ready to accept the creation of its first directories and files, is not done automatically. In order to perform the FFS format operation and then fill the new FFS with whatever directories and files are deemed needed to be present in "fresh" shipping products, the factory production station connects to the just-booted firmware running on the target via the RVT/ETM protocol (see the RVTMUX write-up), and sends "test mode" commands to this running firmware. These "FFS test mode" (or TMFFS) commands include the format operation, an mkdir operation to create directories, and a "file write" operation akin to doing 'cat > /dir/whatever/file', creating files in FFS and storing any desired data in them. The IMEI is assigned and written into FFS in this process, but it is not the only data item that will be unique for each individual device made. Much more important are the RF calibration values: I have yet to learn exactly what is being (or needs to be) measured, how these measurements are performed (under what conditions; what external test equipment is needed), and how these measured and recorded RF calibration values affect GSM device operation, but this TI presentation gives some clues: ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf All of these calibration values are stored in a bunch of files under the /gsm/rf subtree, and these files seem to be "owned" by the L1 code. The latter has RAM data structures which correspond to these files; upon normal boot the initialization code looks in FFS, and if it finds any of the RF calibration files, it reads each present file into the corresponding RAM data structure, overwriting the compiled-in defaults. It appears (slightly uncertain because I have not yet reintegrated the code in question into our own gsm-fw) that the RF calibration files in FFS come into being as follows: * The RF calibration code in L1 (i.e., part of the main GSM fw) performs the measurements and stores results in its RAM data structures as commanded by the production test station through the "test mode" interface; * A final test mode command directs the above L1 code to write its RAM data structures into FFS. Once I actually learn this RF calibration process properly in connection with building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what it would take to recreate these RF calibration values if they are lost. But until then the only advice I can give is to make a backup copy of your modem FFS with fc-loadtool, and to save it securely. Compal and Pirelli differences ============================== The above description refers to TI's vanilla reference version, and it seems like Openmoko (FIC) was the only phone/modem manufacturer who followed it without major deviations. In contrast, both Compal (Mot C1xx) and Foxconn (Pirelli DP-L10) moved the vital per-unit factory data (IMEI and RF calibration) out of the FFS into their own ad hoc flash data structures (which are very difficult to reverse-engineer and make use of, unfortunately), leaving their FFS only for less critical data. In Compal's case (at least on the C139 model with which I have extensive personal experience) the FFS stores only users' personal information and nothing more. One can turn the phone off, use fc-loadtool to erase the FFS sectors, and boot the regular fw back up; the fw will automatically do a new FFS format (it even displays a message on the LCD as it does so) and carry on happily as a "fresh" or "blank", perfectly functional and usable phone. In Pirelli's case, booting their official fw with blank FFS sectors will also result in the FFS being automatically formatted, but their fw expects some static "asset" files to be present in this FFS: UI graphics and language strings, ringtones, firmware images for the WiFi and VoIP processors and some static configuration files, about 3 MiB in total. Thus although the firmware will auto-format the blank FFS sectors, it won't function normally with all of these "asset" files missing. Foxconn's original factory production line station must have uploaded these files to each phone via the TMFFS2 protocol, and our FreeCalypso suite now features a tool that can replicate this feat: fc-fsio. FreeCalypso support for TIFFS ============================= Aside from implementing and using it in our own gsm-fw, FreeCalypso offers the following support for TIFFS: 1. We have a utility for "in vitro" examination of FFS images read out of GSM devices with fc-loadtool. This tiffs utility (along with mokoffs and pirffs wrappers) lives in the ffstools top-level directory of the freecalypso-sw source tree. This TIFFS "in vitro analyzer" utility supplants the earlier mpffs-* tools, and adds some additional examination functionality. It is strictly a "read only" tool, however - it is not designed for "in vitro" editing of TIFFS images. 2. A number of FC tools may be strung together into a kit for editing the FFS content of a GSM device, e.g., for changing the IMEI. The following pieces will be involved: * What is destined to eventually become our totally free GSM fw (the gsm-fw source subtree at the top of freecalypso-sw) does not contain any of the actual GSM protocol stack (or even L1) functionality yet, but it already contains both the FFS code and those components (ETM and TMFFS[12]) which are needed for interfacing an external "test mode shell" to this FFS implementation through the RVTMUX interface. And when our gsm-fw does gain the actual GSM functionality, the ability to build a minimal FFS+ETM-only configuration will still be retained. * The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs entirely from RAM via fc-xram, no flashing), and run on a physical device such as the GTA0x GSM modem via the fc-xram host utility; * After loading the ramImage, fc-xram will immediately exec our rvinterf host utility (see rvinterf/README); * Once the GSM device is running what is effectively an FFS editing agent out of RAM, accessed via rvinterf over the serial channel, the user can run fc-tmsh or fc-fsio, and this "test mode shell" provides commands for writing things to FFS exactly like one would do in the factory production line environment for which TI taylored their tools. The "in vivo" method of editing the FFS content of a GSM device described above will probably sound very convoluted, and you may find yourself asking for a way to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit that image "in vitro" with some utility on your PC, and then use fc-loadtool again to program it back into your device. But consider that an "in vitro" FFS modification would involve erasing and rewriting all sectors of your FFS, whereas an "in vivo" modification of some small file like the IMEI would be just a short flash write operation without any erasures at all, i.e., kinder on the flash. In any case, the "in vivo" method is already available now because all of the components involved therein are also needed for other development uses in the FreeCalypso project, whereas developing a fully-functional "in vitro" alternative (one that can create an FFS image "de novo" from a tree of files and directories a la mkfs.jffs2, or add new files to an existing TIFFS image etc) would be a good amount of extra work which we otherwise don't need - hence the latter is not very likely to be written any time soon. However, if the "in vitro" modification you seek is something trivial like changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without changing its length, you can use the existing "in vitro, read-only" tiffs host utility to find the exact byte location of the file data within the TIFFS image, and then use your favourite hex editor to whack whatever new byte content you like at that offset.