[Historical note: this document was originally written in 2014 when the vision of FreeCalypso was very different from what it is today. Since then we have transitioned from making aftermarket hacks on pre-existing Calypso phones and modems to producing and supporting our own FreeCalypso hardware products, and our firmware work has changed from a nebulous dream to stable production code. The ways in which we approach various technical aspects have changed accordingly, but much of our documentation has been slow to catch up. The documentation is now being updated, but there may still be some passages where the old world view shines through.]All TI GSM firmwares known to this author (Mother Mychaela of FreeCalypso)implement some kind of flash file system, or FFS. Several different FFS codeimplementations, and correspondingly several different on-flash data formats,have been used throughout the history of TI's involvement in the wirelessterminal business. The FFS incarnation of primary interest to the FreeCalypsoproject is the one invented by Mads Meisner-Jensen at TI in the early 2000s(at least according to the comments in the sources available to us), and it isrelevant to us in the following ways:* When targeting the GSM modem in Openmoko's GTA01/02 smartphones, we need to work with the original FFS from the factory (call it MokoFFS), the same FFS as used by the mokoN firmwares: this FFS contains the IMEI and the RF calibration values from the factory, which we most certainly don't want to go without.* The Leonardo firmware semi-src which we are using as the reference for building our own full source, multi-target GSM fw contains a turnkey-working implementation of this very FFS, using the on-flash format in question and providing run-time APIs expected by the rest of the GSM fw suite. Following the principle of ``if it ain't broke, don't fix it'', we use this FFS not only on the gtamodem target, but also on other targets, including those where we are starting from a blank state and thus have the freedom to use whatever FFS we like.* The original proprietary fw on the Pirelli DP-L10 phone also happens to use an FFS in the same format, although Pirelli's FFS does *not* contain the IMEI or any of the RF calibration values. This Pirelli phone was originally a target of high interest for FreeCalypso, as we had high hopes of turning it into a libre phone by way of our aftermarket firmware - but this plan has since been abandoned when it became clear that Pirelli's hw is unsuitable for aftermarket fw development because of the multitude of extra peripheral chips for non-GSM functions which get in the way. In the earlier years of FreeCalypso a lot of effort had been invested into studying all aspects of the Pirelli DP-L10 phone and its original firmware, including the FFS.Naming======I have previously referred to the FFS format in question as Mokopir-FFS orMPFFS, from "Moko" and "Pirelli". I was originally hesitant to call it TIFFS,as lacking the source code, I had no way of knowing whether the FFS format andimplementation were of TI's own invention, or something that TI licensed as ablack box from one of their many proprietary software partners. (I was unableto identify it as any well-known, industry-standard FFS format, but absence ofevidence is not evidence of absence.) But now that we have TI's original sourcecode which implements this FFS (first the MV100-0.1.rar source, then the fullLeonardo one), complete with comments and a HISTORY file, we know that our FFSwas invented and implemented by someone named Mads Meisner-Jensen at TI-DK,apparently their flash chip expert who also wrote FLUID.I am now making a naming transition from MPFFS to TIFFS: there is really nolink between this FFS format and the Openmoko+Pirelli duo, other than thehappenstance of me having first encountered this FFS on these two GSM devicebrands, and the name TIFFS is more neutrally-descriptive.What it is==========In a rare departure from TI's norm (most of TI's GSM firmware and associateddevelopment tools suffer from heavy Windows poisoning), what I call TIFFS isvery Unixy. It is a file system with a hierarchical directory tree structureand with Unixy forward-slash-separated, case-sensitive pathnames; the semanticsof "what is a file" and "what is a directory" are exactly the same as in UNIX;and TIFFS even supports symlinks, although that support is a little under-developed, and apparently no FFS symlinks were ever used in any production GSMdevice. Thus the FFS implemented in TI-based GSM devices (modems and"dumbphone" handsets) is really no different from, for example, JFFS2 inembedded Linux systems.(The only traditional UNIX file system features which are missing in TIFFS are the creation/modification/access timestamps and the ownership/permission fields.)The FFS in a GSM device typically stores two kinds of content:* Factory data: IMEI, RF calibration values, device make/model/revision ID strings etc. These files are expected to be programmed on the factory production line and not changed afterward.* Dynamic data written into the FFS in normal device operation: when you use a "dumbphone" running TI-based firmware, every time you store something "on the phone" or in "non-volatile memory", that item is actually stored in the FFS. (Where else, if you think of it?) That includes contacts and received SMS stored "on the phone" instead of the SIM, any selections you make in the settings/preferences menus which persist across reboots (power cycles), call history etc.It needs to be noted that the "dynamic data" aspect of FFS usage applies notonly to complete phones, but also to modems like the one used in the GTA01/02.One would naively think that non-volatile storage of data in flash outside offactory programming would be needed only in a device with its own UI, and thata modem subservient to external AT commands would be completely statelessacross reboot/power cycles; but that is not the case in actuality. TI-baseedGSM firmwares, including Openmoko's mokoN and our own FreeCalypso, are designedto always "mount" their FFS with read/write access; TI's FFS implementation inthe firmware has no concept of a "read-only mount".I am still investigating just what kinds of data are routinely written into thenon-volatile FFS by the firmware in normal operation on devices like the GTA0xmodem, but there most definitely are some.There is no hard separation between "static" and "dynamic" data in the filesystem structure; TIFFS is thus akin to an embedded Linux system with just asingle root file system containing both "static" files like userland binariesand "dynamic" ones like configuration files under /etc which the user isexpected to edit with vi after logging into the box, or log and similar filescreated by the system itself under /var, for example.Where it lives==============The type of flash memory used in Calypso GSM modems and "dumbphones" is calledNOR flash. This NOR flash memory is physically divided (by the design of theflash chip itself) into units called "sectors" or more descriptively, eraseblocks. The typical NOR flash sector size (in Calypso GSM devices) ranges from64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in theS71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 and in our own FreeCalypsohardware designs (16 MiB of flash total). The key physical property is thatany bit may be changed from a '1' to a '0' at any time, in any combination, butresetting of '0' bits back to ones can be done only on the granularity of theselargish sectors, in an operation called "sector erase".The location of TIFFS within the flash memory of a given GSM device is definedby the firmware design of that device, but is always some integral number ofcontiguous flash sectors. Some examples:* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at flash offset 0x380000.* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies 18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning of the 2nd flash chip select (0x02000000 in the ARM7 address space).* On Motorola/Compal C139/140 phones, the FFS used by the original proprietary fw occupies 5 sectors of 64 KiB each (320 KiB in total), starting at 0x370000. C11x/12x use smaller FFS configurations, whereas C155/156 use a different FFS implementation with a completely different on-flash format - see the new Compal-FFS article for more details.* On our own FreeCalypso hardware family we have put our FFS in the first 8 sectors (of 256 KiB each) in the 2nd flash chip select bank, which appears at 0x01800000 in the ARM7 address space instead of Pirelli's 0x02000000 because we have wired the 2nd flash chip select to nCS2 on the Calypso instead of Pirelli's nCS3.* The smallest real FFS configuration called for by the table in dev.c in TI's original Leonardo fw source is 3 sectors of 64 KiB each; the same table also sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in RAM without real flash).* The largest FFS configuration that has been envisioned by the original designers seems to be somewhere around 128 sectors.Each flash sector used for TIFFS begins with this 6-byte signature:46 66 73 23 10 02The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are theformat version number of 0x0210 in little-endian byte order. The following twobytes give a count of how many times that sector has been erased and rewritten(FF FF in "fresh" or "virgin" FFS images), and the following byte indicatesthat block's role and status in the FFS life cycle.How it works============Just like JFFS2 and other high-quality flash file systems, TIFFS is designed torecover gracefully from any possible power failure or crash: one can yank thebattery from the GSM device (or induce a firmware crash) at the most mis-opportune moment in the middle of an FFS write operation, and the FFS isexpected to recover on the next boot cycle. I won't be able to document hereall gory details of exactly how this goal is achieved, partly because I haven'tstudied the code to the requisite level of depth myself yet, but all of theresponsible code lives under src/cs/drivers/drv_app/ffs in our fc-magnetite andfc-selenite source trees; feel free to study it.In its "normal" or "clean" state (i.e., when not in the middle of a writeoperation or recovery from an ungracefully interrupted one), a TIFFS instanceconsists of the following 3 types of blocks:* One block containing inode records, indicated by AB in its type/flags/status byte in the block header;* N-2 blocks (where N is the total number of flash sectors allocated for the FFS) containing (or waiting to be filled with) data chunks - indicated by BD in the type/flags/status byte;* One "free" block, indicated by BF - destined to become a new AB or a new BD at some point.Each object written into the FFS (file, directory or symlink) consists of a16-byte inode record written into the AB block and a data chunk written intoone of the BD blocks. The data chunk includes the name of the object, henceone is required even for directories. Data chunks are contiguous, uncompressed,and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFSconfiguration. Files larger than this limit are stored in a "segmented" form,giving rise to a 4th inode or object type (after file, directory and symlink):segment. Each segment of a segmented file consists of not only a data chunk,but also an inode record for the segment, which gives the location of the datachunk and ties the segment object into the overall FFS structure, making itaccessible.Because aside from complete sector erasure, flash memory bits can onlytransition from '1' to '0' but not the other way around, overwriting an existingfile with some new content (an operation which any reasonable file system mustimplement in some way) cannot be done in place. Instead like most flash filesystems, TIFFS implements this common operation by writing the new version ofthe file to a new location (previously blank flash) and then invalidating theold version - and doing all that while keeping in mind the possibility of anungraceful crash or powerdown at any moment, and the requirement of recoveringgracefully from any such event.Of course as an FFS receives more write activity, even if one keeps overwritingsome existing files with new content of the same size, without adding to thevisible total content size (think du(1) command), eventually all remaining blankflash space will fill up. At that point (or at some earlier point, dependingon the FFS design and/or configuration) the FFS has to invoke a compaction orreclamation or garbage collection procedure: any "mixed" blocks containing bothvalid and stale data are transitioned into a "stale-only" state by having theactive data moved to a new block, and then the "all stale" blocks are subjectedto sector erasure, becoming new blank sectors. The logic responsible for theseoperations once again needs to be resilient to the possibility of a crash orpowerdown occurring at the most mis-opportune moment, and it also needs toimplement flash wear leveling: there is a physical limit to how many times agiven flash sector can be erased and rewritten before it goes bad.All of the above are common and well-known principles, successfully implementedin well-known flash file systems such as JFFS2 in Linux. TIFFS is absolutelyno different in this regard; for the implementation details, read the sourcecode.TIFFS filename and pathname limits==================================Classic TIFFS, as in the canonical firmware source from TI, imposes thefollowing limits on the content written into FFS:* Each elementary filename or pathname component (the name of each individual file or subdirectory within its parent directory) is limited to 20 characters;* The set of allowed characters in these elementary filenames is limited to [A-Za-z0-9_.,+%$#-];* The maximum pathname depth is limited to 6.As an illustration of the pathname depth limit, the deepest allowed path to anon-directory file is /d1/d2/d3/d4/d5/file. It is also possible to have adirectory nesting of /d1/d2/d3/d4/d5/d6, but in this case the deepest directorycan only be empty.How this FFS comes into being=============================(This section is only relevant to you if you plan on physically producing your own GSM phones or modems on your own factory production line, like we currently do at our family company, or if you simply enjoy knowing how it is done.)To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in theembedded Linux world, or akin to our recently developed tiffs-mkfs, which wouldproduce a TIFFS image complete with some initial directory and file content"in vitro". Instead it appears that the FFS instances found in shipped productssuch as Openmoko phones have been created "in vivo" by TI's firmware running onthe device itself during the "production test" phase.We never got a copy of the original factory production line software that wasused by Openmoko, but we have successfully replicated the process using our ownUnix/Linux-based FreeCalypso host tools, the very same tools that are containedin the present source package you are looking at. The process goes like this:* When the printed circuit board is physically populated with components such as the Calypso chip and the flash chip, the latter can be blank - if the board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM (Openmoko and Pirelli both good on this one, but shame on Compal!), there is no need to preprogram the flash chip with anything prior to populating it on the board, and the device remains fully unbrickable at all times afterward.* When the assembled board is powered up for the first time, with completely blank flash, the Calypso boot ROM will sit there and patiently wait for a code download on either of its two UARTs.* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's fc-loadtool free replacement, the factory production station loads the main firmware image into the flash. Note, it is just the firmware image at this step, and the FFS sectors remain blank.* The board is commanded to reboot (or power-cycled), and the firmware image boots for the first time.* TI's FFS implementation code in their standard firmware reacts to all blank flash in the FFS sectors as follows: it performs what they call the preformat operation, writing the TIFFS signature and a BF state byte into every FFS sector, but the main "format" operation, which sets up the AB/BD block roles, creates the root inode and makes the FFS ready to accept the creation of its first directories and files, is not done automatically.In order to perform the FFS format operation and then fill the new FFS withwhatever directories and files are deemed needed to be present in "fresh"shipping products, the factory production station connects to the just-bootedfirmware running on the target via the RVT/ETM protocol (see the RVTMUXwrite-up), and sends "test mode" commands to this running firmware. These"FFS test mode" (or TMFFS) commands include the format operation, an mkdiroperation to create directories, and a "file write" operation akin to doing'cat > /dir/whatever/file', creating files in FFS and storing any desired datain them.The IMEI is assigned and written into FFS in this process, but it is not theonly data item that will be unique for each individual device made. Much moreimportant are the RF calibration values; the factory calibration procedure doesthe following for each individual unit:* Measures the frequency offset produced by the VCXO as a function of the AFC DAC control value and constructs the afcparams table based on these measurements;* Characterizes the dBm output of the Tx chain as a function of the APC DAC control value and comes up with a set of these DAC values which produce Tx power levels prescribed by the GSM 05.05 spec;* Determines the correction values which need to be applied in order to set the correct Rx path gains and to determine the true Rx signal level from the dBfs power measurements made in the DSP from Rx I&Q samples.These calibration procedures are performed by connecting a suitable RF testinstrument (R&S CMU200 is the industry gold standard) to the GSM device'santenna connector or RF test port and running special calibration programswhich talk both to the CMU200 or other test instrument and to the L1TM (Layer 1test modes) component in the DUT (device under test). In FreeCalypso hardwaremanufacturing we use a CMU200 instrument which is itself maintained in goodcalibration standing, and for the calibration software we use our ownfc-rfcal-tools which talk to the DUT via rvinterf.All of the resulting calibration values are stored in a bunch of files under the/gsm/rf subtree, and these files are "owned" by the L1 code. The latter hasRAM data structures which correspond to these files; upon normal boot theinitialization code looks in FFS, and if it finds any of the RF calibrationfiles, it reads each present file into the corresponding RAM data structure,overwriting the compiled-in defaults. With TI's standard production calibrationprocedure which we have replicated in our FreeCalypso hw manufacturing setup,these RF calibration files in FFS come into being as follows:* The Test Mode support code in L1 (i.e., part of the main GSM fw) performs the measurements and stores results in its RAM data structures as commanded by the production test station through the Test Mode interface;* Certain special test mode commands encoded via the MISC_ENABLE opcode direct the above L1TM code to write its RAM data structures into FFS.See the RF_tables article for more information.Compal and Pirelli differences==============================The above description refers to TI's vanilla reference version, and it seemslike Openmoko (FIC) was the only phone/modem manufacturer (prior to us!) whofollowed it without major deviations. In contrast, both Compal (Motorola C1xxand Sony Ericsson J100) and Foxconn (Pirelli DP-L10) moved their vital per-unitfactory data (IMEI and RF calibration) out of the FFS into their own ad hocflash data structures (which are very difficult to reverse-engineer and makeuse of, unfortunately), leaving their FFS only for less critical data.In Compal's case (all C1xx models and SE J100) the FFS stores only users'personal information and nothing more. One can turn the phone off, usefc-loadtool to erase the FFS sectors, and boot the regular fw back up; the fwwill automatically do a new FFS format (it even displays a message on the LCDas it does so) and carry on happily as a "fresh" or "blank", perfectlyfunctional and usable phone. Please see the new Compal-FFS article for furtherdetails.In Pirelli's case, booting their official fw with blank FFS sectors will alsoresult in the FFS being automatically formatted, but their fw expects somestatic "asset" files to be present in this FFS: UI graphics and languagestrings, ringtones, firmware images for the WiFi and VoIP processors and somestatic configuration files, about 3 MiB in total. Thus although the firmwarewill auto-format the blank FFS sectors, it won't function normally with all ofthese "asset" files missing. Foxconn's original factory production line stationmust have uploaded these files to each phone via the TMFFS2 protocol, and ourFreeCalypso suite now features a tool that can replicate this feat: fc-fsio.Aftermarket FFS for FreeCalypso on Compal & Pirelli targets===========================================================When we run our own FreeCalypso fw on "alien" (not native to us) Mot C1xx andPirelli DP-L10 hardware, we don't use the FFS from their respectively originalfirmwares: those original FFS instances don't contain any bits of interest tous, trying to make our fw use the same FFS as Mot/Compal's or Pirelli's originalfw would be more trouble than benefit, and on one of the target devices in thisfamily (Mot C155/156) the original FFS is in some different format. Instead wecreate our own aftermarket FFS for our FreeCalypso fw on these alien hw targets,using a different flash location from the original so that the original fw's FFScannot be mistaken for our own.On the Pirelli DP-L10 we put our aftermarket FFS in an area of the flash whichthe official fw family uses as a staging area for over-the-air firmware updates,thus as long as you are not doing official fw updates over WLAN (i.e., if youonly run one fixed official fw version or flash different official fw versionswith fc-loadtool without going through "fw update" protocols), our aftermarketFFS used by our run-from-RAM FreeCalypso firmwares should remain undisturbedwhile the phone is in the official fw mode.The situation is different on Mot C1xx phones. The lower-end C1xx modelsincluding the C139 (our primary hw target in this family) have too little RAMto run our FreeCalypso fw entirely out of RAM without flashing; the C155/156subfamily does have enough RAM to allow a complete FC GSM fw image to be loadedand run via fc-xram under some conditions (we previously supported such usagein our now-retired Citrine fw and we also support it in the gcc-built config ofFC Selenite), but there is no place in the flash where we can put ouraftermarket FFS without overwriting some part of the original fw or its data -thus our general procedure for running FreeCalypso on any C1xx model is toconvert the victim phone to FC on a long-term basis by flashing our modifiedbootloader, flashing one of our fw builds and establishing a FreeCalypsoaftermarket FFS in a flash area designated by us.It was already mentioned earlier that the factory RF calibration values on thesealien phones are stored in non-TIFFS flash data structures of Compal's orFoxconn's invention, and our currently supported FreeCalypso firmwares(Magnetite and Selenite) do not contain any code for reading these alien datastructures. (FC Citrine could read directly from Pirelli's factory data block,but none of our fw offerings ever parsed Compal's weird flash records.) Insteadour current approach is to have an external tool extract the bits of interestfrom the alien factory records, convert them to our TI-standard format ifnecessary, and upload them into our FreeCalypso aftermarket FFS. The specificsare as follows:* The Pirelli DP-L10 is a breeze: simply run our special pirelli-magnetite-init command in fc-fsio while connected to a Magnetite or Selenite fw instance running with our aftermarket FFS, and the tool will copy both the IMEI and the RF calibration records from Pirelli's factory data block into our aftermarket FFS.* Mot C1xx phones present a lot more hassle: our current official procedure is to make a dump of the flash prior to the xenotransplantation procedure (also serves as a backup), extract the RF calibration values with our c1xx-calextr tool, and then later in the procedure when you initialize your aftermarket FFS with fc-fsio, upload these extracted and format-converted RF calibration files as one of the several steps involved. You will also need to enter your IMEI manually: fc-loadtool flash compal-imei command can extract the factory IMEI record from the flash chip's protection register and save it into a text file, but you still need to feed it manually to the new firmware with fc-fsio set-imeisv command.FreeCalypso host tools for TIFFS================================Our FC host tools package supports TIFFS in two ways:1) Our primary tool for working with GSM device file systems is fc-fsio. Whenrun against a compatible firmware version (primarily our own, but Pirelli'sproprietary fw is also compatible), fc-fsio allows various read and writeoperations to be performed on the target device FFS. fc-fsio can also be usedtogether with our fc-xram based FFS editing agent described below.2) We have a TIFFS In Vitro Analyzer (IVA) tool for "in vitro" examination ofFFS images that have been read out of raw flash with fc-loadtool. See theTIFFS-IVA-usage article for more information. As a very recent addition, wealso have another "in vitro" tool (tiffs-mkfs) that goes the other way, creatingnew complete TIFFS images from a tree of directories and files.In addition to the above, back in the days of Openmoko (back when the Openmokocommunity was still active and we considered ourselves a part of it) we hadproduced a kit for editing the modem FFS on Openmoko GTA01/02 devices, givingusers an easy way to change their /pcm/IMEI file. Changing IMEIs for no goodreason is completely pointless and is actually detrimental rather than helpfulfor privacy, but whoever came up with the edicts that "the IMEI MUST beimmutable" had obviously failed Human Psychology 101: declaring something to beforbidden causes people to want it simply because it is forbidden and for noother reason - hence the popular demand for IMEI changing tools.Our Openmoko FFS editing kit from early 2014 consisted of a very early versionof what much later became the present FC host tools package (more specifically,it was before fc-fsio, and the set-imeisv command had been hacked into fc-tmshinstead) plus a pair of "in vivo" FFS editing agent target binaries that run onthe target by way of fc-xram. Our current FC host tools fully supplant theancient version in that 2014 kit, and our current replacement for the ancientFFS editing agent is this new version:https://www.freecalypso.org/hg/ffs-editor/The new FFS editing agent linked above is run via fc-xram, while it is runningyou communicate with it via rvinterf (launched directly from fc-xram as the 2ndprogram), and you can run fc-fsio against it to perform whatever actual FFSmanipulations are needed.