Flash file system in Sony Ericsson K2xx phones
==================================================

SE K2xx phones are based on our familiar Calypso chipset, their firmware is
based on the standard chipset reference fw from TI, and they use TIFFS as their
flexible data storage mechanism.  Their TIFFS instance is located in the first
3328 KiB of the 2nd flash bank; the geometry is 256x13 or 64x52 depending on
which flash chip was used in each given phone.  (Some specimen have Spansion
S71PL129NB0 chips with 256 KiB flash sectors, others feature Samsung K5L2931CAM
chips with 64 KiB flash sectors.)

Calibration data and self-regenerating ability
==============================================

SE's FFS life cycle design on these phones is interesting in that vital per-unit
calibration data (RF, AFC and MADC calibrations) are stored in two places:
(1) in the FFS in TI's standard format and locations, and (2) in a dedicated
flash sector outside of FFS that appears to be written once at the factory and
never touched afterward.

SE's firmware design allows the FFS to be fully erased without breaking the
phone: whenever the fw boots in a state where the FFS area of the flash is fully
erased (you can of course erase those FFS sectors with fc-loadtool, but it
appears that some official tools also provide the same operation) but the
separate factory calibration data sector is valid, the firmware formats and
initializes a new FFS, and furthermore copies all essential calibration records
from SE's proprietary sector structure into their standard TIFFS locations!

But it gets even more interesting - when we examine FFS images read out of
various SE K2xx specimen, we see two different occurrences:

1) In some specimen the FFS content is exactly the same as what we would get by
   erasing the FFS with fc-loadtool and letting the firmware regenerate a fresh
   one on its next boot.  We can thus reasonably assume that the FFS in these
   specimen is indeed the product of such intervention at some point in the
   phone's history.

2) In other specimen we see some additional files beyond those that exist with
   the life cycle of the previous paragraph.  One readily noticeable addition
   is the presence of /gsm/rf/tx/ramps.* files, which are absent in the self-
   regenerated FFS where the phone's fw copied calibration records from SE's
   dedicated calibration data sector.  How are these Tx ramp tables different
   from all other RF calibration records?  Answer: in the present SE K2xx
   design, and also in all other Calypso phone and modem designs known to this
   Mother, Tx ramps are calibrated per design and not per unit.  Because the
   correct Tx ramp tables are compiled into the firmware, having them in FFS is
   redundant.

Based on the above evidence, I (Mother Mychaela) hereby make the following
reconstruction of how these two kinds of FFS found in the wild likely came into
being:

* In the case of Compal (Mot C1xx and SE J100) and Pirelli DP-L10 we know by
  simple deductive reasoning that those firmwares must have had their l1_cust.c
  code modified to read calibration records from their respective proprietary
  flash data structures instead of TI's original FFS files.  However, in the
  case of SE K2xx it is plausible that their l1_cust.c did NOT have to be
  likewise modified - they could plausibly still have TI's original l1_cust.c
  code that reads RF tables from FFS on initialization and writes them back
  into FFS upon those special MISC_ENABLE commands that are issued at the end
  of TI's original calibration procedures.

* If my hypothesis above is correct, then it follows that when SE or their ODM
  did their factory production line calibration, the results were initially
  written into FFS and not into SE's separate factory calibration data sector.
  If they are using unmodified l1_cust.c code from TI, then the presence of
  redundant /gsm/rf/tx/ramps.* files is explained - TI's standard calibration
  procedure results in these files being written into FFS because the canonical
  l1_cust.c code classifies them as "Tx calibration" rather than "Tx config" -
  see our RF_tables article.

* In a separate step after all standard calibration procedures, they must have
  copied all important calibration records into their non-standard flash sector
  at 0x01FD0000, specifically to produce a more durable copy that can persist
  through full erasure of FFS.  Tx ramp tables were not included because they
  are relatively large and redundant, not actually calibrated per unit.

* Those SE K2xx specimen whose FFS contains /gsm/rf/tx/ramps.* files and some
  other files that aren't found in auto-regenerated FFS probably exhibit the
  original FFS from the factory production line that was never subjected to
  being blown away and regenerated!

SE's non-standard extension to TIFFS
====================================

There is one aspect of SE K2xx FFS that creates a pain point for our tools:
they made their own non-standard extension to TIFFS in terms of extended UCS-2
filenames, and given our natural desire to be able to use our tiffs tool (TIFFS
IVA) to examine FFS images from these SE phones, we had to extend our tool so
that it parses SE's extension instead of throwing up errors like it did before.

As explained in our TIFFS-Overview article, classic TIFFS requires every
elementary filename to consist of printable ASCII characters from the sensibly
narrow set of [A-Za-z0-9_.,+%$#-], further limited to a maximum of 20
characters.  However, SE K2xx FFS images often contain some files that violate
this constraint; deeper examination reveals that SE (or their ODM) devised a
feature to allow filenames composed of UCS-2 characters, implemented as follows:

* The first "character" of the filename written into TIFFS is \x02, i.e., NOT a
  printable ASCII character;

* The next "character" in the TIFFS object name is another binary byte encoding
  the length of the UCS-2 string: for example, if the "high-level" name consists
  of 6 UCS-2 characters, this second TIFFS object name "character" is \x06;

* These two non-printable-ASCII chars are followed by a long string of [0-9A-F]
  ASCII characters, encoding the "high-level" UCS-2 name in hex;

* The whole arrangement ends with a terminating NUL, allowing the rest of TIFFS
  to work unchanged.

Seen from TIFFS perspective, these extended filenames created by SE K2xx fw
violate traditional canon in two ways:

1) The overall length of the mostly-hex-encoded filename string usually exceeds
   the traditional 20 character limit;

2) The set of allowed characters is grossly violated: not an innocuous extension
   of allowing some additional "sane" characters, but non-printable-ASCII binary
   chars that are barely compatible with C string functions by virtue of not
   containing zero bytes.

Please see the new section titled "Support for SE K2xx extended filenames" in
our TIFFS-IVA-usage article for the explanation of how we handle these extended
filenames in our TIFFS IVA tool.

fc-fsio considerations
======================

In addition to "in vitro" analysis with our tiffs tool, SE K2xx FFS can also be
accessed "in vivo" with fc-fsio in the following two scenarios:

1) SE K2xx firmware does not have RVTMUX enabled by default, but they have
   non-volatile flags in their FFS (which can be set via a hidden menu entered
   via secret MMI code #*87223564#) that enable RVTMUX on the MODEM UART that
   is brought out, and their RVTMUX includes ETM and TMFFS2 protocols.

2) We can run TI's original FFS code against SE K2xx FFS bodies by adding the
   necessary configuration to our fc-xram based FFS editor tool described in
   our TIFFS-Overview article.

The expected behaviour in scenario 2 can be predicted statically by studying
the code, whereas scenario 1 calls for experimentation.  Experiments with
RVTMUX-enabled SE K2xx original fw reveal that it does behave (at least as far
as visible behaviour over TMFFS2) exactly like TI's original FFS code in terms
of readdir output when listing directories with extended filenames in them: as
long as they fit into TMFFS2 string buffer, these extended filenames are
returned just as one would naively expect.

In light of these observations, fc-fsio has also been patched up to behave
gracefully when faced with such previously unexpected readdir results from the
target.  Specifically:

* The short form of 'ls' command will print whatever the target firmware
  returns, including extended filenames; any non-printable-ASCII characters are
  escaped in \x hex form.

* 'ls -l' or 'll' will throw up an error if one attempts to list a directory
  that contains one or more extended filenames, caught either at the level of
  the readdir result not fitting into the buffer sized for standard TIFFS
  filenames or (for SE K2xx extended filenames of 4 or fewer UCS-2 characters)
  at the level of this readdir result containing non-printable-ASCII chars.

Finally, the following limitations need to be kept in mind:

* There is no way to address an FFS object with an extended filename by entering
  a pathname in fc-fsio commands;

* File systems or subtrees containing extended filenames cannot be read out with
  fc-fsio cpout command: it will fail just like ls -l;

* rm-subtree and cleandir commands will likewise fail and stop whenever they
  encounter a filename longer than 20 characters (invalid for the purpose of
  pathname generation based on fixed-length buffers) anywhere in the subtree to
  be deleted.

The last limitation (inability to delete "bad" files) may seem like a serious
flaw in the design of fc-fsio at first, but please realize that the primary
mission of FreeCalypso is to work with our own hardware and firmware, not alien
- and the original FFS code from TI, which is what we use in FreeCalypso, will
never allow file system objects with such bad filenames to be created in the
first place.  Therefore, the limitation of fc-fsio being unable to manipulate
certain files to the extent of not even being able to delete them is specific
to the peculiar scenario of operating on *alien* FFS from Sony Ericsson.