--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/SE-K2x0-FFS	Mon Jan 02 00:50:19 2023 +0000
@@ -0,0 +1,184 @@
+Flash file system in Sony Ericsson K200/220 phones
+==================================================
+
+SE K200/220 phones are based on our familiar Calypso chipset, their firmware is
+based on the standard chipset reference fw from TI, and they use TIFFS as their
+flexible data storage mechanism.  Their TIFFS instance is located in the first
+3328 KiB of the 2nd flash bank; the geometry is 256x13 or 64x52 depending on
+which flash chip was used in each given phone.  (Some specimen have Spansion
+S71PL129NB0 chips with 256 KiB flash sectors, others feature Samsung K5L2931CAM
+chips with 64 KiB flash sectors.)
+
+Calibration data and self-regenerating ability
+==============================================
+
+SE's FFS life cycle design on these phones is interesting in that vital per-unit
+calibration data (RF, AFC and MADC calibrations) are stored in two places:
+(1) in the FFS in TI's standard format and locations, and (2) in a dedicated
+flash sector outside of FFS that appears to be written once at the factory and
+never touched afterward.
+
+SE's firmware design allows the FFS to be fully erased without breaking the
+phone: whenever the fw boots in a state where the FFS area of the flash is fully
+erased (you can of course erase those FFS sectors with fc-loadtool, but it
+appears that some official tools also provide the same operation) but the
+separate factory calibration data sector is valid, the firmware formats and
+initializes a new FFS, and furthermore copies all essential calibration records
+from SE's proprietary sector structure into their standard TIFFS locations!
+
+But it gets even more interesting - when we examine FFS images read out of
+various SE K2x0 specimen, we see two different occurrences:
+
+1) In some specimen the FFS content is exactly the same as what we would get by
+   erasing the FFS with fc-loadtool and letting the firmware regenerate a fresh
+   one on its next boot.  We can thus reasonably assume that the FFS in these
+   specimen is indeed the product of such intervention at some point in the
+   phone's history.
+
+2) In other specimen we see some additional files beyond those that exist with
+   the life cycle of the previous paragraph.  One readily noticeable addition
+   is the presence of /gsm/rf/tx/ramps.* files, which are absent in the self-
+   regenerated FFS where the phone's fw copied calibration records from SE's
+   dedicated calibration data sector.  How are these Tx ramp tables different
+   from all other RF calibration records?  Answer: in the present SE K2x0
+   design, and also in all other Calypso phone and modem designs known to this
+   Mother, Tx ramps are calibrated per design and not per unit.  Because the
+   correct Tx ramp tables are compiled into the firmware, having them in FFS is
+   redundant.
+
+Based on the above evidence, I (Mother Mychaela) hereby make the following
+reconstruction of how these two kinds of FFS found in the wild likely came into
+being:
+
+* In the case of Compal (Mot C1xx and SE J100) and Pirelli DP-L10 we know by
+  simple deductive reasoning that those firmwares must have had their l1_cust.c
+  code modified to read calibration records from their respective proprietary
+  flash data structures instead of TI's original FFS files.  However, in the
+  case of SE K2x0 it is plausible that their l1_cust.c did NOT have to be
+  likewise modified - they could plausibly still have TI's original l1_cust.c
+  code that reads RF tables from FFS on initialization and writes them back
+  into FFS upon those special MISC_ENABLE commands that are issued at the end
+  of TI's original calibration procedures.
+
+* If my hypothesis above is correct, then it follows that when SE or their ODM
+  did their factory production line calibration, the results were initially
+  written into FFS and not into SE's separate factory calibration data sector.
+  If they are using unmodified l1_cust.c code from TI, then the presence of
+  redundant /gsm/rf/tx/ramps.* files is explained - TI's standard calibration
+  procedure results in these files being written into FFS because the canonical
+  l1_cust.c code classifies them as "Tx calibration" rather than "Tx config" -
+  see our RF_tables article.
+
+* In a separate step after all standard calibration procedures, they must have
+  copied all important calibration records into their non-standard flash sector
+  at 0x01FD0000, specifically to produce a more durable copy that can persist
+  through full erasure of FFS.  Tx ramp tables were not included because they
+  are relatively large and redundant, not actually calibrated per unit.
+
+* Those SE K2x0 specimen whose FFS contains /gsm/rf/tx/ramps.* files and some
+  other files that aren't found in auto-regenerated FFS probably exhibit the
+  original FFS from the factory production line that was never subjected to
+  being blown away and regenerated!
+
+SE's non-standard extension to TIFFS
+====================================
+
+There is one aspect of SE K2x0 FFS that creates a pain point for our tools:
+they made their own non-standard extension to TIFFS in terms of extended UCS-2
+filenames, and given our natural desire to be able to use our tiffs tool (TIFFS
+IVA) to examine FFS images from these SE phones, we had to extend our tool so
+that it parses SE's extension instead of throwing up errors like it did before.
+
+As explained in our TIFFS-Overview article, classic TIFFS requires every
+elementary filename to consist of printable ASCII characters from the sensibly
+narrow set of [A-Za-z0-9_.,+%$#-], further limited to a maximum of 20
+characters.  However, SE K2x0 FFS images often contain some files that violate
+this constraint; deeper examination reveals that SE (or their ODM) devised a
+feature to allow filenames composed of UCS-2 characters, implemented as follows:
+
+* The first "character" of the filename written into TIFFS is \x02, i.e., NOT a
+  printable ASCII character;
+
+* The next "character" in the TIFFS object name is another binary byte encoding
+  the length of the UCS-2 string: for example, if the "high-level" name consists
+  of 6 UCS-2 characters, this second TIFFS object name "character" is \x06;
+
+* These two non-printable-ASCII chars are followed by a long string of [0-9A-F]
+  ASCII characters, encoding the "high-level" UCS-2 name in hex;
+
+* The whole arrangement ends with a terminating NUL, allowing the rest of TIFFS
+  to work unchanged.
+
+Seen from TIFFS perspective, these extended filenames created by SE K2x0 fw
+violate traditional canon in two ways:
+
+1) The overall length of the mostly-hex-encoded filename string usually exceeds
+   the traditional 20 character limit;
+
+2) The set of allowed characters is grossly violated: not an innocuous extension
+   of allowing some additional "sane" characters, but non-printable-ASCII binary
+   chars that are barely compatible with C string functions by virtue of not
+   containing zero bytes.
+
+Please see the new section titled "Support for SE K2x0 extended filenames" in
+our TIFFS-IVA-usage article for the explanation of how we handle these extended
+filenames in our TIFFS IVA tool.
+
+fc-fsio considerations
+======================
+
+In addition to "in vitro" analysis with our tiffs tool, SE K2x0 FFS can also be
+accessed "in vivo" with fc-fsio in the following two scenarios:
+
+1) SE K2x0 firmware does not have RVTMUX enabled by default, but they have
+   non-volatile flags in their FFS (which can be set via a hidden menu entered
+   via secret MMI code #*87223564#) that enable RVTMUX on the MODEM UART that
+   is brought out, and their RVTMUX includes ETM and TMFFS2 protocols.
+
+2) We can run TI's original FFS code against SE K2x0 FFS bodies by adding the
+   necessary configuration to our fc-xram based FFS editor tool described in
+   our TIFFS-Overview article.
+
+The expected behaviour in scenario 2 can be predicted statically by studying
+the code, whereas scenario 1 calls for experimentation.  Experiments with
+RVTMUX-enabled SE K2x0 original fw reveal that it does behave (at least as far
+as visible behaviour over TMFFS2) exactly like TI's original FFS code in terms
+of readdir output when listing directories with extended filenames in them: as
+long as they fit into TMFFS2 string buffer, these extended filenames are
+returned just as one would naively expect.
+
+In light of these observations, fc-fsio has also been patched up to behave
+gracefully when faced with such previously unexpected readdir results from the
+target.  Specifically:
+
+* The short form of 'ls' command will print whatever the target firmware
+  returns, including extended filenames; any non-printable-ASCII characters are
+  escaped in \x hex form.
+
+* 'ls -l' or 'll' will throw up an error if one attempts to list a directory
+  that contains one or more extended filenames, caught either at the level of
+  the readdir result not fitting into the buffer sized for standard TIFFS
+  filenames or (for SE K2x0 extended filenames of 4 or fewer UCS-2 characters)
+  at the level of this readdir result containing non-printable-ASCII chars.
+
+Finally, the following limitations need to be kept in mind:
+
+* There is no way to address an FFS object with an extended filename by entering
+  a pathname in fc-fsio commands;
+
+* File systems or subtrees containing extended filenames cannot be read out with
+  fc-fsio cpout command: it will fail just like ls -l;
+
+* rm-subtree and cleandir commands will likewise fail and stop whenever they
+  encounter a filename longer than 20 characters (invalid for the purpose of
+  pathname generation based on fixed-length buffers) anywhere in the subtree to
+  be deleted.
+
+The last limitation (inability to delete "bad" files) may seem like a serious
+flaw in the design of fc-fsio at first, but please realize that the primary
+mission of FreeCalypso is to work with our own hardware and firmware, not alien
+- and the original FFS code from TI, which is what we use in FreeCalypso, will
+never allow file system objects with such bad filenames to be created in the
+first place.  Therefore, the limitation of fc-fsio being unable to manipulate
+certain files to the extent of not even being able to delete them is specific
+to the peculiar scenario of operating on *alien* FFS from Sony Ericsson.

--- a/doc/TIFFS-IVA-usage	Sat Dec 31 22:55:23 2022 +0000
+++ b/doc/TIFFS-IVA-usage	Mon Jan 02 00:50:19 2023 +0000
@@ -190,3 +190,27 @@
 
 Just like regular cat, but takes an inode number instead of a pathname.  Can be
 used to cat the old content of deleted or overwritten files.
+
+Support for SE K2x0 extended filenames
+======================================
+
+As explained in the SE-K2x0-FFS article, Sony Ericsson K200/220 phones use
+TIFFS as their flexible data storage mechanism, but they also made their own
+non-standard extension to TIFFS in terms of extended UCS-2 filenames.
+
+In order to allow listing full FFS trees from SE K2x0 phones with tiffs ls and
+possibly also extracting them with tiffs xtr, we have made the following hackish
+extension to our TIFFS IVA: whenever either of these two commands encounters an
+"extended" filename (one which fails standard TIFFS filename validity checks but
+has a terminating NUL that allows the name and data portions of the chunk to be
+separated) in the visible tree walk, our tool replaces it (for the purpose of
+listed pathname generation or extraction into host file system) with a
+synthesized pseudonym of the form "!bogo%04x", where %04x is the FFS inode
+number in hex.  In the case of tiffs ls, the actual extended filename is then
+printed on a separate line (immediately following the synthesized listing), with
+non-printable-ASCII characters escaped in \x hex form.  We've also extended the
+individual-inode form of lsino command to list these extended filenames when
+encountering such, and we've extended catino (which needs to skip past the name
+to get to the beginning of the data portion of the head chunk) to not reject
+file head chunks with extended filenames in them as long as the needed
+separating NUL is found.