--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/Binary-file-formats	Sun Mar 08 22:15:57 2020 +0000
@@ -0,0 +1,121 @@
+In FreeCalypso we use 3 different file formats for Calypso binary images, i.e.,
+code images to be loaded into either flash or RAM or retrieved flash dumps.
+These 3 different file formats are straight binary (*.bin), moko-style m0 (*.m0)
+and little-endian S-records (*.srec).
+
+Straight binary (*.bin)
+=======================
+
+Straight binary is our preferred format for flash dumps.  It is written in the
+native little-endian byte order of the Calypso ARM7 processor, i.e., the order
+of bytes in the raw binary file directly corresponds to incrementing byte
+addresses as visible to the ARM7 - any ASCII strings in the image thus appear
+naturally.  We also use the same straight binary format in native LE byte order
+for flashable code images generated with the gcc+binutils toolchain (as opposed
+to TI's TMS470), generated with arm-elf-objcopy -O binary - although we don't
+have too many such code images currently given that neither FC Citrine nor FC
+Selenite ever achieved production quality.
+
+Another unrelated use of this straight binary format is for RAM-loadable code
+images that are fed to Compal's bootloader (Motorola C1xx and Sony Ericsson
+J100) as opposed to Calypso boot ROM.  Our generally preferred image format for
+RAM-loadable code pieces is little-endian S-records (*.srec, see below), but
+for Compal's bootloader we use straight binary instead because of the way this
+bootloader protocol works.
+
+moko-style m0
+=============
+
+This format is a variant of Motorola hex (S-records), a variant invented by TI
+rather than by us.  This format is produced by TI's hex470 tool when run with
+-m -memwidth 16 -romwidth 16 options, which is the configuration used by TI in
+the Calypso program, and is read by TI's flash programming tool called FLUID.
+TI used this format not only for flashable firmware images, but also for various
+RAM-loadable code pieces, particularly those that comprise the target-side
+component of FLUID.
+
+The special quirk of this S-record variant format is its peculiar byte order.
+TI viewed it as "16-bit hex", meaning that the image is logically viewed as
+consisting of 16-bit words rather than 8-bit bytes, each S3 record carries an
+even number of bytes to be loaded at an even address, and each 16-bit word
+(4 hex digits) appears in these S3 records with its most-significant hex nibble
+toward the left, just like the address field of each S-record.  But if this
+image gets interpreted by some more naive tool (for example, objcopy from GNU
+binutils) as bytes rather than 16-bit words, the result will be a reversed byte
+order, with all strings etc messed up.
+
+In FreeCalypso we use this moko-style m0 format (our new name for what TI called
+16-bit hex) only for flashable firmware images built with TI's TMS470 toolchain
+(can be our own FC Magnetite or historical ones built by Openmoko or other
+similar historical vendors), but never for any RAM-loadable code pieces - we use
+little-endian SREC for the latter as explained below.
+
+And what about the name?  Why do we call it moko-style m0 rather than just m0?
+The reason is because Compal muddied our waters by introducing their own *.m0
+files that were generated with -memwidth 8 -romwidth 8 instead of -memwidth 16
+-romwidth 16, producing 8-bit hex instead of 16-bit hex.  We do not support
+Compal's different *.m0 files at all, and we needed some name to specifically
+identify TI-style m0 files rather than Compal-style.  We ended up with the name
+moko-style rather than TI-style because we already had our mokosrec2bin program
+going back to 2013-04-15, one of the very first programs written in the
+FreeCalypso family of projects: our very first encounter with this file format
+were mokoN firmware images put out in this *.m0 format by That Company.
+
+Little-endian S-records (*.srec)
+================================
+
+Back at the beginning of FreeCalypso in the spring/summer of 2013 I (Mother
+Mychaela) decided to use S-records instead of straight binary for our
+RAM-loadable code pieces, i.e., code that is loaded into RAM either through the
+Calypso boot ROM (fc-iram) or by chain-loading via loadagent (fc-xram).  I made
+this decision based on two factors:
+
+1) ARM code generated by common toolchains (both TI's TMS470 and gcc+binutils)
+   without special contortions is not position-independent: a code image that
+   was linked for a given address needs to be loaded at that specific address,
+   not some other.
+
+2) An S-record image has its load address embedded in the image itself, whereas
+   a raw binary naturally does not carry any such extra metadata.
+
+With SREC as the standardized hand-off format from code generation tools to
+loadtools, the choice of load address is made entirely on the code generation
+side; loadtools do not impose a fixed load address, nor do they require it to
+be communicated via extra command line arguments or options.
+
+However, the variant of SREC we use for RAM-loadable code pieces is not the same
+as moko-style m0 - the byte order is the opposite, with our RAM-loadable code
+pieces using the native little-endian byte order of the ARM7 processor as the
+byte order within S3 records.  Prior to the introduction of RAM-loadable FC
+Magnetite fw images for Pirelli DP-L10 in late 2016 (and then likewise for our
+own FCDEV3B), the only RAM-loadable code pieces we have had were built with
+gcc+binutils, not with TMS470, and GNU binutils got a different take on the
+S-record format than TI did: they generate byte-oriented SREC files, with the
+byte order being the same as it would be in a straight binary file, matching
+the target processor's memory byte addressing order.  Thus GNU-style SREC has
+been adopted as the format for our RAM-loadable code images for both fc-iram
+and fc-xram, as opposed to TI-style SREC aka moko-style m0.  The convention we
+have adopted is that *.m0 filename suffix means TI-style aka moko-style,
+whereas *.srec means GNU-style.
+
+Besides the S3 record byte order, there is one other difference between TI-built
+*.m0 code images and GNU-built *.srec ones: the final S7 record carries the
+entry point address in GNU-built *.srec images, whereas TI's *.m0 images always
+have a zero dummy address in there.  Our fc-iram and fc-xram tools require the
+real entry point address in the S7 record.
+
+How do we generate ramimage.srec RAM-loadable images for fc-xram in FC
+Magnetite?  Answer: FC Magnetite build system includes a special ad hoc
+converter program that reads ramimage.m0 produced by TI's hex470 tool and
+produces ramimage.srec: it reverses the order of bytes, adds another S3 record
+that writes the boot-ROM-redirected interrupt and exception vectors and
+generates an S7 record with the right entry point address.
+
+This little-endian *.srec format is actively used only for RAM-loadable code
+pieces in FreeCalypso, not for anything that goes into or gets read from flash.
+We do have flash dump2srec and flash program-srec commands in fc-loadtool, they
+were implemented back in the founding stage of FreeCalypso in 2013 for the sake
+of completeness and symmetry (it seemed right to support both binary and
+S-record formats), but they never got any practical use: if you are making a
+flash dump, you would normally want to examine it afterward, and any such
+examination almost always needs a straight binary image, not S-records.

--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/Flash-programming	Sun Mar 08 22:15:57 2020 +0000
@@ -0,0 +1,199 @@
+Our Calypso device flash programming tool fc-loadtool can be used in several
+different paradigms; this article is an attempt to clarify the many available
+modes of usage.  You should also read the companion article Binary-file-formats
+for further background.
+
+Flashing firmware release images
+================================
+
+In conventional forward engineering environments where you develop or maintain
+firmware for hardware made by your own company (meaning no reverse eng, no
+"illicit" aftermarket tinkering on hw made by some other company aeons ago),
+you have a firmware build system that produces fw build images (some of which
+may subsequently be blessed as releases), and you have a tool that flashes
+these fw build images into your hardware, operating as efficiently as possible,
+automated as much as possible, requiring minimal user action for the boring
+repetitive task of flashing a new fw image every time you build one.  And if
+you become lucky enough to produce your hardware in volume, the same objectives
+of maximal efficiency and automation carry over into the production line
+environment as well.
+
+In TI's environment the standardized format for firmware build images which are
+then flashable into hardware targets was a variant of Motorola SREC written in
+*.m0 files, a variant which we have named moko-style m0 after its most famous
+user.  The special quirk of this particular SREC variant is its peculiar byte
+order.  TI's firmware build system produces a *.m0 S-record image as its final
+build product, and TI's official Calypso flash programming tool (FLUID) takes
+these *.m0 files as its input.
+
+Since the beginning of FreeCalypso we have had two ways of flashing TI-built
+firmware images into suitable targets (initially OM GTA02 modem, then many
+others including our own FCDEV3B):
+
+1) Our fc-loadtool has had a flash program-m0 command from the beginning,
+   programming device flash with bits from an m0 file directly and natively.
+   However, prior to fc-host-tools-r12 this command was poorly supported: it
+   ran significantly slower than flash program-bin, had poorer progress
+   indication and did not perform CRC-32 verification at the end, which is an
+   important integrity check.  Also this original flash program-m0 command (as
+   opposed to flash e-program-m0 added in fc-host-tools-r13) does not include a
+   built-in erase step, thus prior to fc-host-tools-r13 the user had to have
+   outside knowledge of how many sectors to erase first with a separate flash
+   erase command.
+
+   The new flash e-program-m0 command added in fc-host-tools-r13 is m0 image
+   flashing finally done right.  It reads in the specified S-record image in
+   moko-style m0 format, builds a map of potentially discontiguous flash
+   regions into which the image deposits bits, erases the set of flash sectors
+   which need to be erased before programming these regions, then programs the
+   new image bits into flash, exactly like TI's own FLUID.
+
+2) The alternative way is to first convert the *.m0 S-record image produced by
+   TI's hex470 post-linker tool into straight binary (*.bin) with a FreeCalypso
+   tool called mokosrec2bin, then program the binary fw image into flash with
+   fc-loadtool command flash program-bin.  This method is the one we've been
+   using since 2017, and our FC Magnetite firmware build system is now set up
+   to produce not only fwimage.m0, but also fwimage.bin (it runs mokosrec2bin),
+   and it also generates an fc-loadtool command script (a text file named
+   flash-script) with two commands in it: a flash erase command with a
+   calculated sector address range and a flash program-bin command to program
+   the accompanying fwimage.bin image.
+
+As of fc-host-tools-r13 both methods work equally well: if you have an official
+FreeCalypso firmware release (containing fwimage.m0, fwimage.bin and
+flash-script files) which you need to flash into a device such as our own
+FCDEV3B or OM GTA02 (but *not* Mot C1xx!), you can execute either
+'exec flash-script' or 'flash e-program-m0 fwimage.m0' at the loadtool> prompt,
+and both ways will produce exactly the same result with equal performance and
+reliability.  And if you need a more special operation such as erasing the
+entire flash (factory production lines) or erasing and reprogramming only a
+certain part of the normally affected sector range, that's what custom command
+scripting ability is for.
+
+For the sake of symmetry, we also have a flash e-program-bin command that is a
+binary image format counterpart to flash e-program-m0: it first erases the
+sectors into which new bits will be programmed, then programs the new bits.
+Thus a third equally good way to flash a new FreeCalypso fw release into a
+target such as FCDEV3B or GTA02 is to execute
+'flash e-program-bin 0 fwimage.bin' - but don't *ever* do it on a Mot C1xx
+phone!
+
+Flash backup and restore
+========================
+
+A completely different paradigm takes place on alien targets such as Motorola
+C1xx and Pirelli DP-L10, made by alien manufacturers, meaning not FreeCalypso,
+not Openmoko and not TI.  The most important flash operation on these alien
+targets is making a flash dump; these dumps can then be used for forensics,
+reverse engineering, or simply as a backup.  When we subsequently write to
+flash on these alien targets (after having saved a backup first), we are not
+flashing an m0 fw image or a binary image made from one with mokosrec2bin,
+instead the most common operations are:
+
+* Flashing a backup image back into the same device it was originally made
+  from (flash restore);
+
+* Changing a device from one firmware version to a different one by programming
+  its flash with firmware bits that were originally read out from some fw-donor
+  unit;
+
+* Surgical manipulations such as erasing FFS sectors or rewriting one specific
+  part of the flash based on reverse-engineered understanding of its structure.
+
+This different paradigm leads to a different mode of usage for fc-loadtool:
+instead of needing a maximally-automated operation that flashes a firmware
+release image with as little user thought involvement as possible, our flash
+manipulations need to be of a more manual peek-n-poke manner.  We provide a
+flash dump2bin command for making and saving flash dumps first and foremost,
+allowing any part of the flash to be dumped and saved selectively if desired,
+including the second flash bank on the Pirelli DP-L10 and likewise on our own
+FCDEV3B.  When it comes to flash write operations, we provide a manual flash
+erase command that allows (and requires) the operator to specify exactly which
+sector range should be erased and a manual flash program-bin command that
+allows any range of 16-bit words to be programmed at any flash address, with
+the bits to be programmed coming from a binary file, either the whole file or
+any specified subrange.
+
+These manual flash erase and flash program-bin commands give full control to
+the operator, allowing every possible flash manipulation which the hardware
+itself allows, at the expense of requiring the operator to think about which
+flash addresses, offsets and lengths need to be operated on, and either enter
+long commands manually or write a command script.
+
+Given our historical origins (long before we got to the point of producing our
+own hardware, we started out by exploring the forbidden GSM realm of devices
+made by alien manufacturers who were hostile to our cause), our original flash
+manipulation support in fc-loadtool had been centered around the manual
+peek-n-poke paradigm, with elementary flash erase and flash program-bin commands
+as our main staple, and no thought had been given originally to producing
+functionality that would work like FLUID or like our current flash e-program-m0
+and e-program-bin commands.  But all actively maintained software evolves, and
+as our FreeCalypso family of projects has matured over the years, we now offer
+richer functionality covering a wider range of use cases.
+
+Binary vs. S-records
+====================
+
+(Please read the companion article Binary-file-formats for background, then
+ come back here.)
+
+If you are exploring and manipulating the flash content of a GSM device in an
+aftermarket fashion, as opposed to flashing your own fw builds into your own hw
+design produced by your own company like Openmoko did in the late 2000s and
+like we do currently at FreeCalypso HQ, then binary is the generally preferred
+format: you make dumps with flash dump2bin, and when you selectively program
+these images back into devices, you use flash program-bin with the right offsets
+and length, along with appropriate flash erase commands.
+
+We also have flash dump2srec and flash program-srec commands in fc-loadtool,
+they were implemented back in the founding stage of FreeCalypso in 2013 for the
+sake of completeness and symmetry (it seemed right to support both binary and
+S-record formats), but they never got any practical use: if you are making a
+flash dump, you would normally want to examine it afterward, and any such
+examination almost always needs a straight binary image, not S-records.
+Furthermore, our flash program-bin command allows you to selectively program
+just a particular portion of a binary image file into flash, at any arbitrary
+flash address, but we don't have the same flexibility with flash program-srec -
+the latter command is really just a sibling of program-m0 with the opposite
+byte order.
+
+Thus the short summary is as follows:
+
+* If you are flashing an official firmware release image into your device, you
+  need to use flash e-program-bin or flash e-program-m0 depending on whether
+  the image is provided in *.bin or *.m0 format, or alternatively our older
+  flash program-bin or flash program-m0 commands preceded by a separate flash
+  erase command with the right sector range, possibly packaged in a supplied
+  fc-loadtool command script.
+
+* If you are restoring a flash dump made with flash dump2bin or performing
+  aftermarket flash manipulations on Mot C1xx or Pirelli DP-L10 phones or other
+  such alien devices, you need to use binary-format-based flash manipulations
+  commands; the specific commands will depend on exactly what you are seeking
+  to do.
+
+* flash program-srec and e-program-srec commands do not currently have a valid
+  use case.
+
+Special considerations for Compal phones
+========================================
+
+Motorola C1xx and Sony Ericsson J100 phones made by Compal have brickable flash:
+the right kind of flash-resident bootloader must always be present at the
+beginning of the flash, or else the phone is unrecoverably bricked.  We have
+special support in fc-loadtool for minimizing the bricking vulnerability window
+when operating on these phones, but this special support requires user
+cooperation, meaning that you must limit your flash manipulations on these
+phones to a narrower subset:
+
+* flash program-m0, program-srec, e-program-m0 and e-program-srec commands are
+  not appropriate for these brickable phones - do not use any of these commands
+  on these targets.
+
+* Flash sector 0 must be manipulated only with the special
+  flash erase-program-boot command, not any of the regular erase or program
+  commands.
+
+* Regular flash erase, flash program-bin and flash e-program-bin commands can
+  and should be used for the rest of the flash starting at offset 0x10000 - but
+  you still need to understand what you are doing.