Our Calypso device flash programming tool fc-loadtool can be used in several
different paradigms; this article is an attempt to clarify the many available
modes of usage.  You should also read the companion article Binary-file-formats
for further background.

Flashing firmware release images
================================

In conventional forward engineering environments where you develop or maintain
firmware for hardware made by your own company (meaning no reverse eng, no
"illicit" aftermarket tinkering on hw made by some other company aeons ago),
you have a firmware build system that produces fw build images (some of which
may subsequently be blessed as releases), and you have a tool that flashes
these fw build images into your hardware, operating as efficiently as possible,
automated as much as possible, requiring minimal user action for the boring
repetitive task of flashing a new fw image every time you build one.  And if
you become lucky enough to produce your hardware in volume, the same objectives
of maximal efficiency and automation carry over into the production line
environment as well.

In TI's environment the standardized format for firmware build images which are
then flashable into hardware targets was a variant of Motorola SREC written in
*.m0 files, a variant which we have named moko-style m0 after its most famous
user.  The special quirk of this particular SREC variant is its peculiar byte
order.  TI's firmware build system produces a *.m0 S-record image as its final
build product, and TI's official Calypso flash programming tool (FLUID) takes
these *.m0 files as its input.

Since the beginning of FreeCalypso we have had two ways of flashing TI-built
firmware images into suitable targets (initially OM GTA02 modem, then many
others including our own FCDEV3B):

1) Our fc-loadtool has had a flash program-m0 command from the beginning,
   programming device flash with bits from an m0 file directly and natively.
   However, prior to fc-host-tools-r12 this command was poorly supported: it
   ran significantly slower than flash program-bin, had poorer progress
   indication and did not perform CRC-32 verification at the end, which is an
   important integrity check.  Also this original flash program-m0 command (as
   opposed to flash e-program-m0 added in fc-host-tools-r13) does not include a
   built-in erase step, thus prior to fc-host-tools-r13 the user had to have
   outside knowledge of how many sectors to erase first with a separate flash
   erase command.

   The new flash e-program-m0 command added in fc-host-tools-r13 is m0 image
   flashing finally done right.  It reads in the specified S-record image in
   moko-style m0 format, builds a map of potentially discontiguous flash
   regions into which the image deposits bits, erases the set of flash sectors
   which need to be erased before programming these regions, then programs the
   new image bits into flash, exactly like TI's own FLUID.

2) The alternative way is to first convert the *.m0 S-record image produced by
   TI's hex470 post-linker tool into straight binary (*.bin) with a FreeCalypso
   tool called mokosrec2bin, then program the binary fw image into flash with
   fc-loadtool command flash program-bin.  This method is the one we've been
   using since 2017, and our FC Magnetite firmware build system is now set up
   to produce not only fwimage.m0, but also fwimage.bin (it runs mokosrec2bin),
   and it also generates an fc-loadtool command script (a text file named
   flash-script) with two commands in it: a flash erase command with a
   calculated sector address range and a flash program-bin command to program
   the accompanying fwimage.bin image.

As of fc-host-tools-r13 both methods work equally well: if you have an official
FreeCalypso firmware release (containing fwimage.m0, fwimage.bin and
flash-script files) which you need to flash into a device such as our own
FCDEV3B or OM GTA02 (but *not* Mot C1xx!), you can execute either
'exec flash-script' or 'flash e-program-m0 fwimage.m0' at the loadtool> prompt,
and both ways will produce exactly the same result with equal performance and
reliability.  And if you need a more special operation such as erasing the
entire flash (factory production lines) or erasing and reprogramming only a
certain part of the normally affected sector range, that's what custom command
scripting ability is for.

For the sake of symmetry, we also have a flash e-program-bin command that is a
binary image format counterpart to flash e-program-m0: it first erases the
sectors into which new bits will be programmed, then programs the new bits.
Thus a third equally good way to flash a new FreeCalypso fw release into a
target such as FCDEV3B or GTA02 is to execute
'flash e-program-bin 0 fwimage.bin' - but don't *ever* do it on a Mot C1xx
phone!

Discontiguous m0 fw images with large gaps
==========================================

One sometimes-advantage of S-record-based formats over straight binary is that
they can have discontiguous filled regions with gaps in between them; all m0 fw
images produced by TI's TCS211 build system have some gaps in them.  In our
current FC Magnetite firmwares these gaps have been reduced to less than 16384
bytes in total, reducing the wasted disk space and image transfer time to
sufficiently insignificant if these m0 images are converted to straight binary
with mokosrec2bin and then flashed as binary as we've been doing.  For
comparison, in Openmoko's classic mokoN firmwares (almost TI's original version
of the memory map) the first flash sector of 65536 bytes had only 5072 bytes of
image payload programmed into it, thus indirect programming via mokosrec2bin
followed by flash program-bin meant sending just over 60k filler 0xFF bytes to
be flashed - becomes noticeable at 115200 baud.

But these gaps were even bigger in TI's classic firmwares (and many vendor
firmwares) built for 8 MiB or 16 MiB flash chips with TI's classic memory map.
They have two code regions: one region is exactly 4 MiB in size but is placed
not quite at the beginning, thus spilling past the chip midpoint, and the other
region starts where the first 4 MiB region ends, going to the end of the flash
chip or to the beginning of the area reserved for FFS.  The way TI's linker
works results in the first of these two program regions being not quite full
before bits start getting thrown into the second one, thus the resulting m0
image can easily have a gap of a megabyte or more between the two regions.
TI's own FLUID handles such discontiguous images with no sweat, but our
fc-loadtool has gained matching capability only as of fc-host-tools-r13 with
our new e-program-m0 command.  And yes, it is smart enough to avoid needlessly
erasing sectors in the big gap, just like FLUID.

Flash backup and restore
========================

A completely different paradigm takes place on alien targets such as Motorola
C1xx and Pirelli DP-L10, made by alien manufacturers, meaning not FreeCalypso,
not Openmoko and not TI.  The most important flash operation on these alien
targets is making a flash dump; these dumps can then be used for forensics,
reverse engineering, or simply as a backup.  When we subsequently write to
flash on these alien targets (after having saved a backup first), we are not
flashing an m0 fw image or a binary image made from one with mokosrec2bin,
instead the most common operations are:

* Flashing a backup image back into the same device it was originally made
  from (flash restore);

* Changing a device from one firmware version to a different one by programming
  its flash with firmware bits that were originally read out from some fw-donor
  unit;

* Surgical manipulations such as erasing FFS sectors or rewriting one specific
  part of the flash based on reverse-engineered understanding of its structure.

This different paradigm leads to a different mode of usage for fc-loadtool:
instead of needing a maximally-automated operation that flashes a firmware
release image with as little user thought involvement as possible, our flash
manipulations need to be of a more manual peek-n-poke manner.  We provide a
flash dump2bin command for making and saving flash dumps first and foremost,
allowing any part of the flash to be dumped and saved selectively if desired,
including the second flash bank on the Pirelli DP-L10 and likewise on our own
FCDEV3B.  When it comes to flash write operations, we provide a manual flash
erase command that allows (and requires) the operator to specify exactly which
sector range should be erased and a manual flash program-bin command that
allows any range of 16-bit words to be programmed at any flash address, with
the bits to be programmed coming from a binary file, either the whole file or
any specified subrange.

These manual flash erase and flash program-bin commands give full control to
the operator, allowing every possible flash manipulation which the hardware
itself allows, at the expense of requiring the operator to think about which
flash addresses, offsets and lengths need to be operated on, and either enter
long commands manually or write a command script.

Given our historical origins (long before we got to the point of producing our
own hardware, we started out by exploring the forbidden GSM realm of devices
made by alien manufacturers who were hostile to our cause), our original flash
manipulation support in fc-loadtool had been centered around the manual
peek-n-poke paradigm, with elementary flash erase and flash program-bin commands
as our main staple, and no thought had been given originally to producing
functionality that would work like FLUID or like our current flash e-program-m0
and e-program-bin commands.  But all actively maintained software evolves, and
as our FreeCalypso family of projects has matured over the years, we now offer
richer functionality covering a wider range of use cases.

Binary vs. S-records
====================

(Please read the companion article Binary-file-formats for background, then
 come back here.)

If you are exploring and manipulating the flash content of a GSM device in an
aftermarket fashion, as opposed to flashing your own fw builds into your own hw
design produced by your own company like Openmoko did in the late 2000s and
like we do currently at FreeCalypso HQ, then binary is the generally preferred
format: you make dumps with flash dump2bin, and when you selectively program
these images back into devices, you use flash program-bin with the right offsets
and length, along with appropriate flash erase commands.

We also have flash dump2srec and flash program-srec commands in fc-loadtool,
they were implemented back in the founding stage of FreeCalypso in 2013 for the
sake of completeness and symmetry (it seemed right to support both binary and
S-record formats), but they never got any practical use: if you are making a
flash dump, you would normally want to examine it afterward, and any such
examination almost always needs a straight binary image, not S-records.
Furthermore, our flash program-bin command allows you to selectively program
just a particular portion of a binary image file into flash, at any arbitrary
flash address, but we don't have the same flexibility with flash program-srec -
the latter command is really just a sibling of program-m0 with the opposite
byte order.

Thus the short summary is as follows:

* If you are flashing an official firmware release image into your device, you
  need to use flash e-program-bin or flash e-program-m0 depending on whether
  the image is provided in *.bin or *.m0 format, or alternatively our older
  flash program-bin or flash program-m0 commands preceded by a separate flash
  erase command with the right sector range, possibly packaged in a supplied
  fc-loadtool command script.

* If you are restoring a flash dump made with flash dump2bin or performing
  aftermarket flash manipulations on Mot C1xx or Pirelli DP-L10 phones or other
  such alien devices, you need to use binary-format-based flash manipulation
  commands; the specific commands will depend on exactly what you are seeking
  to do.

* flash program-srec and e-program-srec commands do not currently have a valid
  use case.

Special considerations for Compal phones
========================================

Motorola C1xx and Sony Ericsson J100 phones made by Compal have brickable flash:
the right kind of flash-resident bootloader must always be present at the
beginning of the flash, or else the phone is unrecoverably bricked.  We have
special support in fc-loadtool for minimizing the bricking vulnerability window
when operating on these phones, but this special support requires user
cooperation, meaning that you must limit your flash manipulations on these
phones to a narrower subset:

* flash program-m0, program-srec, e-program-m0 and e-program-srec commands are
  not appropriate for these brickable phones - do not use any of these commands
  on these targets.

* Flash sector 0 must be manipulated only with the special
  flash erase-program-boot command, not any of the regular erase or program
  commands.

* Regular flash erase, flash program-bin and flash e-program-bin commands can
  and should be used for the rest of the flash starting at offset 0x10000 - but
  you still need to understand what you are doing.