+
−Our Calypso device flash programming tool fc-loadtool can be used in several
+
−different paradigms; this article is an attempt to clarify the many available
+
−modes of usage.  You should also read the companion article Binary-file-formats
+
−for further background.
+
−
+
−Flashing firmware release images
+
−================================
+
−
+
−In conventional forward engineering environments where you develop or maintain
+
−firmware for hardware made by your own company (meaning no reverse eng, no
+
−"illicit" aftermarket tinkering on hw made by some other company aeons ago),
+
−you have a firmware build system that produces fw build images (some of which
+
−may subsequently be blessed as releases), and you have a tool that flashes
+
−these fw build images into your hardware, operating as efficiently as possible,
+
−automated as much as possible, requiring minimal user action for the boring
+
−repetitive task of flashing a new fw image every time you build one.  And if
+
−you become lucky enough to produce your hardware in volume, the same objectives
+
−of maximal efficiency and automation carry over into the production line
+
−environment as well.
+
−
+
−In TI's environment the standardized format for firmware build images which are
+
−then flashable into hardware targets was a variant of Motorola SREC written in
+
−*.m0 files, a variant which we have named moko-style m0 after its most famous
+
−user.  The special quirk of this particular SREC variant is its peculiar byte
+
−order.  TI's firmware build system produces a *.m0 S-record image as its final
+
−build product, and TI's official Calypso flash programming tool (FLUID) takes
+
−these *.m0 files as its input.
+
−
+
−Since the beginning of FreeCalypso we have had two ways of flashing TI-built
+
−firmware images into suitable targets (initially OM GTA02 modem, then many
+
−others including our own FCDEV3B):
+
−
+
−1) Our fc-loadtool has had a flash program-m0 command from the beginning,
+
−   programming device flash with bits from an m0 file directly and natively.
+
−   However, prior to fc-host-tools-r12 this command was poorly supported: it
+
−   ran significantly slower than flash program-bin, had poorer progress
+
−   indication and did not perform CRC-32 verification at the end, which is an
+
−   important integrity check.  Also this original flash program-m0 command (as
+
−   opposed to flash e-program-m0 added in fc-host-tools-r13) does not include a
+
−   built-in erase step, thus prior to fc-host-tools-r13 the user had to have
+
−   outside knowledge of how many sectors to erase first with a separate flash
+
−   erase command.
+
−
+
−   The new flash e-program-m0 command added in fc-host-tools-r13 is m0 image
+
−   flashing finally done right.  It reads in the specified S-record image in
+
−   moko-style m0 format, builds a map of potentially discontiguous flash
+
−   regions into which the image deposits bits, erases the set of flash sectors
+
−   which need to be erased before programming these regions, then programs the
+
−   new image bits into flash, exactly like TI's own FLUID.
+
−
+
−2) The alternative way is to first convert the *.m0 S-record image produced by
+
−   TI's hex470 post-linker tool into straight binary (*.bin) with a FreeCalypso
+
−   tool called mokosrec2bin, then program the binary fw image into flash with
+
−   fc-loadtool command flash program-bin.  This method is the one we've been
+
−   using since 2017, and our FC Magnetite firmware build system is now set up
+
−   to produce not only fwimage.m0, but also fwimage.bin (it runs mokosrec2bin),
+
−   and it also generates an fc-loadtool command script (a text file named
+
−   flash-script) with two commands in it: a flash erase command with a
+
−   calculated sector address range and a flash program-bin command to program
+
−   the accompanying fwimage.bin image.
+
−
+
−As of fc-host-tools-r13 both methods work equally well: if you have an official
+
−FreeCalypso firmware release (containing fwimage.m0, fwimage.bin and
+
−flash-script files) which you need to flash into a device such as our own
+
−FCDEV3B or OM GTA02 (but *not* Mot C1xx!), you can execute either
+
−'exec flash-script' or 'flash e-program-m0 fwimage.m0' at the loadtool> prompt,
+
−and both ways will produce exactly the same result with equal performance and
+
−reliability.  And if you need a more special operation such as erasing the
+
−entire flash (factory production lines) or erasing and reprogramming only a
+
−certain part of the normally affected sector range, that's what custom command
+
−scripting ability is for.
+
−
+
−For the sake of symmetry, we also have a flash e-program-bin command that is a
+
−binary image format counterpart to flash e-program-m0: it first erases the
+
−sectors into which new bits will be programmed, then programs the new bits.
+
−Thus a third equally good way to flash a new FreeCalypso fw release into a
+
−target such as FCDEV3B or GTA02 is to execute
+
−'flash e-program-bin 0 fwimage.bin' - but don't *ever* do it on a Mot C1xx
+
−phone!
+
−
+
−Discontiguous m0 fw images with large gaps
+
−==========================================
+
−
+
−One sometimes-advantage of S-record-based formats over straight binary is that
+
−they can have discontiguous filled regions with gaps in between them; all m0 fw
+
−images produced by TI's TCS211 build system have some gaps in them.  In our
+
−current FC Magnetite firmwares these gaps have been reduced to less than 16384
+
−bytes in total, reducing the wasted disk space and image transfer time to
+
−sufficiently insignificant if these m0 images are converted to straight binary
+
−with mokosrec2bin and then flashed as binary as we've been doing.  For
+
−comparison, in Openmoko's classic mokoN firmwares (almost TI's original version
+
−of the memory map) the first flash sector of 65536 bytes had only 5072 bytes of
+
−image payload programmed into it, thus indirect programming via mokosrec2bin
+
−followed by flash program-bin meant sending just over 60k filler 0xFF bytes to
+
−be flashed - becomes noticeable at 115200 baud.
+
−
+
−But these gaps were even bigger in TI's classic firmwares (and many vendor
+
−firmwares) built for 8 MiB or 16 MiB flash chips with TI's classic memory map.
+
−They have two code regions: one region is exactly 4 MiB in size but is placed
+
−not quite at the beginning, thus spilling past the chip midpoint, and the other
+
−region starts where the first 4 MiB region ends, going to the end of the flash
+
−chip or to the beginning of the area reserved for FFS.  The way TI's linker
+
−works results in the first of these two program regions being not quite full
+
−before bits start getting thrown into the second one, thus the resulting m0
+
−image can easily have a gap of a megabyte or more between the two regions.
+
−TI's own FLUID handles such discontiguous images with no sweat, but our
+
−fc-loadtool has gained matching capability only as of fc-host-tools-r13 with
+
−our new e-program-m0 command.  And yes, it is smart enough to avoid needlessly
+
−erasing sectors in the big gap, just like FLUID.
+
−
+
−Flash backup and restore
+
−========================
+
−
+
−A completely different paradigm takes place on alien targets such as Motorola
+
−C1xx and Pirelli DP-L10, made by alien manufacturers, meaning not FreeCalypso,
+
−not Openmoko and not TI.  The most important flash operation on these alien
+
−targets is making a flash dump; these dumps can then be used for forensics,
+
−reverse engineering, or simply as a backup.  When we subsequently write to
+
−flash on these alien targets (after having saved a backup first), we are not
+
−flashing an m0 fw image or a binary image made from one with mokosrec2bin,
+
−instead the most common operations are:
+
−
+
−* Flashing a backup image back into the same device it was originally made
+
−  from (flash restore);
+
−
+
−* Changing a device from one firmware version to a different one by programming
+
−  its flash with firmware bits that were originally read out from some fw-donor
+
−  unit;
+
−
+
−* Surgical manipulations such as erasing FFS sectors or rewriting one specific
+
−  part of the flash based on reverse-engineered understanding of its structure.
+
−
+
−This different paradigm leads to a different mode of usage for fc-loadtool:
+
−instead of needing a maximally-automated operation that flashes a firmware
+
−release image with as little user thought involvement as possible, our flash
+
−manipulations need to be of a more manual peek-n-poke manner.  We provide a
+
−flash dump2bin command for making and saving flash dumps first and foremost,
+
−allowing any part of the flash to be dumped and saved selectively if desired,
+
−including the second flash bank on the Pirelli DP-L10 and likewise on our own
+
−FCDEV3B.  When it comes to flash write operations, we provide a manual flash
+
−erase command that allows (and requires) the operator to specify exactly which
+
−sector range should be erased and a manual flash program-bin command that
+
−allows any range of 16-bit words to be programmed at any flash address, with
+
−the bits to be programmed coming from a binary file, either the whole file or
+
−any specified subrange.
+
−
+
−These manual flash erase and flash program-bin commands give full control to
+
−the operator, allowing every possible flash manipulation which the hardware
+
−itself allows, at the expense of requiring the operator to think about which
+
−flash addresses, offsets and lengths need to be operated on, and either enter
+
−long commands manually or write a command script.
+
−
+
−Given our historical origins (long before we got to the point of producing our
+
−own hardware, we started out by exploring the forbidden GSM realm of devices
+
−made by alien manufacturers who were hostile to our cause), our original flash
+
−manipulation support in fc-loadtool had been centered around the manual
+
−peek-n-poke paradigm, with elementary flash erase and flash program-bin commands
+
−as our main staple, and no thought had been given originally to producing
+
−functionality that would work like FLUID or like our current flash e-program-m0
+
−and e-program-bin commands.  But all actively maintained software evolves, and
+
−as our FreeCalypso family of projects has matured over the years, we now offer
+
−richer functionality covering a wider range of use cases.
+
−
+
−Binary vs. S-records
+
−====================
+
−
+
−(Please read the companion article Binary-file-formats for background, then
+
− come back here.)
+
−
+
−If you are exploring and manipulating the flash content of a GSM device in an
+
−aftermarket fashion, as opposed to flashing your own fw builds into your own hw
+
−design produced by your own company like Openmoko did in the late 2000s and
+
−like we do currently at FreeCalypso HQ, then binary is the generally preferred
+
−format: you make dumps with flash dump2bin, and when you selectively program
+
−these images back into devices, you use flash program-bin with the right offsets
+
−and length, along with appropriate flash erase commands.
+
−
+
−We also have flash dump2srec and flash program-srec commands in fc-loadtool,
+
−they were implemented back in the founding stage of FreeCalypso in 2013 for the
+
−sake of completeness and symmetry (it seemed right to support both binary and
+
−S-record formats), but they never got any practical use: if you are making a
+
−flash dump, you would normally want to examine it afterward, and any such
+
−examination almost always needs a straight binary image, not S-records.
+
−Furthermore, our flash program-bin command allows you to selectively program
+
−just a particular portion of a binary image file into flash, at any arbitrary
+
−flash address, but we don't have the same flexibility with flash program-srec -
+
−the latter command is really just a sibling of program-m0 with the opposite
+
−byte order.
+
−
+
−Thus the short summary is as follows:
+
−
+
−* If you are flashing an official firmware release image into your device, you
+
−  need to use flash e-program-bin or flash e-program-m0 depending on whether
+
−  the image is provided in *.bin or *.m0 format, or alternatively our older
+
−  flash program-bin or flash program-m0 commands preceded by a separate flash
+
−  erase command with the right sector range, possibly packaged in a supplied
+
−  fc-loadtool command script.
+
−
+
−* If you are restoring a flash dump made with flash dump2bin or performing
+
−  aftermarket flash manipulations on Mot C1xx or Pirelli DP-L10 phones or other
+
−  such alien devices, you need to use binary-format-based flash manipulation
+
−  commands; the specific commands will depend on exactly what you are seeking
+
−  to do.
+
−
+
−* flash program-srec and e-program-srec commands do not currently have a valid
+
−  use case.
+
−
+
−Special considerations for Compal phones
+
−========================================
+
−
+
−Motorola C1xx and Sony Ericsson J100 phones made by Compal have brickable flash:
+
−the right kind of flash-resident bootloader must always be present at the
+
−beginning of the flash, or else the phone is unrecoverably bricked.  We have
+
−special support in fc-loadtool for minimizing the bricking vulnerability window
+
−when operating on these phones, but this special support requires user
+
−cooperation, meaning that you must limit your flash manipulations on these
+
−phones to a narrower subset:
+
−
+
−* flash program-m0, program-srec, e-program-m0 and e-program-srec commands are
+
−  not appropriate for these brickable phones - do not use any of these commands
+
−  on these targets.
+
−
+
−* Flash sector 0 must be manipulated only with the special
+
−  flash erase-program-boot command, not any of the regular erase or program
+
−  commands.
+
−
+
−* Regular flash erase, flash program-bin and flash e-program-bin commands can
+
−  and should be used for the rest of the flash starting at offset 0x10000 - but
+
−  you still need to understand what you are doing.