In the summer of 2019 we (FreeCalypso) had recovered a copy of TI's original
FLUID package including source, but this TI original FLUID source targets only
Windows, no Unix or Linux support included.  Back in 2007 or earlier Openmoko
had made their own port of FLUID to Linux and we have their Linux/ARM binary,
but the source for that Linux port appears to have been lost.

The present work is an independent (non-Openmoko) port of TI's FLUID to Linux
(can be further ported to other Unixes with some additional work, see below)
made by Mother Mychaela N. Falconia of FreeCalypso, named fluid-mnf in order to
distinguish it from any other Unix/Linux ports of FLUID that may have been made
by unknown other parties.

Purpose and scope
=================

The present fluid-mnf port has been produced as an act of restorative justice,
righting the wrong of Closedmoko not releasing theirs when they made it in 2007
or earlier.  The present fluid-mnf port is NOT intended to replace fc-loadtool,
thus a strict limit was put on the extent of work that was done on fluid-mnf:
just enough to right Moko's wrong, plus the minimum extra work that was needed
in order to satisfy my natural curiosity in the area of FLUID working with TI's
own D-Sample board, but no more.

As a result of these strict limits on the scope of the work, neither this
fluid-mnf nor any other known version of FLUID will work on FCDEV3B boards with
Spansion S71PL129N flash - instead our boards with this flash are supported only
by our own fc-loadtool, not FLUID.  The ONLY Calypso targets that are expected
to be supported by fluid-mnf (same as TI's original FLUID for Windows) are:

* TI's own D-Sample and Leonardo boards;
* A certain non-TI development board nicknamed Caramel which has a Leonardo
  clone as its core but has a D-Sample-like physical form factor;
* Openmoko GTA01 and GTA02 modems.

The present fluid-mnf release has been tested and confirmed to work on the
Mother's D-Sample C05 and Caramel boards, as well as on Openmoko GTA02.  It is
NOT expected to work on most other Calypso devices out in the wild.

What is FLUID
=============

FLUID stands for Flash Loader Utility Independent of Device.  It is not totally
clear to us (FreeCalypso) exactly what the "independent of device" part is
referring to; it can refer to any or all of the following:

* One can run FLUID, often with the exact same command line, against several
  different TI-based GSM device targets, specifically against three different
  DBB chip generations (ancient pre-Calypso chips, our familiar Calypso, and
  Calypso+), using either the boot ROM protocol on Calypso and Calypso+ or
  FLUID's own bootloader (either flash-resident or loaded via JTAG) with its
  own protocol, and FLUID goes out of its way to try to autodetect and
  automagically support all these target chip and boot entry method variations.

* The flash chip type is autodetected and does not need to be manually selected
  by the user.  If multiple variations exist of a given product where the
  factory has populated different flash chips for whatever part availability
  and pricing reasons, but it is otherwise the same product running the same
  firmware, one can use FLUID to flash fw updates without worrying about the
  flash chip type.  Our recent versions of fc-loadtool (since fc-host-tools-r11)
  feature the same quality.

* Whenever someone needs to add support for a new flash chip, this addition is
  done by editing an ASCII text configuration file (devices.txt) that is read
  by FLUID at run time, as opposed to needing to edit the source code and
  recompile the program.  This distinction was significant in the Windows world,
  a culture of software in binary form, but makes absolutely no practical
  difference for our fluid-mnf port for the Unix/Linux culture, a much healthier
  culture where all software is distributed in source code form and where
  compiling from source is a standard part of end user duties.

TI's internal reference platforms on which FLUID was developed and which FLUID
supports as its most native targets are B-Sample through E-Sample:

B-Sample: seems to be Ulysse(s) chipset, very little known
C-Sample: seems to be early Calypso C05 rev A, little known
D-Sample: Calypso C05 rev B or Calypso C035, Iota ABB, Clara RF
E-Sample: Calypso+, apparently existed in both "secure" and "non-secure"
variants

FLUID supports Hercules/Ulysse(s), Calypso and Calypso+ chipsets as follows:

* Hercules and Ulysse(s) predate our familiar Calypso and have no boot ROM.
  The standard supported way to recover from blank or bricked flash was to use
  JTAG: TI's original FLUID package (fluid-2.27.zip) contains an ulyboot.out
  COFF image which you load via JTAG using TI's XDS510 or XDS560 hardware and
  CCS software, then FLUID makes its entry via the protocol provided by this
  bootloader.  Standard firmwares included a flash-resident version of the same
  bootloader; as long as you don't brick sector 0, you can then reload newer fw
  versions without JTAG.

* On our familiar Calypso, both the old way (JTAG for cold loading, flash-
  resident bootloader for warm reloads) and the new way (Calypso boot ROM) are
  supported.  However, the old way (FLUID bootloader) is supported ONLY on
  13 MHz platforms like D-Sample (Clara RF), and is NOT supported on the more
  common Calypso+Iota+Rita platforms with 26 MHz clock input to the Calypso.

* We know very little about Calypso+, but apparently it does not offer a
  sensible boot ROM protocol like plain Calypso does: instead the choices are
  either anti-user-freedom cryptographically restricted boot ("secure" E-Sample
  boards) or no boot ROM at all, going back to brickable flash and needing JTAG
  for cold loading on "non-secure" E-Sample boards.  FLUID (the tool) supports
  the old FLUID bootloader on those "non-secure" E-Sample boards, and that
  Calypso+ version of the JTAG-loadable FLUID bootloader (calpboot.out) includes
  the fix for 26 MHz clock input (E-Sample has Rita RF) - but the plain Calypso
  version (calboot.out) does NOT support 26 MHz platforms.

What we have done in fluid-mnf
==============================

Almost no change in functionality has been made - the objective was to port TI's
tool from Windows to Linux with as little change as possible, changing only
those parts where the Unix/Linux way of life is strictly different from the
Windows way.

The most principal change made in fluid-mnf is the way you specify the serial
port to use to connect to the target.  The command line interface design in TI's
original version was thoroughly tied to the Windows model of numbered COM ports:
there was a -p option to select the serial port by number (defaulting to COM1),
but absolutely no provision to specify the serial port by name.  Examining
Closedmoko's sans-source ARM/Linux fluid.exe version through a combination of
static analysis and running under strace, we see that they did two things:

1) They changed the numbered "COM" port name generation from "COM%d" to
   "/dev/ttySAC%d" - but the command line parser still restricts -p numbers to
   the range of [1,24], thus the correct modem tty port /dev/ttySAC0 still
   cannot be specified in this manner.  The default with no -p option and no
   FLUID_PORT= environment variable is /dev/ttySAC1, which is garbage because
   that port has the GPS receiver connected to it, not the GSM modem.

2) They added a hack whereby if a FLUID_PORT= environment variable is defined,
   its value overrides the sprintf-constructed numbered "COM" port name.
   Setting FLUID_PORT=/dev/ttySAC0 is the only way to select the correct modem
   tty port with their stupid version.

The following approach has been implemented in fluid-mnf, seeking to be both
forward-looking in the Unix/Linux culture way and backward-compatible with OM's
version:

* -p option in fluid-mnf takes a string argument instead of a number, and this
  string argument is the tty port name.

* If no -p option is given, the FLUID_PORT= environment variable is consulted.

* If both -p and FLUID_PORT= are given, the -p option takes precedence.

* If neither -p nor FLUID_PORT= is specified, it is a hard error - there is no
  default tty port.

The present port uses Linux-specific tty ioctl calls instead of generic termios
for serial port control, thus it won't compile or run under any other Unixes
without further porting.  It was done this way because of non-standard baud
rates: FLUID supports Calypso high baud rates of 203125, 406250 and 812500 bps,
as well as D-Sample XXO (eXternal Xtal Oscillator) baud rates of 230400, 460800
and 921600 bps, and because of POSIX stupidity, supporting all of these baud
rates in a Unix application requires non-portable hacks.

All original FLUID code that supports targets other than our familiar Calypso
(namely, Hercules/Ulysse(s) and Calypso+) has been left completely unchanged;
while none of this code exhibited any problems compiling for Linux, it is
completely untested: we have neither the hardware nor any real knowledge of
what it is and how it is supposed to work.  For this reason, Calypso is the
only target platform which we can really vouch for as being supported by
fluid-mnf.  See below regarding what works on D-Sample and what works on the
more common Calypso+Iota+Rita platforms.

Target-side components of FLUID
===============================

Like any other tool performing the same function, FLUID consists of not only
the PC or Unix/Linux application, but also target-side code pieces which are
fed either to the Calypso boot ROM or to FLUID's other bootloader
(flash-resident or loaded via JTAG) and which run on the target during the
flash programming or reading process.  In the case of FLUID, these target-side
components are cmd.m0 and flash "method" drivers (amd.m0, intel.m0 etc),
performing essentially the same function as loadagent.srec does for our own
fc-loadtool.

TI's original FLUID package fluid-2.27.zip contains both the sources for these
components and the deployable *.m0 target binaries.  Openmoko left these *.m0
target components of FLUID completely unchanged in their port, and we are doing
likewise in fluid-mnf: aside from our cmd39.m0 concoction explained further
below, all *.m0 files in the target-bin subdirectory in this fluid-mnf release
are unchanged from TI's original delivery.  I did not even bother with setting
up an environment to recompile them from source with TI's TMS470 toolchain.

Search path for helper files
============================

FLUID needs to find the just-mentioned *.m0 target binaries as well as the
devices.txt file as helper files.  TI's original code looks in the current
directory, and also tries to look in the directory where the fluid.exe binary
is located by extracting the path from argv[0] - but the latter method works
only in the Windows culture, not Unix/Linux.

Openmoko kept this helper file search logic completely unchanged in their
version, and they installed both their fluid.exe binary and the helper files in
/usr/sbin.  But simply running OM's fluid.exe (never mind the non-sensical .exe
suffix on a Linux binary) from whatever current directory relying on the shell's
PATH search does not work: argv[0] will be equal to just "fluid.exe" without
any path, and TI's Windows-minded code unchanged by OM won't ever figure out
that it needs to look in /usr/sbin for the needed helper files.  OM's official
instructions were to cd to /usr/sbin and run fluid.exe from there - unbelievable
bogosity.

The present fluid-mnf port is made more Unix-proper in this regard: we put all
needed helper files in a designated installation directory
(/opt/freecalypso/fluid), and the code has been modified to look there instead
of the usually-null path extracted from argv[0].  The tool still looks in the
current directory first for every file: changes to TI's FLUID code architecture
have been kept to a minimum.

Calypso C05 vs. C035
====================

For all of its claims of device-independence and automagically supporting all
of various targets, TI's original FLUID as we got it in fluid-2.27.zip fails to
correctly support older Calypso boards with early Calypso C05 chips on them.
First Calypso generations (C05 rev A and C05 rev B) have a maximum ARM7 clock
frequency of 39 MHz, while on the later Calypso C035 (found in most mass-
produced phones and modems) this maximum frequency has been lifted to 52 MHz.

In TI's FLUID architecture, the target component 'cmd' (cmd.m0 built from cmd.c)
configures and enables the Calypso DPLL as soon as the host tool tells it that
the active target is Calypso rather than Ulysse(s).  The responsible function
hardware_init_calypso() in target/cmd.c originally had code that configured
this DPLL at 39 MHz - then Calypso C035 came along, and someone at TI merrily
changed that DPLL setup constant from 39 MHz to 52 MHz.  So what happens then
if someone runs TI's FLUID version 2.27 on an older C-Sample, D-Sample or
Leonardo board that has a Calypso C05 chip on it?  Answer: it will produce an
overclocked chip with unknown consequences - I am not willing to try it on our
one and only D-Sample C05 board.

Trying to improve the architecture of FLUID is strictly beyond the scope of what
I set out to do in this fluid-mnf port, instead I just needed a way to safely
run it on our D-Sample C05 board with minimal changes.  So I went for the
minimally-invasive surgical approach:

* cmd39.m0 is a modified version of cmd.m0, produced by copying cmd.m0 and
  manually patching the one offending DPLL setup constant, restoring the older
  39 MHz config.

* fluid-mnf has an added option -oC that causes it to use cmd39.m0 instead of
  regular cmd.m0.

FLUID operating on D-Sample
===========================

For anyone lucky to have an original TI D-Sample board (either C05 or C035),
the way FLUID works on it is really awesome:

* Both boot entry methods work: fluid-mnf -oo goes through the Calypso boot ROM
  (the older boot ROM version present in Calypso C05 is good enough), whereas
  fluid-mnf -oO goes through the other FLUID bootloader, either flash-resident
  or loaded via JTAG.  (I haven't tried the JTAG way, but fluid-mnf -oO works
  on the D-Sample when the flash contains a build of our FC Magnetite
  l1reconst-bl configuration.)  The extra -oC option is needed for D-Sample C05,
  but won't be needed for D-Sample C035, if anyone has one.

* FLUID (both TI's original and our fluid-mnf port) supports Calypso high baud
  rates of 203125, 406250 and 812500 bps: just select the desired baud rate with
  the -b option.  The baud rate switch is effected when the command interpreter
  ('cmd' target code piece) is running on the target, which is itself loaded at
  115200 baud.

* The D-Sample board has an extra 14.745600 MHz crystal oscillator and a special
  circuit (controlled by bits in a register mapped into Calypso nCS3 address
  space) that can switch the Calypso clock input from the regular 13 MHz coming
  from the RF section to this special 14.745600 MHz clock.  Of course no GSM
  functions can work in this state, but feeding this special clock to Calypso
  allows its UARTs to produce "standard" baud rates of 230400, 460800 and 921600
  bps!  This feature is apparently called XXO for eXternal Xtal Oscillator, and
  is supported by FLUID, including our fluid-mnf port: just select the desired
  baud rate with the -b option, and if the target is D-Sample, it will just
  magically work.  (On more "mere mortal" Calypso targets the result will be a
  spectacular failure instead.)

* The D-Sample board also has a block of 16 debug LEDs controlled by another
  register mapped into Calypso nCS3 address space, and FLUID's target-side
  component (which we haven't modified except for the 39 MHz fix) displays
  pretty dancing patterns on these LEDs as it does its work.

FLUID operating on Openmoko devices
===================================

While nowhere near as sexy as on TI's own D-Sample, our fluid-mnf port works
well on Openmoko modems:

* Only fluid-mnf -oo mode works on Openmoko devices, NOT fluid-mnf -oO, same as
  with Leonardo or Caramel boards or any other Calypso 26 MHz platform.  We have
  incriminating evidence that Openmoko once made fluid -oO mode work on their
  platform by witchcraft (bending the known laws of physics), but we were never
  able to reproduce that paranormal feat - see this FreeCalypso community
  mailing list thread:

  https://www.freecalypso.org/pipermail/community/2020-March/000743.html

* Don't try D-Sample XXO baud rates of 230400, 460800 or 921600 bps - they
  won't work.  However, Calypso high baud rates of 203125, 406250 and 812500 bps
  do work if you are going through the external headset jack and have one of
  the better USB-serial cables, either FTDI or appropriately programmed CP2102.

Target boot control
===================

The code we got from TI in fluid-2.27.zip includes a rather bizarre provision
for some ancient way of doing target boot control: right after opening the
target serial port but before sending periodic beacons seeking to interrupt and
divert the boot path either in the Calypso boot ROM or in a flash-resident
FLUID bootloader, FLUID does some manipulation of the host UART's DTR and RTS
outputs, as well as sending a break on the main data line.  These manipulations
do absolutely nothing on any ordinary Calypso hardware: the DTR line goes only
to a GPIO if anywhere at all; the host's RTS output line will normally be
connected to Calypso CTS_MODEM input if it's the MODEM UART, but these flow
control lines are completely ignored by both the Calypso boot ROM and various
flash-resident bootloaders.  But apparently there once existed some special
cable, interfaced to some unknown (probably very early and TI-internal) Calypso
or before-Calypso targets in some unknown way, and that arrangement produced a
target reset (probably a predecessor of Iota nTESTRESET on whatever before-Iota
ABB) when FLUID did this wiggle.

In any case, that hardware no longer exists and cannot be recreated because we
have no idea what it was like and how it is supposed to work.  But the logic in
question is still there in this fluid-mnf port; running OM's fluid.exe under
strace reveals that they had retained this logic as well, although they broke
the code that generates a break on the serial data line - how ironic.

Aside from this non-understood UART control line wiggling, there is no effective
target boot control in FLUID: not in TI's original version, not in OM's port
and not in the present fluid-mnf.  You just run FLUID against a serial port, it
sends beacons and waits forever for the selected bootloader (boot ROM or FLUID
bl) to respond, and you have to cause the Calypso target to go through its boot
path by your own external means.  Most importantly, there is no provision for
automation, i.e., no provision for the process to exit with an error code
instead of hanging forever if you got some target boot control implemented, but
the bootloader fails to respond as expected - put another way, there is no
equivalent to loadtools -t option which we've added as of fc-host-tools-r13.

Performance
===========

Even though it cannot be a replacement for fc-loadtool in most use cases, FLUID
is very aggressively optimized for speed in ways that would be hard to match in
our fc-loadtool architecture:

* FLUID's serial protocol between the host tool and the target-side component
  includes compression (some form of LZ77) for blocks of data destined for
  flash;

* The process of programming flash is parallelized: the bits to be programmed
  are serially transferred into a large RAM buffer on the target in parallel
  with the execution of flash erase and program operations on the target;
  specifically, flash erase and program operation functions call the UART Rx
  handler function (which handles the incoming serial stream) as they poll the
  flash chip to complete its operation!

For these reasons, on those targets which are supported by both FLUID and
fc-loadtool, flash programming with FLUID is faster.  Here are the performance
numbers obtained on the Mother's Slackware 14.2 host system, flashing an
Openmoko GTA02 modem with firmware version moko-new-fw-20190128, going through
the external headset jack, UberWaves CP2102 Professional cable:

fluid-mnf   at 115200 baud: 2m28s
fc-loadtool at 115200 baud: 4m02s
fluid-mnf   at 812500 baud: 0m40s
fc-loadtool at 812500 baud: 1m12s

With both tools the m0 version of the firmware image (fwimage.m0) was being
flashed; in the case of fc-loadtool the new flash e-program-m0 command was used.
The version of fc-loadtool used for this test is the one that is about to be
released with fc-host-tools-r13; previous versions would be even slower.  In
the case of fluid-mnf the -C option was used to disable delta downloads, making
the test operation independent of previous flash state.

Where to go from here
=====================

So now we have two competing tools for operating on the flash memory of Calypso
GSM devices: fc-loadtool and fluid-mnf.  Which one should you use?  In most
practical use cases only fc-loadtool will work:

* FLUID only works on D-Sample, Caramel and Openmoko targets.  If you are
  working with an FCDEV3B, any Mot C1xx phone, Pirelli DP-L10 or a GTM900 module
  with the newer Samsung K5L3316CAM flash chip, then fc-loadtool is the only
  choice.

* Both fc-loadtool and fluid-mnf can read flash dumps, saving them in either
  binary or S-record format.  However, FLUID only accepts *.m0 S-record files
  (16-bit moko-style by default, but 8-bit and 32-bit variants can be used too)
  as the data source for flash programming.  Binary format is much more
  convenient for flash backups, but restoring such backups with FLUID would be
  a complicated matter, requiring an external converter that extracts the
  desired subrange out of a binary flash dump file and makes an m0 image out of
  it.

* fc-loadtool and its loadagent back end are only one part of a larger loadtools
  and target-utils suite that supports many other operations besides flash
  manipulation.

On the other hand, if your target is one of the few which are supported by FLUID
and your desired operation is to flash a firmware image built in m0 format,
then you can use fluid-mnf and enjoy faster speed - or enjoy the dancing LED
patterns if the target is a TI D-Sample board.

We got into the present situation we find ourselves in because of the culture
of suppression that was built around our dear Calypso by the Closed Moko
Company: back when we started FreeCalypso in 2013, the source for FLUID was
wrongfully withheld from Humanity, thus producing a from-scratch replacement
like our own fc-loadtool was the only viable option.  And because the project
was essentially a battle for the liberation of Calypso, performance was not a
major concern at that time, which is how we ended up with a tool that is not
optimized for speed like FLUID is.

If I (Mother Mychaela) had been a Calypso modem engineer for a truly Open kind
of Moko instead of those NDA-worshipping Germans back in 2006-2008, with access
to all documentation, firmware sources and FLUID from day 0, and working in a
fully open culture with everything freely published like in current FreeCalypso,
I would have initially produced something very similar to the present fluid-mnf
port, just to get the existing and known working tool running under Linux
instead of Windows.  If the project had stayed with Samsung K5A32xx or Spansion
S71PL-J flash chips, such a Linux port of FLUID would have been sufficient.  If
new requirements came along to support S71PL-N flash, second flash bank
operations, significantly different Calypso targets like Mot C1xx, or anything
else that does not fit well into FLUID architecture, then my solution would
have been to produce a new tool that is free of unmaintainable FLUID gunk
(support for hw that no longer exists and thus can't be tested, old FLUID
bootloader protocol), but keeps those parts of FLUID architecture that make it
so fast.  But as they say, history does not know subjunctive mood.

So what can we do in the present?  We have targets and operations which would
be very difficult to shoehorn into the architecture of FLUID, but which are
well-supported by fc-loadtool - how can we make them as fast as FLUID?  One
promising approach would be to extend our loadagent yet again, adding a flash
programming protocol similar to FLUID, with parallelized flash operations and
serial stream Rx and with LZ77 decompression, and change fc-loadtool to use
this new protocol for flash programming while keeping all other functionality
as it already works.  But given the sorry state of FreeCalypso funding, this
idea may never get implemented. :-(