FreeCalypso > hg > freecalypso-reveng
view fluid-mnf/README @ 401:4b6b595ae0a0
compal/boot/code-deriv: new analysis
author | Mychaela Falconia <falcon@freecalypso.org> |
---|---|
date | Sat, 14 Jan 2023 23:59:23 +0000 |
parents | f888ae294b1b |
children |
line wrap: on
line source
In the summer of 2019 we (FreeCalypso) had recovered a copy of TI's original FLUID package including source, but this TI original FLUID source targets only Windows, no Unix or Linux support included. Back in 2007 or earlier Openmoko had made their own port of FLUID to Linux and we have their Linux/ARM binary, but the source for that Linux port appears to have been lost. The present work is an independent (non-Openmoko) port of TI's FLUID to Linux (can be further ported to other Unixes with some additional work, see below) made by Mother Mychaela N. Falconia of FreeCalypso, named fluid-mnf in order to distinguish it from any other Unix/Linux ports of FLUID that may have been made by unknown other parties. Purpose and scope ================= The present fluid-mnf port has been produced as an act of restorative justice, righting the wrong of Closedmoko not releasing theirs when they made it in 2007 or earlier. The present fluid-mnf port is NOT intended to replace fc-loadtool, thus a strict limit was put on the extent of work that was done on fluid-mnf: just enough to right Moko's wrong, plus the minimum extra work that was needed in order to satisfy my natural curiosity in the area of FLUID working with TI's own D-Sample board, but no more. As a result of these strict limits on the scope of the work, neither this fluid-mnf nor any other known version of FLUID will work on FCDEV3B boards with Spansion S71PL129N flash - instead our boards with this flash are supported only by our own fc-loadtool, not FLUID. The ONLY Calypso targets that are expected to be supported by fluid-mnf (same as TI's original FLUID for Windows) are: * TI's own D-Sample and Leonardo boards; * A certain non-TI development board nicknamed Caramel which has a Leonardo clone as its core but has a D-Sample-like physical form factor; * Openmoko GTA01 and GTA02 modems. The present fluid-mnf release has been tested and confirmed to work on the Mother's D-Sample C05 and Caramel boards, as well as on Openmoko GTA02. It is NOT expected to work on most other Calypso devices out in the wild. What is FLUID ============= FLUID stands for Flash Loader Utility Independent of Device. It is not totally clear to us (FreeCalypso) exactly what the "independent of device" part is referring to; it can refer to any or all of the following: * One can run FLUID, often with the exact same command line, against several different TI-based GSM device targets, specifically against three different DBB chip generations (ancient pre-Calypso chips, our familiar Calypso, and Calypso+), using either the boot ROM protocol on Calypso and Calypso+ or FLUID's own bootloader (either flash-resident or loaded via JTAG) with its own protocol, and FLUID goes out of its way to try to autodetect and automagically support all these target chip and boot entry method variations. * The flash chip type is autodetected and does not need to be manually selected by the user. If multiple variations exist of a given product where the factory has populated different flash chips for whatever part availability and pricing reasons, but it is otherwise the same product running the same firmware, one can use FLUID to flash fw updates without worrying about the flash chip type. Our recent versions of fc-loadtool (since fc-host-tools-r11) feature the same quality. * Whenever someone needs to add support for a new flash chip, this addition is done by editing an ASCII text configuration file (devices.txt) that is read by FLUID at run time, as opposed to needing to edit the source code and recompile the program. This distinction was significant in the Windows world, a culture of software in binary form, but makes absolutely no practical difference for our fluid-mnf port for the Unix/Linux culture, a much healthier culture where all software is distributed in source code form and where compiling from source is a standard part of end user duties. TI's internal reference platforms on which FLUID was developed and which FLUID supports as its most native targets are B-Sample through E-Sample: B-Sample: seems to be Ulysse(s) chipset, very little known C-Sample: seems to be early Calypso C05 rev A, little known D-Sample: Calypso C05 rev B or Calypso C035, Iota ABB, Clara RF E-Sample: Calypso+, apparently existed in both "secure" and "non-secure" variants FLUID supports Hercules/Ulysse(s), Calypso and Calypso+ chipsets as follows: * Hercules and Ulysse(s) predate our familiar Calypso and have no boot ROM. The standard supported way to recover from blank or bricked flash was to use JTAG: TI's original FLUID package (fluid-2.27.zip) contains an ulyboot.out COFF image which you load via JTAG using TI's XDS510 or XDS560 hardware and CCS software, then FLUID makes its entry via the protocol provided by this bootloader. Standard firmwares included a flash-resident version of the same bootloader; as long as you don't brick sector 0, you can then reload newer fw versions without JTAG. * On our familiar Calypso, both the old way (JTAG for cold loading, flash- resident bootloader for warm reloads) and the new way (Calypso boot ROM) are supported. However, the old way (FLUID bootloader) is supported ONLY on 13 MHz platforms like D-Sample (Clara RF), and is NOT supported on the more common Calypso+Iota+Rita platforms with 26 MHz clock input to the Calypso. * We know very little about Calypso+, but apparently it does not offer a sensible boot ROM protocol like plain Calypso does: instead the choices are either anti-user-freedom cryptographically restricted boot ("secure" E-Sample boards) or no boot ROM at all, going back to brickable flash and needing JTAG for cold loading on "non-secure" E-Sample boards. FLUID (the tool) supports the old FLUID bootloader on those "non-secure" E-Sample boards, and that Calypso+ version of the JTAG-loadable FLUID bootloader (calpboot.out) includes the fix for 26 MHz clock input (E-Sample has Rita RF) - but the plain Calypso version (calboot.out) does NOT support 26 MHz platforms. What we have done in fluid-mnf ============================== Almost no change in functionality has been made - the objective was to port TI's tool from Windows to Linux with as little change as possible, changing only those parts where the Unix/Linux way of life is strictly different from the Windows way. The most principal change made in fluid-mnf is the way you specify the serial port to use to connect to the target. The command line interface design in TI's original version was thoroughly tied to the Windows model of numbered COM ports: there was a -p option to select the serial port by number (defaulting to COM1), but absolutely no provision to specify the serial port by name. Examining Closedmoko's sans-source ARM/Linux fluid.exe version through a combination of static analysis and running under strace, we see that they did two things: 1) They changed the numbered "COM" port name generation from "COM%d" to "/dev/ttySAC%d" - but the command line parser still restricts -p numbers to the range of [1,24], thus the correct modem tty port /dev/ttySAC0 still cannot be specified in this manner. The default with no -p option and no FLUID_PORT= environment variable is /dev/ttySAC1, which is garbage because that port has the GPS receiver connected to it, not the GSM modem. 2) They added a hack whereby if a FLUID_PORT= environment variable is defined, its value overrides the sprintf-constructed numbered "COM" port name. Setting FLUID_PORT=/dev/ttySAC0 is the only way to select the correct modem tty port with their stupid version. The following approach has been implemented in fluid-mnf, seeking to be both forward-looking in the Unix/Linux culture way and backward-compatible with OM's version: * -p option in fluid-mnf takes a string argument instead of a number, and this string argument is the tty port name. * If no -p option is given, the FLUID_PORT= environment variable is consulted. * If both -p and FLUID_PORT= are given, the -p option takes precedence. * If neither -p nor FLUID_PORT= is specified, it is a hard error - there is no default tty port. The present port uses Linux-specific tty ioctl calls instead of generic termios for serial port control, thus it won't compile or run under any other Unixes without further porting. It was done this way because of non-standard baud rates: FLUID supports Calypso high baud rates of 203125, 406250 and 812500 bps, as well as D-Sample XXO (eXternal Xtal Oscillator) baud rates of 230400, 460800 and 921600 bps, and because of POSIX stupidity, supporting all of these baud rates in a Unix application requires non-portable hacks. All original FLUID code that supports targets other than our familiar Calypso (namely, Hercules/Ulysse(s) and Calypso+) has been left completely unchanged; while none of this code exhibited any problems compiling for Linux, it is completely untested: we have neither the hardware nor any real knowledge of what it is and how it is supposed to work. For this reason, Calypso is the only target platform which we can really vouch for as being supported by fluid-mnf. See below regarding what works on D-Sample and what works on the more common Calypso+Iota+Rita platforms. Target-side components of FLUID =============================== Like any other tool performing the same function, FLUID consists of not only the PC or Unix/Linux application, but also target-side code pieces which are fed either to the Calypso boot ROM or to FLUID's other bootloader (flash-resident or loaded via JTAG) and which run on the target during the flash programming or reading process. In the case of FLUID, these target-side components are cmd.m0 and flash "method" drivers (amd.m0, intel.m0 etc), performing essentially the same function as loadagent.srec does for our own fc-loadtool. TI's original FLUID package fluid-2.27.zip contains both the sources for these components and the deployable *.m0 target binaries. Openmoko left these *.m0 target components of FLUID completely unchanged in their port, and we are doing likewise in fluid-mnf: aside from our cmd39.m0 concoction explained further below, all *.m0 files in the target-bin subdirectory in this fluid-mnf release are unchanged from TI's original delivery. I did not even bother with setting up an environment to recompile them from source with TI's TMS470 toolchain. Search path for helper files ============================ FLUID needs to find the just-mentioned *.m0 target binaries as well as the devices.txt file as helper files. TI's original code looks in the current directory, and also tries to look in the directory where the fluid.exe binary is located by extracting the path from argv[0] - but the latter method works only in the Windows culture, not Unix/Linux. Openmoko kept this helper file search logic completely unchanged in their version, and they installed both their fluid.exe binary and the helper files in /usr/sbin. But simply running OM's fluid.exe (never mind the non-sensical .exe suffix on a Linux binary) from whatever current directory relying on the shell's PATH search does not work: argv[0] will be equal to just "fluid.exe" without any path, and TI's Windows-minded code unchanged by OM won't ever figure out that it needs to look in /usr/sbin for the needed helper files. OM's official instructions were to cd to /usr/sbin and run fluid.exe from there - unbelievable bogosity. The present fluid-mnf port is made more Unix-proper in this regard: we put all needed helper files in a designated installation directory (/opt/freecalypso/fluid), and the code has been modified to look there instead of the usually-null path extracted from argv[0]. The tool still looks in the current directory first for every file: changes to TI's FLUID code architecture have been kept to a minimum. Calypso C05 vs. C035 ==================== For all of its claims of device-independence and automagically supporting all of various targets, TI's original FLUID as we got it in fluid-2.27.zip fails to correctly support older Calypso boards with early Calypso C05 chips on them. First Calypso generations (C05 rev A and C05 rev B) have a maximum ARM7 clock frequency of 39 MHz, while on the later Calypso C035 (found in most mass- produced phones and modems) this maximum frequency has been lifted to 52 MHz. In TI's FLUID architecture, the target component 'cmd' (cmd.m0 built from cmd.c) configures and enables the Calypso DPLL as soon as the host tool tells it that the active target is Calypso rather than Ulysse(s). The responsible function hardware_init_calypso() in target/cmd.c originally had code that configured this DPLL at 39 MHz - then Calypso C035 came along, and someone at TI merrily changed that DPLL setup constant from 39 MHz to 52 MHz. So what happens then if someone runs TI's FLUID version 2.27 on an older C-Sample, D-Sample or Leonardo board that has a Calypso C05 chip on it? Answer: it will produce an overclocked chip with unknown consequences - I am not willing to try it on our one and only D-Sample C05 board. Trying to improve the architecture of FLUID is strictly beyond the scope of what I set out to do in this fluid-mnf port, instead I just needed a way to safely run it on our D-Sample C05 board with minimal changes. So I went for the minimally-invasive surgical approach: * cmd39.m0 is a modified version of cmd.m0, produced by copying cmd.m0 and manually patching the one offending DPLL setup constant, restoring the older 39 MHz config. * fluid-mnf has an added option -oC that causes it to use cmd39.m0 instead of regular cmd.m0. FLUID operating on D-Sample =========================== For anyone lucky to have an original TI D-Sample board (either C05 or C035), the way FLUID works on it is really awesome: * Both boot entry methods work: fluid-mnf -oo goes through the Calypso boot ROM (the older boot ROM version present in Calypso C05 is good enough), whereas fluid-mnf -oO goes through the other FLUID bootloader, either flash-resident or loaded via JTAG. (I haven't tried the JTAG way, but fluid-mnf -oO works on the D-Sample when the flash contains a build of our FC Magnetite l1reconst-bl configuration.) The extra -oC option is needed for D-Sample C05, but won't be needed for D-Sample C035, if anyone has one. * FLUID (both TI's original and our fluid-mnf port) supports Calypso high baud rates of 203125, 406250 and 812500 bps: just select the desired baud rate with the -b option. The baud rate switch is effected when the command interpreter ('cmd' target code piece) is running on the target, which is itself loaded at 115200 baud. * The D-Sample board has an extra 14.745600 MHz crystal oscillator and a special circuit (controlled by bits in a register mapped into Calypso nCS3 address space) that can switch the Calypso clock input from the regular 13 MHz coming from the RF section to this special 14.745600 MHz clock. Of course no GSM functions can work in this state, but feeding this special clock to Calypso allows its UARTs to produce "standard" baud rates of 230400, 460800 and 921600 bps! This feature is apparently called XXO for eXternal Xtal Oscillator, and is supported by FLUID, including our fluid-mnf port: just select the desired baud rate with the -b option, and if the target is D-Sample, it will just magically work. (On more "mere mortal" Calypso targets the result will be a spectacular failure instead.) * The D-Sample board also has a block of 16 debug LEDs controlled by another register mapped into Calypso nCS3 address space, and FLUID's target-side component (which we haven't modified except for the 39 MHz fix) displays pretty dancing patterns on these LEDs as it does its work. FLUID operating on Openmoko devices =================================== While nowhere near as sexy as on TI's own D-Sample, our fluid-mnf port works well on Openmoko modems: * Only fluid-mnf -oo mode works on Openmoko devices, NOT fluid-mnf -oO, same as with Leonardo or Caramel boards or any other Calypso 26 MHz platform. We have incriminating evidence that Openmoko once made fluid -oO mode work on their platform by witchcraft (bending the known laws of physics), but we were never able to reproduce that paranormal feat - see this FreeCalypso community mailing list thread: https://www.freecalypso.org/pipermail/community/2020-March/000743.html * Don't try D-Sample XXO baud rates of 230400, 460800 or 921600 bps - they won't work. However, Calypso high baud rates of 203125, 406250 and 812500 bps do work if you are going through the external headset jack and have one of the better USB-serial cables, either FTDI or appropriately programmed CP2102. Target boot control =================== The code we got from TI in fluid-2.27.zip includes a rather bizarre provision for some ancient way of doing target boot control: right after opening the target serial port but before sending periodic beacons seeking to interrupt and divert the boot path either in the Calypso boot ROM or in a flash-resident FLUID bootloader, FLUID does some manipulation of the host UART's DTR and RTS outputs, as well as sending a break on the main data line. These manipulations do absolutely nothing on any ordinary Calypso hardware: the DTR line goes only to a GPIO if anywhere at all; the host's RTS output line will normally be connected to Calypso CTS_MODEM input if it's the MODEM UART, but these flow control lines are completely ignored by both the Calypso boot ROM and various flash-resident bootloaders. But apparently there once existed some special cable, interfaced to some unknown (probably very early and TI-internal) Calypso or before-Calypso targets in some unknown way, and that arrangement produced a target reset (probably a predecessor of Iota nTESTRESET on whatever before-Iota ABB) when FLUID did this wiggle. In any case, that hardware no longer exists and cannot be recreated because we have no idea what it was like and how it is supposed to work. But the logic in question is still there in this fluid-mnf port; running OM's fluid.exe under strace reveals that they had retained this logic as well, although they broke the code that generates a break on the serial data line - how ironic. Aside from this non-understood UART control line wiggling, there is no effective target boot control in FLUID: not in TI's original version, not in OM's port and not in the present fluid-mnf. You just run FLUID against a serial port, it sends beacons and waits forever for the selected bootloader (boot ROM or FLUID bl) to respond, and you have to cause the Calypso target to go through its boot path by your own external means. Most importantly, there is no provision for automation, i.e., no provision for the process to exit with an error code instead of hanging forever if you got some target boot control implemented, but the bootloader fails to respond as expected - put another way, there is no equivalent to loadtools -t option which we've added as of fc-host-tools-r13. Performance =========== Even though it cannot be a replacement for fc-loadtool in most use cases, FLUID is very aggressively optimized for speed in ways that would be hard to match in our fc-loadtool architecture: * FLUID's serial protocol between the host tool and the target-side component includes compression (some form of LZ77) for blocks of data destined for flash; * The process of programming flash is parallelized: the bits to be programmed are serially transferred into a large RAM buffer on the target in parallel with the execution of flash erase and program operations on the target; specifically, flash erase and program operation functions call the UART Rx handler function (which handles the incoming serial stream) as they poll the flash chip to complete its operation! For these reasons, on those targets which are supported by both FLUID and fc-loadtool, flash programming with FLUID is faster. Here are the performance numbers obtained on the Mother's Slackware 14.2 host system, flashing an Openmoko GTA02 modem with firmware version moko-new-fw-20190128, going through the external headset jack, UberWaves CP2102 Professional cable: fluid-mnf at 115200 baud: 2m28s fc-loadtool at 115200 baud: 4m02s fluid-mnf at 812500 baud: 0m40s fc-loadtool at 812500 baud: 1m12s With both tools the m0 version of the firmware image (fwimage.m0) was being flashed; in the case of fc-loadtool the new flash e-program-m0 command was used. The version of fc-loadtool used for this test is the one that is about to be released with fc-host-tools-r13; previous versions would be even slower. In the case of fluid-mnf the -C option was used to disable delta downloads, making the test operation independent of previous flash state. Where to go from here ===================== So now we have two competing tools for operating on the flash memory of Calypso GSM devices: fc-loadtool and fluid-mnf. Which one should you use? In most practical use cases only fc-loadtool will work: * FLUID only works on D-Sample, Caramel and Openmoko targets. If you are working with an FCDEV3B, any Mot C1xx phone, Pirelli DP-L10 or a GTM900 module with the newer Samsung K5L3316CAM flash chip, then fc-loadtool is the only choice. * Both fc-loadtool and fluid-mnf can read flash dumps, saving them in either binary or S-record format. However, FLUID only accepts *.m0 S-record files (16-bit moko-style by default, but 8-bit and 32-bit variants can be used too) as the data source for flash programming. Binary format is much more convenient for flash backups, but restoring such backups with FLUID would be a complicated matter, requiring an external converter that extracts the desired subrange out of a binary flash dump file and makes an m0 image out of it. * fc-loadtool and its loadagent back end are only one part of a larger loadtools and target-utils suite that supports many other operations besides flash manipulation. On the other hand, if your target is one of the few which are supported by FLUID and your desired operation is to flash a firmware image built in m0 format, then you can use fluid-mnf and enjoy faster speed - or enjoy the dancing LED patterns if the target is a TI D-Sample board. We got into the present situation we find ourselves in because of the culture of suppression that was built around our dear Calypso by the Closed Moko Company: back when we started FreeCalypso in 2013, the source for FLUID was wrongfully withheld from Humanity, thus producing a from-scratch replacement like our own fc-loadtool was the only viable option. And because the project was essentially a battle for the liberation of Calypso, performance was not a major concern at that time, which is how we ended up with a tool that is not optimized for speed like FLUID is. If I (Mother Mychaela) had been a Calypso modem engineer for a truly Open kind of Moko instead of those NDA-worshipping Germans back in 2006-2008, with access to all documentation, firmware sources and FLUID from day 0, and working in a fully open culture with everything freely published like in current FreeCalypso, I would have initially produced something very similar to the present fluid-mnf port, just to get the existing and known working tool running under Linux instead of Windows. If the project had stayed with Samsung K5A32xx or Spansion S71PL-J flash chips, such a Linux port of FLUID would have been sufficient. If new requirements came along to support S71PL-N flash, second flash bank operations, significantly different Calypso targets like Mot C1xx, or anything else that does not fit well into FLUID architecture, then my solution would have been to produce a new tool that is free of unmaintainable FLUID gunk (support for hw that no longer exists and thus can't be tested, old FLUID bootloader protocol), but keeps those parts of FLUID architecture that make it so fast. But as they say, history does not know subjunctive mood. So what can we do in the present? We have targets and operations which would be very difficult to shoehorn into the architecture of FLUID, but which are well-supported by fc-loadtool - how can we make them as fast as FLUID? One promising approach would be to extend our loadagent yet again, adding a flash programming protocol similar to FLUID, with parallelized flash operations and serial stream Rx and with LZ77 decompression, and change fc-loadtool to use this new protocol for flash programming while keeping all other functionality as it already works. But given the sorry state of FreeCalypso funding, this idea may never get implemented. :-(