Firmware debug: handover to the community

[Spacefalcon the Outlaw]

Das Signal <das.signal at freecalypso.org> wrote:

> I'd like to report on the state of my attempts to debug gsm-fw. So far I've
> been able to install the TI toolchain and recompile both sotovik and leo2moko
> (debug version) without any issues,

Nice!

> The sotovik build is
> a bit too large (5 MB) but I assume some parameter must be tweaked -- it's
> not a big deal anyway.

Look at the linker script templates under chipsetsw/system/template:
TI's original configuration (gsm_ds_amd8_lj3.template) was/is set up
for the 8 MiB flash chip on their D-Sample and Leonardo boards, and
even though the total code size in the pdt_2091 configuration is just
under 2.25 MiB, TI's linker still puts some sections in the upper
4 MiB half.

Note that these are linker script *templates*.  The final linker
script that is actually fed to the linker
(g23m/__out__/gsm_<blah>/lib/lnk__<blah>.cmd) is generated from the
selected template via a perl script.  It looks like the Original Modem
company fixed the fw build process to fit into their 4 MiB flash by
patching that lnk__<blah>.cmd output rather than the template from
which it is generated; I didn't like that approach because it's too
fragile (reset the timestamps and the patched downstream file will get
blown away and rebuilt incorrectly), and I wanted to bring the fw
build process as close as possible to being "from source" - hence I
looked at the Original Modem company's patches to lnk__<blah>.cmd and
made a functionally equivalent patch to the linker script template.

The patched linker script template in leo2moko is gsm_ds_k5a3281.template
(Samsung K5A3281 is the flash+RAM MCP we need to fit into when targeting
the GTA02 modem); I patched pdt_2091.mak to use it instead of the
original gsm_ds_amd8_lj3.template.

> I first used the original leo2moko (gsm-fw.m0) as a reference: it successfully
> connected to my test network, and the phone received SMS without issues.

Do you use your own test SIM with your test network?

> Then
> I flashed my compiled leo2moko-debug, flashed it with fc-loadtool and ran
> qtmoko-neo. Again everything worked fine, the phone being able to attach and
> receive SMS, including when in deep sleep.

Nice!

> Finally I compiled gsm-fw, based on configs/gtamodem-gsm then flashed
> flashImage.bin. Running qt-moko blocks on the PIN input screen,

Does your test SIM require a PIN?  I admit that I have yet to try
testing any of our firmwares with a SIM that requires a PIN; so far
all my testing has been on Operator 310260's live network (I know, I
know, I'm a bad girl for doing this) and their SIMs don't require a
PIN, at least not by default.

> rvinterf also
> does not show the usual frequency/sch burst that appears with leomoko-debug.

It means that the modem initialization process hasn't progressed to
the point of attempting to connect to the network.

> Now since qtmoko, or more specifically a process called "qpe", is a bit of a
> black-box, I switched to fc-shell

QtMoko is probably too high-level for our current need of debugging
our fw in terms of most basic functionality.  I've been talking AT
commands to the GTA02 modem manually long before we had fc-shell;
without this recent AT-over-RVTMUX addition one would talk AT commands
to the modem by running some simple terminal emulator program on the
Freerunner's Linux AP, pointed at /dev/ttySAC0.  But the AT-over-RVTMUX
addition in leo2moko-debug combined with fc-shell makes the process a
little bit easier.

The logs I posted earlier contain the correct sequence of AT commands
for bringing the modem up and dialing an outbound (MO) test call: just
grep for 'Sent to ATI' to see them explicitly.

I believe that the proper debug approach should be to boot both
leo2moko-debug and full FC firmwares in the same way, i.e., power the
modem up *without* QtMoko running, and run the same sequence of
fc-shell AT command against both fws.  Having QtMoko in the picture
with either fw will just muddy the waters.

> AT+COPS=? returns with ERROR immediately
> whereas; it works as expected in leo2moko-debug.

Hmm, with classic TCS211 fw (mokoN or leo2moko*) if one powers up the
modem without QtMoko running and gives it an AT+COPS=? command right
off the bat, it will also respond with ERROR without doing or trying
to do anything.  Instead the following two commands need to be issued
first:

AT+CFUN=1
AT+COPS

AT+COPS makes it connect (register) to the default network.  AT+COPS=?
is a query for the list of available networks; common sense says that
one should be able to make this query without having to register to
the default network first, but for some reason with TCS211 fw AT+COPS=?
fails until one does a plain AT+COPS first.

If plain AT+COPS succeeds, one can do an AT+COPS? query (notice the
lack of '=') to see which network it registered to.

> So at this point I'm enabling more traces to identify where this problem comes
> from. One thing I've noticed is that leo2moko unfortunately uses this id to
> string translation mechanism (str2ind.tab), which makes comparing traces with
> gsm-fw a bit difficult, so I'm looking into ctracedec to get a better output. 

That's why I wrote that ctracedec utility. :)

ACI does get compiled from real source in leo2moko though, so
hypothetically we could turn off str2ind when compiling the ACI code -
but it was more surgery on the TCS211 build system than I was willing
to get into - you are welcome to try it though!  And of course we
still have the binary blobs for the rest of G23M that contain str2ind
indices compiled into them. :-(

> Well, as it turns out I forgot to run AT+CFUN=1. Silly me ;)
> Now AT+COPS=? works as expected!

Wait a moment, are you saying that with our TCS3.2-based FC GSM fw you
could issue AT+CFUN=1 and then AT+COPS=? works *without* an intervening
plain AT+COPS?  I ask because I haven't tried it myself, and right now
I'm away from my Freerunner.  Perhaps someone at TI realized that
their TCS211 behaviour of not allowing AT+COPS=? without registering
first was a defect and fixed it in TCS3.2.

Speaking of which, ACI is the one component of TI's core modem
functionality which we got in full source form with both of our
starting points: both the TCS211 semi-src and the TCS3.2/LoCosto
source.  For the rest of G23M and L1, the TCS3.2 version is the only
real source (unless we also count the much older and heavily
divergent TSM30), hence TCS3.2 is the version we had to use here.

In the case of ACI, I had the choice of either going with the version
from TCS211 (which is known to work but could have any number of
incompatibilities with the rest of G23M being from TCS3.2) or the
version from TCS3.2 (which is of unknown quality but matches the rest
of G23M and L1).  The version of ACI currently in FreeCalypso is the
one from TCS3.2: I figured that having to discover and fix whatever
bugs and issues we may have in this code is a lesser evil that having
a version mismatch boundary fall inside G23M - after all, ACI is still
a part of G23M, and uses cdginc headers.

Also comparing the two versions of ACI, I could see that the newer
TCS3.2 code does have some functional improvements over TCS211 - thus
once we shake the bugs out of it, it is entirely possible that our
TCS3.2-based fw may work even better than TCS211.  But we need to
shake all those bugs out first, as right now it is non-functional. :(

Happy hacking,
SF