Firmware debug: handover to the community

[Spacefalcon the Outlaw]

Das Signal <das.signal at freecalypso.org> wrote:

> I do use my own SIM (without PIN). This way I can do a complete
> authentication procedure, as if I were a real network.

Nice!  One big advantage of having a setup like yours is that you
should be able to see the mobile station's behavior as the network
sees it.  What do you see from the network side when running our
entroubled gsm-fw?  Do you see it registering like it should, or does
it do something bogus?  What does the network see when our entroubled
gsm-fw attempts to dial a call or send SMS?

> So I've followed your advice and used this combination of sp/AT commands:
> sp MM TRACECLASS FFFFFFFF  (mobility management)
> sp RR TRACECLASS FFFFFFFF  (radio resource)
> sp L1 TRACECLASS FFFFFFFF  (layer1 control of dsp)

Are you sure if the last one makes any difference?  As far as I could
tell, our version of L1 (both FC and TCS211) does not emit any traces
through GPF, only through its own trace path, so it seems to me that
it won't be affected one way or the other by the TRACECLASS setting in
GPF.

> AT+CMGF=1  (send SMS in text mode)
> AT+CMGS="+..."
> ATsomemessage

Ah nice, I see you are testing SMS.  I know it works perfectly well in
leo2moko-debug, but what happens when you try to send SMS (with the
commands you were using) from our entroubled gsm-fw?  I haven't tried
it myself yet, I only tried dialing a MO voice call, and seeing that
it goes berzerk with GPF memory alloc errors even when deep sleep is
disabled, I concluded that we must have some serious bugs in that code
path which need to be shaken out before it will make sense to play
with other stuff.  But thinking about it more, since you have a test
network and can therefore test stuff without messing with a real one,
testing what happens with SMS (as opposed to calls) should an easy to
acquire extra data point.

BTW, if you need to send a string to ATI from fc-shell that does not
begin with "AT" or "at" (i.e., to send SMS string input that doesn't
begin with those characters), the command for doing so is 'str'.

Oh, and have you tried dialing a call (ATD<number>;) with gsm-fw?  Are
you able to reproduce the failing behavior seen here:

https://www.freecalypso.org/traces/20150607-fc-bigsleep.txt

(That log shows MO voice call failing with gsm-fw even though deep
 sleep is disabled.)

Or does the same erratic behavior (including GPF memory alloc errors)
occur when attempting MO SMS too and not just voice?

> I have been able to reproduce this bug with gsm-fw,

Which one: deep sleep resulting in failing burst Rx and loss of network
registration, or the berzerk activity when trying to dial a MO call
regardless of deep sleep state?

> I'm still busy analyzing the traces, but one thing was apparent: I'm
> getting 138 "Trc Abort" messages in gsm-fw, whereas leo2moko only got 2.
> One reason a trace can be aborted is a problem with memory allocation,
> so I'm tempted to examine deeper the memory allocation mechanism.

Yes, you should look into those memory and trace issues.  I never got
any of those "Trc Abort" messages myself, but then I wasn't enabling
any verbose GPF traces (just looking at the ones from L1 which are
always enabled), and we probably will need those GPF traces in order
to troubleshoot the issues we are seeing.

> One further idea would be the development of a small gdb stub that would
> be able to trace the execution of the firmware, using the rvt protocol.
> I'm not completely sure this is doable, because the code to be traced,
> unless in RAM, would be read-only.

The idea does sound nice...  Meanwhile, until we can implement that
sweet idea, take a look at my lldbg hack: freecalypso-sw/gsm-fw/lldbg,
there is a README in there.  In particular, if no one else makes any
progress on the MO call error behavior while I'm away from this
subproject, the way I was going to debug it would be to insert an
lldbg_entry() "breakpoint" in vsi_m_new() where that error is reported,
and try to backtrace from there.

Happenings on my side: I've been away from the FC project for a little
bit to catch up on general life chores, but I expect to be done with
those chores in another few days, and then I'll start tackling the
hardware subproject of FC like I said I was going to earlier.

Happy hacking,
SF