Firmware debug: handover to the community

[Spacefalcon the Outlaw]

Das Signal <das.signal at freecalypso.org> wrote:

> Thanks for the detailed update! I will try to follow in your footsteps
> and debug the firmware in the directions you have mentioned. It could
> take a couple weeks though, as I'm not very familiar with the code base.

Take your time: it will take me more than a couple of weeks to design
and build the proposed GSM development board. :)

David wrote:

> Are you guys aware that there was a deep sleep hardware problem on the
> freerunner? 

Thanks for bringing it up - I know about this issue full well, but DS
may not, so we do need to discuss the hardware aspect of deep sleep
troubles before DS wastes a ton of time chasing after it.

The issue was/is known as the infamous bug #1024, and it manifested
itself in Openmoko devices losing network registration when entering
deep sleep - exact same behaviour which we now see with my backport of
LoCosto L1 to Calypso, except that on those Om devices that were
affected by bug #1024 the problem would occur with the official mokoN
firmwares based on the official TCS211 code from TI.

Not every Om-produced unit was affected (it seemed to be a law of
chance), and on those units that were affected the behaviour changed
with temperature and possibly other environmental factors.  Om's first
round of chasing after this problem was in 2008, and that was a year
before TI exited the baseband chipset business and ceased providing
support, so Om still had support from TI at that time.  They tried a
few firmware patches and then came to the conclusion that the problem
was in hw rather than fw.

A year later, in early 2009 (just as TI was closing shop) Om's hired
contractor Dieter Spaar did another round of going after this bug, and
after a few false leads he found that the problem went away when the
capacitor at reference designator C1009 was bumped up from 10 uF to a
higher value, either by replacing it with a 22 uF cap or by connecting
another 10 uF cap in parallel.  Look at this schematic to see what I'm
talking about:

http://dos.openmoko.pl/GTA02_Schematic_MB_A5_1220.pdf

(Also mirrored on ftp.ifctf.org, of course.)

Dieter blamed the original problem on a poor PCB layout in that Iota's
VSDBB ball (sense input) is connected directly to VRDBB (LDO regulator
output), as opposed to having a separate trace coming back from the
point of load (one of Calypso's power input balls), but I suspect that
Om-added 0R resistors at R1003 and R1004 (look on the same schematic)
may be to blame instead: the original Leonardo schematics show star
connection points there instead, and turning them into physical 0R
resistors (0402s) must have been Om's idea.

For our own FreeCalypso hardware, starting with the development board
discussed earlier, my plan is to implement star connection points in
the PCB layout *without* physical 0R resistors, but also populate a
22 uF cap instead of 10 uF (same 0805 footprint) just to be safe.

> I'm not sure this is accurate, but AFAIK the problem was (supposedly)
> fixed in later versions of the hardware.

AFAIK they never did a new factory production run with the problem
fixed, instead the major distributors sent their stock to a contracted
shop for rework; the latter consisted of increasing the capacitance at
C1009 as described above.  The device I have (bought from GolDeliCo in
late 2011) is one of those reworked units.

> I say supposedly, coz there was a software test to check the status
> of your handset;

AFAIK the test consists of detecting the "bouncing" behaviour (network
registration getting lost and then coming back) and switching to
AT%SLEEP=2 (big sleep instead of deep sleep) when that behaviour is
seen.

Oh, and just to clarify what deep sleep is: it is the sleep mode in
which the 26 MHz VCXO is physically stopped and powered off, so that
the only oscillator that remains running is the 32.768 kHz one.  The
system wakes up from deep sleep and turns the main clock source back
on when the programmed number of 32.768 kHz cycles have elapsed; the
programming of this cycle count by the fw is based on the knowledge of
when the MS needs to wake up to listen to the paging channel.

> If this may be to the point, do you want me to try and find the
> documentation of this issue?

http://docs.openmoko.org/trac/ticket/1024
http://lists.openmoko.org/pipermail/hardware/2009-May/001192.html

However, the point that matters for FreeCalypso is as follows: on my
GTA02 unit (which has the capacitor fix to the best of my knowledge)
deep sleep works just fine with TCS211 firmware (both mokoN and
leo2moko), but fails with our FC firmware in which L1 has been taken
from TI's LoCosto (TCS3.2) source and back-ported to Calypso by yours
truly, hence we know that our FC version of L1 code is currently
defective with respect to deep sleep irrespective of any hardware
issues.

But it is also true that hardware issues with deep sleep do complicate
matters, and we don't know the hardware fix status of the unit which
DS recently obtained.  Therefore, my recommendation to DS and anyone
else who would like to try his or her hand at fw debugging is to start
with deep sleep disabled, troubleshoot the other show-stopping issues
first, without deep sleep in the picture (remember that our fw still
goes hayware when I try to dial a call, even with deep sleep disabled),
and then revisit deep sleep later when the rest seems to work OK.

Happy hacking,
SF