FCDEV3B sleep mode bug update

[Mychaela Falconia]

Hello FreeCalypso community,

A couple of days ago I finally performed a long-delayed experiment: I
tried increasing the bypass capacitor values on the V-FLASH and V-SRAM
power nets to see if doing so would make the sleep mode bug go away
while keeping the high-capacity flash+pSRAM chip S71PL129NC0HFW4B.
The result: upping these capacitor values did not change the behaviour
in any way, and the following scenario still consistently produces an
aberrant self-reboot:

* Put FC Magnetite l1reconst firmware in the flash (must be flashed,
  not fc-xram, see below);
* Enable small sleep (AT%SLEEP=1) or all sleep (AT%SLEEP=4);
* Try to bring up the SIM with AT+CFUN=1 - bam, the board reboots.

With the power nets ruled out, I went thinking what else could possibly
be causing this sleep-triggered misbehaviour to occur with our high-
capacity Spansion MCP (copied from Pirelli DP-L10) but not with the
smaller Samsung MCP used by Openmoko, and I got a new working
hypothesis.

The Calypso chip has a poorly documented output called FDP.  TI's
CAL000 document gives it a one-line description of "Flash deep
low-power", but there is no further explanation anywhere that I could
find.  TI's Leonardo reference schematics connect this Calypso output
to the flash chip's reset input, and this same connection is present
in Openmoko's PCB.  We then copied it for the FCDEV3B.

I have no proof yet, but my current working hypothesis is that
Calypso's FDP output goes low (corresponding to flash reset asserted)
not only in deep sleep, but also in small and big sleep modes which
involve the ARM7 being stopped, but not the VC(TC)XO.  The proper
scientific way to test this hypothesis would be to probe this signal
with an oscilloscope on a live board and see what happens in different
sleep modes, but it would be difficult to do this experiment on an
FCDEV3B or a GTA02 because in our (previously OM's) PCB the signal in
question goes from one BGA to another (from the Calypso to the memory
chip) without any accessible test points.  However, on Pirelli's PCB
the signal that comes from Calypso's FDP output goes to an unpopulated
resistor footprint that can be used as a test point, hence I plan on
using a decased Pirelli motherboard for this experiment - but it will
take me some time to make the necessary arrangements.

So here is my current working hypothesis as to the sleep mode bug: if
we suppose that FDP does go low during small sleep, then if the
running firmware is flashed (not RAM-based) and has the FFFF:FB10
register configured so that interrupt vectors go to flash rather than
Calypso's internal ROM (vectors in flash is the configuration used in
TI's original firmwares and in FC Magnetite), then the interrupt event
that causes wakeup from sleep is immediately followed by the Calypso's
memory interface (MEMIF) doing a read from address 0x0000_0018, which
is the ARM IRQ vector location.  The flash chip sees an asserted reset
signal during the sleep, then upon wakeup it sees this reset being
negated, immediately followed by a read cycle.

Now suppose that the reset pulse seen by the flash chip is too short
because the sleep duration was very short, or perhaps the time between
FDP going high (reset negation) and the immediate read cycle for the
IRQ vector is shorter than the minimum required by the flash chip.
Under such circumstances it is plausible that the flash chip may
ignore the read cycle or return the wrong data.  Because this read is
for the IRQ vector which contains an ARM branch instruction, if the
Calypso reads wrong bits from the data bus on this cycle, the flow of
control will go haywire, and a reboot can plausibly occur if the
control falls into the RESET entry code nearby.

I have no way of telling if TI's approach of connecting the FDP output
to the flash chip's reset input was ever a good idea for any flash
chip, but it appears that the high-capacity Spansion MCP which we
copied from the Pirelli may be less forgiving of such abuse than its
predecessors used by TI or the Samsung MCP used by Openmoko, and while
this arrangement worked fine for OM and possibly on TI's Leonardo
boards (of which I have never seen one with my own eyes), the newer
Spansion flash chip gets unhappy with the peculiar reset timing it
gets subjected to.  If this hypothesis is correct, then it would
follow that the very strange behaviour that was seen on only one
FCDEV3B board whereby the board would boot from flash only in mode 0
but not in mode 1 probably has the same root cause as the sleep mode
bug, and it just happened that on that one board the flash chip is a
bit more sensitive and gets unhappy with the watchdog reset that
happens as part of flash boot mode 1.

I can only assume that TI connected Calypso's FDP output to the flash
chip's reset input on the reasoning that having the reset input
asserted ought to put the flash chip in its lowest current draw state,
and like I already said, I don't know if this reasoning was ever
correct for any flash chip - maybe it was, maybe it wasn't.  But the
datasheet for Samsung K5A3280 (the closest I could find to Openmoko's
K5A3281CTM) gives exactly the same Icc (current from Vcc) spec both
for the reset held down state and for the regular standby state (no
reset asserted, but no chip select activity), hence with this chip
having or not having the reset input asserted during sleep ought to
make no difference.  And with Spansion high-capacity flash it is the
opposite according to the datasheet we have: Icc in the reset held
down state is higher than Icc in regular standby!

At this point the logical thing to do was to see how this flash reset
signal is connected on the Pirelli DP-L10, so I traced it using
steve-m's PCB grind-down scans.  And lo and behold: on the Pirelli
phone the reset input to this same Spansion flash chip is NOT
connected to the FDP output from the Calypso!  Instead it is connected
to a pair of 0402 resistor footprints, making it configurable: there
is one resistor that, if populated, would connect the flash reset line
to FDP - but that resistor is not populated - and the other resistor,
which is populated, connects it to Calypso's TCXOEN output.  TCXOEN
enables or disables the 26 MHz oscillator (and on the Pirelli it
actually is an external VCTCXO), and it goes low during deep sleep -
but not during small or big sleep.  Wakeup from deep sleep has
programmable timings (unlike wakeup from small sleep), and I can only
reason that Foxconn/Pirelli must have configured these timings so that
the flash chip's reset timing requirements are satisfied.

Another experiment I did was to try running a RAM build of the same
Magnetite l1reconst fw that consistently triggers the sleep mode bug
when flashed, and guess what: when I run the fw in RAM, I can enable
all sleep modes with AT%SLEEP=4, then do AT+CFUN=1 and the latter
command completes without rebooting!  I then connected to the GSM
network with AT+COPS=0, and I saw the fw go into both big and deep
sleep while camped on a cell and listening for paging.  This was my
first time ever seeing deep sleep on an FCDEV3B with L1 in I_MODE
(idle, camped on a cell) as opposed to CS_MODE0 (the power-up state
before any AT commands when no network bring-up is attempted), and I
am pleased to report that we do NOT have the kind of issues with deep
sleep which Openmoko experienced as their infamous bug #1024.

My next experiment was to hack together a firmware image that boots
and runs out of flash, but routes the ARM7 interrupt and exception
vectors through Calypso's internal ROM and RAM just like we do with
our run-from-RAM fw builds.  I was hoping that this change would make
the sleep mode problem go away by inserting an additional delay
between the wakeup event and the first subsequent flash access, and
thus give us a workable software solution for allowing sleep modes in
flashed fw on our current FCDEV3B hw, but no such luck: the exact
behaviour changed a little, but the same AT+CFUN=1 operation that
always caused a self-reboot before still causes some kind of crash or
hang.

I am now wondering if this sleep mode bug can be fixed in the hw by
disconnecting the flash chip's reset input from Calypso's FDP output
and simply tying it high instead.  There will be no more external
power-on reset to the flash chip with this arrangement, but I reason
that the flash chip probably does not need such an external reset and
can instead ensure the correct power-up state on its own just from the
Vcc ramp-up - after all, many flash chips I have worked with don't
have an external reset pin at all.  I plan to proceed as follows:

1. Do some experiments on a decased Pirelli motherboard to test my
   hypothesis regarding Calypso's FDP output going low in all sleep
   modes.

2. If my FDP hypothesis proves correct, try a very delicate surgical
   experiment on an FCDEV3B: remove the flash+pSRAM chip (BGA rework
   station to the rescue), cut the trace between the D5 ball pad
   (flash reset) and the microvia it goes to, make a solder bridge or
   some other hacky connection to the C5 ball pad next to it (WP#/ACC
   input, connected to flash Vcc through a pull-up resistor), and then
   put a new S71PL129NC0HFW4B flash+pSRAM chip back on (BGA rework
   station once again).  The effect will be to disconnect the flash
   chip's reset input from FDP and tie it high instead.  See if this
   hw change makes the sleep mode bug go away.

3. If the previous step succeeds, make the same change in the Altium
   PCB layout, make new gerbers with this change, and fab a new batch
   of boards (call them FCDEV3Bv2) with this change incorporated.

The last step will probably cost me somewhere in the 3-4 kUSD range,
hence the preceding steps to make very sure that my proposed change
will do the trick.

Hasta la Victoria, Siempre,
Mychaela aka The Mother