How NOR flash memory really works
=================================

The type of flash memory used in Calypso GSM devices is formally known as NOR
flash.  Most embedded software programmers and tinkerers know the fundamental
principle of how NOR flash works: any bit can be transitioned from a '1' to a
'0' at any time in any combination (an operation called programming), but the
opposite transition (from '0' to '1' bits, an operation called erasure) can only
be done on fairly large sectors - you can erase a sector and make it all 1s,
but changing bits from 0 to 1 individually or in any smaller granularity
(smaller than a sector) is impossible.

What many "software-minded" programmers and tinkerers don't realize, however,
is that sector erasure is not an elementary or atomic operation that magically
"makes all bits 1s" in one motion.  Instead it is a complex process with two or
three substeps:

1) Before starting the physical process of erasure, one has to go through all
   bits in the to-be-erased sector and make them all 0s.  Any bits that are in
   '1' state when the sector erase operation is commanded MUST be programmed to
   '0' state before the actual erasure begins!  In the language of flash chip
   industry, this step is called preprogramming.  In the case of flash chips
   that are used in Calypso GSM devices (all known ones), this preprogramming
   step is done internally by the chip, so that you as the user or software
   developer are not aware of it - but it is there nonetheless.  The chip does
   NOT magically "wave" all bits in the sector into '1' state, instead it first
   makes them all '0' internally, and only then erases.

2) Once every bit in the sector is in '0' state, the real physics of erasure
   begins.  All bit cells in the sector are physically acted upon at once in
   this step, and because it is a probabilistic process involving a Gaussian
   distribution, all bit cells need to be in the fully programmed state before
   they begin their shared journey toward the erased state.

3) The step of preprogramming every bit to 0 prior to erasure prevents the
   absolutely unacceptable condition of gross overerasure - but given the
   Gaussian distribution, some bit cells may still get a little overerased.
   Many (most? all? not sure) flash chips therefore implement a third internal
   step before the software-visible "erase" operation is declared complete:
   they go through all bit cells in the just-erased sector, check for
   overerasure, and "soft-program" (move slightly to the right in the Vt
   distribution) any overerased cells.  This step is called post-erase
   conditioning or recovery.

The above process was originally explained to me (Mother Mychaela) some years
ago (around 2008, IIRC) by a Spansion support engineer on a conference call at
my then day job - it was a project for a customer who was big and powerful
enough to get top-tier support from chip vendors.  More recently, however, some
other flash vendors have posted public documents that provide the same
explanation - here is one from Renesas/Adesto:

https://www.freecalypso.org/pub/embedded/flash/REN_an500_APN_20210702_1.pdf

Even though the above document was written by Renesas (or more precisely, the
part that was originally Adesto), the theory described therein applies just as
well to Intel, Spansion and Samsung flash chips that are used in Calypso GSM
devices.  For anyone who wishes to know how NOR flash memory really works, I
strongly recommend reading that Renesas appnote - it is a good description.

Additional note on terminology: describing the two states of a flash memory cell
as '0' and '1', like I did above, is only a convenience for software-minded
people.  A more proper view is to think in terms of a "programmed state" and an
"erased state" for each bit cell.  History and tradition are such that flash
chips return '0' on read in the programmed state and '1' in the erased state
(this tradition probably originates from the fact that the actual NV storage
element, a transistor, conducts read current in the erased state), at least for
the main flash array - however, when flash memory elements are used for
additional purposes such as write protection controls, it is best to think
natively in terms of programmed and erased states.  For the latter kind of
special applications, an opposite polarity may be applied in read-bit values.

One straightforward take-away from this theory is that flash endurance is really
about program-erase cycles, rather than number of program or number of erase
operations.  Every time you give a sector erase command, every bit in that
sector cycles through the fully programmed (0) state first before becoming
erased (1), irrespective of whether or not you programmed into it on your own!
Hence every bit-cell of the affected sector always goes through a full
program-erase cycle, and all bits in a given sector are always cycled equally,
irrespective of whether they get written with mostly-0s or mostly-1s in between
erase cycles.

Another situation where this raw physics gets exposed to the user is the case
of special-purpose non-volatile bits in flash chips outside of the main flash
memory array - for example, Persistent Protection Bits (PPBs) in some Spansion
and Samsung flash chips.  While program and erase commands for the main flash
array invoke chip-internal mechanisms that take care of everything and present
a sane model of 0s and 1s to software, Spansion PL-J PPB program and erase
commands expose raw guts: there is a command that applies a raw program pulse
to a single PPB, and there is a command that applies a raw erase pulse to the
NV memory element (like a little sector of its own) that holds all PPBs.
Applying the erase pulse without preprogramming every PPB first would be very
bad (see Renesas appnote about the badness of overerasure) - hence in a seeming
paradox, one has to explicitly lock every sector before applying PPB erase
pulses that will eventually unlock everything!

Our flash ppb-erase-all command does implement the preprogramming step before
actual erasure, and the present document (hopefully) explains why.