It has been discovered that the DSP ROM in the Calypso GSM baseband processor
makes it possible to "tap" into speech traffic on GSM traffic channels (TCH):

1) In the downlink direction, the signal processing chain which every GSM MS
   must implement includes a GSM 05.03 channel decoder, operating in one of
   several variants as necessary for each supported TCH mode, followed by speech
   decoders for each supported codec.  TI's DSP naturally implements this
   required signal processing chain, and this implementation includes one nifty
   feature: the bits that make up the internal interface from GSM 05.03 channel
   decoder output to the input of speech decoders are written into the NDB API
   RAM page that is also accessible to the ARM core, and these bits can be
   externally read out.  The act of reading these bits is completely
   non-invasive (we are only reading bits that are already there, not modifying
   anything), thus we can sniff TCH downlink on any voice call in real time
   without disrupting or impacting standard type-approved GSM MS operation in
   any way.

2) In the uplink direction, there is a reverse signal processing chain in which
   the output of the internal speech encoder for the selected codec feeds into
   the input of the corresponding GSM 05.03 channel encoder.  In this direction
   there are two tapping possibilities:

2a) There is a buffer in the NDB API RAM page from which one can read the bits
    that pass from the speech encoder output to the channel encoder input -
    let's call this form of TCH tap "uplink sniffing";

2b) There is a special mode in which the output of the internal speech encoder
    is effectively suppressed and the input to the channel encoder comes from
    another NDB API RAM buffer that needs to be filled by ARM firmware - let's
    call this form of TCH tap "uplink substitution".

Sources of knowledge about these DSP functions
==============================================

For the functions of TCH DL sniffing (tap 1 in the above summary) and TCH UL
substitution (tap 2b in the above summary), the primary source of knowledge is
the defunct '#if TRACE_TYPE==3' code in TSM30 and LoCosto L1 sources.  I call
this code defunct because the TRACE_TYPE preprocessor symbol is set to 4 (not 3)
in both TCS211 and LoCosto versions, and appears to be set to 0 (all trace
disabled) in the ancient TSM30 build.  This code appears to be some very old
test mode, apparently sending some test bit patterns into TCH UL and expecting
the same bit patterns back on TCH DL, presumably with a test instrument such as
CMU200 providing a loopback from UL to DL on this test TCH, and has only
survived in an incomplete form:

* There are '#if TRACE_TYPE==3' stanzas in l1_cmplx.c, in both TSM30 and LoCosto
  versions, that implement DSP buffer writing for TCH UL substitution (TCH/F
  only) and timing control for TCH DL buffer reading (both TCH/F and TCH/H),
  calling a function named play_trace() for the latter.

* There is no play_trace() code in the LoCosto source. but there is an
  hw_debug.c source module in the TSM30 code drop under MCU/Layer1/L1c/Src,
  and it contains (presumed) TI-legacy play_trace() and play_diagnostics()
  functions, once again under '#if (TRACE_TYPE==3)'.  play_trace() reads the
  DSP's TCH DL buffer and saves the bits in an ARM firmware RAM buffer, and
  then play_diagnostics() analyzes the captured booty - and studying the second
  function is how we learn the apparent original intent of doing test bit
  patterns on TCH.

* The code that feeds "UL play" test bit patterns to the earlier-mentioned
  '#if TRACE_TYPE==3' TCH UL substitution code in l1_cmplx.c (apparently once
  hacked into dll_read_dcch() and tx_tch_data()) has not been found anywhere.

For TCH tap 2a in our summary at the beginning of this article (non-invasive
sniffing of TCH UL bits produced by the internal speech encoder) there does not
exist any authoritative source of knowledge.  It naturally follows from
otherwise-known Calypso DSP architecture that these internally produced TCH UL
bits should reside in the "main" a_du_0 buffer (or in a_du_1 when TCH/H
subchannel 1 is active), and I (Mother Mychaela) have heard an anecdotal report
(from someone who once worked with Calypso in a non-community-based manner) that
these UL bits could indeed be read out of this buffer - but in the absence of
an authoritative source, we don't know when would be the correct time to read
this buffer.

In our current state of knowledge, only TCH DL sniffing can be exercised safely:
for UL sniffing we don't know the correct time when the buffer would need to be
read, while active UL substitution is obviously an invasive hack involving a DSP
debug or test feature that is never used in standard GSM MS operation.

Support for different speech codecs
===================================

When it comes to passively sniffing TCH DL and/or UL, we are merely reading bits
that are already there, and basic reasoning tells us that the DSP's DL and UL
buffers involved in this exercise exist in all speech TCH modes supported by
the DSP: FR1, HR1, EFR and AMR.  However:

* The ancient '#if TRACE_TYPE==3' reference code exists only for FR1, HR1 and
  EFR - it clearly predates the addition of AMR in the later Calypso DSP
  versions.

* FR1, HR1 and EFR are the only codecs for which we (FreeCalypso community) know
  the format in which TCH DL bits appear in the DSP's a_dd_0 and a_dd_1 buffers.

* I (Mother Mychaela) have heard an anecdotal report (from the same
  non-community-based party mentioned earlier) that TCH DL bits could be read
  out of a_dd_0 buffer in TCH/AFS (AMR) mode - but I never got any details.

In contrast with passive sniffing, active TCH UL substitution requires explicit
support from the DSP - and this explicit DSP support is known to exist for
certain only for TCH/FS and TCH/EFS channel modes, i.e., for FR1 and EFR codecs
only.  In the case of TCH/HS channel mode (HR1 codec), it *appears* that the DSP
supports UL substitution in this mode too, but this combination has only been
exercised by OsmocomBB people (the original '#if TRACE_TYPE==3' code for UL play
only supports TCH/F), and FreeCalypso policy is to treat everything coming out
of OBB as highly suspect.

What about AMR?  The anecdotal report (from the same already-mentioned party) is
that TCH UL substitution that works for FR1 and EFR appears to NOT work for AMR
- that's all I know - but frankly speaking, given that it's a weird DSP debug
mode that is never needed in standard GSM MS operation, I find it more
surprising that it works for FR1 and EFR than the observation that it doesn't
work for AMR.

FreeCalypso support for TCH tap functions
=========================================

TCH DL sniffing and UL substitution provisions were initially implemented in
FreeCalypso back in 2016, but only in the Citrine version, which was deemed to
be a dead end later that same year.  However, this functionality is now being
resurrected, and it has been incorporated into our production FC Tourmaline
firmware as of 2022-12-13.

In order to activate the function of TCH DL sniffing and save the recording of a
TCH DL session into a file, one needs to use the fc-shell utility from FC host
tools, specifically the tch record command in an interactive fc-shell session.
The format in which TCH DL tap traffic is passed over RVTMUX (an original
FreeCalypso invention) has changed in a slight but incompatible way between the
original hackish version from 2016 and the new production version as of 2022,
and capturing TCH DL with new firmware requires the updated version of fc-shell
that was released as part of fc-host-tools-r18.  The current (developed in late
2022) incarnation of FreeCalypso TCH DL sniffing feature supports FR1, HR1 and
EFR codecs; initially only FR1 and EFR were tested, but in 2024 HR1 joined the
tested and known-working set.

The function of TCH UL substitution is currently implemented in FC Tourmaline
only for FR1 and EFR (no HR1, no AMR), and it likewise requires running an
interactive fc-shell session in which you would invoke the tool's tch play
command.  In the case of TCH UL play feature there has been NO change in the
RVTMUX transport format between 2016 and 2022 versions.

TCH DL DSP buffers and capture format
=====================================

The DSP's NDB API page has two buffers in which TCH DL bits appear: a_dd_0 and
a_dd_1.  All TCH/F modes use a_dd_0, but TCH/H uses one buffer or the other
depending on the subchannel: subchannel 0 uses a_dd_0, subchannel 1 uses a_dd_1.
(It is certainly a strange design - the DSP won't be able to receive and decode
the "wrong" subchannel because it doesn't know the ciphering key for the other
MS - but perhaps the designers of this DSP architecture aeons ago found this
design to somehow flow more naturally with their scheduling of DSP tasks.)  Each
buffer consists of 22 16-bit words - they were originally 20 words, but then
extended to 22 words to support CSD 14.4 kbps mode.

Each TCH buffer in the DSP's NDB API page consists of 3 status or header words
followed by N words of payload, where N depends on TCH mode: 17 for TCH/FS and
TCH/EFS, 8 for TCH/HS, and not-yet-studied for AMR and CSD.  Let's begin our
analysis with the 3 status words that make up the buffer header:

Status word 0 (a_dd_0[0] or a_dd_1[0]) is a word of flag bits.  We don't know
the meaning of every bit in this word, but here are the bit meanings we've been
able to figure out so far, looking at TCH/FS, TCH/EFS and more recently TCH/HS
DL captures:

* Bit 15 (B_BLUD) is a "buffer filled" or "data present" flag.  This flag is
  observed as 1 in *almost* every 20 ms window in which a traffic frame is
  expected (fn_report_mod13_mod4 == 0 in l1s_read_dedic_dl(), case TCHTF),
  except for certain instances early in the call setup process which remain to
  be studied.

* Bit 14 (B_AF) will be set if the block of 8 half-bursts (block diagonal
  interleaving of GSM 05.03) corresponding to this buffer was channel-decoded
  as speech rather than as FACCH - see further analysis below.

* Bit 9 (B_ECRC) has only ever been observed as 1 when B_AF is set, i.e., when
  the speech-not-FACCH channel decoder was invoked.  In the case of TCH/EFS this
  bit is set to 1 if the EFR-added CRC-8 was bad, and cleared if this CRC-8 was
  good; in the case of TCH/FS this bit has always been observed as 1 and should
  be ignored because there is no CRC-8 in TCH/FS.

  (Update for TCH/HS: in this channel mode this bit has always been observed
   as 0.)

* Bit 7 has always been observed as 1 wheneven B_BLUD is set but B_AF is
  cleared, i.e., whenever the block was channel-decoded in FACCH rather than
  speech mode.

* Bits 6:5 indicate the result of FIRE decoding in the event that the FACCH
  decoder was invoked.

* Bits 4:3 carry the ternary SID flag encoded as in section 6.1.1 of GSM 06.31
  and 06.81, but only when the speech-not-FACCH channel decoder was invoked as
  indicated by B_AF.

  (Update for TCH/HS: these bits apply to HR codec too, following GSM 06.41
   section 6.1.1 now.)

* Bit 2 is BFI as defined in section 6.1.1 of GSM 06.31 and 06.81.  Whenever
  the block was decoded as FACCH (bit 14 clear, bit 7 set), bit 2 has always
  been observed as set, agreeing with the stipulation in GSM 06.31 and 06.81
  that BFI=1 whenever a FACCH frame has been received.  However, in the case of
  TCH/EFS it appears that CRC-8 status (reported in bit 9) is NOT factored into
  the logic that sets bit 2 - it appears that the subsequent speech decoding
  logic is expected to OR bits 2 and 9 together to get the BFI flag for the Rx
  DTX handler of GSM 06.81.

  (Update for TCH/HS: bit 2 is BFI in HR codec too.)

* Bit 1 is seen only in TCH/HS, and appears to be the elusive BCI flag whose
  existence has been figured out from ETSI GSM 06.06 reid.c source.  See this
  article for more information:

  https://osmocom.org/projects/retro-gsm/wiki/HRv1_error_flags

* Bit 0 is UFI for TCH/HS only.

In the case of 20 ms blocks (reassembled from 8 half-bursts) that were channel-
decoded as speech rather than FACCH, the observed behavior is that bits 15 and
14 are set, the payload portion of the buffer is filled with the output from the
channel decoder, and bits 4:3 are set from this payload by the bit-counting rule
of section 6.1.1 of GSM 06.31 and 06.81 irrespective of the good-or-bad status
in bits 2 and 9.  However, when bit 14 is clear and bit 7 is set, indicating
that the block (from 8 half-bursts) was channel-decoded in FACCH mode, the
following additional behavior is observed:

* The payload portion of the buffer remains unchanged from its previous content,
  last written when a frame was channel-decoded in speech-not-FACCH mode;

* Bit 2 is set, bit 9 is cleared;

* Bits 4:3 are cleared even when they previously indicated SID based on the bit
  pattern in the payload portion of the buffer, even when that SID-encoding
  payload is still there.

In the standard TCH DL signal processing chain, GSM 05.03 channel decoding is
followed by the Rx DTX handler of GSM 06.31 or 06.81 for TCH/FS or TCH/EFS,
respectively.  It appears that the Rx DTX handler implemented in TI's DSP is
driven by this status word 0 at the head of the buffer, and we can only guess
as to its exact logic.  At this point it bears reminding that the functions of
the Rx DTX handler are not rigidly prescribed in the specs: in the case of EFR
the bit-exact reference implementation is normative only in certain aspects
(e.g., comfort noise generation after receiving SID), but is considered a non-
normative example in some other key aspects (all GSM 06.61 functions, including
what happens when a FACCH block was received when speech frames were expected),
and in the case of FR1 there is no bit-exact reference implementation at all,
only general guidance.

Having the curiosity of a cat, I (Mother Mychaela) naturally desire to know
exactly how the Rx DTX handler (the bridge between the channel decoder and the
speech decoder) works in TI's DSP.  A full static reversing job on the DSP ROM
would provide complete answers, but is a very daunting proposition, thus I am
also looking at the idea of behavioral analysis: the output of the speech
decoder can be captured from MCSI on FCDEV3B hardware, or from the VSP tap on
FC Venus if we ever build that board, and if we combine that speech decoder
output capture with the currently-discussed capture of TCH DL buffers, we may
be able to glean some insight into the workings of the Rx DTX handler block: we
could implement a candidate Rx DTX handler clone in software and compare the
output (of this proposed handler followed by the spec-defined speech decoder)
against the actual speech output from the DSP.

Back to our exposition of TCH DL buffer content:

Status word 1 (a_dd_0[1] or a_dd_1[1]) is some kind of DSP measurement or count
which Calypso ARM fw does not need to look at, except when debugging - the only
code which I (Mother Mychaela) could find that does anything with this DSP
status word is the ancient play_diagnostics() code in the TSM30 version
(obviously never included in any production fw); this code looks at the unknown
word in question and calls it "D_MACC".  This play_diagnostics() code compares
the D_MACC reading against a threshold, and if the per-block reading is below
the threshold, an error message is printed.  That's all we know!

Status word 2 (a_dd_0[2] or a_dd_1[2]) is a bit error count: the code in
l1s_read_dedic_dl() reads this error count and uses it for RXQUAL computation
for measurement reports.

If one's area of interest is in replicating Rx DTX handling and speech decoding
that happens in the DSP, status words 1 and 2 can probably be ignored - instead
the important parts are status word 0 (extensively covered above) and the
payload portion of the buffer.

The payload portion of the buffer consists of some number of 16-bit words: 17
of them for TCH/FS and TCH/EFS, or 8 of them for TCH/HS.  The DSP does not have
any notion of 8-bit bytes, instead it operates on 16-bit words as its elementary
data unit.  The ordering of bits within these 16-bit words (in the payload
portion of TCH buffers) is from the most-significant bit toward the least-
significant bit, thus when these TCH buffers are transferred via octet-oriented
interfaces, the upper byte of each word should be transferred first, even though
this byte order is counter to the little-endian byte order of the Calypso ARM
core.

In the case of TCH/FS and TCH/EFS, the fill order of bits in the payload words
is as follows, starting with the most-significant bit of buffer word 3 (first
word of the payload portion):

* 182 bits of class 1;

* 4 dummy bits (always observed as 0);

* 78 bits of class 2;

* the last 8 bits of a_dd_0[19] are unused.

In the case of TCH/HS, the fill order is similar, but modified as appropriate
for TCH/HS:

* 95 bits of class 1;

* 4 dummy bits;

* 17 bits of class 2;

* the last 12 bits of a_dd_0[10] or a_dd_1[10] are unused.

Aside from the insertion of 4 extra dummy bits at the boundary between class 1
and class 2, the overall bit order is that of GSM 05.03 Figure 1 interface 1.

In the case of TCH/EFS, the following additional considerations apply:

* Bits [65:73] in all received DL frames, where CRC-8 would go in the 260-bit
  frame of GSM 05.03 interface 1 for EFR, are always observed as 0, whether
  this CRC-8 was good (a_dd_0[0] bit 9 clear) or bad (a_dd_0[0] bit 9 set).

* The handling of repetition bits (4 bits of 244-bit EFR codec frame, each of
  which is triplicated in the channel encoding for transmission) is unclear.

Further detail regarding the repetition bits of TCH/EFS: distinct bit positions
exist in the 260-bit frame of GSM 05.03 interface 1 (which is the frame format
in the TCH buffers of TI's DSP) for each of the 3 copies of each of the 4
triplicated bits.  It is obvious that correct decoding of these triplicated bits
requires a majority-vote function just like the one implemented in TMR systems
in space gear - but it is not absolutely and unquestionably obvious where this
TMR voting function is implemented in the Rx processing chain of TI's DSP.  It
*appears* that this majority-vote function has already been performed by the DSP
function that writes a_dd_0, and that the first bit position out of each group
of 3 holds the output of this voting function, so that the subsequent speech
decoder only needs to use those "cooked" bits - but there is this mystery:

* At certain times, particularly during the main part of a test call, TCH DL
  buffer readouts contain zeros in the "extra" repetition bit positions: for
  each group of 3 bits, the first will contain 0 or 1, but the other two will
  always be 0.

* At other times, seemingly in the beginning and ending parts of test calls,
  TCH DL buffer readouts contain matching bit values in all 3 positions: for
  each group of 3 bits, if the first bit is 0, the other two will also be 0, or
  if the first bit is 1, then the other two will also be 1.

One possibility is that the DSP applies the required majority-voting function,
writes its output into the first bit position of each group of 3, but then
sometimes (and not at other times) applies another function that writes the
voting function output into the remaining bit positions, perhaps for loopback
of TCH DL into TCH UL.  More study is needed in this area.

Appearance of FACCH in TCH/HS downlink
--------------------------------------

All mentions of FACCH handling in the previous description refer only to TCH/F.
In the case of TCH/HS, the following differences in FACCH handling are seen:

* Bits [7:5] described earlier for the case of FACCH/F are never seen set in
  TCH/HS DL captures.

* In the first 20 ms frame position stolen for FACCH/H, status word 0 is seen
  as 0xC007 (looks like regular BFI), status word 1 is set to 0 (instead of
  whatever D_MACC is), and it is not clear whether or not status word 2 is
  still filled with the bit error count from the convolutional decoding step.
  (We've only seen 0 in that word.)  The payload bits are filled with what
  looks like garbage, in contrast with FACCH/F behavior of retaining previous
  buffer content.

* In the second 20 ms frame position stolen for FACCH/H, status word 0 is seen
  as 0, status word 1 looks like correct D_MACC after 456 rather than 228 bits
  have been decoded, and status word 2 contains 0xFFFF.  The "garbage" in the
  payload bits stays the same from the first half of FACCH/H.

FreeCalypso file format for TCH DL captures
===========================================

The file format written by fc-shell tch record command is ASCII hex, line-based,
with one line for every captured 20 ms window.  The new format as of 2022 is:

* Each line begins with an FR, HR or EFR keyword indicating which variant of
  TCH DL has been captured;

* This keyword is followed by 3 space-separated DSP status words, each written
  as 4 hex digits;

* The main body of the frame is written as 33 (TCH/FS & TCH/EFS) or 15 (TCH/HS)
  hex bytes, produced from the payload portion of the TCH DL buffer by turning
  each 16-bit word into 2 bytes (MSB first) and discarding the last byte that
  is unused (always 0);

* Each line ends with a frame number in decimal, specifically the value of
  fn_mod_104 variable in the l1s_read_dedic_dl() function when the DSP buffer
  was read.

The addition of the frame number field allows these TCH DL captures to be
reconciled against the SACCH multiframe structure, which matters for the rules
of DTX.

TCH UL substitution: open questions
===================================

Moving from the mostly-understood realm of TCH DL capture into the much more
experimental realm of TCH UL substitution, we have some open questions: how does
this DSP special mode really work?  Here is what we know: if we load externally
sourced speech frames into otherwise-unused a_du_1 DSP buffer at the time of
(fn_report_mod13_mod4 == 3), which is the same time when FACCH or CSD UL would
be expected, and set B_PLAY_UL bit in DSP NDB API word d_tch_mode, the speech
frame stream going to the other end of the call will be the one we feed into
a_du_1 instead of the one produced from the microphone input by the internal
speech encoder.  But here are the parts we don't know:

* If one were to set B_PLAY_UL in d_tch_mode but not feed external UL input
  into a_du_1 buffer at the needed time, what will happen?

* Vice-versa, if one were to load a_du_1 and set its B_BLUD bit without setting
  B_PLAY_UL in d_tch_mode, what will happen?

* Can the frame stream fed into a_du_1 be encoded in DTX-enabled mode, including
  SID frames?  If this possibility is allowed, what magic bits would need to be
  set where in order to get the correct behavior from the DSP's subsequent
  burst-by-burst DTX logic?

TCH UL substitution: implemented PoC
====================================

Back in 2016 we implemented a proof-of-concept TCH UL play feature in
FreeCalypso (only for TCH/FS and TCH/EFS), and the same PoC has been retained
when the overall TCH tap facility has been mainlined in late 2022.  Having this
highly experimental (not fit for production use) TCH UL play code present in our
current production fw is deemed acceptable because this code will never be
invoked unless the user sends TCH_ULBITS_REQ packets to the running fw via
RVTMUX - and if you do send such packets (via tch play command in an fc-shell
session or by any other means), you are leaving the realm of production-approved
functionality and entering the realm of wild experimentation.

The PoC TCH UL play mechanism consists of a small buffer (holding up to 4 FR1 or
EFR frames) implemented in the ARM firmware; this buffer is filled by arriving
TCH_ULBITS_REQ packets and drained by the tchf_substitute_uplink() function
called from l1s_ctrl_tchtf().  Specifically, a flag named tch_ul_play_mode is
set when TCH_ULBITS_REQ input is received, telling l1s_ctrl_tchtf() to start
calling tchf_substitute_uplink() when (fn_report_mod13_mod4 == 3); the called
function drains an uplink frame from the ring buffer, writes it into the DSP's
a_du_1 buffer, sets B_PLAY_UL in d_tch_mode and sends a TCH_ULBITS_CONF packet
back to the host.  If the ring buffer is empty, the function clears both
B_PLAY_UL and the firmware's tch_ul_play_mode flag, ending the special TCH UL
play mode.

This PoC mechanism is meant to be exercised with tch play command in an
interactive fc-shell session: this command reads an ASCII line-based uplink data
file and sends it to the firmware frame by frame, paced by TCH_ULBITS_CONF
packets from the target.  The input to this command is a line-based ASCII hex
file similar to the format written by tch record, but simplified: each line is
just the 33-byte frame to be sent (in TI DSP buffer format, following GSM 05.03
interface 1), without any flags or status words or frame numbers.