It has been discovered that the DSP ROM in the Calypso GSM baseband processor
implements one nifty feature which is not used at all in standard phone or modem
operation, but which can be used for all kinds of interesting hacks: the traffic
channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink
direction (to be fed to the channel mode-appropriate speech decoder) can be read
out of the DSP's API RAM in real time, and in the uplink direction the user can
feed her own bits to the input of the GSM 05.03 channel encoder, effectively
suppressing the output of the internal vocoder.

The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel
modes, corresponding to FR1 and EFR codecs; it also appears to work for TCH/HS
(HR1 codec), but we (FreeCalypso) haven't tested it because almost no one uses
that infamous HR1 codec - the commercial GSM network in our part of the world
gives you a full-rate channel if your phone does not support AMR.  It would be
possible to implement HR1 in our own test GSM network, but the effort that would
be required is difficult to justify.  Exploring TCH tap modes with AMR or CSD
traffic channels is likewise a subject for further study.

In order to make use of this TCH bit access feature, one needs 3 things:

1) Firmware on the Calypso (the ARM part) that reads downlink bits from the DSP
   and writes uplink bits into it while doing everything else that a GSM fw must
   do in order to operate the MS;

2) Some protocol for passing these TCH bits into and out of the Calypso device;

3) A source for TCH UL bits and a sink for TCH DL bits on the external host.

In the case of FreeCalypso, we have defined our own protocol for passing TCH
bits into and out of Calypso GSM devices running one of our firmwares in the
form of an extension to TI's RVTMUX interface, i.e., we have defined a new
RVTMUX channel for this TCH interface and defined the packet types and formats
to be sent over the wire.  On the Calypso side the special functionality in
question was originally implemented in FC Citrine firmware in 2016 and then set
aside for some years; when the right time came to resurrect this feature in late
2022, it turned out that the original implementation from 2016 was slightly
incorrect, and the new implementation in FC Tourmaline fw is slightly different.
On the host tools side the RVTMUX-based TCH interface is supported in rvinterf
and fc-shell; the new version as of fc-host-tools-r18 supports both 2016 and
2022 versions of this over-the-wire interface.

The TCH bit access mechanism in FreeCalypso has been designed with an objective
of presenting to the user exactly what TI's DSP presents to us, i.e., standing
out of the way as much as possible.  TI's DSP presents TCH downlink bits and
accepts TCH uplink bits in the form of an array of 16-bit words; the bit order
within these words corresponds to the GSM 05.03 channel encoder bit order (with
a couple of TI-specific quirks documented below) and NOT that of the GSM 06.10
or EFR codecs.  On the RVTMUX serial interface between the Calypso device and
the external host we transfer each TCH frame as a block of 33 bytes; our Calypso
firmwares translate between these bytes and the DSP's 16-bit words, but do not
reorder or change the bits in any way.

On the host tools side our fc-shell utility provides user commands to save TCH
DL bits into a file and to play TCH UL bits from a file; in the present version
these files are written and read in an ASCII-based hex format.  In these ASCII
files each TCH frame is represented as a string of 66 hexadecimal digits, and
these hex digits correspond directly to the 33 bytes being read out of or
written into DSP API words.  Therefore, in order to generate and/or interpret
these hexadecimal strings correctly, you (the user) need to understand the bit
order and mapping used by TI's implementation of the GSM 05.03 channel encoder.

As of late 2022, there is a new TCH-tap-modes article in our freecalypso-docs
repository that covers in detail the format of TI's DSP buffers for TCH DL and
UL bits, as well as all known information about TCH DL status words and bit
flags.  But here is our original description from 2016:

Recall from the GSM specs that the 260 bits which comprise one speech frame are
not all treated equally, instead they are divided into 182 class 1 bits which
are protected by a convolutional encoder and 78 class 2 bits which are
transmitted without any forward error correction.  Furthermore, the first 50 of
the class 1 bits are also protected by a CRC.  The order in which the bits
appear on TI's DSP interface corresponds to this division, known as the order of
subjective importance.

Now let's look at the actual bit order and mapping which you need to understand
in order to make sense of the hex strings in tch record and tch play files.
The bit numbering is from the most significant bit to the least significant bit,
i.e., in our string of 66 hex digits the most significant bit of the leftmost
digit is bit 0 and the least significant bit of the rightmost digit is bit 263.
TI's DSP assigns these bits as follows:

* Bits 0 through 181 correspond to the 182 protected (class 1) bits in the
  standard GSM 05.03 order;

* Bits 182 through 185 are unused - put zeros there if you are generating them
  yourself;

* Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the
  standard GSM 05.03 order.

TCH DL recording
================

When you are in an established voice call in fc-shell, you can record the
downlink TCH bits as follows:

tch record <name of file to put the recording into>

If you would like to record an entire call from the beginning, issue the
tch record command as above before the ATD or ATA command that dials or answers
the call.  Either way, whether you are recording a call from the beginning or
from the middle, you need to eventually stop your recording with this command:

tch record stop

You can issue this stop command before or after the call is terminated, but
until you issue this tch record stop command, the output file is not closed and
thus may not be written to the file system.

The recording is written in an ASCII line-based format with one line for every
received TCH DL frame, but the exact format of each written line will depend on
which firmware version is in use.  If you are running ancient Citrine firmware
that emits its TCH DL output in the old format from 2016 (now known to be
incomplete and thus unusable for proper decoding), fc-shell will likewise write
its ASCII output in the old format, which won't be covered further as it is
deprecated and not practically useful.  However, if you are running current
FreeCalypso firmware with the resurrected (late 2022) version of the TCH tap
feature, each TCH DL frame will be sent by the fw and received by fc-shell in
the new over-the-wire format, and fc-shell will write the recording file in the
new ASCII format documented in the TCH-tap-modes article in freecalypso-docs.

Once you have captured a TCH DL recording, what can you do with it?  If the
recording came from an FR1 call, you will need to pass it through an Rx DTX
handler for FR1 (see GSM 06.11, 06.12 and 06.31 specs) before you can pass it
to a naive GSM 06.10 decoder such as classic Unix libgsm, and if the recording
came from an EFR call, you will need to pass it to a proper EFR (not AMR!)
decoder that includes the necessary EFR Rx DTX handler.  Neither of the two
just-mentioned library pieces (neither the Rx DTX handler for FR1 nor a proper,
not-same-as-AMR implementation of GSM EFR) could be found among the existing
body of FOSS as of 2022, thus we (FreeCalypso and Themyscira Wireless)
implemented our own.  Please look for our GSM codec libraries & utilities
package, which is expected to reach its first official release some time in
early 2023.

Inside our gsm-codec-lib package you will find gsmfr-dlcap-* and gsmefr-dlcap-*
utilities that read TCH downlink capture files written by fc-shell tch record
and perform various decoding operations - please refer to further documentation
within that package.

Please don't use the old fc-tch2fr utility - the function it performs is now
known to be a bogo-transform, and it can't grok the new TCH DL recording format
which you will get with current FreeCalypso fw.

TCH UL play
===========

The uplink sending mechanism can be exercised as follows:

1. If you are going to be in an FR1 call, prepare a speech sample in the GSM
   06.10 codec format using any Unix/Linux audio tool that can write the de
   facto standard libgsm format.  For example, using SoX:

   rec -c1 recording.gsm

   SoX will write the recording in the GSM 06.10 libgsm format based on the
   .gsm suffix at the end of the recording file name; the -c1 option is needed
   to disable stereo, otherwise the recording will be slowed down 2x.
   Alternatively, you can use our new gsmfr-encode utility (gsm-codec-lib
   package) to encode from WAV into GSM 06.10, or gsmfr-encode-r for raw BE
   input instead of WAV.

   OTOH, if you are going to be in an EFR call rather than FR1, you will need to
   prepare a speech sample in the EFR codec format instead.  You will need to
   use Themyscira gsmefr-encode or gsmefr-encode-r utilities, or convert from
   AMR (MR122 mode only, no DTX) with our gsm-amr2efr utility.

2. Convert your speech sample from libgsm standard format (FR1) or Themyscira
   gsmx format (EFR) into our ad hoc hex strings for playing into a TCH uplink:

   fc-fr2tch recording.gsm recording.tch-ul

   or

   fc-efr2tch recording.gsmx recording.tch-ul

3. In fc-shell, when you are in an established voice call, issue this command:

   tch play recording.tch-ul

You should now hear the speech sample you recorded in step 1 above on the other
end of the GSM call.  Needless to say, the TCH mode of the call (TCH/FS or
TCH/EFS) needs to match the codec in which your to-be-played recording was
prepared, otherwise the other end of the call will receive garbage!

Controlling the selection of speech codec for calls
===================================================

One very obvious shortcoming of the present facilities for voice TCH redirection
is that we only support FR1 and EFR codecs, but not AMR.  However, most GSM
networks prefer to use AMR when the MS supports it - and in regular operation
with a speaker & mic user connection (as opposed to TCH tap modes), our current
FreeCalypso firmwares do support AMR when running on Calypso C035 silicon with
DSP ROM version 3606.  (DSP ROM version 3416 together with the respective patch
version also appears to have working AMR, at least in light testing, although
of course we do NOT recommend it for production use.)  Therefore, if you wish
to play with EFR, you need to tell the network (via the Bearer Capabilities
information element in CC messages) that your MS does not support AMR, and if
you wish to play with FR1, you need to tell the network that your MS only
supports FR1 and no others.

The outstanding issue here is that we need to implement some mechanism in our
FreeCalypso firmwares, probably a custom AT command, that will modify the
Bearer Capabilities IE (artificially restrict the set of supported codecs) on a
per-call basis.  Until we implement the necessary support, the only current way
to create such codec-restricted operation is by writing a /pcm/MSCAP file into
FFS with the desired bit mask of supported codecs - but this method is hugely
inconvenient because this file is read only on fw boot, thus every modification
requires a full reboot cycle.