It has been discovered that the DSP ROM in the Calypso GSM baseband processor
implements one nifty feature which is not used at all in standard phone or modem
operation, but which can be used for all kinds of interesting hacks: the traffic
channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink
direction (to be fed to one of several speech decoders) can be read out of the
DSP's API RAM in real time, and in the uplink direction the user can feed her
own bits to the input of the GSM 05.03 channel encoder, effectively suppressing
the output of the internal vocoder.

The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel
modes, corresponding to the FR and EFR codecs; it is not currently known if it
can also be used in TCH/HS, data traffic or AMR modes.  I am aware though of an
anecdotal report that someone tried to make this feature work with AMR, but was
unsuccessful - hence we should be prepared for the possibility that the hack
is not possible with AMR.

In order to make use of this TCH bit access feature, one needs 3 things:

1) Firmware on the Calypso (the ARM part) that reads downlink bits from the DSP
   and writes uplink bits into it while doing everything else that a GSM fw must
   do in order to operate the MS;

2) Some protocol for passing these TCH bits into and out of the Calypso device;

3) A source for TCH UL bits and a sink for TCH DL bits on the external host.

In the case of FreeCalypso, we have defined our own protocol for passing TCH
bits into and out of Calypso GSM devices running one of our firmwares in the
form of an extension to TI's RVTMUX interface, i.e., we have defined a new
RVTMUX channel for this TCH interface and defined the packet types and formats
to be sent over the wire.  On the Calypso side this interface is implemented in
FreeCalypso GSM firmwares (currently only in Citrine, but we can also make a
special version of TCS211 with this feature added if the need arises), and on
the host tools side there is support in rvinterf and fc-shell.

The TCH bit access mechanism in FreeCalypso has been designed with an objective
of presenting to the user exactly what TI's DSP presents to us, i.e., standing
out of the way as much as possible.  TI's DSP presents TCH downlink bits and
accepts TCH uplink bits in the form of an array of 16-bit words; the bit order
within these words corresponds to the GSM 05.03 channel encoder bit order (with
a couple of TI-specific quirks documented below) and NOT that of the GSM 06.10
or EFR codecs.  On the RVTMUX serial interface between the Calypso device and
the external host we transfer each TCH frame as a block of 33 bytes; our Calypso
firmwares translate between these bytes and the DSP's 16-bit words, but do not
reorder or change the bits in any way.

On the host tools side our fc-shell utility provides user commands to save TCH
DL bits into a file and to play TCH UL bits from a file; in the present version
these files are written and read in an ASCII-based hex format.  In these ASCII
files each TCH frame is represented as a string of 66 hexadecimal digits, and
these hex digits correspond directly to the 33 bytes being read out of or
written into DSP API words.  Therefore, in order to generate and/or interpret
these hexadecimal strings correctly, you (the user) need to understand the bit
order and mapping used by TI's implementation of the GSM 05.03 channel encoder.

Recall from the GSM specs that the 260 bits which comprise one speech frame are
not all treated equally, instead they are divided into 182 class 1 bits which
are protected by a convolutional encoder and 78 class 2 bits which are
transmitted without any forward error correction.  Furthermore, the first 50 of
the class 1 bits are also protected by a CRC.  The order in which the bits
appear on TI's DSP interface corresponds to this division, known as the order of
subjective importance.

Now let's look at the actual bit order and mapping which you need to understand
in order to make sense of the hex strings in tch record and tch play files.
The bit numbering is from the most significant bit to the least significant bit,
i.e., in our string of 66 hex digits the most significant bit of the leftmost
digit is bit 0 and the least significant bit of the rightmost digit is bit 263.
TI's DSP assigns these bits as follows:

* Bits 0 through 181 correspond to the 182 protected (class 1) bits in the
  standard GSM 05.03 order;

* Bits 182 through 185 are unused - put zeros there if you are generating them
  yourself;

* Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the
  standard GSM 05.03 order.

Uplink testing
==============

The uplink sending mechanism can be exercised as follows:

1. Record a speech sample in the GSM 06.10 codec format using any Unix/Linux
   audio tool that can write the de facto standard libgsm format.  For example,
   using SoX:

   rec -c1 recording.gsm

   SoX will write the recording in the GSM 06.10 libgsm format based on the
   .gsm suffix at the end of the recording file name; the -c1 option is needed
   to disable stereo, otherwise the recording will be slowed down 2x.

2. Convert it from libgsm standard format into our ad hoc hex strings for
   playing into the TCH uplink:

   fc-fr2tch recording.gsm recording.tch-ul

3. In fc-shell, when you are in an established voice call, issue this command:

   tch play recording.tch-ul

You should now hear the speech sample you recorded in step 1 above on the other
end of the GSM call.  Please note, though, that for this example to work, the
call must be connected in the FR codec mode, not EFR.  If you try it on an EFR
call, the vocoder on the other end of the call will try to interpret your FR
codec bits per EFR, resulting in garbage.  In order to make this procedure work
properly with EFR, you would need to generate a speech sample in EFR format and
then put it in the correct hex string form with the correct bit order for
feeding to TI's GSM 05.03 channel encoder implementation, and at the present
moment the tools to do this feat are lacking.

The fc-fr2tch step above is new with the current version of FreeCalypso host
tools.  My original implementation of the tch play command in fc-shell read
libgsm GSM 06.10 speech files directly, but I changed it to read hex strings
which correspond literally to what goes into TI's implementation of the 05.03
channel encoder to make it more general:

1) to support playing EFR speech bits into TCH/EFS channels;
2) to support people who may want to pass totally non-speech data over traffic
   channels in TFO mode - see below.

Downlink testing
================

When you are in an established voice call in fc-shell, you can record the
downlink TCH bits as follows:

tch record <name of file to put the recording into>

If you would like to record an entire call from the beginning, issue the
tch record command as above before the ATD or ATA command that dials or answers
the call.  Either way, whether you are recording a call from the beginning or
from the middle, you need to eventually stop your recording with this command:

tch record stop

You can issue this stop command before or after the call is terminated, but
until you issue this tch record stop command, the output file is not closed and
thus may not be written to the file system.

The recording is written in an ASCII format that is similar but not identical
to what the tch play command takes as its input.  Here is an example of what
you might get from tch record:

C214 281D 0063 81C008000008200046000000000000000000000000000007056608060B0A010B09
C204 0540 003A 9391480D3051F4BD81DF5EB35069BEBC4AEDF756351C4C19689BB1CA4EA5D4F5F5
C204 05A2 0047 65B80F9E690F7C8CE4ED80DEF69DA6436518AB99ABCE5815E6B562C5CE4EAC5DC5
C204 04CA 0044 A4483744B04371ED0334ECB350AF28C639B7F095519EF0242D299B6405124F77A5
C214 2544 0066 83800400000C1000E80000000000000000000000000000300674A07070F080D0B2
C200 4E8A 0000 07071DF0F83B9521EE61CFF095AA8C0E560300F6A5573C31F3E00601ED4AAE7E2F
C200 4C94 0000 077385ADE20450B2E410961D6C5B0A173ACF9E2D38D77C28CED8495D88AA4DE72E
C200 4ABC 0000 077F0CF86004132DA5C0A6D5A4B0BD4B28159A07D8F4282DC6AAAB27503BC02701

The example above is quoted from an actual recording made from a call to WWV at
+1-303-499-7111, a time of day service.  This example shows 5 garbled frames
followed by 3 good ones.

In each line the first 3 space-separated 16-bit words are status words from the
DSP, and the rest is a hex string of 66 digits giving the 260-bit frame with 4
extra dummy bits at the boundary between the protected and unprotected portions.

Now here comes the unpleasant part: radio systems are inherently error-prone,
hence if you are naively expecting every received downlink frame to be a valid
speech frame that matches exactly what the other end transmitted, you will be
disappointed.  The downlink frame stream will always consist of both valid and
invalid frames, and in the standard processing chain inside TI's DSP the speech
decoder block that receives these frames from the channel decoder will look at
the status words written by the latter in order to decide what to do with each
frame.  But unfortunately I don't know the details, as the DSP is a mostly
undocumented black box.

Our FreeCalypso firmwares forward the downlink frame status words from the DSP
to the external host and my current implementation of the tch record command
writes them into the file because it is valuable information which should not
be discarded, but unfortunately I don't know their meaning.  The first status
word consists of bit flags, the last one seems to be some kind of error count
(what kind of errors? I don't know), and the middle one is totally unknown.

There is also an fc-tch2fr utility that does sort of an inverse of fc-fr2tch,
except that fc-tch2fr expects files written by tch record which contain the
status words at the beginning - the files written by fc-fr2tch and read by
tch play don't have these status words; they just have 66 hex digits per line
giving the bits to be fed to the GSM 05.03 block in the DSP.  The fc-tch2fr
utility works by disregarding the status words; the result will be a .gsm file
which you can play with SoX etc, but it will produce horrible sounds wherever
some garbled frames were received on the call downlink.

Passing non-speech data over TCH
================================

If you are incredibly lucky and you happen to live in a part of the world where
the local GSM network implements TFO (tandem-free operation, see GSM 02.53),
i.e., if your GSM network transparently passes codec frames from one end of the
call to the other without transcoding back and forth in the middle, you might
be able to send arbitrary non-speech data bits over this TCH connection.  If
you are going to attempt such a feat, you are mostly on your own as the GSM
network in my part of the world does not support TFO, but here are some general
ideas:

* Heed the difference between the 182 protected bits and the 78 unprotected
  bits of each 260-bit frame.  The bits in the first class are less likely to
  be corrupted by radio errors.  Also the first 50 out of these class 1 bits
  are protected by a CRC, so if they get corrupted, there will be a flag bit
  somewhere in the DSP status words - but I don't know which bit it is...

* Given the lack of detailed information about the exact meaning of the DSP
  status words, implement some error checking of your own, e.g., an in-band
  CRC within your frame payload so you can check on the receiving end if you
  got an uncorrupted frame from your data source or not.

* Expect to lose some frames - even if your radio environment is pristine and
  error-free, some frames will still be lost because their timeslots had to be
  stolen by FACCH on either end of the call.  You will need to implement
  acknowledgements, retransmissions and retries - the usual stuff.

EFR differences
===============

TCH/FS and TCH/EFS channel modes corresponding to the FR and EFR codecs,
respectively, share the same channel encoder that takes 260-bit frames as input.
An EFR speech frame is only 244 bits, but as the GSM 05.03 spec explains, each
EFR frame gets expanded to 260 bits by adding some CRC and repetition bits.
One would thus expect that the behaviour of the channel encoder block should be
strictly identical between TCH/FS and TCH/EFS channel modes, but it appears that
TI's implementation exhibits one difference in behaviour between the two modes.

It appears that TI's implementation of the GSM 05.03 channel encoder computes
the EFR-specific CRC bits internally, such that bits [65,72] of each uplink
frame fed to the DSP in the TCH/EFS mode need to be 0 and will be replaced by
the DSP with CRC bits computed per the standard.  It also appears that their
implementation of the channel decoder verifies this CRC, sets a status bit
indicating whether the check succeeded or failed, and then zeroes these bits
out, i.e., the original bits in these CRC bit positions are lost.  No problem
if what you are passing over TCH/EFS is indeed EFR speech frames, but it will
be a problem if you are passing non-speech data over your TCH and put something
else in those bits which the spec allocates to CRC.