--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/TCH-bit-access	Tue Jun 14 07:13:46 2016 +0000
@@ -0,0 +1,238 @@
+It has been discovered that the DSP ROM in the Calypso GSM baseband processor
+implements one nifty feature which is not used at all in standard phone or modem
+operation, but which can be used for all kinds of interesting hacks: the traffic
+channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink
+direction (to be fed to one of several speech decoders) can be read out of the
+DSP's API RAM in real time, and in the uplink direction the user can feed her
+own bits to the input of the GSM 05.03 channel encoder, effectively suppressing
+the output of the internal vocoder.
+
+The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel
+modes, corresponding to the FR and EFR codecs; it is not currently known if it
+can also be used in TCH/HS, data traffic or AMR modes.  I am aware though of an
+anecdotal report that someone tried to make this feature work with AMR, but was
+unsuccessful - hence we should be prepared for the possibility that the hack
+is not possible with AMR.
+
+In order to make use of this TCH bit access feature, one needs 3 things:
+
+1) Firmware on the Calypso (the ARM part) that reads downlink bits from the DSP
+   and writes uplink bits into it while doing everything else that a GSM fw must
+   do in order to operate the MS;
+
+2) Some protocol for passing these TCH bits into and out of the Calypso device;
+
+3) A source for TCH UL bits and a sink for TCH DL bits on the external host.
+
+In the case of FreeCalypso, we have defined our own protocol for passing TCH
+bits into and out of Calypso GSM devices running one of our firmwares in the
+form of an extension to TI's RVTMUX interface, i.e., we have defined a new
+RVTMUX channel for this TCH interface and defined the packet types and formats
+to be sent over the wire.  On the Calypso side this interface is implemented in
+FreeCalypso GSM firmwares (currently only in Citrine, but we can also make a
+special version of TCS211 with this feature added if the need arises), and on
+the host tools side there is support in rvinterf and fc-shell.
+
+The TCH bit access mechanism in FreeCalypso has been designed with an objective
+of presenting to the user exactly what TI's DSP presents to us, i.e., standing
+out of the way as much as possible.  TI's DSP presents TCH downlink bits and
+accepts TCH uplink bits in the form of an array of 16-bit words; the bit order
+within these words corresponds to the GSM 05.03 channel encoder bit order (with
+a couple of TI-specific quirks documented below) and NOT that of the GSM 06.10
+or EFR codecs.  On the RVTMUX serial interface between the Calypso device and
+the external host we transfer each TCH frame as a block of 33 bytes; our Calypso
+firmwares translate between these bytes and the DSP's 16-bit words, but do not
+reorder or change the bits in any way.
+
+On the host tools side our fc-shell utility provides user commands to save TCH
+DL bits into a file and to play TCH UL bits from a file; in the present version
+these files are written and read in an ASCII-based hex format.  In these ASCII
+files each TCH frame is represented as a string of 66 hexadecimal digits, and
+these hex digits correspond directly to the 33 bytes being read out of or
+written into DSP API words.  Therefore, in order to generate and/or interpret
+these hexadecimal strings correctly, you (the user) need to understand the bit
+order and mapping used by TI's implementation of the GSM 05.03 channel encoder.
+
+Recall from the GSM specs that the 260 bits which comprise one speech frame are
+not all treated equally, instead they are divided into 182 class 1 bits which
+are protected by a convolutional encoder and 78 class 2 bits which are
+transmitted without any forward error correction.  Furthermore, the first 50 of
+the class 1 bits are also protected by a CRC.  The order in which the bits
+appear on TI's DSP interface corresponds to this division, known as the order of
+subjective importance.
+
+Now let's look at the actual bit order and mapping which you need to understand
+in order to make sense of the hex strings in tch record and tch play files.
+The bit numbering is from the most significant bit to the least significant bit,
+i.e., in our string of 66 hex digits the most significant bit of the leftmost
+digit is bit 0 and the least significant bit of the rightmost digit is bit 263.
+TI's DSP assigns these bits as follows:
+
+* Bits 0 through 181 correspond to the 182 protected (class 1) bits in the
+  standard GSM 05.03 order;
+
+* Bits 182 through 185 are unused - put zeros there if you are generating them
+  yourself;
+
+* Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the
+  standard GSM 05.03 order.
+
+Uplink testing
+==============
+
+The uplink sending mechanism can be exercised as follows:
+
+1. Record a speech sample in the GSM 06.10 codec format using any Unix/Linux
+   audio tool that can write the de facto standard libgsm format.  For example,
+   using SoX:
+
+   rec -c1 recording.gsm
+
+   SoX will write the recording in the GSM 06.10 libgsm format based on the
+   .gsm suffix at the end of the recording file name; the -c1 option is needed
+   to disable stereo, otherwise the recording will be slowed down 2x.
+
+2. Convert it from libgsm standard format into our ad hoc hex strings for
+   playing into the TCH uplink:
+
+   fc-fr2tch recording.gsm recording.tch-ul
+
+3. In fc-shell, when you are in an established voice call, issue this command:
+
+   tch play recording.tch-ul
+
+You should now hear the speech sample you recorded in step 1 above on the other
+end of the GSM call.  Please note, though, that for this example to work, the
+call must be connected in the FR codec mode, not EFR.  If you try it on an EFR
+call, the vocoder on the other end of the call will try to interpret your FR
+codec bits per EFR, resulting in garbage.  In order to make this procedure work
+properly with EFR, you would need to generate a speech sample in EFR format and
+then put it in the correct hex string form with the correct bit order for
+feeding to TI's GSM 05.03 channel encoder implementation, and at the present
+moment the tools to do this feat are lacking.
+
+The fc-fr2tch step above is new with the current version of FreeCalypso host
+tools.  My original implementation of the tch play command in fc-shell read
+libgsm GSM 06.10 speech files directly, but I changed it to read hex strings
+which correspond literally to what goes into TI's implementation of the 05.03
+channel encoder to make it more general:
+
+1) to support playing EFR speech bits into TCH/EFS channels;
+2) to support people who may want to pass totally non-speech data over traffic
+   channels in TFO mode - see below.
+
+Downlink testing
+================
+
+When you are in an established voice call in fc-shell, you can record the
+downlink TCH bits as follows:
+
+tch record <name of file to put the recording into>
+
+If you would like to record an entire call from the beginning, issue the
+tch record command as above before the ATD or ATA command that dials or answers
+the call.  Either way, whether you are recording a call from the beginning or
+from the middle, you need to eventually stop your recording with this command:
+
+tch record stop
+
+You can issue this stop command before or after the call is terminated, but
+until you issue this tch record stop command, the output file is not closed and
+thus may not be written to the file system.
+
+The recording is written in an ASCII format that is similar but not identical
+to what the tch play command takes as its input.  Here is an example of what
+you might get from tch record:
+
+C214 281D 0063 81C008000008200046000000000000000000000000000007056608060B0A010B09
+C204 0540 003A 9391480D3051F4BD81DF5EB35069BEBC4AEDF756351C4C19689BB1CA4EA5D4F5F5
+C204 05A2 0047 65B80F9E690F7C8CE4ED80DEF69DA6436518AB99ABCE5815E6B562C5CE4EAC5DC5
+C204 04CA 0044 A4483744B04371ED0334ECB350AF28C639B7F095519EF0242D299B6405124F77A5
+C214 2544 0066 83800400000C1000E80000000000000000000000000000300674A07070F080D0B2
+C200 4E8A 0000 07071DF0F83B9521EE61CFF095AA8C0E560300F6A5573C31F3E00601ED4AAE7E2F
+C200 4C94 0000 077385ADE20450B2E410961D6C5B0A173ACF9E2D38D77C28CED8495D88AA4DE72E
+C200 4ABC 0000 077F0CF86004132DA5C0A6D5A4B0BD4B28159A07D8F4282DC6AAAB27503BC02701
+
+The example above is quoted from an actual recording made from a call to WWV at
++1-303-499-7111, a time of day service.  This example shows 5 garbled frames
+followed by 3 good ones.
+
+In each line the first 3 space-separated 16-bit words are status words from the
+DSP, and the rest is a hex string of 66 digits giving the 260-bit frame with 4
+extra dummy bits at the boundary between the protected and unprotected portions.
+
+Now here comes the unpleasant part: radio systems are inherently error-prone,
+hence if you are naively expecting every received downlink frame to be a valid
+speech frame that matches exactly what the other end transmitted, you will be
+disappointed.  The downlink frame stream will always consist of both valid and
+invalid frames, and in the standard processing chain inside TI's DSP the speech
+decoder block that receives these frames from the channel decoder will look at
+the status words written by the latter in order to decide what to do with each
+frame.  But unfortunately I don't know the details, as the DSP is a mostly
+undocumented black box.
+
+Our FreeCalypso firmwares forward the downlink frame status words from the DSP
+to the external host and my current implementation of the tch record command
+writes them into the file because it is valuable information which should not
+be discarded, but unfortunately I don't know their meaning.  The first status
+word consists of bit flags, the last one seems to be some kind of error count
+(what kind of errors? I don't know), and the middle one is totally unknown.
+
+There is also an fc-tch2fr utility that does sort of an inverse of fc-fr2tch,
+except that fc-tch2fr expects files written by tch record which contain the
+status words at the beginning - the files written by fc-fr2tch and read by
+tch play don't have these status words; they just have 66 hex digits per line
+giving the bits to be fed to the GSM 05.03 block in the DSP.  The fc-tch2fr
+utility works by disregarding the status words; the result will be a .gsm file
+which you can play with SoX etc, but it will produce horrible sounds wherever
+some garbled frames were received on the call downlink.
+
+Passing non-speech data over TCH
+================================
+
+If you are incredibly lucky and you happen to live in a part of the world where
+the local GSM network implements TFO (tandem-free operation, see GSM 02.53),
+i.e., if your GSM network transparently passes codec frames from one end of the
+call to the other without transcoding back and forth in the middle, you might
+be able to send arbitrary non-speech data bits over this TCH connection.  If
+you are going to attempt such a feat, you are mostly on your own as the GSM
+network in my part of the world does not support TFO, but here are some general
+ideas:
+
+* Heed the difference between the 182 protected bits and the 78 unprotected
+  bits of each 260-bit frame.  The bits in the first class are less likely to
+  be corrupted by radio errors.  Also the first 50 out of these class 1 bits
+  are protected by a CRC, so if they get corrupted, there will be a flag bit
+  somewhere in the DSP status words - but I don't know which bit it is...
+
+* Given the lack of detailed information about the exact meaning of the DSP
+  status words, implement some error checking of your own, e.g., an in-band
+  CRC within your frame payload so you can check on the receiving end if you
+  got an uncorrupted frame from your data source or not.
+
+* Expect to lose some frames - even if your radio environment is pristine and
+  error-free, some frames will still be lost because their timeslots had to be
+  stolen by FACCH on either end of the call.  You will need to implement
+  acknowledgements, retransmissions and retries - the usual stuff.
+
+EFR differences
+===============
+
+TCH/FS and TCH/EFS channel modes corresponding to the FR and EFR codecs,
+respectively, share the same channel encoder that takes 260-bit frames as input.
+An EFR speech frame is only 244 bits, but as the GSM 05.03 spec explains, each
+EFR frame gets expanded to 260 bits by adding some CRC and repetition bits.
+One would thus expect that the behaviour of the channel encoder block should be
+strictly identical between TCH/FS and TCH/EFS channel modes, but it appears that
+TI's implementation exhibits one difference in behaviour between the two modes.
+
+It appears that TI's implementation of the GSM 05.03 channel encoder computes
+the EFR-specific CRC bits internally, such that bits [65,72] of each uplink
+frame fed to the DSP in the TCH/EFS mode need to be 0 and will be replaced by
+the DSP with CRC bits computed per the standard.  It also appears that their
+implementation of the channel decoder verifies this CRC, sets a status bit
+indicating whether the check succeeded or failed, and then zeroes these bits
+out, i.e., the original bits in these CRC bit positions are lost.  No problem
+if what you are passing over TCH/EFS is indeed EFR speech frames, but it will
+be a problem if you are passing non-speech data over your TCH and put something
+else in those bits which the spec allocates to CRC.