--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/RTP-BFI-extension	Mon Nov 21 12:17:55 2022 -0800
@@ -0,0 +1,159 @@
+We (Themyscira Wireless) have invented our own non-standard extension to the
+generally accepted standard for RTP-based transport of GSM FR and EFR traffic
+within a GSM RAN, on stretches running from a BTS to a TRAU-like component.
+
+The fundamental question is: when the radio subsystem of the BTS does not have
+any good traffic frame to send in a given 20 ms window, what should it do?  The
+generally accepted standard behavior is that no packet is sent, an intentional
+gap is created in the RTP stream (the next time an RTP packet does go out, the
+timestamp increments over the gap while the sequence number increments only by
+1, indicating an intentional gap rather than packet loss), and apparently the
+intent was/is that this gap in the RTP stream serves as the BFI (bad frame
+indication).
+
+The problem with this generally accepted gap-as-BFI approach is that it deprives
+the downstream transcoding MGW (a "soft TRAU" of sorts) of its timing source.
+If the TRAU-like entity on the receiving end of the RTP stream originating from
+the BTS were an RTP to TDM gateway, there would be no problem - such a gateway
+would have to buffer received RTP packets in order to synchronize to fixed TDM
+timing, and the absence of an RTP packet arriving in time would serve just fine
+as the BFI marker, signaling BFI condition to the Rx DTX handler.  But what if
+the G.711 interface on the 64 kbps side of the TRAU is also an RTP stream, this
+time going to a PSTN-via-SIP connectivity provider?  Now the TRAU-like component
+becomes a transcoding RTP forwarding MGW without any inherently fixed timing.
+
+If the desire is to implement a traditional TRAU in every way except for an
+RTP-based implementation instead of TDM-based, i.e., if the desire is to emit a
+fully continuous G.711 RTP stream from the MGW toward PSTN with comfort noise
+generation and in-band DTMF insertion happening inside the MGW, rather than
+emit gaps in the outgoing stream or punt CN generation (and DTMF) to VoIP
+network elements, this task becomes dramatically easier if the BTS can be
+forced to send an RTP packet in every 20 ms window, be it rain or shine,
+conveying either a good traffic frame or a BFI marker.
+
+Representing BFI markers in an RTP stream
+=========================================
+
+In the case of AMR codec, the existing standard RTP payload format already
+provides an obvious way to send a BFI marker: it is the NO_DATA frame type,
+i.e., FT=15 - see RFC 4867 section 4.3.2.  That same section also categorizes
+what we seek to do here as a "SHOULD NOT":
+
+   Note that packets containing only NO_DATA frames SHOULD NOT be
+   transmitted in any payload format configuration, [...]
+
+However, the just-quoted directive is a SHOULD NOT rather than a MUST NOT,
+and RFC 2119 states:
+
+   SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
+   there may exist valid reasons in particular circumstances when the
+   particular behavior is acceptable or even useful, but the full
+   implications should be understood and the case carefully weighed
+   before implementing any behavior described with this label.
+
+Our situation is just that: in our particular circumstance (desire to implement
+a traditional GSM TRAU in an RTP-to-RTP environment with no TDM network to act
+as a rigid timing governor) a valid reason exists why this "SHOULD NOT" behavior
+is not only acceptable, but becomes necessary.  Thus in the case of AMR, we are
+good - there is no need to invent our own totally non-standard extensions to
+RTP payload format, it just needs to be a configurable option in the IP-based
+BTS or in OsmoMGW converting from an E1-based BTS to RTP.
+
+But what about the older FR and EFR codecs?  In the case of existing standard
+RTP payload formats for FR and EFR, there is no defined way to represent a BFI
+condition as distinct from any possible good traffic frame, and there lies our
+challenge.
+
+Inventing an RTP BFI marker for FR and EFR
+==========================================
+
+The existing code in osmo-bts-trx (but not in the osmo-bts-sysmo version of
+interest to us) already contains a partial implementation of what we seek to do
+here: it runs its own ECU instance in the case of a BFI from the channel
+decoding layer, and if there is still no luck, there is code present to send a
+BFI packet.  The implemented behavior is not useful for us because RTP output
+is still fully suppressed when the uplink is expected to be in DTX, and there
+is a higher-level check in common/l1sap.c (l1sap_tch_ind() function) that also
+suppresses RTP output, but still, the point is that someone did already write
+code for sending an RTP packet intended to serve as a BFI.  In the case of AMR,
+that code sends out the expected NO_DATA (aka AMR_BAD) frame type - but what
+about FR and EFR?
+
+The existing code in osmo-bts-trx sends its FR codec BFI as a valid-looking FR
+frame with all 260 content bits set to 0, and it sends its EFR codec BFI as a
+valid-looking EFR frame with all 244 content bits set to 0.  I (Mother Mychaela)
+have given consideration to using this all-zeros in-band BFI representation as
+our RTP BFI marker for ThemWi, but then rejected this idea and decided to
+implement our own non-standard extension to RTP payload format instead,
+described further below.
+
+The fundamental philosophical problem which I (Mother Mychaela) have with this
+in-band BFI representation is that in the world of ETSI and 3GPP standards, BFI
+has always been meant to be out-of-band, not in-band.  In the TRAU frame format
+defined in GSM 08.60 there is an explicit control bit that carries BFI - the
+condition is NOT to be derived from the 260 or 244 traffic frame bits carried
+in data bit positions.  Abusing one particular bit pattern within the regular
+260-bit or 244-bit frame, even if it happens to be all zeros, goes against the
+spirit of classic GSM and 3GPP.  Per the specs, an FR codec frame of all zeros
+would be a SID frame with all LAR coefficients set to 0, and standards-compliant
+FR decoders would accept it as a valid SID frame, not as BFI.  The situation is
+likely to be even worse with EFR, where a frame of all zeros would not be
+treated as SID (EFR SID code word is 95 ones instead of 95 zeros) and would
+probably produce garbage at the decoder output.
+
+Themyscira Wireless implemented solution
+========================================
+
+We have invented our own non-standard extension to RTP payload format for GSM
+FR and EFR codecs.  Our extension is as follows: wherever a BTS needs to send a
+BFI marker in the place of a traffic frame, instead of sending a 33-byte payload
+beginning with 0xD nibble or a 31-byte payload beginning with 0xC nibble, it
+needs to send a 2-byte payload formatted as follows:
+
+byte 0: 0xBF signature;
+byte 1: least-significant bit encoding TAF per GSM 06.31 or GSM 06.81,
+        section 6.1.1 in both documents; other bits are reserved.
+
+In the uplink direction, with an RTP stream going from a BTS to our "soft TRAU"
+MGW, our themwi-mgw recognizes these BFI packets and acts accordingly, feeding
+BFI and TAF to the spec-prescribed Rx DTX handler for FR or EFR.  However, if a
+BTS receives these BFI marker packets in the downlink direction as a result of
+TrFO (the RTP stream comes from the uplink of another GSM call), it simply
+discards them without any processing - because a BTS always runs on its own TDMA
+timing, there is no difference between receiving a BFI packet vs receiving no
+RTP packet at all for that 20 ms frame.
+
+Our patch to osmo-bts
+=====================
+
+Our current BTS is a sysmoBTS, as opposed to a T1/E1-based BTS interfaced via
+OsmoBSC and OsmoMGW, hence the software component in need of patching is
+osmo-bts.  The patch in osmo-patches/osmo-bts-rtp-bfi.patch in the present
+repository applies to the common part of osmo-bts, hence it should theoretically
+work with all models, but it has only been tested with osmo-bts-sysmo.  If
+anyone wishes to play with this patch in an osmo-bts-trx environment, they will
+probably need to remove TRX-specific ECU and BFI code from
+osmo-bts-trx/sched_lchan_tchf.c, otherwise the two implementations will "fight":
+the TRX-specific code will kick in on "unexpected" bad frames and times of FACCH
+stealing, whereas the common code will fill in the new form of BFI during times
+of uplink DTX.
+
+Aside from the just-described incompatibility with osmo-bts-trx version, our
+current patch has the following outstanding issues which will need to be fixed
+before it can be proposed for merging:
+
+* Setting of TAF based on the frame number remains to be implemented - right
+  now we always send TAF=0.
+
+* The code will need to be extended to support AMR, and to skip out gracefully
+  in the case of totally unsupported codecs like HR1 - our current patch always
+  emits our FR/EFR BFI, implicitly assuming that the speech codec is either FR
+  or EFR, as that is how we currently run our network.
+
+* A configuration option will need to be implemented, to be controlled from
+  OsmoBSC, enabling or disabling the non-standard behavior of sending explicit
+  BFI packets instead of gaps in the RTP stream.  It would probably be best
+  implemented as a tristate option, with the choices being no BFI packets at
+  all (default), sending BFI packets only for AMR (where a standard encoding
+  exists) or sending BFI packets for AMR and also for FR & EFR, using
+  Themyscira encoding for the latter.