--- a/doc/RTP-BFI-extension	Thu Apr 06 20:51:30 2023 -0800
+++ b/doc/RTP-BFI-extension	Thu Apr 06 21:30:33 2023 -0800
@@ -1,3 +1,12 @@
+NOTE: This document describes the old BFI representation format that was used
+by Themyscira Wireless for a few months from late 2022 into early 2023.  We no
+longer use this format, instead we have a new, bolder extended payload format
+for FR & EFR codecs, defined and described in the new RTP-TRAUlike-format
+document.
+
+Original document follows
+=========================
+
 We (Themyscira Wireless) have invented our own non-standard extension to the
 generally accepted standard for RTP-based transport of GSM FR and EFR traffic
 within a GSM RAN, on stretches running from a BTS to a TRAU-like component.

--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/RTP-TRAUlike-format	Thu Apr 06 21:30:33 2023 -0800
@@ -0,0 +1,417 @@
+TRAU-UL-like RTP transport format for FR & EFR codecs
+=====================================================
+
+The generally accepted industry standard format for RTP transport of FR and EFR
+codec frames in an IP-based GSM RAN is given in ETSI TS 101 318; the same format
+is also codified in IETF RFC 3551.  However, when compared to the classic
+TRAU-UL format of 3GPP TS 48.060, the standard RTP format of RFC 3551 exhibits
+the following two shortcomings:
+
+1) no way to indicate a BFI condition and still send frame data bits;
+2) no way to transport the Time Alignment Flag (TAF).
+
+Both of these shortcomings will be explained in detail further in this document;
+however, the primary purpose of this document is to propose a new, regrettably
+non-standard, RTP transport format for FR & EFR codecs, for use only within a
+GSM RAN and the immediately attached CN transcoder ("soft TRAU"), that provides
+the same functionality as the classic TRAU-UL format of TS 48.060, but is
+carried over RTP in IP rather than a 16 kbps TDM subchannel.
+
+The non-standard RTP transport format presented in this document is implemented
+in OsmoBTS on a private feature branch:
+
+https://cgit.osmocom.org/osmo-bts/log/?h=falconia/rtp_traulike
+
+OsmoBTS versions that include this code always accept TRAUlike FR/EFR packets
+on their RTP input, following the principle of being liberal in what you accept
+while being conservative in what you send, but emit such packets on their RTP
+output only when this non-default vty config option is given:
+
+rtp fr-efr-traulike
+
+The recently added (mainline) "rtp continuous-streaming" vty config option also
+needs to be enabled.
+
+The present document serves as the formal specification for the TRAUlike RTP
+transport format for FR and EFR.
+
+Detailed description of shortcomings of standard RTP transport for FR & EFR
+===========================================================================
+
+These shortcomings are solved in the TRAUlike RTP transport format defined in
+this document; understanding these shortcomings provides the essential rationale
+for TRAU-like RTP.
+
+Indicating BFI along with data bits
+-----------------------------------
+
+The only way to indicate a BFI condition in standard RTP (for FR/EFR) is to
+either send no packet at all in the 20 ms window in question (industry standard
+behavior and OsmoBTS default) or send an RTP packet with a zero-length payload
+("rtp continuous-streaming" option in OsmoBTS).  The latter option provides a
+timing tick for a CN-attached transcoder relying on the BTS-originating RTP
+stream as its timing source, but there is still no way to send a frame of
+marked-erroneous data bits.  Contrast with TS 48.060 TRAU-UL format: in this
+format the Dn bits carrying FR or EFR frame bits and the C12 bit carrying BFI
+are orthogonal.
+
+Why would one care about known-bad or deemed-to-be-bad frame data bits?  They
+do matter at least in the case of EFR: the official reference C-source EFR
+decoder from ETSI makes use of the "fixed codebook excitation pulses" portion
+of its EFR frame bits input (140 bits out of 244) even when BFI=1.  This
+portion of reference C-source behavior is declared to be a non-normative example
+by the text of GSM 06.61 spec, thus there may be other compliant EFR decoder
+implementations that never look at marked-erroneous data bits - but given the
+ease of simply using the C code from ETSI as-is, or recoding it more efficiently
+but keeping unchanged all bit-exact algorithms, including non-normative ones,
+we should expect that the behavior of ETSI reference code is retained in many
+production implementations and deployments.
+
+Consider the case where a traditional E1-based BTS with a classic TRAU interface
+is attached to an IP-based Osmocom RAN by way of OsmoMGW, and the resulting RTP
+stream then (after passing through another OsmoMGW instance at the MSC) goes to
+a "soft TRAU" transcoder (TC) in the CN.  The TC will feed its RTP input to FR
+and EFR decoders, and at least the EFR decoder makes use of "fixed codebook
+excitation pulses" bits from erroneous frames.  Furthermore, the TC may
+implement in-band TFO (3GPP TS 28.062) inside its G.711 RTP output, in which
+case it will need to insert a slightly modified TRAU-UL frame into that output.
+The bits that would ideally be fed to the ETSI EFR decoder and emitted to the
+outside world in TFO frames already exist at the output of the E1-based BTS,
+but they get lost in the RTP transport when the industry standard RTP payload
+format is used.
+
+Consider another case where OsmoBTS does have an FR/EFR traffic frame that
+could potentially be sent out, but it is suppressed by the
+(tch_ind->lqual_cb >= bts->min_qual_norm) check in l1sap_tch_ind() in
+src/common/l1sap.c.  In this case it would be ideal to send out that frame
+along with a BFI=1 indication, if the RTP transport format were to allow such
+representation.
+
+Lack of TAF bit in standard RTP transport
+-----------------------------------------
+
+The TRAU-UL frame format of TS 48.060 for FR and EFR includes a bit called TAF,
+for Time Alignment Flag.  Per the specs (TS 48.060 refers to TS 46.031 for
+definition and coding of frame indicators) this bit shall be set to 1 in one
+particular position in the 480 ms SACCH multiframe (the particular 20 ms frame
+position in which a valid frame is always transmitted, even during DTX pauses)
+and set to 0 in all other frames.  This flag factors into the Rx DTX handler
+logic prescribed in GSM 06.31 and 06.81 specs for FR and EFR, respectively, and
+there exist production decoders for these codecs that implement their Rx DTX
+handler function exactly to the letter of the specs, including the use of TAF
+bit when deciding what to do with a BFI=1 frame received in the comfort noise
+generation state.  (These spec-compliant decoders include the reference ETSI
+C-source decoder for EFR and Themyscira libgsmfrp for FR.)
+
+This TAF bit does not exist in the standard RTP transport for FR & EFR.  The
+lack of this TAF bit causes the following problems for the CN-attached "soft
+TRAU" transcoder:
+
+1) The ability to implement spec-compliant handling of GSM 06.11 or 06.61
+   section 5.4 requirement (same section in both specs) is lost;
+
+2) The TC won't know when to set the TAF bit in its outgoing TFO frames, if it
+   implements in-band TFO per 3GPP TS 28.062.
+
+The TFO problem is particularly concerning because these TFO frames are emitted
+to the outside world, outside of administrative and technical control of the
+party implementing the Osmocom-based GSM network and the TC at its edge.  The
+resulting G.711 octet stream with TFO frames embedded inside can be carried
+half-way around the world by the international toll telephone network, and there
+is no telling what kind of implementation may be receiving and decoding these
+bits on the other end.  For this reason, "poor man's" workarounds in the
+RTP-fed, TFO-generating TC are very unattractive:
+
+* If the TC were to set TAF=0 in all TFO frames it generates, the receiver's
+  expectation of seeing TAF=1 in every 24th frame will be violated.
+
+* If the TC were to arbitrarily set TAF=1 in every 24th frame by its own free-
+  running count, without knowledge of the actual SACCH alignment in the original
+  GSM call leg, these TAF-marked frames won't coincide with those frame
+  positions where the MS sends its SID frames, and the resulting TFO frame
+  stream will be invalid to the receiving Rx DTX handler on the far end.
+
+The knowledge of which frames need to be marked with TAF=1 exists inside the
+entity that generates the FR/EFR RTP stream: if this entity is a converter from
+E1-based Abis to RTP, the TRAU-UL frames from the BTS contain this TAF bit, and
+if the RTP-generating entity is a native IP BTS, it knows the frame number for
+which it generates each RTP packet.  The only problem is that there is no place
+to insert this TAF bit in the standard RTP transport format of TS 101 318.
+
+Why TRAU-UL and not TRAU-DL
+===========================
+
+The present document argues the case that the industry standard RTP transport
+format for FR & EFR is functionally crippled compared to the TRAU-UL transport
+format of 3GPP TS 48.060, and defines an alternative RTP transport format that
+can be used by those who desire TRAU-UL-like functionality badly enough to
+accept the price of going totally non-standard in their IP RAN transport.  The
+new RTP transport format defined in this document explicitly mimics the
+functionality and semantics of TS 48.060 TRAU-UL for FR and EFR.
+
+At this point a reader may reasonably ask: why TRAU-UL and not TRAU-DL?  The
+answer is TFO: 3GPP TS 28.062 and its predecessor GSM 08.62 define the TFO frame
+format as being based on TRAU-UL frames with only a few bits changed, and no
+change in semantics of any of the frame indicator bits of TRAU-UL (C12 through
+C17).  Whereas the Abis interface is inherently asymmetric (TRAU-UL frames in
+one direction, TRAU-DL frames in the other direction), end-to-end TFO is
+directionally symmetric.  If we imagine a TFO call between Alice in America and
+Bob in Britain, there will be TRAU-UL frames flowing in both directions of the
+trans-oceanic G.711 toll connection, one set coming almost unchanged from
+Alice's BTS CCU and the other coming almost unchanged from Bob's BTS CCU.  Of
+course each party's GSM call DL will require TRAU-DL frames to be fed to it,
+not TRAU-UL, but the necessary UL-to-DL conversion is the responsibility of the
+TFO receiver on each end.
+
+The general rules for turning a TRAU-UL frame into one for TRAU-DL are specified
+in TS 28.062 section C.3.2.1.1; it should be noted that this section spells out
+the requirements of what the UL-to-DL converter must do, but does not specify
+exactly how to do it algorithmically - the wording it uses is "subject to
+manufacturer dependent future improvements and is not part of this
+recommendation."  Implementing all of these section C.3.2.1.1 rules (hereafter
+called C3211 rules for short) exactly to the letter is quite easy for the FR
+codec (Themyscira libgsmfrp does everything that is needed, and is a simple and
+lightweight FLOSS function library), but much harder for EFR.  At the present
+time it is unclear to the author of this document whether real historical T1/E1
+TRAU implementations for which GSM 08.62 TFO was originally specified really did
+implement C3211 rules to the letter, particularly for EFR, or if they cut some
+corners.
+
+Because the TRAUlike RTP transport format defined in this document is
+semantically equivalent to TRAU-UL, any entity that receives such RTP packets
+but internally needs to generate either TRAU-DL or some private functional
+equivalent thereof will need to perform the same UL-to-DL conversion as called
+for in TFO.  The lack of a readily available function library that implements
+the onerous rules of C3211 for EFR is certainly an obstacle, but it is also
+possible to "cut corners" by doing the following:
+
+1) Ignore Table C.3.2.1-1 case 1 and treat it like case 2, at least for EFR:
+   whenever SID frames are received on the incoming TRAU-UL or TRAUlike RTP
+   interface, forward them to call leg B even when that destination call leg
+   has no DTXd.  Given that DTX and SID support has been an integral part of
+   the EFR codec from the beginning, as opposed to an after-addition in the
+   case of FR, every GSM MS that supports EFR can be expected to understand
+   SID frames on the downlink.
+
+2) During speech pauses following transmission of a SID frame on call leg B DL,
+   if real DTXd (turning off Tx) is not allowed, do "fake DTXd" by transmitting
+   dummy FACCH with an L2 fill frame in the same 20 ms traffic frame windows in
+   which real DTXd would have been exercised if it were allowed.
+
+3) Whenever a BFI condition is encountered in the incoming TRAU-UL or TRAUlike
+   RTP frame stream outside of SID, i.e., the case described in the first
+   paragraph of section C.3.2.1.1, induce an intentional BFI condition in the
+   receiving GSM MS by transmitting a dummy FACCH frame as above, instead of
+   trying to devise a parameter-level ECU for EFR.
+
+It should be noted that the just-outlined "cut corners" method is exactly what
+OsmoBTS (and a "pure" Osmocom network in general) does currently, hence nothing
+is lost and no regression is introduced by continuing to do the same.
+
+Seen another way, by making our RTP transport semantically equivalent to
+TRAU-UL, we achieve harmonization between TFO and TrFO.  TrFO (Transcoder-Free
+Operation) is a scenario in which the RTP output from one IP BTS for call leg A
+goes directly to the RTP input of another IP BTS for call leg B, possibly
+passing through simple RTP forwarders like OsmoMGW, but never passing through
+any transcoder.  TrFO is what happens in a self-contained Osmocom network
+without any external MNCC connected to OsmoMSC.  The principal rules of what
+transformations are inherently necessary in order to produce a fully proper DL
+for call leg B from the UL of call leg A remain the same whether the transport
+in between is old-fashioned TFO or modern TrFO, hence the same conversions that
+are codified in TS 28.062 section C.3.2.1.1 are still needed - the only question
+is where in the network are they to be performed.  The original TDM-based GSM
+designers at ETSI gave us a superb architecture end to end; by employing an RTP
+transport that is semantically equivalent to TRAU-UL, we can preserve that whole
+architecture fully intact in an all-IP implementation.
+
+Specification for TRAUlike RTP payload format for FR and EFR
+============================================================
+
+The modified RTP payload format shall consist of a single octet called TRAUlike
+Extension Header (TEH), followed (most of the time) by the standard (same as in
+RFC 3551) 33 octets for FR or 31 octets for EFR.  The TEH octet has the
+following structure:
+
+         +----+----+----+----+----+----+----+----+
+Hex mask |       0xF0        |0x08|0x04|0x02|0x01|
+         +----+----+----+----+----+----+----+----+
+Meaning  |     signature     |DTXd|NDF |BFI |TAF |
+         +----+----+----+----+----+----+----+----+
+
+(Bit numbers are identified by hex masks in order to avoid getting into an
+ argument over which bit numbering convention should be used.)
+
+The following bit fields are defined within the TEH octet:
+
+signature: the upper nibble of the TEH octet shall be set to 0xE.  This
+signature allows RTP packet receivers to identify the payload format by the
+upper nibble of the first octet: if it equals 0xC, the format is EFR without
+TEH, if it equals 0xD, the format is FR without TEH, and if it equals 0xE, then
+the first octet is TEH.
+
+DTXd: this bit is strictly identical with TRAU-UL frame bit C17.
+
+No_Data flag (NDF): this bit shall be set to 1 if the TRAUlike payload consists
+solely of TEH, with the standard 33-octet FR frame or 31-octet EFR frame
+entirely omitted, and shall be 0 otherwise.
+
+BFI: this bit is strictly identical with TRAU-UL frame bit C12.
+
+TAF: this bit is strictly identical with TRAU-UL frame bit C15.
+
+There are two possibilities for full composition of a TRAUlike RTP payload:
+
+Possibility 1: TEH with NDF=0 is followed by a standard 33-octet FR frame or a
+standard 31-octet EFR frame.  The signature in the upper nibble of the octet
+immediately following TEH shall be correct: 0xD for FR or 0xC for EFR.
+
+Possibility 2: TEH with NDF=1 constitutes the entirety of the RTP payload for
+the 20 ms time window in question.
+
+If the No_Data flag is set, BFI must also be set: the combination of NDF=1 and
+BFI=0 is invalid.
+
+Per this specification, the sender of a BFI packet has the choice of sending it
+in one of two forms: with or without presumed-erroneous frame bits.  If the
+TRAUlike RTP packet is generated from bits received in an actual TRAU-UL frame
+(E1 Abis or TFO), erroneous frame bits shall be included, unchanged from the
+TRAU-UL source.  However, if the entity generating the TRAUlike RTP packet is
+the ultimate point of origin (e.g., a native IP BTS), then it shall choose one
+form or the other based on the situation at hand:
+
+a) if the sender does have an FR or EFR frame "on hand" but that frame is
+   considered to be erroneous (for example, the link quality check in
+   l1sap_tch_ind() in OsmoBTS), the long form of BFI shall be sent, with the
+   presumed-erroneous frame bits included.
+
+b) if the sender does not have any FR or EFR frame at all that could be sent
+   (for example, if the reason for the BFI condition is because FACCH was
+   successfully received and decoded instead of a traffic frame), then the
+   No_Data form of BFI shall be sent.
+
+The option of No_Data BFI is provided in this RTP transport format specification
+because if this option were disallowed, senders would be tasked with an
+additional burden of having to artificially generate dummy or "garbage" frame
+bits.  This task is slightly complicated, as explained in the following section,
+and the present design moves that task from all senders to only those receivers
+that need it.
+
+Lack of SID classification bits matching TRAU-UL C13 & C14
+----------------------------------------------------------
+
+TRAU-UL frame format includes two bits C13 & C14 that carry the ternany SID flag
+(0, 1 or 2) as defined in GSM 06.31 and 06.81 section 6.1.1 (same section in
+both specs).  No equivalent bits are included in the TRAUlike RTP transport
+format as defined by this specification - however, these bits are redundant.
+The rules of section 6.1.1 in GSM 06.31 and 06.81, hereafter called S611 rules,
+specify a strictly deterministic, unambiguous formula by which these C13 & C14
+bits derive their values from the bit content of the FR/EFR frame payload -
+thus if a TRAU-UL frame is received in which these C13 & C14 bits fail to match
+the S611 value derived from the contained payload, then that TRAU-UL frame is
+defective.  There is no need to include such redundant bits in our TRAUlike RTP
+format, only to create confusion for receivers as to which source of SID S611
+classification they should use.
+
+Feeding received TRAUlike BFI frames to an EFR decoder
+======================================================
+
+If an EFR decoder implementation is based on the reference C source from ETSI,
+this decoder requires that _some_ frame bits input be fed to it at all times,
+even when BFI=1.  But what if the BFI packet came in as No_Data?  In that case
+the receiver must synthesize its own fake "bad data" bits to feed to the
+standard decoder.  When synthesizing "bad data" bits in this manner, the
+following rules should be observed:
+
+* The 140 bits corresponding to "fixed codebook excitation pulses" (35 bits in
+  each of the 4 subframes) shall be filled using a PRNG.  These bits are the
+  ones used by the standard decoder when its internal state, based on previous
+  good frames, puts it in GSM 06.61 substitution/muting mode as opposed to
+  GSM 06.62 comfort noise generation mode.
+
+* The remaining 104 bits of the EFR frame shall be set to 0.  These bits are
+  never used by the standard decoder under the condition of BFI=1, and setting
+  them to 0 prevents the possibility of S611 rules classifying the frame as SID
+  even if the PRNG output in the other 140 bits happens to be all 1s in those
+  SID codeword bit positions (70 out of 140) that fall within the "fixed
+  codebook excitation pulses" portion.
+
+Converting from TRAU-UL to TRAUlike RTP
+=======================================
+
+There will be a need to convert from standard TS 48.060 TRAU-UL frames to our
+TRAUlike RTP format in the following two scenarios:
+
+1) When interfacing an E1 BTS to Osmocom RAN, when and if such support is to be
+   added to OsmoMGW;
+
+2) In the CN transcoder operating in TFO mode, when forwarding received TFO
+   frames to the local RAN.
+
+In both cases the conversion is straightforward:
+
+* Always generate full-length TRAUlike RTP payloads, never generate No_Data in
+  the case of a properly received TRAU-UL speech (not idle) frame.
+
+* Forward the payload bits directly from TRAU-UL to TRAUlike RTP, for both good
+  and bad frames.
+
+* Directly forward BFI, TAF and DTXd frame indicator bits from TRAU-UL C-bits
+  to TEH octet bits.
+
+* Ignore TRAU-UL C13 & C14 bits.
+
+Converting from TRAUlike RTP to TRAU-UL
+=======================================
+
+This direction of conversion will need to be performed in the CN transcoder when
+emitting TFO frames toward the outside world.  The following rules will need to
+be applied:
+
+* If the incoming TRAUlike RTP payload is full-length, as opposed to No_Data,
+  simply copy the payload bits into the constructed TRAU-UL frame, for both
+  good (BFI=0) and bad (BFI=1) frames.
+
+* If the incoming TRAUlike RTP payload is No_Data, put the following filler in
+  the data bits portion of the TRAU-UL frame:
+
+  - For FR codec, use the silence frame of 3GPP TS 46.011 Table 1 as the filler.
+
+  - For EFR codec, perform the same PRNG procedure as detailed earlier in this
+    document for the case of feeding a No_Data BFI packet to the standard ETSI
+    decoder for EFR.  Given that a TFO-frame-emitting transcoder still needs to
+    run its regular speech decoder in order to fill the upper 6 bits of each
+    outgoing G.711 sample octet, the same No_Data PRNG handler will typically
+    be run just once for both internal decoding and TFO frame output.
+
+* Algorithmically set C13 & C14 bits in the generated TRAU-UL frame per the
+  rules of S611.  This step can be done using osmo_{fr,efr}_sid_classify()
+  functions proposed in this Gerrit patch submission:
+
+  https://gerrit.osmocom.org/c/libosmocore/+/32183
+
+  or using equivalent functions in Themyscira libgsmefr and libgsmfrp.
+
+* Directly forward BFI, TAF and DTXd frame indicator bits from TEH octet bits
+  to TRAU-UL C12, C15 and C17, respectively.
+
+Mixing standard RFC 3551 and TRAUlike RTP payloads
+==================================================
+
+An RTP stream receiver for FR/EFR codecs that supports the present non-standard
+extension to the RTP payload format shall behave gracefully when it receives a
+mixture of standard RFC 3551 payloads and TRAUlike payloads in the same RTP
+stream.  A receiver that has no interest in the additional information carried
+in the TRAUlike Extension Header shall simply strip the TEH octet when one is
+received, reducing the received payload to standard RFC 3551; if a BFI or
+No_Data payload is received, treat it the same as if nothing at all was
+received.  A receiver that is interested in the TRAUlike Extension Header but
+receives an FR/EFR payload without one should behave as if it received a TEH
+with BFI=0, TAF=0, and a received zero-length RTP payload should be treated the
+same as receiving a No_Data TRAUlike payload with TAF=0.
+
+There may even be cases when an RTP sender may alternate between sending
+standard RFC 3551 payloads and TRAUlike payloads in the same session: for
+example, a TFO-supporting CN transcoder may emit "plain" RFC 3551 payloads when
+supplying the output of its free-running speech encoder, but switch to sending
+TRAUlike payloads when it switches to forwarding bits received in TFO frames
+from the far end.