annotate doc/TW-TS-005 @ 581:e2d5cad04cbf

libgsmhr1 RxFE: store CN R0+LPC separately from speech In the original GSM 06.06 code the ECU for speech mode is entirely separate from the CN generator, maintaining separate state. (The main intertie between them is the speech vs CN state variable, distinguishing between speech and CN BFIs, in addition to the CN-specific function of distinguishing between initial and update SIDs.) In the present RxFE implementation I initially thought that we could use the same saved_frame buffer for both ECU and CN, overwriting just the first 4 params (R0 and LPC) when a valid SID comes in. However, I now realize it was a bad idea: the original code has a corner case (long sequence of speech-mode BFIs to put the ECU in state 6, then SID and CN-mode BFIs, then a good speech frame) that would be broken by that buffer reuse approach. We could eliminate this corner case by resetting the ECU state when passing through a CN insertion period, but doing so would needlessly increase the behavioral diffs between GSM 06.06 and our version. Solution: use a separate CN-specific buffer for CN R0+LPC parameters, and match the behavior of GSM 06.06 code in this regard.
author Mychaela Falconia <falcon@freecalypso.org>
date Thu, 13 Feb 2025 10:02:45 +0000
parents d9f6b3125259
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
549
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
1 The original set of Themyscira Wireless utilities for FR and EFR codecs uses an
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
2 ad hoc binary file format to represent streams of FR or EFR codec frames - see
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
3 Binary-file-format article. However, a newer hexadecimal format has now been
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
4 standardized as Themyscira Wireless Technical Specification TW-TS-005:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
5
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
6 https://www.freecalypso.org/specs/tw-ts-005-v010003.txt
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
7
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
8 The standard has two annexes intended for practical use:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
9
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
10 * TW-TS-005 Annex A defines a representation format for FR and EFR codecs;
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
11 * TW-TS-005 Annex B defines a representation format for HR codec.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
12
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
13 The present version of ThemWi GSM codec libraries & utilities suite includes
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
14 some utilities that operate on TW-TS-005 Annex A hex files; support for Annex B
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
15 will appear in a future version when our work on GSM-HR codec integration
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
16 progresses further.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
17
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
18 TW-TS-005 Annex A vs gsmx binary format
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
19 =======================================
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
20
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
21 For working with FR and EFR codecs, our original binary file format has one
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
22 major defect: it cannot represent bad traffic frames (in GSM 06.31 & 06.81
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
23 definition, i.e., BFI=1) that have payload data bits included, as happens in
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
24 well-designed GSM networks that use GSM 08.60 TRAU-UL frames or TW-TS-001
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
25 enhanced RTP transport. This file format deficiency leads to the following
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
26 downstream defects:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
27
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
28 * The combination of "bad traffic frame" and "accepted SID frame" (again,
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
29 GSM 06.31 & 06.81 terminology) gets incorrectly treated as "unusable frame"
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
30 rather than "invalid SID frame" as the specs decree.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
31
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
32 * In the case of EFR, the reference decoder C code that forms the basis for
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
33 Themyscira libgsmefr makes use of "fixed codebook excitation pulses" portion
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
34 of bad frames during speech (as opposed to comfort noise) state - but these
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
35 bits were lost to file format shortcoming.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
36
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
37 The new hexadecimal format of TW-TS-005 Annex A solves this shortcoming: each
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
38 frame is stored as a hex line that directly corresponds to a single RTP payload,
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
39 hence the full capabilities of TW-TS-001 extended RTP format are made available
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
40 in a file at rest.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
41
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
42 Because we have so many existing utilities that read and write gsmx binary
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
43 files, and this binary format is so entrenched in Themyscira development
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
44 environment, we are not doing a "forklift" migration of all of our tools to the
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
45 new format. Instead we are taking a more tempered approach:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
46
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
47 * For the decoding operation (taking a frame stream from an Rx Radio Subsystem
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
48 and producing linear PCM output) that is most affected by the shortcomings of
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
49 gsmx format, we have new utilities that read TW-TS-005 Annex A input, while
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
50 the old gsmx-reading utilities are still preserved and maintained;
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
51
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
52 * For most other workflows (for example, encoding of new speech) conversion
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
53 utilities between the two formats (described below) are deemed sufficient;
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
54
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
55 * New developments such as TFO transform use TW-TS-005 Annex A format natively.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
56
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
57 Human-readable dump decoding of TW-TS-005 hex files
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
58 ===================================================
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
59
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
60 A line-based hexadecimal file format with one line per stored codec frame is
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
61 inherently more human-readable than a binary file, but we also desire a more
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
62 complete decoding such as that produced by gsmrec-dump, showing all codec
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
63 parameters and frame metadata flags. tw5a-dump produces such decoding for
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
64 TW-TS-005 Annex A hex files; there will also be a corresponding tw5b-dump
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
65 utility for TW-TS-005 Annex B when we finish integrating GSM-HR codec support.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
66
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
67 Conversion utilities (FR and EFR codecs)
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
68 ========================================
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
69
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
70 gsmx-to-tw5a and tw5a-to-gsmx utilities do what their names suggest: convert
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
71 FR/EFR speech recordings or test sequences between gsmx (binary) and TW-TS-005
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
72 Annex A (hex) formats. Important semantic notes:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
73
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
74 * gsmx-to-tw5a emits basic RTP format (no TEH) for all good frames, while each
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
75 BFI marker record is converted to a TEH-only No_Data frame.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
76
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
77 * tw5a-to-gsmx is the lossy conversion: distinction between basic and extended
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
78 RTP formats is lost, ditto for TAF without BFI, all BFIs become BFI-no-data.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
79
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
80 A conversion from gsmx to tw5a back to gsmx is lossless, but not the other way
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
81 around.