FreeCalypso > hg > gsm-codec-lib

annotate doc/TW-TS-005 @ 553:ebcf414b7d99

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc/TFO-transform: describe details for FRv1, both modes
author Mychaela Falconia <falcon@freecalypso.org>
date Mon, 07 Oct 2024 08:24:24 +0000
parents d9f6b3125259
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
549
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
1 The original set of Themyscira Wireless utilities for FR and EFR codecs uses an
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
2 ad hoc binary file format to represent streams of FR or EFR codec frames - see
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
3 Binary-file-format article. However, a newer hexadecimal format has now been
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
4 standardized as Themyscira Wireless Technical Specification TW-TS-005:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
5
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
6 https://www.freecalypso.org/specs/tw-ts-005-v010003.txt
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
7
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
8 The standard has two annexes intended for practical use:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
9
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
10 * TW-TS-005 Annex A defines a representation format for FR and EFR codecs;
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
11 * TW-TS-005 Annex B defines a representation format for HR codec.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
12
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
13 The present version of ThemWi GSM codec libraries & utilities suite includes
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
14 some utilities that operate on TW-TS-005 Annex A hex files; support for Annex B
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
15 will appear in a future version when our work on GSM-HR codec integration
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
16 progresses further.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
17
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
18 TW-TS-005 Annex A vs gsmx binary format
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
19 =======================================
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
20
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
21 For working with FR and EFR codecs, our original binary file format has one
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
22 major defect: it cannot represent bad traffic frames (in GSM 06.31 & 06.81
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
23 definition, i.e., BFI=1) that have payload data bits included, as happens in
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
24 well-designed GSM networks that use GSM 08.60 TRAU-UL frames or TW-TS-001
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
25 enhanced RTP transport. This file format deficiency leads to the following
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
26 downstream defects:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
27
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
28 * The combination of "bad traffic frame" and "accepted SID frame" (again,
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
29 GSM 06.31 & 06.81 terminology) gets incorrectly treated as "unusable frame"
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
30 rather than "invalid SID frame" as the specs decree.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
31
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
32 * In the case of EFR, the reference decoder C code that forms the basis for
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
33 Themyscira libgsmefr makes use of "fixed codebook excitation pulses" portion
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
34 of bad frames during speech (as opposed to comfort noise) state - but these
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
35 bits were lost to file format shortcoming.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
36
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
37 The new hexadecimal format of TW-TS-005 Annex A solves this shortcoming: each
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
38 frame is stored as a hex line that directly corresponds to a single RTP payload,
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
39 hence the full capabilities of TW-TS-001 extended RTP format are made available
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
40 in a file at rest.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
41
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
42 Because we have so many existing utilities that read and write gsmx binary
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
43 files, and this binary format is so entrenched in Themyscira development
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
44 environment, we are not doing a "forklift" migration of all of our tools to the
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
45 new format. Instead we are taking a more tempered approach:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
46
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
47 * For the decoding operation (taking a frame stream from an Rx Radio Subsystem
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
48 and producing linear PCM output) that is most affected by the shortcomings of
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
49 gsmx format, we have new utilities that read TW-TS-005 Annex A input, while
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
50 the old gsmx-reading utilities are still preserved and maintained;
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
51
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
52 * For most other workflows (for example, encoding of new speech) conversion
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
53 utilities between the two formats (described below) are deemed sufficient;
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
54
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
55 * New developments such as TFO transform use TW-TS-005 Annex A format natively.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
56
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
57 Human-readable dump decoding of TW-TS-005 hex files
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
58 ===================================================
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
59
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
60 A line-based hexadecimal file format with one line per stored codec frame is
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
61 inherently more human-readable than a binary file, but we also desire a more
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
62 complete decoding such as that produced by gsmrec-dump, showing all codec
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
63 parameters and frame metadata flags. tw5a-dump produces such decoding for
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
64 TW-TS-005 Annex A hex files; there will also be a corresponding tw5b-dump
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
65 utility for TW-TS-005 Annex B when we finish integrating GSM-HR codec support.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
66
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
67 Conversion utilities (FR and EFR codecs)
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
68 ========================================
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
69
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
70 gsmx-to-tw5a and tw5a-to-gsmx utilities do what their names suggest: convert
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
71 FR/EFR speech recordings or test sequences between gsmx (binary) and TW-TS-005
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
72 Annex A (hex) formats. Important semantic notes:
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
73
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
74 * gsmx-to-tw5a emits basic RTP format (no TEH) for all good frames, while each
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
75 BFI marker record is converted to a TEH-only No_Data frame.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
76
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
77 * tw5a-to-gsmx is the lossy conversion: distinction between basic and extended
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
78 RTP formats is lost, ditto for TAF without BFI, all BFIs become BFI-no-data.
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
79
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
80 A conversion from gsmx to tw5a back to gsmx is lossless, but not the other way
d9f6b3125259 document TW-TS-005 utilities
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
81 around.