FreeCalypso > hg > fc-magnetite

comparison cdg3/README @ 16:c15047b3d00d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
cdg3: import from freecalypso-citrine/cdg
author Mychaela Falconia <falcon@freecalypso.org>
date Tue, 27 Sep 2016 16:27:34 +0000
parents
children
comparison
equal deleted inserted replaced
15:c8bdae60fcb1 16:c15047b3d00d
1 There exists a set of C header files, needed for TI's GSM firmware to build,
2 called cdginc. They are not needed for building everything up to and including
3 GPF and L1, but they are needed in order to build the core G23 protocol stack
4 components.
5
6 However, these cdginc headers are generated files, not human-written sources,
7 and the process by which they get created is very messy:
8
9 1. Contained as part of the ultimate source for the firmware, there is a set of
10 XML files which give definitions for Air Interface Messages (AIMs) and
11 Service Access Points (SAPs).
12
13 2. Each *.aim file is "compiled" into an MDF (message definition file) and each
14 *.sap file gets similarly "compiled" into a PDF (primitive definition file).
15 (Despite sharing the same acronym and filename suffix, these "primitive
16 definition files" have nothing to do with PDF as in Portable Document
17 Format.) This "compilation" is done by way of XSLT: an XSLT processor is
18 invoked, its inputs being the source *.{aim,sap} files and a set of *.xsl
19 files defining the transformation. (The latter can be found under
20 gpf/util/sape/xslt in the "peek" LoCosto source.)
21
22 3. A special TI/Condat-developed program called ccdgen (which we only have in
23 the form of an M$ Windows binary sans source, ccdgen.exe) reads the *.mdf
24 and *.pdf files produced in the previous step, performs some unknown
25 processing (unknown because we have no source for this tool), and writes out
26 most of the C header files which appear in the cdginc directory. (The
27 exception is that a few of these header files seem to be produced directly
28 by the XSLT step.)
29
30 ==================
31 XSLT and Java woes
32 ==================
33
34 When I (Space Falcon) tried to reproduce the above steps for FreeCalypso,
35 problems began at the XSLT step. The XSLT processor used in TI's build flow is
36 an old version of Xalan-J from Apache. J stands for Java - yikes. Thus TI's
37 build flow actually runs java with a set of *.jar files which comprise Xalan-J.
38
39 I looked to see if the use of Xalan-J (and thus of Java) was required, or if
40 one could use any XSLT processor, including non-Java implementations. Not so
41 fast: TI's *.xsl files for the needed transformation call some functions
42 (please forgive my probably incorrect terminology: both XSLT and Java are as
43 foreign and unfamiliar to me as Japanese or Arabic or ... - you get the idea)
44 which seem to have been implemented by TI as custom Java classes, falling under
45 com.ti.xslt.extension - the latter live in gpf/tools/lib/java/xalan-ext.jar in
46 TI's semi-source trees (both Leonardo and LoCosto), if you would like to see
47 for yourself.
48
49 That xalan-ext.jar file with TI's "XSLT extension" classes contains Java
50 bytecode, not source. Thus one of the required pieces for the *.{aim,sap} ->
51 *.{mdf,pdf} build step effectively exists only in the form of compiled code
52 sans source. It is of course an impairment to freedom, and as I quickly
53 discovered, not only in philosophical terms, but also in practice: as I will
54 show in a moment, there appears to be a bug in there which we lack the ability
55 to fix.
56
57 Of course TI ran their java invokation for XSLT under Winblows. (As a side
58 note, I successfully ran TI's entire Winblows environment, including this step,
59 under Wine when I did leo2moko - but I wasn't trying to extract any individual
60 step and get it to run by itself, instead I ran TI's *entire Winblows env*
61 under a single top-level wine invokation.) But Java bytecode is supposed to be
62 platform-independent, right? So I tried running the java command from pdt_*.mak
63 makefiles in the Leonardo version, using their set of XSLT/xalan jars as-is,
64 under Slackware Linux without Wine, using the Linux-native version of Java that
65 came with Slackware.
66
67 I started with the AIM->MDF part. The operation succeeded on a few of the
68 files, but then failed on others. The error had something to do with filename
69 and pathname manipulation. Some of the com.ti.xslt.extension functions called
70 by TI's xslt transforms seem to be responsible for turning short filenames into
71 absolute pathnames and then into file:// URLs, and it appears that TI's
72 implementation of these functions assumes that absolute pathnames will have
73 Weendoze drive letters, and breaks on Unix absolute pathnames which lack that
74 nonsense. And the part responsible for the bug is a piece of Java bytecode in
75 a jar sans source, remember? I didn't get as far as trying the SAP->PDF part.
76
77 I reason that someone who knows the world of Java could probably reverse-eng
78 that bytecode and fix the bug with a binary patch, or rewrite an alternate
79 implementation. Reversing Java bytecode might not even be necessary: someone
80 who understands XSLT could probably figure out what functionality is expected
81 from these extension functions, and then reimplement that (most likely trivial)
82 functionality anew. But XSLT is just as foreign to me as Java; they both might
83 as well be Japanese or Arabic or some other super-hard foreign language.
84
85 Given that my goal is to produce free GSM firmware, I decided that taking a
86 very long detour to learn XSLT and Java just so we can regenerate TI's *.[mp]df
87 files from *.{aim,sap} is not worth it, and imported prebuilt *.[mp]df files
88 from the LoCosto source along with *.{aim,sap}.
89
90 ============================
91 Different versions of cdginc
92 ============================
93
94 Most of the hard work in the FreeCalypso project involves reconciliation between
95 our two reference versions: TCS211 (aka Leonardo) and LoCosto. Our TCS211
96 reference version (in the form of leo2moko) already runs on one of our target
97 platforms and works beautifully, but it has the entire GSM protocol stack in
98 binary-only libs. The LoCosto version is full source (aside from Nucleus and
99 some parts of GPF which have already been taken care of), but targets the wrong
100 chipset, and has that nasty SBuild crap instead of pdt_*.mak.
101
102 The versions of cdginc used in the TCS211 and LoCosto semi-src trees differ in
103 the following ways:
104
105 1. The starting *.aim and *.sap files are different: the LoCosto versions are
106 newer.
107
108 2. Slightly different versions of ccdgen.exe are used: the version featured in
109 the TCS211 version from Sotovik identifies itself as 2.5.5, whereas the one
110 featured in the LoCosto "peek" find identifies itself as 2.5.5A. Aside from
111 some cosmetic differences, one substantive difference was found in the
112 generated output: the so-called mtx tables (don't ask me what they are, as I
113 don't understand it myself) are emitted in a different format. (Ccdgen
114 version 2.5.5A generates the new format by default, and has a command line
115 option to revert to the old format.)
116
117 These "mtx" generated header files are included by some ccddata modules (see
118 ../ccd/README for more info), and the only C source we have for these modules
119 (for all of CCD, in fact) comes from the LoCosto version. This version of
120 the ccddata C source expects "mtx" cdginc headers in the new format, hence
121 that is the format we need to use.
122
123 3. One additional input to ccdgen besides the *.[mp]df files is a "settings"
124 file called fflags.h. It has the form of a C header file with #define and
125 #undef lines (the rest is just comments), but as far as I can tell, it never
126 gets fed to a C preprocessor, only to ccdgen. The starting *.{aim,sap} files
127 contain some "options", and for each of these options, fflags.h must give a
128 yes/no answer in the form of $define or #undef. This "settings" file is
129 mandatory: if it is not given on the ccdgen command line, or if it has
130 neither a #define nor a #undef for some "option" defined in the *.{aim,sap}
131 files, ccdgen aborts with an error.
132
133 It appears that the version of *.{aim,sap} featured in the TCS211 fw has only
134 one option named TI_DUAL_MODE, which needs to be disabled, as the version of
135 fflags.h in that tree has only one non-comment line:
136
137 #undef TI_DUAL_MODE
138
139 But the LoCosto version of fflags.h (which appears here as fflags-locosto.h) is
140 much more extensive, and all of the options listed therein appear in the
141 *.{aim,sap} files and are in need of explicit enabling or disabling - as I
142 found out when I tried running ccdgen on LoCosto *.[mp]df files with the TCS211
143 version of fflags.h - it immediately failed with a bunch of errors about certain
144 options not being set one way or the other.
145
146 ============
147 New features
148 ============
149
150 It appears that all of the options enabled in the LoCosto version of fflags.h
151 correspond to new features of the G23 protocol stack which do not exist at all
152 in our TCS211 reference version (leo2moko). How can we tell what features are
153 present or absent in our TCS211 version if all we have is binary libs sans
154 source? For a long time I thought the problem was unsolvable, but then I found
155 the answer in an obscure place: the "relic" pdt_*.mak files *other than* the
156 actively used pdt_2091.mak present in the source from Sotovik. The source we
157 have is "sanitized" in that the C source files for all of L1 and G23 have been
158 removed, and the makefile in pdt_2091.mak has no compilation stanzas for these
159 modules either: this generated makefile is set up to take the corresponding
160 binary-only libs as "sources". However, the other (unused) pdt_*.mak files
161 have been built (back in TI's development environment, presumably) in the "full
162 source present" configuration, and list not only the names and paths for all of
163 the deleted source files, but also their complete compilation lines with all -I
164 directories and -D options!
165
166 Looking at the latter compilation lines, one can see that none of the options
167 related to GSM REL99 or TI's new "multiband" stuff (seen both in fflags.h and
168 throughout the L1 and G23 sources we are back-porting from the LoCosto "peek"
169 find) are present at all in our TCS211/leo2moko reference version.
170
171 ===============================
172 Approach chosen for FreeCalypso
173 ===============================
174
175 I have not yet figured out whether the new apparently-high-level GSM PS features
176 found in the LoCosto version we are working with actually depend on some LoCosto
177 hardware or DSP ROM code feature not present on the Calypso, or if they can be
178 back-ported to the Calypso just fine. I also currently have very little
179 understanding as to their merit, i.e., practical value or usefulness for a GSM
180 cellphone end user.
181
182 The approach I'm taking for the initial version is to recreate the TCS211
183 configuration that already works on the Neo Freerunner in the form of leo2moko
184 as closely as possible, and that means setting all of the newer options to the
185 disabled state. Or at least that is the approach I am following until and
186 unless I run into some problem with it; if and when the latter happens, I'll
187 re-evaluate my course.
188
189 However, simply using the set of cdginc files from TCS211 (or even regenerating
190 them from the TCS211 versions of *.[mp]df with the new ccdgen.exe to get mtx
191 tables in the format expected by our version of CCD source) with the G23
192 protocol stack C sources from the LoCosto version will likely lead to problems,
193 or least more hassle than it's worth, hence I decided to bite the bullet and
194 use the *.{aim,sap} and *.[mp]df files from LoCosto.
195
196 Using the LoCosto versions of *.[mp]df, I ran ccdgen.exe (version 2.5.5A, set
197 to generate mtx tables in the new format) with two different fflags.h
198 configurations: "locosto" (unchanged from the "peek" find) and "conservative".
199 The latter is a configuration in which every existing option is set to disabled:
200 I took fflags-locosto.h and changed every #define to #undef.
201
202 The symlink is currently set to compile the GSM fw using the "conservative"
203 version of cdginc headers. As I said above, I'll proceed with this
204 configuration until and unless I hit a problem, and then re-evaluate this
205 course if need be.
206
207 =====================
208 ccdgen binary problem
209 =====================
210
211 We only have a ccdgen.exe binary, but no source. The Winblows binary in
212 question happens to run fine under Wine, at least on my Slackware machine, but
213 needless to say, asking every FreeCalypso user who wishes to compile her own
214 GSM fw from source to run a sans-source Weendoze binary under Wine and pray
215 that it works is not an attractive proposition. Therefore, as a workaround I
216 have checked the generated cdginc files into Hg as if they were source files.
217
218 Of course this workaround is not a proper solution either, but it is the best
219 we can do for the time being, until and unless someone either finds the missing
220 source for ccdgen or figures out its logic and writes a from-scratch functional
221 replacement.