+
−There exists a set of C header files, needed for TI's GSM firmware to build,
+
−called cdginc.  They are not needed for building everything up to and including
+
−GPF and L1, but they are needed in order to build the core G23 protocol stack
+
−components.
+
−
+
−However, these cdginc headers are generated files, not human-written sources,
+
−and the process by which they get created is very messy:
+
−
+
−1. Contained as part of the ultimate source for the firmware, there is a set of
+
−   XML files which give definitions for Air Interface Messages (AIMs) and
+
−   Service Access Points (SAPs).
+
−
+
−2. Each *.aim file is "compiled" into an MDF (message definition file) and each
+
−   *.sap file gets similarly "compiled" into a PDF (primitive definition file).
+
−   (Despite sharing the same acronym and filename suffix, these "primitive
+
−   definition files" have nothing to do with PDF as in Portable Document
+
−   Format.)  This "compilation" is done by way of XSLT: an XSLT processor is
+
−   invoked, its inputs being the source *.{aim,sap} files and a set of *.xsl
+
−   files defining the transformation.  (The latter can be found under
+
−   gpf/util/sape/xslt in the "peek" LoCosto source.)
+
−
+
−3. A special TI/Condat-developed program called ccdgen (which we only have in
+
−   the form of an M$ Windows binary sans source, ccdgen.exe) reads the *.mdf
+
−   and *.pdf files produced in the previous step, performs some unknown
+
−   processing (unknown because we have no source for this tool), and writes out
+
−   most of the C header files which appear in the cdginc directory.  (The
+
−   exception is that a few of these header files seem to be produced directly
+
−   by the XSLT step.)
+
−
+
−==================
+
−XSLT and Java woes
+
−==================
+
−
+
−When I (Space Falcon) tried to reproduce the above steps for FreeCalypso,
+
−problems began at the XSLT step.  The XSLT processor used in TI's build flow is
+
−an old version of Xalan-J from Apache.  J stands for Java - yikes.  Thus TI's
+
−build flow actually runs java with a set of *.jar files which comprise Xalan-J.
+
−
+
−I looked to see if the use of Xalan-J (and thus of Java) was required, or if
+
−one could use any XSLT processor, including non-Java implementations.  Not so
+
−fast: TI's *.xsl files for the needed transformation call some functions
+
−(please forgive my probably incorrect terminology: both XSLT and Java are as
+
−foreign and unfamiliar to me as Japanese or Arabic or ... - you get the idea)
+
−which seem to have been implemented by TI as custom Java classes, falling under
+
−com.ti.xslt.extension - the latter live in gpf/tools/lib/java/xalan-ext.jar in
+
−TI's semi-source trees (both Leonardo and LoCosto), if you would like to see
+
−for yourself.
+
−
+
−That xalan-ext.jar file with TI's "XSLT extension" classes contains Java
+
−bytecode, not source.  Thus one of the required pieces for the *.{aim,sap} ->
+
−*.{mdf,pdf} build step effectively exists only in the form of compiled code
+
−sans source.  It is of course an impairment to freedom, and as I quickly
+
−discovered, not only in philosophical terms, but also in practice: as I will
+
−show in a moment, there appears to be a bug in there which we lack the ability
+
−to fix.
+
−
+
−Of course TI ran their java invokation for XSLT under Winblows.  (As a side
+
−note, I successfully ran TI's entire Winblows environment, including this step,
+
−under Wine when I did leo2moko - but I wasn't trying to extract any individual
+
−step and get it to run by itself, instead I ran TI's *entire Winblows env*
+
−under a single top-level wine invokation.)  But Java bytecode is supposed to be
+
−platform-independent, right?  So I tried running the java command from pdt_*.mak
+
−makefiles in the Leonardo version, using their set of XSLT/xalan jars as-is,
+
−under Slackware Linux without Wine, using the Linux-native version of Java that
+
−came with Slackware.
+
−
+
−I started with the AIM->MDF part.  The operation succeeded on a few of the
+
−files, but then failed on others.  The error had something to do with filename
+
−and pathname manipulation.  Some of the com.ti.xslt.extension functions called
+
−by TI's xslt transforms seem to be responsible for turning short filenames into
+
−absolute pathnames and then into file:// URLs, and it appears that TI's
+
−implementation of these functions assumes that absolute pathnames will have
+
−Weendoze drive letters, and breaks on Unix absolute pathnames which lack that
+
−nonsense.  And the part responsible for the bug is a piece of Java bytecode in
+
−a jar sans source, remember?  I didn't get as far as trying the SAP->PDF part.
+
−
+
−I reason that someone who knows the world of Java could probably reverse-eng
+
−that bytecode and fix the bug with a binary patch, or rewrite an alternate
+
−implementation.  Reversing Java bytecode might not even be necessary: someone
+
−who understands XSLT could probably figure out what functionality is expected
+
−from these extension functions, and then reimplement that (most likely trivial)
+
−functionality anew.  But XSLT is just as foreign to me as Java; they both might
+
−as well be Japanese or Arabic or some other super-hard foreign language.
+
−
+
−Given that my goal is to produce free GSM firmware, I decided that taking a
+
−very long detour to learn XSLT and Java just so we can regenerate TI's *.[mp]df
+
−files from *.{aim,sap} is not worth it, and imported prebuilt *.[mp]df files
+
−from the LoCosto source along with *.{aim,sap}.
+
−
+
−============================
+
−Different versions of cdginc
+
−============================
+
−
+
−Most of the hard work in the FreeCalypso project involves reconciliation between
+
−our two reference versions: TCS211 (aka Leonardo) and LoCosto.  Our TCS211
+
−reference version (in the form of leo2moko) already runs on one of our target
+
−platforms and works beautifully, but it has the entire GSM protocol stack in
+
−binary-only libs.  The LoCosto version is full source (aside from Nucleus and
+
−some parts of GPF which have already been taken care of), but targets the wrong
+
−chipset, and has that nasty SBuild crap instead of pdt_*.mak.
+
−
+
−The versions of cdginc used in the TCS211 and LoCosto semi-src trees differ in
+
−the following ways:
+
−
+
−1. The starting *.aim and *.sap files are different: the LoCosto versions are
+
−   newer.
+
−
+
−2. Slightly different versions of ccdgen.exe are used: the version featured in
+
−   the TCS211 version from Sotovik identifies itself as 2.5.5, whereas the one
+
−   featured in the LoCosto "peek" find identifies itself as 2.5.5A.  Aside from
+
−   some cosmetic differences, one substantive difference was found in the
+
−   generated output: the so-called mtx tables (don't ask me what they are, as I
+
−   don't understand it myself) are emitted in a different format.  (Ccdgen
+
−   version 2.5.5A generates the new format by default, and has a command line
+
−   option to revert to the old format.)
+
−
+
−   These "mtx" generated header files are included by some ccddata modules (see
+
−   ../ccd/README for more info), and the only C source we have for these modules
+
−   (for all of CCD, in fact) comes from the LoCosto version.  This version of
+
−   the ccddata C source expects "mtx" cdginc headers in the new format, hence
+
−   that is the format we need to use.
+
−
+
−3. One additional input to ccdgen besides the *.[mp]df files is a "settings"
+
−   file called fflags.h.  It has the form of a C header file with #define and
+
−   #undef lines (the rest is just comments), but as far as I can tell, it never
+
−   gets fed to a C preprocessor, only to ccdgen.  The starting *.{aim,sap} files
+
−   contain some "options", and for each of these options, fflags.h must give a
+
−   yes/no answer in the form of $define or #undef.  This "settings" file is
+
−   mandatory: if it is not given on the ccdgen command line, or if it has
+
−   neither a #define nor a #undef for some "option" defined in the *.{aim,sap}
+
−   files, ccdgen aborts with an error.
+
−
+
−It appears that the version of *.{aim,sap} featured in the TCS211 fw has only
+
−one option named TI_DUAL_MODE, which needs to be disabled, as the version of
+
−fflags.h in that tree has only one non-comment line:
+
−
+
−#undef	TI_DUAL_MODE
+
−
+
−But the LoCosto version of fflags.h (which appears here as fflags-locosto.h) is
+
−much more extensive, and all of the options listed therein appear in the
+
−*.{aim,sap} files and are in need of explicit enabling or disabling - as I
+
−found out when I tried running ccdgen on LoCosto *.[mp]df files with the TCS211
+
−version of fflags.h - it immediately failed with a bunch of errors about certain
+
−options not being set one way or the other.
+
−
+
−============
+
−New features
+
−============
+
−
+
−It appears that all of the options enabled in the LoCosto version of fflags.h
+
−correspond to new features of the G23 protocol stack which do not exist at all
+
−in our TCS211 reference version (leo2moko).  How can we tell what features are
+
−present or absent in our TCS211 version if all we have is binary libs sans
+
−source?  For a long time I thought the problem was unsolvable, but then I found
+
−the answer in an obscure place: the "relic" pdt_*.mak files *other than* the
+
−actively used pdt_2091.mak present in the source from Sotovik.  The source we
+
−have is "sanitized" in that the C source files for all of L1 and G23 have been
+
−removed, and the makefile in pdt_2091.mak has no compilation stanzas for these
+
−modules either: this generated makefile is set up to take the corresponding
+
−binary-only libs as "sources".  However, the other (unused) pdt_*.mak files
+
−have been built (back in TI's development environment, presumably) in the "full
+
−source present" configuration, and list not only the names and paths for all of
+
−the deleted source files, but also their complete compilation lines with all -I
+
−directories and -D options!
+
−
+
−Looking at the latter compilation lines, one can see that none of the options
+
−related to GSM REL99 or TI's new "multiband" stuff (seen both in fflags.h and
+
−throughout the L1 and G23 sources we are back-porting from the LoCosto "peek"
+
−find) are present at all in our TCS211/leo2moko reference version.
+
−
+
−===============================
+
−Approach chosen for FreeCalypso
+
−===============================
+
−
+
−I have not yet figured out whether the new apparently-high-level GSM PS features
+
−found in the LoCosto version we are working with actually depend on some LoCosto
+
−hardware or DSP ROM code feature not present on the Calypso, or if they can be
+
−back-ported to the Calypso just fine.  I also currently have very little
+
−understanding as to their merit, i.e., practical value or usefulness for a GSM
+
−cellphone end user.
+
−
+
−The approach I'm taking for the initial version is to recreate the TCS211
+
−configuration that already works on the Neo Freerunner in the form of leo2moko
+
−as closely as possible, and that means setting all of the newer options to the
+
−disabled state.  Or at least that is the approach I am following until and
+
−unless I run into some problem with it; if and when the latter happens, I'll
+
−re-evaluate my course.
+
−
+
−However, simply using the set of cdginc files from TCS211 (or even regenerating
+
−them from the TCS211 versions of *.[mp]df with the new ccdgen.exe to get mtx
+
−tables in the format expected by our version of CCD source) with the G23
+
−protocol stack C sources from the LoCosto version will likely lead to problems,
+
−or least more hassle than it's worth, hence I decided to bite the bullet and
+
−use the *.{aim,sap} and *.[mp]df files from LoCosto.
+
−
+
−Using the LoCosto versions of *.[mp]df, I ran ccdgen.exe (version 2.5.5A, set
+
−to generate mtx tables in the new format) with two different fflags.h
+
−configurations: "locosto" (unchanged from the "peek" find) and "conservative".
+
−The latter is a configuration in which every existing option is set to disabled:
+
−I took fflags-locosto.h and changed every #define to #undef.
+
−
+
−The symlink is currently set to compile the GSM fw using the "conservative"
+
−version of cdginc headers.  As I said above, I'll proceed with this
+
−configuration until and unless I hit a problem, and then re-evaluate this
+
−course if need be.
+
−
+
−=====================
+
−ccdgen binary problem
+
−=====================
+
−
+
−We only have a ccdgen.exe binary, but no source.  The Winblows binary in
+
−question happens to run fine under Wine, at least on my Slackware machine, but
+
−needless to say, asking every FreeCalypso user who wishes to compile her own
+
−GSM fw from source to run a sans-source Weendoze binary under Wine and pray
+
−that it works is not an attractive proposition.  Therefore, as a workaround I
+
−have checked the generated cdginc files into Hg as if they were source files.
+
−
+
−Of course this workaround is not a proper solution either, but it is the best
+
−we can do for the time being, until and unless someone either finds the missing
+
−source for ccdgen or figures out its logic and writes a from-scratch functional
+
−replacement.