# HG changeset patch # User Michael Spacefalcon # Date 1411691474 0 # Node ID c2c4c6b5a5c51f209f6b532ccc3bbd5f0494c719 # Parent 46e5c90fd0b89300e66746a0282aa045def1bd21 gsm-fw/cdg/README: written diff -r 46e5c90fd0b8 -r c2c4c6b5a5c5 gsm-fw/cdg/README --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/gsm-fw/cdg/README Fri Sep 26 00:31:14 2014 +0000 @@ -0,0 +1,221 @@ +There exists a set of C header files, needed for TI's GSM firmware to build, +called cdginc. They are not needed for building everything up to and including +GPF and L1, but they are needed in order to build the core G23 protocol stack +components. + +However, these cdginc headers are generated files, not human-written sources, +and the process by which they get created is very messy: + +1. Contained as part of the ultimate source for the firmware, there is a set of + XML files which give definitions for Air Interface Messages (AIMs) and + Service Access Points (SAPs). + +2. Each *.aim file is "compiled" into an MDF (message definition file) and each + *.sap file gets similarly "compiled" into a PDF (primitive definition file). + (Despite sharing the same acronym and filename suffix, these "primitive + definition files" have nothing to do with PDF as in Portable Document + Format.) This "compilation" is done by way of XSLT: an XSLT processor is + invoked, its inputs being the source *.{aim,sap} files and a set of *.xsl + files defining the transformation. (The latter can be found under + gpf/util/sape/xslt in the "peek" LoCosto source.) + +3. A special TI/Condat-developed program called ccdgen (which we only have in + the form of an M$ Windows binary sans source, ccdgen.exe) reads the *.mdf + and *.pdf files produced in the previous step, performs some unknown + processing (unknown because we have no source for this tool), and writes out + most of the C header files which appear in the cdginc directory. (The + exception is that a few of these header files seem to be produced directly + by the XSLT step.) + +================== +XSLT and Java woes +================== + +When I (Space Falcon) tried to reproduce the above steps for FreeCalypso, +problems began at the XSLT step. The XSLT processor used in TI's build flow is +an old version of Xalan-J from Apache. J stands for Java - yikes. Thus TI's +build flow actually runs java with a set of *.jar files which comprise Xalan-J. + +I looked to see if the use of Xalan-J (and thus of Java) was required, or if +one could use any XSLT processor, including non-Java implementations. Not so +fast: TI's *.xsl files for the needed transformation call some functions +(please forgive my probably incorrect terminology: both XSLT and Java are as +foreign and unfamiliar to me as Japanese or Arabic or ... - you get the idea) +which seem to have been implemented by TI as custom Java classes, falling under +com.ti.xslt.extension - the latter live in gpf/tools/lib/java/xalan-ext.jar in +TI's semi-source trees (both Leonardo and LoCosto), if you would like to see +for yourself. + +That xalan-ext.jar file with TI's "XSLT extension" classes contains Java +bytecode, not source. Thus one of the required pieces for the *.{aim,sap} -> +*.{mdf,pdf} build step effectively exists only in the form of compiled code +sans source. It is of course an impairment to freedom, and as I quickly +discovered, not only in philosophical terms, but also in practice: as I will +show in a moment, there appears to be a bug in there which we lack the ability +to fix. + +Of course TI ran their java invokation for XSLT under Winblows. (As a side +note, I successfully ran TI's entire Winblows environment, including this step, +under Wine when I did leo2moko - but I wasn't trying to extract any individual +step and get it to run by itself, instead I ran TI's *entire Winblows env* +under a single top-level wine invokation.) But Java bytecode is supposed to be +platform-independent, right? So I tried running the java command from pdt_*.mak +makefiles in the Leonardo version, using their set of XSLT/xalan jars as-is, +under Slackware Linux without Wine, using the Linux-native version of Java that +came with Slackware. + +I started with the AIM->MDF part. The operation succeeded on a few of the +files, but then failed on others. The error had something to do with filename +and pathname manipulation. Some of the com.ti.xslt.extension functions called +by TI's xslt transforms seem to be responsible for turning short filenames into +absolute pathnames and then into file:// URLs, and it appears that TI's +implementation of these functions assumes that absolute pathnames will have +Weendoze drive letters, and breaks on Unix absolute pathnames which lack that +nonsense. And the part responsible for the bug is a piece of Java bytecode in +a jar sans source, remember? I didn't get as far as trying the SAP->PDF part. + +I reason that someone who knows the world of Java could probably reverse-eng +that bytecode and fix the bug with a binary patch, or rewrite an alternate +implementation. Reversing Java bytecode might not even be necessary: someone +who understands XSLT could probably figure out what functionality is expected +from these extension functions, and then reimplement that (most likely trivial) +functionality anew. But XSLT is just as foreign to me as Java; they both might +as well be Japanese or Arabic or some other super-hard foreign language. + +Given that my goal is to produce free GSM firmware, I decided that taking a +very long detour to learn XSLT and Java just so we can regenerate TI's *.[mp]df +files from *.{aim,sap} is not worth it, and imported prebuilt *.[mp]df files +from the LoCosto source along with *.{aim,sap}. + +============================ +Different versions of cdginc +============================ + +Most of the hard work in the FreeCalypso project involves reconciliation between +our two reference versions: TCS211 (aka Leonardo) and LoCosto. Our TCS211 +reference version (in the form of leo2moko) already runs on one of our target +platforms and works beautifully, but it has the entire GSM protocol stack in +binary-only libs. The LoCosto version is full source (aside from Nucleus and +some parts of GPF which have already been taken care of), but targets the wrong +chipset, and has that nasty SBuild crap instead of pdt_*.mak. + +The versions of cdginc used in the TCS211 and LoCosto semi-src trees differ in +the following ways: + +1. The starting *.aim and *.sap files are different: the LoCosto versions are + newer. + +2. Slightly different versions of ccdgen.exe are used: the version featured in + the TCS211 version from Sotovik identifies itself as 2.5.5, whereas the one + featured in the LoCosto "peek" find identifies itself as 2.5.5A. Aside from + some cosmetic differences, one substantive difference was found in the + generated output: the so-called mtx tables (don't ask me what they are, as I + don't understand it myself) are emitted in a different format. (Ccdgen + version 2.5.5A generates the new format by default, and has a command line + option to revert to the old format.) + + These "mtx" generated header files are included by some ccddata modules (see + ../ccd/README for more info), and the only C source we have for these modules + (for all of CCD, in fact) comes from the LoCosto version. This version of + the ccddata C source expects "mtx" cdginc headers in the new format, hence + that is the format we need to use. + +3. One additional input to ccdgen besides the *.[mp]df files is a "settings" + file called fflags.h. It has the form of a C header file with #define and + #undef lines (the rest is just comments), but as far as I can tell, it never + gets fed to a C preprocessor, only to ccdgen. The starting *.{aim,sap} files + contain some "options", and for each of these options, fflags.h must give a + yes/no answer in the form of $define or #undef. This "settings" file is + mandatory: if it is not given on the ccdgen command line, or if it has + neither a #define nor a #undef for some "option" defined in the *.{aim,sap} + files, ccdgen aborts with an error. + +It appears that the version of *.{aim,sap} featured in the TCS211 fw has only +one option named TI_DUAL_MODE, which needs to be disabled, as the version of +fflags.h in that tree has only one non-comment line: + +#undef TI_DUAL_MODE + +But the LoCosto version of fflags.h (which appears here as fflags-locosto.h) is +much more extensive, and all of the options listed therein appear in the +*.{aim,sap} files and are in need of explicit enabling or disabling - as I +found out when I tried running ccdgen on LoCosto *.[mp]df files with the TCS211 +version of fflags.h - it immediately failed with a bunch of errors about certain +options not being set one way or the other. + +============ +New features +============ + +It appears that all of the options enabled in the LoCosto version of fflags.h +correspond to new features of the G23 protocol stack which do not exist at all +in our TCS211 reference version (leo2moko). How can we tell what features are +present or absent in our TCS211 version if all we have is binary libs sans +source? For a long time I thought the problem was unsolvable, but then I found +the answer in an obscure place: the "relic" pdt_*.mak files *other than* the +actively used pdt_2091.mak present in the source from Sotovik. The source we +have is "sanitized" in that the C source files for all of L1 and G23 have been +removed, and the makefile in pdt_2091.mak has no compilation stanzas for these +modules either: this generated makefile is set up to take the corresponding +binary-only libs as "sources". However, the other (unused) pdt_*.mak files +have been built (back in TI's development environment, presumably) in the "full +source present" configuration, and list not only the names and paths for all of +the deleted source files, but also their complete compilation lines with all -I +directories and -D options! + +Looking at the latter compilation lines, one can see that none of the options +related to GSM REL99 or TI's new "multiband" stuff (seen both in fflags.h and +throughout the L1 and G23 sources we are back-porting from the LoCosto "peek" +find) are present at all in our TCS211/leo2moko reference version. + +=============================== +Approach chosen for FreeCalypso +=============================== + +I have not yet figured out whether the new apparently-high-level GSM PS features +found in the LoCosto version we are working with actually depend on some LoCosto +hardware or DSP ROM code feature not present on the Calypso, or if they can be +back-ported to the Calypso just fine. I also currently have very little +understanding as to their merit, i.e., practical value or usefulness for a GSM +cellphone end user. + +The approach I'm taking for the initial version is to recreate the TCS211 +configuration that already works on the Neo Freerunner in the form of leo2moko +as closely as possible, and that means setting all of the newer options to the +disabled state. Or at least that is the approach I am following until and +unless I run into some problem with it; if and when the latter happens, I'll +re-evaluate my course. + +However, simply using the set of cdginc files from TCS211 (or even regenerating +them from the TCS211 versions of *.[mp]df with the new ccdgen.exe to get mtx +tables in the format expected by our version of CCD source) with the G23 +protocol stack C sources from the LoCosto version will likely lead to problems, +or least more hassle than it's worth, hence I decided to bite the bullet and +use the *.{aim,sap} and *.[mp]df files from LoCosto. + +Using the LoCosto versions of *.[mp]df, I ran ccdgen.exe (version 2.5.5A, set +to generate mtx tables in the new format) with two different fflags.h +configurations: "locosto" (unchanged from the "peek" find) and "conservative". +The latter is a configuration in which every existing option is set to disabled: +I took fflags-locosto.h and changed every #define to #undef. + +The symlink is currently set to compile the GSM fw using the "conservative" +version of cdginc headers. As I said above, I'll proceed with this +configuration until and unless I hit a problem, and then re-evaluate this +course if need be. + +===================== +ccdgen binary problem +===================== + +We only have a ccdgen.exe binary, but no source. The Winblows binary in +question happens to run fine under Wine, at least on my Slackware machine, but +needless to say, asking every FreeCalypso user who wishes to compile her own +GSM fw from source to run a sans-source Weendoze binary under Wine and pray +that it works is not an attractive proposition. Therefore, as a workaround I +have checked the generated cdginc files into Hg as if they were source files. + +Of course this workaround is not a proper solution either, but it is the best +we can do for the time being, until and unless someone either finds the missing +source for ccdgen or figures out its logic and writes a from-scratch functional +replacement.