Solution for high baud rates with FTDI adapters without kernel patch

Fri Sep 22 07:39:17 UTC 2017

Hello FreeCalypso community,

I now have what may be a solution to the problem of high baud rates
with FTDI adapters without kernel patches or other ugly hacks.

Short review for the newcomers: in a typical GSM chipset like the
Calypso all timing comes from a 13 MHz master clock (48 times the GSM
bit rate), and the Calypso UARTs are no exception - they are clocked
with the very same 13 MHz.  A UART clocked at 13 MHz can produce close
approximations to the standard PC baud rates (115200 and below), but
it can also produce what I call GSM baud rates of 203125, 406250 and
812500 bps - I call them GSM baud rates because they are the direct
product of a 13 MHz UART which is a GSM-ism.

These Calypso high baud rates of 203125, 406250 and 812500 bps do not
correspond to any standard PC baud rates.  Their use is entirely
optional, i.e., you can make full use of a Calypso device like our
FCDEV3B without ever using these non-standard high baud rates, but
transferring firmware images over 115200 baud gets painfully slow,
especially when you do it a lot, hence the ability to use the highest
Calypso baud rate of 812500 bps is highly desirable.

When Calypso UARTs (one or both of them) are connected to a PC or
laptop, this connection is usually made by way of USB, and there are
two types of USB-serial adapters which can support these non-standard-
to-a-PC high baud rates: CP2102 and the FTDI family.

CP2102 adapters have a baud rate table in a non-volatile EEPROM inside
the chip and the use of this EEPROM table is required by the chip (one
cannot set a baud rate that is not in the table), thus support for GSM
baud rates requires appropriate programming of this EEPROM.  Some
people see this EEPROM programming requirement as an inconvenience,
but for FreeCalypso this quirk has always worked in our favor instead:
the CP2102 adapters we work with (built into the Pirelli DP-L10 or in
a cable made by a professional GSM cable vendor like Sysmocom or
UberWaves) already have their EEPROM correctly programmed, and the
remapping performed by the EEPROM baud rate table means that we can
request B230400, B460800 and B921800 from termios without any extra
sweat, and they magically turn into the 203125, 406250 and 812500 baud
rates we want.

But FTDI adapters don't have such remapping magic, thus in order to
get 812500 bps with an FTDI adapter one needs to actually request
812500 bps from the userspace process somehow.  That last part has
been our biggest difficulty with GSM baud rates on FTDI adapters: as
it turns out, there does exist a sensible and fairly clean way to
request arbitrary serial baud rates under Linux in a driver-
independent manner, and has existed for about 10 y now, but its
existence appears to be a closely guarded secret, as I have only
discovered it earlier this week.

There exists an extremely ugly but fairly well-known method that
involves the TIOCSSERIAL ioctl, the ASYNC_SPD_CUST flag and setting
the desired baud rate not in the form of the natural bps integer, but
in the form of a divisor from a baud base.  This method is particularly
ugly when used with the ftdi_sio driver, as the latter driver does not
use the userspace-provided divisor directly, but instead performs a
double conversion first to an integer bps rate, and then to the real
divisor that goes into the chip.  Because the divisions aren't exact,
this double conversion needlessly increases the offset error on the
non-standard baud rates for which the hack is needed in the first
place.  OsmocomBB uses this method to get high baud rates with FTDI
adapters, and our dear community member Das Signal once produced an
unofficial patch for FC host tools that does the same, but this
ASYNC_SPD_CUST hack is just too ugly for me to use it myself or to
officially recommend it to others.  Furthermore, this ASYNC_SPD_CUST
method is absolutely not compatible with CP2102, hence the userspace
code needs to differentiate between CP2102 and FTDI adapters - more
ugliness.

But then I took a look at the ftdi_sio.c driver code in the current
Linux tree at kernel.org, and saw that the code that handles the
ASYNC_SPD_CUST hack is now preceded by this comment:

	/*
	 * Observe deprecated async-compatible custom_divisor hack, update
	 * baudrate if needed.
	 */

Hmm - this hack is now deprecated?  But what is then the proper, non-
deprecated way to request custom serial baud rates that aren't in the
predefined once-and-for-all termios Bnnn list?  Just prior to that
comment the code in the ftdi_sio driver calls the tty_get_baud_rate()
function, so I looked in there, and saw the following interesting bit:

#ifdef BOTHER
	/* Magic token for arbitrary speed via c_ispeed/c_ospeed */
	if (cbaud == BOTHER)
		return termios->c_ospeed;
#endif

Now I knew what to search for, and found other people's experiences:

https://www.downtowndougbrown.com/2013/11/linux-custom-serial-baud-rates/
https://stackoverflow.com/questions/12646324/how-to-set-a-custom-baud-rate-on-linux
https://www.toradex.com/community/questions/7445/custom-baudrate-on-vybrid-soc.html

As it turns out, Linux kernel support for arbitrary serial baud rate
setting in the form of bps integers directly from userspace (not
ASYNC_SPD_CUST divisors) has been implemented *and mainlined* about
10 y ago (in response to pressure from the embedded systems community,
no doubt), but glibc never caught up apparently - to this day, 10 y
later, if you wish to exercise this sensible and fairly clean
arbitrary serial baud rate setting capability of the Linux kernel, you
have to bypass standard libc termios facilities and use raw <asm/...>
Linux header files and raw ioctl calls instead, and deal with some
nasty include file conflicts in the process.  Portability goes out the
window, there is no official documentation or support anywhere for
this method, and as my experience shows, the very existence of this
capability is quite difficult to discover.

So what does this discovery mean for FreeCalypso?  It means that we
now have official support in FC host tools for the high baud rates of
203125, 406250 and 812500 bps that should work with both CP2102 and
FTDI adapters, with no need for the user to communicate the adapter
type to the tools and with no more need for kernel patches in the case
of FTDI.  More specifically:

* The serial port and baud rate handling code that has previously been
  triplicated between loadtools, rvinterf and fc-serterm has now been
  factored out into a common library called libserial.

* The current code in freecalypso-tools Hg repository has two versions
  of libserial: libserial-orig and libserial-newlnx.  The -orig version
  corresponds to what we had before: standard POSIX termios, no Linux-
  specific hacks, support for high baud rates depends on the existence
  of a magic genie somewhere below (CP2102 EEPROM or a patched ftdi_sio
  kernel driver) that remaps termios baud rates B230400, B460800 and
  B912800 to 203125, 406250 and 812500 bps, respectively.  The -newlnx
  version uses Linux-specific <asm/...> headers and raw ioctl calls
  instead of standard termios, and in the case of our high GSM baud
  rates, libserial-newlnx actually requests 203125/406250/812500 baud,
  no more B230400/B460800/B921600.

* The libserial symlink currently points to libserial-orig.  To try
  out the new code, you need to change this symlink in your local
  copy.  You also need to do a full make clean and a full recompile of
  the entire freecalypso-tools tree after changing this symlink, as
  the different versions of baudrate.h (included from loadtools
  outside of libserial itself) have different definitions of struct
  baudrate.

The new libserial-newlnx code is able to use the high GSM baud rates
with FTDI adapters (needed for FCDEV3B with FT2232D) with the standard
unpatched ftdi_sio kernel driver, and the same code still works with
CP2102 adapters (used in the Pirelli DP-L10 and in the official
FreeCalypso serial cables for Openmoko devices), at least on my system.
In the case of CP2102 adapters the cp210x kernel driver now sees the
userspace requesting 203125/406250/812500 bps instead of 230400,
460800 or 912600, but at least with the version of the driver in my
elderly linux-2.6.37.6 kernel the actual bits going out to the CP2102
chip remain the same, and everything still works.

The next necessary step is further testing by the FreeCalypso
community.  Before I can change the libserial symlink to point to
libserial-newlnx and make the new code official, I need to hear some
reports from other community members.  Does this libserial-newlnx code
compile on newer distros?  If it passes compilation, do the resulting
binaries work with the newer kernels?  Is the libserial-newlnx version
of fc-loadtool able to use every baud rate with both CP2102 and FTDI
adapters going through the standard unpatched drivers in recent
kernels?

This issue is getting my attention currently because I am trying to
get our FCDEV3B boards out in the world, and because our FCDEV3B
conveniently brings out both Calypso UARTs, not just one, working with
FCDEV3B practically requires an FT2232D (or FT2232H) adapter, not
CP2102.  (But CP2102 adapters still need to be fully supported because
we still need to support Openmoko and Pirelli targets.)  And the users
of our FCDEV3B boards with FT2232D adapters need to be able to use the
high baud rates without resorting to ugly hacks if possible - and now
that we know that it *is* possible, we need to get this new way
working solidly.

It should also be obvious that the new libserial-newlnx code is
absolutely specific to Linux.  I haven't heard of anyone running our
FC host tools on FreeBSD or illumos or any other alternative-to-Linux
OS, but if someone does wish to use an FCDEV3B with an FT2232D or
FT2232H adapter with, say, FreeBSD, then the answer should be obvious:
s/he will need to make an appropriately ported libserial-freebsd or
whatever, and preferably post a link to that code here so we can add
it to the official repo for the benefit of other users of the same OS.

Hasta la Victoria, Siempre,
Mychaela aka The Mother