--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/User-phone-tools	Sun Mar 11 06:32:00 2018 +0000
@@ -0,0 +1,373 @@
+FreeCalypso User Phone Tools are a new software addition to the FreeCalypso
+family.  These tools are programs that run on a Unix host computer such as a
+GNU/Linux PC or laptop and communicate with a FreeCalypso phone or modem via
+the standard AT command interface, rather than any of the formerly proprietary
+interfaces specific to TI's internal architecture.  The following tools are
+currently available:
+
+fcup-at		Issues an arbitrary AT command given on the command line.
+
+fcup-settime	Issues AT+CCLK command to the target to set its clock to the
+		host computer's notion of local time.
+
+fcup-smdump	Retrieves a dump of SMS records (received, sent or stored
+		messages) from the FC device's SMS storage (currently SIM
+		storage; ME storage may be implemented in the future),
+		optionally deleting them from the severely space-limited
+		SIM/ME storage afterward.
+
+fcup-smsend*	Tools for sending outgoing SMS from a host computer through a
+		FreeCalypso phone or modem and/or writing such outgoing SMS
+		into the FC device's SMS storage.
+
+fcup-smwrite	Debug and development tool: writes arbitrary message records
+		into the FC device's SMS storage (currently SIM storage) in any
+		of the possible 4 states, with arbitrary incoming or outgoing
+		SMS PDU content.
+
+Because these tools communicate with the target via standards-defined AT
+commands, in theory they ought to work with any AT-command-speaking 3GPP phone
+or modem and not just our own FreeCalypso.  However, experience has shown that
+in the case of the common proprietary implementations, practice does not match
+theory: when I (Mychaela) tried these same AT commands against a random
+off-the-shelf proprietary modem (Huawei E303 USB stick modem for 3G), the
+following problems were seen:
+
+* The essential AT+CMGL=4 command for retrieving the full set of SMS records
+  from SIM storage in PDU mode appears to be broken: all I got was a hang.
+  Its text mode counterpart AT+CMGL="ALL" produces incomplete output.
+
+* Qualcomm/Huawei's implementation of the AT command interface does not allow
+  AT+CSCS to be set to "HEX"; our fcup-smdump implementation uses this setting
+  so that the phonebook names returned along with SMS PDUs in the +CMGL
+  responses can be parsed reliably no matter what weird characters they might
+  contain.
+
+* Setting AT+CSCS to "8859-1" is not supported either; this setting is used by
+  our fcup-smsend and fcup-smsendmult tools when sending in text mode.
+
+* Sending outgoing SMS with fcup-smsend in PDU mode (which does not touch
+  AT+CSCS) works in that the message goes out, but the tool complains afterward
+  because the echo after the ^Z is different from what our tools expect.
+
+Because of these quirks, our FC User Phone Tools officially work only with our
+own FreeCalypso phones and modems, and are not expected to work against various
+proprietary implementations.  Let us not forget that the broken and buggy nature
+of the common proprietary implementations is the very reason why we need
+FreeCalypso in the first place.
+
+Target interface options
+========================
+
+Our fcup-* tools can communicate with the AT-command-speaking target in one of
+two ways:
+
+* The default is the standard AT command interface over a dedicated UART.  As
+  of this writing, the only FreeCalypso device that provides a full-featured AT
+  command interface of this kind is our FCDEV3B modem, but the ultimate goal of
+  the project is to build our own end user phone handset (a Libre Dumbphone)
+  that will also provide a full-featured AT command interface on its USB port
+  via a built-in CP2102 or FT232R chip.
+
+* As a dirty hack, one can run FreeCalypso GSM fw on some alien hw targets,
+  currently Motorola C139 and Pirelli DP-L10.  In this hacked-up configuration
+  there is no dedicated UART available for a standard AT command interface, but
+  there is a hack that allows a limited subset of AT commands to be passed over
+  the RVTMUX binary packet interface provided by the running FreeCalypso GSM fw.
+  Our fcup-* tools can work with such targets to a limited extent.
+
+The AT-over-RVTMUX mechanism was originally invented back in 2015 as a
+development aid, and was never intended for production use or to support any
+kind of end user functionality.  One of its limitations is that the strings
+that are sent to ATI via this interface are limited to 254 characters, whereas
+sending or writing SMS in hex format requires longer strings.  As a result, SMS
+sending and writing functionality via fcup-smsend* is limited when a crippled
+Motorola or Pirelli hw target is used instead of proper FreeCalypso hardware.
+
+All fcup-* tools take the following common command line options for selecting
+the AT command target interface:
+
+-B baud		Valid only when -p is also given; selects a different baud rate
+		than the default 115200 bps.
+
+-n		Dry run debug mode with no target interface at all: the AT
+		commands which would otherwise be sent to the target are simply
+		printed on stdout.
+
+-p ttyport	Names the serial port to be used to talk to the target.
+
+-R		Use the AT-over-RVTMUX interface instead of the standard AT
+		command interface over a dedicated UART.
+
+-X program	Use the specified external program as the AT target
+		communication back-end; read the source code for the details.
+
+-R and -p options interact as follows:
+
+Neither -R	The standard dedicated AT command interface is used;
+nor -p		FC_GSM_DEVICE= environment variable needs to be set
+		to point to the serial port.
+
+-p only		The standard dedicated AT command interface is used;
+		the serial port is named with the -p option.
+
+-R only		AT-over-RVTMUX interface is used; the fcup-* tool connects
+		to an already running rvinterf process.
+
+-R and -p	AT-over-RVTMUX interface is used; a new rvinterf process
+		is launched to talk RVTMUX on the specified serial port.
+
+Retrieving and decoding stored SMS
+==================================
+
+As of this writing, our current FreeCalypso GSM firmware supports only SIM
+storage for SMS, i.e., there is no working mechanism currently for storing SMS
+records (received and sent messages) in the phone's or modem's own flash file
+system.  The capacity of this SIM SMS storage is determined by the SIM issuer,
+but it is typically quite limited, on the order of 20 to 30 messages.
+
+The model adopted for FreeCalypso is that incoming (and possibly saved outgoing)
+messages initially accumulate in the SIM storage as they come in, and then the
+user periodically transfers them to her larger host computer, simultaneously
+deleting them from the SIM storage to reclaim the limited space.  The retrieval
+of stored SMS from FreeCalypso GSM devices is accomplished with our fcup-smdump
+utility; unlike SMS sending/writing, this operation works exactly the same
+whether the FC GSM device offers a full-featured AT command interface or only
+AT over RVTMUX.  SMS retrieval is always done in PDU mode, and the output from
+fcup-smdump contains raw SMS PDUs in the form of long hex strings.  A separate
+utility called sms-pdu-decode then does what its name says.
+
+The intended mode of usage is something like this:
+
+fcup-smdump -d >> long-term-sms-log
+
+The -d option to fcup-smdump tells it to delete the retrieved messages from the
+SIM or future ME storage; this option should only be used when the output is
+redirected into some kind of longer-term storage.  In the above model the file
+named long-term-sms-log becomes what its name says as new messages retrieved
+from the FC GSM device get added to it; the format will look like this:
+
+Received message:
+XXXXXX...
+
+Received message:
+XXXXXX...
+
+Sent message:
+XXXXXX...
+
+Stored unsent message:
+XXXXXX...
+
+Received message:
+XXXXXX...
+
+Each of the "XXXXXX..." lines will be a long hex string giving an SMS PDU.  The
+idea is that the complete record of all received and sent messages should be
+stored on the user's big computer in raw PDU form, rather than decoded, and the
+decoding utility sms-pdu-decode should be invoked by the user (with the message
+log file as input) as needed for reading these messages.
+
+The message decoding utility sms-pdu-decode does its best to decode and show
+everything without dropping any bits: in addition to the actual decoded message
+characters and the From/To address (the "end user" content of the message), it
+decodes and shows the SC address, the first octet, the MR octet for outgoing
+messages, PID and DCS octets, the SC timestamp or the validity period fields,
+and the UDH bytes if present.  However, some bits can still be lost in the
+decoding, which is why it is important to archive messages in the raw PDU form:
+
+* Padding bits used to round the From/To address and septet-based user data to
+  an octet boundary and to round any UDH to a septet boundary are not decoded.
+
+* If the user data portion of the message is 8-bit or compressed data (per the
+  DCS octet), it is shown as a raw hex dump, which is lossless, but if it is
+  GSM7 or UCS-2 text (GSM 03.38 character encodings), the characters are
+  converted to the user's character set (plain ASCII only by default) for
+  display, and some characters may not be displayable.
+
+Character sets and encodings
+----------------------------
+
+By default, sms-pdu-decode only emits 7-bit ASCII characters in its output; any
+GSM7 or UCS-2 characters which fall outside of this plain ASCII repertoire are
+displayed as the '?' error character and the presence of such decoding errors
+is indicated in the Length: header.  This conservative default behaviour can be
+modified as follows:
+
+-e option extends the potential output character repertoire from 7-bit ASCII to
+8-bit ISO 8859-1.  Any 8859-1 high characters are emitted as single bytes,
+i.e., are NOT encoded in UTF-8 - this option is intended for non-UTF-8
+environments.
+
+-u option extends the potential output character repertoire to the entire Basic
+Multilingual Plane of Unicode, and changes the output encoding to UTF-8.
+
+-h option causes the user data portion of every message to be displayed as a
+raw hex dump; in the case of GSM7-encoded messages, this hex dump shows the
+unpacked septets.
+
+Composing and sending outgoing SMS
+==================================
+
+When used with a FreeCalypso GSM device that offers the full AT command
+interface (currently only the FCDEV3B modem), the primary SMS sending/writing
+tool fcup-smsend offers the following capabilities:
+
+* Sending outgoing messages in either GSM7 or UCS-2 encoding;
+* Sending either single or long (concatenated) SMS;
+* Message body input in ASCII, ISO 8859-1 or UTF-8;
+* Message body input either on the command line or on stdin;
+* Any messages sent through this tool (single or concatenated) may be
+  multiline, i.e., may contain embedded newlines;
+* Messages sent in GSM7 encoding can contain ASCII characters [\]^ and {|}~
+  - the tool is smart enough to do the necessary escape encoding.
+
+The default and preferred AT command interface mode for sending/writing SMS is
+PDU mode, which works great when the GSM device provides a proper AT command
+interface.  However, when a message of maximum or near-maximum length is being
+submitted to the modem in PDU mode, the hex string that needs to be sent is
+longer than what the crippled AT-over-RVTMUX mechanism can handle, thus if you
+are using crippled Motorola or Pirelli hardware, you need to give the -t option
+to fcup-smsend or fcup-sendmult, telling these tools to use text mode instead
+of PDU mode on the AT command interface.  In this text (-t) mode the following
+restrictions apply:
+
+* Only single SMS can be sent, not concatenated;
+* Only GSM7-encoded messages can be sent, not UCS-2;
+* No multiline messages can be sent, i.e., no newlines in the message body;
+* ASCII characters [\]^ and {|}~ won't be sent correctly - GSM 07.05 text mode
+  drops them.
+
+But if you have to use FreeCalypso on crippled hardware, the -t option does
+allow you to send GSM7-encoded SMS of the full maximum length of 160 characters.
+If you attempt to use PDU mode (no -t option) with an AT-over-RVTMUX back-end
+(-R option), the send or write operation will fail if the generated PDU is
+longer than 127 octets; the length of the generated PDU depends not only on the
+message body length, but also on the length of the destination address.
+
+The invokation syntax is as follows:
+
+fcup-smsend [options] dest-addr [message]
+
+The destination address must be given on the command line; the address digits
+may be optionally followed by a comma and an address type byte, either decimal
+or hexadecimal with 0x prefix.  The default address type is 0x91 if the number
+begins with a '+' or 0x81 otherwise.  If the message body is given on the
+command line, it must be given as a single argument; if no message body argument
+is given, the message body will be read from stdin.  Any trailing newlines are
+stripped before SMS encoding.
+
+The following options are supported, in addition to the common target interface
+options listed earlier:
+
+-c		Enables concatenated SMS.  Concatenated SMS will be sent only
+		if the message body exceeds 160 GSM7 or 70 UCS-2 characters,
+		otherwise plain SMS will be sent whether -c is given or not -
+		but the -c option enables the possibility of sending
+		concatenated SMS.
+
+-C refno	Enables concatenated SMS like -c, but also explicitly sets the
+		concatenated SMS reference number to be used.  The number can
+		be either decimal or hexadecimal with 0x prefix.
+
+-q		Concatenated SMS quiet mode.  If -c is given without -q, the
+		tool prints a message on stdout indicating whether the message
+		was sent as single or concatenated, and in how many parts.
+		-q suppresses this additional output.
+
+-t		Use text mode instead of PDU mode on the AT command interface.
+		This option is incompatible with -c and with -U, and introduces
+		other restrictions listed above.
+
+-u		By default, if the message body input contains any 8-bit
+		characters, they are interpreted as ISO 8859-1.  With -u they
+		are interpreted as UTF-8 instead.  This option is only relevant
+		for GSM7 output encoding, and it is implemented by converting
+		the input first from UTF-8 to 8859-1, and then from 8859-1 to
+		GSM7 - thus all UTF-8 input characters must fall into the
+		8859-1 repertoire, and it is not currently possible to send
+		GSM7-encoded messages containing the few Greek letters or the
+		Euro currency symbol allowed by GSM 03.38 encoding.
+
+-U		Send message in UCS-2 encoding instead of GSM7.  Any 8-bit
+		characters in the message body input are interpreted as UTF-8,
+		and the entire Basic Multilingual Plane of Unicode is allowed.
+
+-w		By default the outgoing message is sent out on the GSM network
+		with the AT+CMGS command.  With this -w option, the message is
+		first written into SIM or future ME SMS storage with AT+CMGW,
+		then sent out on the GSM network with AT+CMSS.
+
+-W		Write only, not send: the message is written into storage with
+		AT+CMGW and no further action is taken.  The modem's +CMGW:
+		responses with message storage indices are forwarded to stdout.
+		With this option the destination address argument can be a null
+		string or omitted altogether.
+
+Concatenated SMS reference numbers
+----------------------------------
+
+Every concatenated SMS transmission needs a reference number, and this number
+needs to increment from one concatenated SMS to the next, to help message
+recipients sort out which is which.  If the reference number is not given
+explicitly with -C, fcup-smsend creates (opens with O_RDWR|O_CREAT) a file
+named .concat_sms_refno in the invoking user's $HOME directory; automatically
+incrementing reference numbers are maintained in this file.  The initial seed
+is an XOR of all bytes of the current time returned by gettimeofday(2),
+followed by simple linear incrementing; these reference numbers do not need to
+be random in any kind of cryptographically secure sense.
+
+fcup-smsendmult
+===============
+
+As an alternative to sending concatenated SMS, one can use the fcup-smsendmult
+utility to send several single (no UDH) messages in one batch.  This utlity
+supports both text and PDU modes (PDU mode is still the preferred default when
+it can be used), and when PDU mode is used, it supports both GSM7 and UCS-2
+output encodings just like fcup-smsend.  The messages to be sent are read from
+stdin, and each input line produces a new message.
+
+The entire batch of messages can be sent to a single recipient, or each message
+in the batch can have its own individual destination address.  If the
+destination address is given on the command line, each input line read from
+stdin is just a message body; if no destination address is given on the command
+line, each input line must have the following format:
+
+<dest addr><white space><message body>
+
+-t, -u, -U, -w and -W command line options are unchanged from fcup-smsend.
+
+If you have to use FreeCalypso on crippled hardware, fcup-smsendmult -t can be
+a viable alternative to sending concatenated SMS, as each message in the batch
+can be up to the maximum limit of 160 characters.
+
+fcup-smsendpdu
+==============
+
+This utility sends out SMS PDUs that have been prepared externally; it only
+works in PDU mode, which limits its usefulness to high-end FreeCalypso hardware
+with a full AT command interface.  The PDUs to be sent out are read from stdin,
+one long hex string PDU per line; one can send either a single message or a
+batch.  Because the destination address and all content details are encoded in
+the PDU, the tool does not care if the messages are going to the same recipient
+or to different recipients, nor does it care if they constitute a concatenated
+SMS transmission or not.  -w and -W options work the same way as in fcup-smsend
+and fcup-smsendmult.
+
+fcup-smwrite
+============
+
+This utility is a debug and development tool; it differs from fcup-smsendpdu in
+the following ways:
+
+* fcup-smsendpdu can send messages out with AT+CMGS, write them into memory
+  with AT+CMGW, or do a write-then-send sequence (-w option) with AT+CMGW
+  followed by AT+CMSS.  fcup-smwrite only issues AT+CMGW commands.
+
+* fcup-smwrite passes a second argument to AT+CMGW that sets the message state
+  to any of the possible 4 values; fcup-smsend* -W put them in the "stored
+  unsent" state.
+
+* The input to fcup-smsendpdu is just PDU hex strings; the input to
+  fcup-smwrite needs to have the same format as fcup-smdump output in order to
+  indicate what state each message should be written in.