Connecting to serial ports Jan 2016

The main mechanisms for communicating between devices around the house in JET are via serial communication and via Ethernet (LAN or WLAN) - often in the form of USB devices acting as virtual serial interfaces. For simple radio modules used for a Wireless Sensor Network, we can then use a USB or WiFi “bridge”.

Other (less common) options are: I2C and SPI hardware interfaces, and direct I/O via either digital or analog “pins”. For these special-purpose interfaces, there is the WiringPi library, which is often also ported to other boards than the Raspberry Pi for which it was originally conceived. In this case, a small C program can be used as bridge to MQTT.

Network connections are simple in Linux, and in particular in Go, and will not be covered here right now. Besides, with MQTT in the mix, network access is essentially solved if the other end knows how to act as MQTT client. Getting data into JET via the network is easy - even if some massaging is needed, it can be done later by including some custom code for it in a JET pack.

Serial ports are a different story. There are many serial conventions in use, with all sorts of settings that you need to get just right to send messages across: baudrate, parity, stop bits, h/w or s/w handshake - it can be quite a puzzle. Just enter the man stty command on Linux to see how many different configuration choices have been added over time…

Fortunately, a few common serial interface conventions will cover the majority of today’s cases, when it comes to “hooking up” a serial device such as an USB-to-serial “dongle”. And Linux tends to have excellent support out of the box for all the different brands and vendors out there. All we need is some glue code in the hub, and we should be able to get serial data in and out.

And yet that’s just step one in this story. Welcome to the “interfacing to the real world” puzzle!

We also need to match the serial data-format choices of the device. Is it text? Is each line one message? Or if it’s binary data: how do we know where a message ends and the next one begins? There are so many different “framing” and other protocol conventions, that this is probably best handled in a custom pack - at least for more complex cases. But even then we need a serial driver which is able to pass all the information faithfully across to that JET pack.

Another area of concern is with the I/O pins other than the send and receive lines: do we need to connect any of the RTS, CTS, DTR, DSR pins? Do they need to be in a certain state?

Eventually, many of these use cases will need to be addressed. For now, let’s just focus on a basic subset and aim for the following scenario:

What about the actual serial data I/O?

This is where MQTT’s pub-sub can help a lot: we can subscribe to a fixed/known topic for each interface, and pass each incoming message to the serial port. The advantage over plain serial, is that any number of processes can do so - if more than one send at the same time, the output will get inter-mixed, but that’s fine as long as each message is a self-contained outbound “packet”.

On the incoming side, there are two uses cases:

  1. pick up each message and pass in along to anyone interested - this is the most natural mode for MQTT and matches exactly with its pub-sub semantics

  2. briefly claim the output while a client “takes control”, which it then relinquishes once done - this is useful for an “uploader”: switch the attached device to firmware-upgrade mode, send new firmware, after which normal operation resumes, with all listeners getting new incoming data

Here is a design, currently being implemented, which supports both modes:

Serial port open requests

Request format, in JavaScript notation:

  device: <path>            // the serial device to open
  baud: <baudrate>          // connection speed, default is 57600 baud
  init: [<commands...>]     // initialisation commands and settings
  sendto: <topic>           // the topic to forward incoming data to

Example (JSON):


New open requests implicitly close the serial port first, if previously open.

To close the serial port and not re-open it, we can send an empty message. This is not valid JSON, but will be recognised as a special “close-only” request.

Serial interface requests

Interface change requests are inside a JSON array and get processed in order of appearance:

(note: “¬” denotes a pin with inverted logic: “1” is inactive, “0” is active!)

Additional requests could be added later, e.g. to switch between text and binary mode, to set a receive timeout, and to encode/decode hex, base64, etc.

Power-up behaviour

While the above is sufficient to use serial ports, it does not address what happens on power-up or after a system restart. Ah, but wait… meet MQTT’s “RETAIN” flag:

In other words: to configure our system, we merely need to send the proper open requests for each serial port once - with the RETAIN flag set, i.e. the MQTT server now acts as persistent configuration store for each of these settings.

Other messages, without the RETAIN flag set, pass through as is - they won’t affect the storage of prior retained messages. Normal outbound data should therefore be published without the RETAIN flag. Likewise for interface change requests: they must be processed, but not stored.

To permanently open a serial port in a different way, we can simply send a new open request message, with RETAIN set, and replace the previous one.

Incoming data

The open request includes a sendto: field, which specifies the topic where every incoming message is sent. In the initial implementation, data is expected to come in line-by-line, and each line will be re-published to the given topic.

By using open requests without the RETAIN flag, we can play tricks and briefly re-open a serial port for a special case, with a different sendto value. Then, once ready, we simply re-open again with the original topic, and data will start getting dispatched as before.

As already mentioned, the above mechanisms are currently being implemented and will be included in the hub once the software is working and stable. For real code, see GitHub.

Weblog © Jean-Claude Wippler. Generated by Hugo.