An introduction to JET/Hub Jan 2016
This is the start of a series describing “JET v4”, and in particular the “hub” subsystem. JET is a system which is intended to bring together all the home monitoring and automation tasks here at JeeLabs, for the next 10 years…
As mentioned before - and as its name indicates - “JET/Hub” is the centrepiece of this architecture. It’s the switchboard which sits between the various (and changing) components of the system, including: from serial ports, bridges to Wireless Sensor Networks, and directly-attached I/O hardware, to software-based functionality, e.g. real-time data collection of sensor readings, sending out commands to control lights, heating, and appliances in the house, calculating trends and other statistics, presenting this information in a web browser. And eventually also the management of potentially large rule sets to guard and even control certain actions in and around the house. The hub is where everything passes through, it’s also the autonomous “always-on” part of JET.
That’s quite a mouthful. In a nutshell: JET is for home monitoring and automation, whereby JET/Hub takes care of orchestrating the entire system.
To expand a bit on an earlier set of requirements, the hub should:
- make minimal demands on the hardware and run well on a small Linux board
- allow continuous development on a “live” system, without constant restarts
- be flexible enough to support a very wide range of home sensors and actuators
- make installation very straightforward, and likewise for subsequent updates
- support remote management and avoid the need to log in and enter commands
Yet Another HAS
There are many Home Automation Systems. Even at JeeLabs there have been several major iterations: JeeMon/JeeBus (Tcl), HouseMon 0.7 (Node.js, with “Briqs”), HouseMon 0.8 (Node.js, using “Primus”), and HouseMon 0.9 (with Go, using “Flow”). But hey, that’s just the JeeLabs code - there are tons (hundreds?) of other OSS automation projects on GitHub alone.
The reason for JET is really that nothing else seems to fit the (perhaps not totally conventional) requirements here at JeeLabs:
- it should be extremely lightweight, to run on a small Linux system with very low energy demands (50W for an always-on system still wastes 430 KWh each year)
- it should support continuous development, since the system needs to remain useful - and usable - for at least a decade, and preferably well beyond that
- it should make few assumptions in the core about technology, since the needs and available solutions are bound to change drastically over time
JET is not about making the flashiest choices today - it’s about picking a limited set of design guidelines and adopting “minimally viable” conventions. And based on that: implementing a small core to keep around for a long time.
JET’s design choices
It’s time to cut through the vague hand-waving fog, and make some hard choices:
- all subsystems of JET will run as separate processes: 1 “hub” and N “packs”
- the hub stays running 24 / 7, and manages the lifetimes of all the JET packs
- all communication between hub and packs goes through an MQTT server (broker)
- MQTT’s “topics” lend themselves well to designing a clear naming hierarchy
- the payload of messages going through MQTT should be JSON in most cases
The MQTT server of choice right now is Mosquitto, which is open source, highly standardised, and well-tested. Furthermore, it scales well and it’s widely available on all major platforms.
The hub subsystem is implemented in the Go language, which is also open source, portable, in active development yet very robust, and extremely well-suited for network-oriented applications. Being statically compiled (yet supporting flexible dynamic typing) means that the hub code can be built and installed as executable with zero external package dependencies.
The major functions included in the hub (as opposed to being implemented in JET packs) are:
- simple communication with the MQTT server/broker
- connecting to serial ports (incl. USB) to capture data and emit commands
- a built-in fast and scalable key-value data store
- a built-in web server with websockets, for efficient web browser access
- installing / removing JET packs, and a way of starting and stopping them
- registration and discovery services to let packs work together
- a robust upgrade mechanism for packs, as well as for the hub itself
- supervising all running packs, with configurable automatic restart-on-failure
- watchdogs to detect system anomalies and report / act upon them
- basic system and error logging to stdout/stderr
- everything else is configurable, and can evolve massively over time…
- as long as the hub can launch it, a pack can be built in any way you like
A brief attempt to include the MQTT broker inside the hub as well has been abandoned for now, since the SurgeMQ package is not quite ready for prime time. For now, JET will rely on both the hub and the MQTT broker running alongside each other as separate processes.
Oh, and by the way: this will be called “JET version 4.0” … gotta start somewhere, right?