JET and Forth, some thoughts Mar 2016

The JET project is about “creating an infrastructure for home monitoring and automation” (it’s actually considerably more, but this is a big-enough bone to chew on already…).

Note that JET is not about individual µCs or boards, it’s about managing an entire set of nodes, warts and all, and heterogenous from the start. JET is about bringing together (and bridging) lots of technologies, and about interfacing to existing ones as well. It’s also about evolution and long-term use - a decade or more - because redoing everything all the time is wasteful.

The JET infrastucture is not necessarily centralised, altough it will be in the first iterations: a “hub”, a variety of remote “nodes”, and browser access to use and administer it all. This is easy to map onto actual hardware, at least for a simple setup:

Nothing new so far, this story has not changed much in the past years, other than exploring different µC options and trying out some self-powered Micro Power Snitch ideas.

The hub is a recent introduction: a portable application written in Go, in combination with the Mosquitto MQTT server. It has been running here at JeeLabs for a few months now, dutifully collecting home monitoring data, mostly from room nodes, the smart meter, and the solar inverter. The hub itself does very little, but it provides a way to add “Jet Packs” to run arbitrary processes which can tie into the system.

The browser side of JET has not changed: it’ll continue to be written in JavaScript (ES6) and will most likely use ReactJS and PureCSS as foundations. A lot of software development time will end up going there - this is not different from any other modern web-based development.

The nodes have all been running Arduino IDE based C/C++ code, most of this is available in JeeLib on GitHub, as far as ATmega- and ATtiny-based JeeNodes are concerned. Some newer experimental code for ARM has been presented in the past year on the weblog, some for LPC8xx µCs, but recently more for STM32 µCs. That code can be found in the Embello repository, see for example the RF Node Watcher.

But that’s where things are about to change - drastically!

A new beginning

From now on, new work on remote nodes will be done in Forth. Since Mecrisp Forth has proven itself to be very stable and flexible, it’ll be flashed onto every node - very much like a fancy boot loader. This is equivalent to making each node speak Forth on its serial port, once and for all.

This approach has been chosen, because Forth (in particular Mecrisp Forth):

Do I have to learn Forth?

There are several possible answers to that question:

To explain that second answer: for trying out things written in Forth and made available on GitHub, you don’t need to program in Forth, you can just enter commands like “8686 42 rf-init”, “somevar @ .”, and such - easy stuff (once properly documented and explained!).

Is everything going to be in Forth?

Nooooo! - Forth is still merely an implementation language for little µCs. The plan is to use it to implement a compact dataflow engine for remote nodes, which will then present a “gadgets and circuits” model, somewhat like NoFlo, Node-RED, and Pure Data. All data-driven.

Once such a basic dataflow engine exists, we will have a considerably more abstract conceptual framework to build with and on. There will be gadgets to tie into actual hardware (pins, digital & analog I/O, timers, but also the RF driver), and gadgets to perform generic tasks (periodic execution, filtering, arithmetic, range checking, conditional execution, etc). These can then be combined into larger circuits, and sent to a node as definition of the behaviour of that node.

The reason why Forth looks like a perfect fit for this task, is that it allows growing a node’s functionality in small steps, once the Mecrisp core has been flashed into its flash memory. There will need to be a first layer to tie into the RFM69 radio modules (the RF69 driver already exists), and a way to robustly add and remove additional Forth code over the air. After that, we’ll need a dataflow core. Then, tons of “gadgets”, either coded in Forth or combined from existing ones.

At the end of the day/road/tunnel, Forth will end up being simply a low-level implementation language for manually coding only the bottom layers, with everything else generated from a visual diagram editor in the browser. The long-term goal is not to expose, but to bury Forth!

Yes, it will take a lot of work to get there - JET was never meant to be built in a day…

Weblog © Jean-Claude Wippler. Generated by Hugo.