Computing stuff tied to the physical world

It’s all about MOM

In Software on Dec 17, 2012 at 00:01

Home monitoring and home automation have some very obvious properties:

  • a bunch of sensors around the house are sending out readings
  • with actuators to control lights and appliances, driven by secure commands
  • all of this within and around the home, i.e. in a fairly confined space
  • we’d like to see past history, usually in the form of graphs
  • we want to be able to control the actuators remotely, through control panels
  • and lastly, we’d like to automate things a bit, using configurable rules

In information processing terms, this stuff is real-time, but only barely so: it’s enough if things happen within say a tenth of a second. The amount of information we have to deal with is also quite low: the entire state of a home at any point in time is probably no more than perhaps a kilobyte (although collected history will end up being a lot more).

The challenge is not the processing side of things, but the architecture: centralised or distributed, network topology for these readings and commands, how to deal with a plethora of physical interfaces and devices, and how to specify and manage the automation rules. Oh, and the user interface. The setup should also be organic, in that it allows us to grow and evolve all the aspects of our system over time.

It’s all about state and messages: the state of the home, current, sensed, and desired, and the events which change that state, in the form of incoming and outgoing messages.

What we need is MOM, i.e. Message-oriented middleware: a core which represents that state and interfaces through messages – both incoming and generated. One very clean model is to have a core process which allows some processes to “publish” messages to it and other to “subscribe” to specific changes. This mechanism is called pubsub.

Ideally, the core process should be launched once and then kept running forever, with all the features and functions added (at least initially) as separate processes, so that we can develop, add, fix, refine, and even tear down the different functions as needed without literally “bringing down the house” at every turn.

There are a couple of ways to do this, and as you may recall, I’ve been exploring the option of using ZeroMQ as the core foundation for all message exchanges. ZeroMQ bills itself as “the intelligent transport layer” and it supports pubsub as well as several other application interconnect topologies. Now, half a year later, I’m not so sure it’s really what I want. While ZeroMQ is definitely more than flexible and scalable enough, it also is fairly low-level in many ways. A lot will need to be built on top, even just to create that central core process.

Another contender which seems to be getting a lot of traction in home automation these days is MQTT, with an open source implementation of the central core called Mosquitto. In MOM terms, this is called a “broker”: a process which manages incoming message traffic from publishers by re-routing it to the proper subscribers. The model is very clean and simple: there are “channels” with hierarchical names such as perhaps “/kitchen/roomnode/temperature” to which a sensor publishes its temperature readings, and then others can subscribe to say “/+/+/temperature” to get notified of each temperature report around the house, the moment it comes in.

MQTT adds a lot of useful functionality, and optionally supports a quality-of-service (QoS) level as a way to handle messages that need reliable delivery (QoS level 0 messages use best-effort delivery, but may occasionally get dropped). The “retain” feature can hold on to the last message sent on each channel, so that when the system shuts down and comes back up or when a connection has been interrupted, a subscriber immediately learns about the last value. The “last will and testament” lets a publisher prepare a message to be sent out to a channel (not necessarily the same one) when it drops out for any reason.

All very useful, but I’m not convinced this is a good fit. In my perception, state is more central than messages in this context. State is what we model with a home monitoring and automation system, whereas messages come and go in various ways. When I look at the system, I’m first of all interested in the state of the house, and only in the second place interested in how things have changed until now or will change in the future. I’d much rather have a database as the centre of this universe. With excellent support for messages and pubsub, of course.

I’ve been looking at Redis lately, a “key-value store” which is not only small and efficient, but which also has explicit support for pubsub built in. So the model remains the same: publishers and subscribers can find each other through Redis, with wildcards to support the same concept of channels as in MQTT. But the key difference is that the central setup is now based on state: even without any publishers active, I can inspect the current temperature, switch setting, etc. – just like MQTT’s “retain”.

Furthermore, with a database-centric core, we automatically also have a place to store configuration settings and even logic, in the form of scripts, if needed. This approach can greatly simplify publishers and subscribers, as they no longer need local storage for configuration. Not a big deal when everything lives on a single machine, but with a central general-purpose store that is no longer a necessity. Logic can run anywhere, yet operate off the same central configuration.

The good news is that with any of the above three options, programming language choice is irrelevant: they all have numerous bindings and interfaces. In fact, because interconnections take place via sockets, there is not even a need to use C-based interface code: even the language itself can be used to handle properly-formatted packets.

I’ve set up a basic installation on the Mac, using Homebrew. The following steps are not 100% precise, but this is more or less all that’s needed on Windows, MacOSX, or Linux:

    brew install node redis
    npm install -g express
    redis-server               (starts the database server)
    express                    (creates a demo app)
    npm install connect-redis
    node app.js                (starts the demo app on port 3000)

There are many examples around the net on how to get started, such as this one, which is already a bit dated.

Let’s see where this leads to…

  1. I’ve spent man-years fooling with MOM implementations. My company’s internal data acquisition infrastructure uses MQTT (arduino, python, C, and php clients). At our testing peak, our mosquitto server (running ubuntu 10.04 on 1.6GHz single core) was processing and persisting about 180,000 key/value messages/day.

    For our worldwide field deployments, I selected running STOMP transport within Apache ActiveMQ, http://activemq.apache.org/stomp.html.

    Its easy to collect data and distribute it when you’re small. When you want to induce authentication/authorization into the data services mix, mosquitto management will become troublesome.

    Having Camel and Esper in the backend container really gives you enterprise level deployment. Plus all the high-availability stuff from ActiveMQ. Its a big topic and I’m just touching the surface here.

    Good luck and keep us apprised of your progress.

    • The rate you describe is about 2 messages per second – not too far from what a busy home could have, I expect. But the rest seems a bit higher end than needed in this context – although requirements may differ, I don’t really intend to send out all (or any, in fact) of my data outside the LAN here. A Raspberry Pi seems like a suitable baseline for such a setup, and should be able to easily handle the load with any of the approaches described above.

  2. What I like about ZMQ is the simple on-wire transport protocol. It should be quite easy to send ZMQ messages using a JeeNode with an Ethernet board. There is also an example about how to use ZMQ for a distributed key-value store: http://zguide.zeromq.org/page:all#Representing-State-as-Key-Value-Pairs

    Redis looks interesting too. Do you have any idea if it’ll ever be possible to implement Redis on a ATmega?

  3. I cannot disagree that message oriented programs might be a serious approach for automation. another usual approach used in PLC is the concept of cycle, loop in which we have multiple basic statements like { if Input then output} the key benefit I see is the reliability of the code, you always pass trough this test and then whats going to happen to your Output is very redictable, right ? with MOM you go in an “on exception basis” type of approach.

    in a recent thinking on how to replace my own home automation stuff, I came to the idea of bringing easy multitasking and IO handling by providing a “comprehensive” scheduler library and IO management. this is a real alternative to messages. you create a task which deals with some IO and you can rely on its behaviour, because its writeen as a loop with limited equation somehow. I m thinking to introduce message in it as well for cross task talking. for those who like to experience this, let me share the playground url for it http://arduino.cc/forum/index.php/topic,137801.0.html I d be honored if you like to give your feedback on it :) keep us posted on MOM !

    • Thanks for the pointer to SCoop – which looks like an RTOS to me. I’m looking into RTOS’es as well (yeah, many things to keep me off the streets in these cold winter days), and there definitely are a couple which can scale down to ATmega size µC’s. On remote notes, I’m hesitant to go there as long as something like the Scheduler class in JeeLib can be used, because of the ultra-low power needs. While it can be done with an RTOS, I suspect that it’s a bit harder to accomplish this – which is why the roomNode code uses the scheduler. On central nodes and anything with permanent power and an ATmega in it, RTOS’es such as SCoop sound like a great field to explore.

      Note that I’m not suggesting to use MOM on wireless nodes, just on the Linux/etc system used to manage all the incoming and outgoing data.

  4. Redis should also be possible on a JeeNode (see: http://redis.io/topics/protocol). There is also a nice introduction here: http://openmymind.net/redis.pdf

    • It should be feasible to send a packet from a µC – it’s quite similar to Bencode on the wire: no total byte count to send up front, so you can send a fixed prefix (for the key), the data value (with its byte cound), and a fixed suffix.

  5. Seeing as you investigating MQTT, what are you thoughts on MQTT-S for sensor node network?

    • Thanks for the tip – I’ve looked through the PDF, but at 12 KB per node to implement it, it’s really an order of magnitude too complex for me. I couldn’t use it for JNµ’s, for example. And to be honest, I don’t quite see the point of bringing a MOM infrastructure all the way into a WSN – other than perhaps as a gateway on a central JeeLink.

  6. For my home automation system I’m using Mosquitto,a couple of bridges and an application for logging and mapping.

    Every bridge posts to a subtopic of the ‘/raw/’ root topic. For example, a Jeenode connected to my smart meter posts every 10 seconds to /raw/jeelabs/212/3/{kwh_normal,kwh_low,current_tariff,etc}.

    The map&log application listens to every sub-topic of /raw/ and maps the raw values to one or more location specific topics; for example: - /raw/jeelabs/212/3/kwh_normal is mapped to /house/P1/kwh_normal

    • /raw/jeelabs/212/2/temperature is mapped to /house/livingroom/temperature

    By doing so I remove the tight coupling between a sensor and the location of the sensor. When I move a roomnode from the livingroom to the kitchen, it’s historic data doesn’t move with it.

  7. Here is the implementation MQTT-S for Jeenode end JeeLink http://ab-log.ru/forum/viewtopic.php?f=1&t=383

  8. JeeNodes, raspberries, javascript/node.js, database – i guess redis is good although i am a bit fond of sqlite, and events. I like where this is going. Now throw in a process to drive user defined state machines with the events to control “resources” and it’s done. I need nothing else :-)

  9. You might have mixed up MQTT’s retain with persistence. The broker will keep a message in memory that’s published with the retain flag on. If a new subscription is established on the topic, the last retained message on this topic will be sent out immediately. A power loss will however cause retained messages to be lost. Mosquitto (and other MQTT brokers) do offer optional persistence much like Redis’ RDB snapshots.

    Re state being more central than messages: I wonder if putting a DB at the core of HA is really the cure. Take a switchable, Jeenode-enabled plug for example. You could switch the plug on/off locally on the plug through a button or remotely over the air. So where does the state of the plug reside? If you turn on the plug locally, but there’s a messaging hick-up reporting the state change to the Redis/MQTT/… store, is the plug’s state on or off? Or both? Furthermore, if your Redis node should go through a restart before consistency has been re-established, which state should prevail? Keeping state in at least two separate locations means you’ll have to deal with the CAP theorem and eventual consistency. A sensor network could be regarded as a distributed data (state) store on its own, even without an explicit store sitting in the middle.

    • Hello Bart – thx for clearing up retain vs state in MQTT. Yes, the “Heisenberg-like” implications of the CAP theorem are inevitable. State in the software can only be an approximation – with desired, known, and actual state all differing at times.

      For sensors such as temperature, the known state lags the actual state. Whether through packet loss, redis snapshot delays, or heck, even packet transmission delays. I don’t see problems with it, even across power outages.

      For actuators such as switches, simple unchecked one-way packets are not good enough when changed centrally, but ACKs can handle that. For remotely operated switches, it becomes the same case as temperature (“sensing the state”), and although things may lag, they will eventually lead to consistent state.

      As for Redis, it has snapshots which lag, but it can also write out all changes to disk as they occur. That still leaves room for lost state if power goes down, but the gap closes and you could even use a UPS or a backup battery to deal with that.

      A few years ago, a group of students built a home automation prototype called “JeeBus”, which used distributed state (not messages) as its central design. The power of being able to create rules based on past, current, and desired state has convinced me that it’s an excellent way to model “the home” (and other contexts). In the past, I built several large scale projects (using a Tcl package called Tequila) which all used this same approach and again, having state turned out to be really useful.

      State and messages are really two sides of the same coin: distance can only be overcome with messages, and a model of the state is what what you get once you store message payloads in variables or a database.

  10. Hi Jean-Claude!

    Indeed. Messages as a transport for state when shared memory is not an option. With all the complexity that it entails.

    What I find peculiar about Redis is that they did not integrate snapshotting (RDB) and journaling (AOF) from the outset. You keep a journal of state changes since the last snapshot. If a new snapshot is taken, flush the journal. Just like file systems. You just need the option to turn each or both of them off as a speed vs. consistency knob.

    Do you have some pointers for reading up on the concepts behind distributed state and how this was tackled in JeeBus? I need a pastime for these cold, dark winter days.

    • Sorry, JeeBus is no longer on-line. Got dropped during the last Redmine upgrade. Let me add that it was mostly intended as prototype, and as university “class assignment”.

      Distributed state is all around us – even browsers trying to keep up with info on a central server are an example of that, really. So it’s not even just about state distributed across a WSN and a central information processing unit – similar issues will pop up with browsers as control panel for a home automation system, I expect.

      The way I see it, we need to maintain a “model of the home” (i.e. its state) in one spot, and have everything else act on it. That spot could be a Redis store, running on a dedicated always-on Linux box. With WSN’s and other devices feeding it with updates, and browsers displaying the state and offering a way to also make changes (i.e. turn on a light). A secondary process would be a rule engine which “invents” state changes based on rules and the state it currently sees.

      None of this needs to be 100% centralised, in that state could be segmented over different boxes, or one box could act as backup for another one – although in the home, I suspect that people just want to keep one unit going (perhaps battery-backed). And then expand it ad infinitum – both wired and wirelessly.

  11. A lot of the concepts discussed here (message bus, state) have been realized in OpenHAB (http://code.google.com/p/openhab/wiki/Architecture), but I realize that it has a bigger footprint. Nevertheless I am running it on my RaspberryPi (the 256Mb version).

Comments are closed.