Some things are quite surprising in JavaScript / CoffeeScript:

    $ coffee
    > '1'+2
    '12'
    > '1'-2
    -1
    > 1 < null
    false
    > 1 > null
    true
    > 1 < undefined
    false
    > 1 > undefined
    false
    > a = [1,2,3]
    [ 1, 2, 3 ]
    > a.b = 4
    4
    > (x for x in a)
    [ 1, 2, 3 ]
    > (x for x of a)
    [ '0', '1', '2', 'b' ]
    > (x for x in 'abc')
    [ 'a', 'b', 'c' ]
    > (x for x of 'abc')
    [ '0', '1', '2' ]
    > 

It makes sense once you know it … but that’s the whole thing with being a newbie, eh?

That array-with-poperties behaviour is actually very useful, because it lets you create collections which can be looped over, while still offering members in an object-like manner. Very Lua-ish. The same can be done with Object.defineProperty, but that’s more involved.

For the full story on “array-like” objects, see this detailed page. It gets real messy inside – but then again, so does any other language. As long as its easy to find answers, I’m happy. And with JavaScript / CoffeeScript, finding answers and suggestions on the web is trivial.

On another note: Redis is working well for storing data, but there is a slight impedance mismatch with JavaScript. Storing nested objects is trivial by using JSON, but then you lose the ability to let Redis do a lot of nifty sorting and extraction. For now, I’m getting good mileage with two hashes per collection: one for key-to-id lookup, one for id-based storage of each object in JSON format. But I’m really using only a tiny part of Redis this way.

Drat, I can’t stop – this Node.js and AngularJS stuff sure is addictive! …

I’m a big fan of test-driven development, an approach where test code is often treated as more important than the code you’re writing to actually do the work. It may sound completely nuts – the thought of writing tons of tests which can only get in the way of making changes to your newly-written beautiful code later, right?

In fact, TDD goes even further: the idea is to write the test code as soon as you decide to add a new feature or write a new bug, and only then start writing the real stuff. Weird!

But having used TDD in a couple of projects in the past, I can only confirm that it’s in fact a fantastic way to “grow” software. Because adding new tests, and then seeing them pass, constantly increases the confidence in the whole project. Ever had the feeling that you don’t want to mess with a certain part of your code, because you’re afraid it might break in some subtle way? Yeah, well… I know the feeling. It terrifies me too :)

With test scaffolding around your code, something very surprising happens: you can tear down the guts and bring them back up in a different shape, because the test code will help drive that process! And re-writing / re-factoring usually leads to a better architecture.

There’s a very definite hurdle in each new project to start using TDD (or BDD). It’s a painful struggle to spend time thinking about tests, when all you want to do is write the darn code and start using it! Especially at the start of a project.

I started coding HouseMon without TDD/BDD on my plate, because I first want to expose myself to coding in CoffeeScript. But I’m already checking out the field in the world of Node.js and JavaScript. Fell off my chair once again… so many good things going on!

Here’s an example, using the Mocha and should packages with CoffeeScript:

And here’s some sample output, in one of many possible output formats:

I’m using kind of a mix of TDD’s assertion-style testing, and BDD’s requirement-style approach. The T + F tricks suit me better, for their brevity, but the “should.throw” stuff is useful to handle exceptions when they are intentional (there’s also “should.not.throw”).

But that’s just the tip of the iceberg. Once you start thinking in terms of testing the code you write, it becomes a challenge to make sure that every line of that code has been verified with tests, i.e. code coverage testing. And there too, Mocha makes things simple. I haven’t tried it, but here is some sample output from its docs:

On the right, a summary of the test coverage of each of the source files, and in the main window an example of a few lines which haven’t been traversed by any of the tests.

Tools like these sure make it tempting to start writing tests!

Ok, ok, I’ll stop.

Phew! You don’t want to know how many examples, frameworks, and packages I’ve been downloading, trying out, and browsing lately… all to find a good starting point for further software development here at JeeLabs.

Driving this is my desire to pick modern tools, actively being worked on, with an ecosystem which allows me to get up to speed, leverage all the amazing stuff people keep coming up with, and yet stay firmly on the simple side of things. Because good stuff is what you end up with when you weed out the bad, and the result is purity and elegance … as I’ve seen confirmed over and over again.

A new insight I didn’t start out from, is that server-side web page generation is no longer needed. In fact, with clients being more performant than servers (i.e. laptops, tablets, and mobile phones served from a small Linux system), it makes more and more sense to serve only static files (HTML, CSS, JS) and generic data (JSON, usually). The server becomes a file system + a database + a relatively low-end rule engine for stuff that needs to run at all times… even when no client is connected.

Think about the change in mindset: no server-side templating… n o n e !

Anyway – here are all the pieces I intend to use:

Node.js – JavaScript on the server, based on the V8 engine which is available for Intel and ARM architectures. A high-performance standards-compliant JavaScript engine.

The npm package manager comes with Node.js and is a phenomenal workhorse when it comes to getting lots of different packages to work together. Used from the command line, I found npm help to be great starting point for figuring out its potential.

SocketStream is a framework for building real-time apps. It wraps things like socket.io (now being replaced by the simpler engine.io) for bi-directional use of WebSockets between the clients and the server. Comes with lots of very convenient features for development.

AngularJS is a great tool to create very dynamic and responsive client-side applications. This graphic from a post on bennadel.com says it all. The angularjs.org site has good docs and tutorials, this matters because there’s quite a bit of – fantastic! – stuff to learn.

Connect is what makes the HTTP webserver built into Node.js incredibly flexible and powerful. The coin dropped once I read an excellent introduction on project70.com. If you’ve ever tried to write your own HTTP webpage server, then you’ll appreciate the elegance of this timeout module example on senchalabs.org.

Redis is a memory-based key-value store, which also saves its data on file, so that restarts can resume where it left off. An excellent way to cache file contents and “live” data which represents the state of a running system. Keeping data in a separate process is great for development, especially with automatic server restarts when any source file changes. IOW, the basic idea is to keep Redis running at all times, so that everything else can be restarted and reloaded at will during development.

Foundation is a set of CSS/HTML choices to support uniform good-looking web pages.

CoffeeScript is a “JavaScript dialect” which gets transformed into pure JavaScript on the fly. I’m growing quite used to it already, and enjoy its clean syntax and conciseness.

Jade is a shorthand notation which greatly reduces the amount of text (code?) needed to define HTML page structures (i.e. elements, tags, attributes). Like CoffeeScript, it’s just a dialect – the output is standard HTML.

Stylus is again just a dialect, for CSS this time. Not such a big deal, but I like the way all these notations let me see the underlying structure and nesting at a glance.

That’s ten choices, for Ten Terrific Technologies. Here are the links again:

• Node.js – https://github.com/joyent/node – S
• npm – https://github.com/isaacs/npm – S
• SocketStream – https://github.com/socketstream/socketstream – S+C
• AngularJS – https://github.com/angular/angular.js – C
• Connect – https://github.com/senchalabs/connect – S
• Redis – https://github.com/antirez/redis – S
• Foundation – https://github.com/zurb/foundation – C
• CoffeeScript – https://github.com/jashkenas/coffee-script – S+C (T)
• Jade – https://github.com/visionmedia/jade – C (T)
• Stylus – https://github.com/LearnBoost/stylus – C (T)

(S = server-side, C = client-side, S+C = both, T = translated on the server)

All as non-restrictive open source, and with all development taking place on GitHub.

It’s been an intense two weeks, trying to understand enough of all this to be able to make practical decisions. And now I can hardly wait to find out where this will take me!

Update – Bootstrap has been replaced by Foundation – mostly as a matter of taste.

After yesterday’s reading and decoding exploration, here’s some code which will happily play back my daily log files, of which I now have over 4 years worth …

Sample output:

As you can see, this supports scanning entire log files, both plain text and gzipped. In fact, JeeMonLogParser@parseStream should also work fine with sockets and pipes:

The beauty – again – is total modularity: both the real serial interface module and this log-replay module generate the same events, and can therefore be used interchangeably. As the decoders work independent of either one, there is no dependency (“coupling”) whatsoever between these modules.

Not to worry: from now on I won’t bore you with every new JavaScript / CoffeeScript snippet I come up with – just wanted to illustrate how asynchronous I/O and events are making this code extremely easy to develop and try out in small bites.

I wish you a Guten Rutsch ins neue Jahr and a safe, healthy, and joyful 2013!

I’m starting to understand how things work in Node.js. Just wrote a little module to take serial output from the RF12demo sketch and decode its “OK …” output lines:

Sample output:

This quick demo only has decoders for nodes 3 and 9 so far, but it shows the basic idea.

This relies on the EventEmitter class, which offers a very lightweight mechanism of passing around objects on channels, and adding listeners to get called when such “events” happen. A very efficient in-process pub-sub mechanism, in effect!

Here is the “serial-rf12demo” module which does the rest of the magic:

And that’s really all there is to it – very modular!

It’s all about dynamics, really. When software becomes so dynamic that you see the data, then all that complex code will vanish into the background:

This is the transformation we saw a long time ago when going from teletype-based interaction to direct manipulation with the mouse, and the same is happening here: if the link between physical devices and the page shown on the web browser is immediate, then the checkboxes and indicators on the web page become essentially the same as the buttons and the LED’s. The software becomes invisible – as it should be!

That demo from a few days back really has that effect. And of course then networking kicks in to make this work anywhere, including tablets and mobile phones.

But why stop there? With Tcl, I have always enjoyed the fact that I can develop inside a running process, i.e. modify code on a live system – by simply reloading source files.

With JavaScript, although the mechanism works very differently, you can get similar benefits. When launching the Node.js based server, I use this command:

    nodemon app.coffee

This not only launches a web server on port 3000 for use with a browser, it also starts watching the files in the current directory for changes. In combination with the logic of SocketStream, this leads to the following behavior during development:

• when I change a file such as app.coffee or any file inside the server/ directory, nodemon will stop and relaunch the server app, thus picking up all the changes – and SocketStream is smart enough to make all clients re-connect automatically
• when changing a file anywhere inside the clients/ area, the server sends a special request via WebSockets for the clients, i.e. the web browser(s), to refresh themselves – again, this causes all client-side changes to be picked up
• when changing CSS files (or rather, the Stylus files that generate it), the same happens, but in this case the browser state does not get lost – so you can instantly view the effects of twiddling with CSS

Let me stress that: the browser updates on each save, even if it’s not the front window!

The benefits for the development workflow are hard to overstate – it means that you can really build a full client-server application in small steps and immediately see what’s going on. If there is a problem, just insert some “console.log()” calls and watch the server-side (stdout) or client-side (browser console window).

There is one issue, in that browser state gets lost with client-side code changes (current contents of input boxes, button state, etc), but this can be overcome by moving more of this state into Redis, since the Redis “store” can just stay running in the background.

All in all, I’m totally blown away by what’s possible and within reach today, and by the way this type of software development can be done. Anywhere and by anyone.

Onwards!

As shown yesterday, the SocketStream framework takes care of a lot of things for real-time web-based apps. It’s at version 0.3 (0.4 coming up), but already pretty effective.

Here’s what I did to get the “ss-blink” demo app off the ground on my Mac notebook:

• no wallet needed: everything is either free (Xcode) or open source (the rest)
• make sure the Xcode command-line dev tools have been installed (gcc, make, etc)
• install the Homebrew package manager using this scary-looking one-liner:
  ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"
• using HomeBrew, install Node.js – brew install node
• that happens to include NPM, the Node Package Manager, all I had to do was add the NPM bin dir to my PATH (in .bash_profile, for example), so that globally installed commands will be found – PATH=/usr/local/share/npm/bin:$PATH

Not there yet, but I wanted to point out at this point that Xcode plus Homebrew (on Mac, other platforms have their own variants), with Node.js and NPM as foundation for everything else. Once you have those installed and working smoothly, everything else is a matter of obtaining packages through NPM as needed and running them with Node.js – a truly amazing software combo. NPM can also handle uninstalls & cleanup.

Let’s move on, shall we?

• install SocketStream globally – npm install -g socketstream
  (the “-g” is why PATH needs to be properly set after this point)
• install the nodemon utility – npm install -g nodemon
  (this makes development a breeze, by reloading the server whenever files change)
• create a fresh app, using – socketstream new ss-blink
• this creates a dir called ss-blink, so first I switched to it – cd ss-blink
• use npm to fetch and build all the dependencies in ss-blink – npm install
• that’s it, start it up – nodemon app.js (or node app.js if you insist)
• navigate to and you should see a boilerplate chat app
• open a second browser window on the same URL, and marvel at how a chat works :)

So there’s some setup involved, and it’s bound to be a bit different on Windows and Linux, but still… it’s not that painful. There’s a lot hidden behind the scenes of these installation tools. In particular npm is incredibly easy to use, and the workhorse for getting tons and tons of packages from GitHub or elsewhere into your project.

The way this works, is that you add one line per package you want to the “package.json” file inside the project directory, and then simply re-run “npm install”. I did exactly that – adding “serialport” as dependency, which caused npm to go out, fetch, and compile all the necessary bits and pieces.

Note that none of the above require “root” privileges: no superuser == better security.

For yesterday’s demo, the above was my starting point. However, I did want to switch to CoffeeScript and Jade instead of JavaScript and HTML, respectively – which is very easy to do with the js2coffee and html2jade tools.

These were installed using – npm install -g js2coffee html2jade

And then hours of head-scratching, reading, browsing the web, watching video’s, etc.

But hey, it was a pretty smooth JavaScript newbie start as far as I’m concerned!

There are tons of ways to make web pages dynamic, i.e. have them update in real-time. For many years, constant automatic full-page refreshes were the only game in town.

But that’s more or less ignoring the web evolution of the past decade. With JavaScript in the browser, you can manipulate the DOM (i.e. the structure underlying each web page) directly. This has led to an explosion of JavaScript libraries in recent years, of which the most widespread one by now is probably JQuery.

In JQuery, you can easily make changes to the DOM – here is a tiny example:

And sure enough, the result comes out as:

But there is a major problem – with anything non-trivial, this style quickly ends up becoming a huge mess. Everything gets mixed up – even if you try to separate the JavaScript code into its own files, you still need to deal with things like loops inside the HTML code (to create a repeated list, depending on how many data items there are).

And there’s no automation – the more little bits of dynamic info you have spread around the page, the more code you need to write to keep all of them in sync. Both ways: setting items to display as well as picking up info entered via the keyboard and mouse.

There are a number of ways to get around this nowadays – with a very nice overview about seven of the mainstream solutions by Steven Sanderson.

I used Knockout for the RFM12B configuration generator to explore its dynamics. And while it does what it says, and leads to delightfully dynamically-updating web pages, I still found myself mixing up logic and presentation and having to think about template expansion more than I wanted to.

Then I discovered AngularJS. At first glance, it looks like just another JavaScript all-in-the-browser library, with all the usual expansion and looping mechanisms. But there’s a difference: AngularJS doesn’t mix concepts, it embeds all the information it needs in HTML elements and attributes.

AngularJS manipulates the DOM structure (better than XSLT did with XML, I think).

Here’s the same example as above, in Angular (with apologies for abusing ng-init a bit):

The “ng-app” attribute is the key. It tells AngularJS to go through the element tree and do its magic. It might sound like a detail, but as a result, this page remains 100% HTML – it can still be created by a graphics designer using standard HTML editing tools.

More importantly, this sort of coding can grow without ever becoming a mix of concepts and languages. I’ve seen my share of JavaScript / HTML mashups and templating attempts, and it has always kept me from using JavaScript in the browser. Until now.

Here’s a better example (live demo):

Another little demo I just wrote can be seen here. More physical-computing related. As with any web app, you can check the page source to see how it’s done.

For an excellent introduction about how this works, see John Lindquist’s 15-minute video on YouTube. There will be a lot of new stuff here if you haven’t seen AngularJS before, but it shows how to progressively create a non-trivial app (using WebStorm).

If you’re interested in this, and willing to invest some hours, there is a fantastic tutorial on the AngularJS site. As far as I’m concerned (which doesn’t mean much) this is just about the best there is today. I don’t care too much about syntax (or even languages), but AngularJS absolutely hits the sweet spot in the browser, on a conceptual level.

AngularJS is from Google, with MIT-licensed source on GitHub, and documented here.

And to top it all off, there is now also a GitHub demo project which combines AngularJS on the client with SocketStream on the server. Lots of reading and exploring to do!

As I dive into JavaScript, and prompted by a recent comment on the weblog, it occurred to me that it might be useful to create a small list of books resources, for those of you interested in going down the same rabbit hole and starting out along a similar path.

Grab some nice food and drinks, you’re gonna’ need ’em!

First off, I’m assuming you have a good basis in some common programming language, such as C, C++, or Java, and preferably also one of the scripting languages, such as Lua, Perl, Python, Ruby, or Tcl. This isn’t a list about learning to program, but a list to help you dive into JavaScript, and all the tools, frameworks, and libraries that come with it.

Because JavaScript is just the enabler, really. My new-found fascination with it is not the syntax or the semantics, but the fast-paced ecosystem that is evolving around JS.

One more note before I take off: this is just my list. If you don’t agree, or don’t like it, just ignore it. If there are any important pointers missing (of course there are!), feel free to add tips and suggestions in the comments.

JavaScript

There’s JavaScript (the language), and there are the JavaScript environments (in the browser: the DOM, and on the server: Node). You’ll want to learn about them all.

• JavaScript: The Good Parts by Douglas Crockford
  2008, 1st ed, 176 pages, ISBN 0596517742
• JavaScript: The Definitive Guide by David Flanagan
  2011, 6th ed, 1100 pages, ISBN 0596805527

Videos: again by Douglas Crockford, there’s an excellent list at Stack Overflow. Going through them will take many hours, but they are really excellent. I watched all of these.

Don’t skim over prototypes, “==” vs “===”, and how “this” gets passed to functions.

Being able to understand every single notation in JavaScript is essential. Come back here if you can’t. Or google for stuff. Just don’t cut corners – it’s bound to bite you.

If you want to dive in really deep, check out this page about JavaScript and Scheme.

In the browser

Next on the menu: the DOM, HTML, and CSS. This is the essence of what happens inside a browser. Can be consumed in small doses, as the need arises. Simply start with the just-mentioned Wikipedia links.

Not quite sure what to recommend here – I’ve picked this up over the years. Perhaps w3schools this or this. Focus on HTML5 and CSS3, as these are the newest standards.

On the server

There are different implementations of JavaScript, but on the server, by far the most common implementation seems to be Node.js. This is a lot more than “just some JS implementation”. It comes with a standard API, full of useful functions and objects.

Node.js is geared towards asynchronous & event-driven operation. Nothing blocks, not even a read from a local disk – because in CPU terms, blocking takes too long. This means that you tend to call a “read” function and give it a “callback” function which gets called once the read completes. Very very different frame of mind. Deeply frustrating at times, but essential for any non-trivial app which needs to deal with networking, disks, and other “slow” peripherals. Including us mortals.

• Learning Node by Shelley Powers
  2012, 1st ed, 396 pages, ISBN 1449323073

See also this great (but fairly long) list of tutorials, videos, and books at Stack Overflow.

SPA and MVC

Note that JavaScript on the server replaces all sorts of widespread approaches: PHP, ASP, and such. Even advanced web frameworks such as Rails and Django don’t play a role here. The server no longer acts as a templating system generating dynamic web pages – instead it just serves static HTML, CSS, JavaScript, and image files, and responds to requests via Ajax or WebSockets (often using JSON in both directions).

The term for this is Single-page web application, even though it’s not about staying on a single page (i.e. URL) at all costs. See this website for more background – also as PDF.

The other concepts bound to come up are MVC and MVVM. There’s an article about MVC at A List Apart. And here’s an online book with probably more than you want to know about this topic and about JavaScript design patterns in general.

In a nutshell: the model is the data in your app, the view is its presentation (i.e. while browsing), and the controller is the logic which makes changes to the model. Very (VERY!) loosely speaking, the model sits in the server, the view is the browser, and the controller is what jumps into action on the server when someone clicks, drags, or types something. This simplification completely falls apart in more advanced uses of JS.

Dialects

I am already starting to become quite a fan of CoffeeScript, Jade, and Stylus. These are pre-processors for JavaScript, HTML, and CSS, respectively. Totally optional.

Here are some CoffeeScript tutorials and a cookbook with recipes.

CoffeeScript is still JavaScript, so a good grasp of the underlying semantics is important.

It’s fairly easy to read these notations with only minimal exposure to the underlying language dialects, in my (still limited) experience. No need to use them yourself, but if you do, the above links are excellent starting points.

Just the start…

The above are really just pre-requisites to getting started. More on this topic soon, but let me just stress that good foundational understanding of JavaScript is essential. There are crazy warts in the language (which Douglas Crockford frequently points out and explains), but they’re a fact of life that we’ll just have to live with. This is what you get with a language which has now become part of every major web browser in the world.

Graphs used to be made with gnuplot or RRDtool. Both generated on the server and then presented as images in the browser. This used to be called state of the art!

But that sooo last-century …

Then came JavaScript libraries such as Flot, which uses the HTML5 Canvas, allowing you to draw the graph in the browser. The key benefit is that these graphs can be made dynamic (updating through real-time data feeds) and interactive (so you can zoom in and show details).

But that sooo last-decade …

Now there is this, using the latest HTML5 capabilities and resolution-independent SVG:

See http://selection.datavisualization.ch/ (click through on each one to get details).

That picture doesn’t really do justice to the way some of these tools adjust dynamically and animate on change. All in the web browser. Stunning – in features and in variety!

I’ve been zooming in a bit (heh) on tools such as Rickshaw and NVD3 – both with lots of fascinating examples. Some parts are just window dressing, but the dynamics and real-time behaviour will definitely help gain more insight into the underlying datasets. Which is what all the visualisation should be about, of course.

For an interesting project using SocketStream, Flot, and DataTables, see DaisyCentral. There’s a great write-up on the Architectural Overview, and another page on graphically setting up automation rules:

This editor is based on jsPlumb for drawing.

Another interesting project is Dashku, based on SocketStream and Raphaël. It’s a way to build a live dashboard – the essence only became clear to me after seeing this YouTube video. As you build and adjust it in edit mode, you can keep a second view open which shows the final result. Things automatically get synced, due to SocketStream.

Now, if only I knew how to build up my fu-level and find a way into all this magic…

It all starts with baby steps. Let me just say that it feels very awkward and humbling to stumble around in a new programming language without knowing how things should be done. Here’s the sort of gibberish I’m currently writing:

This must be the ugliest code I’ve ever written. Not because the language is bad, but because I’m trying to convert existing code in a hurry, without knowing how to do things properly in JavaScript / CoffeeScript. Of course it’s unreadable, but all I care for right now, is to get a real-time data source up and running to develop the rest with.

I’m posting this so that one day I can look back and laugh at all this clumsiness :)

The output appears in the browser, even though all this is running on the server:

Ok, so now there’s a “feed” with readings coming in. But that’s just the tip of the iceberg:

• What should the pubsub naming structure be, i.e. what are the keys / topic names?
• Should readings be managed per value (temperature), or per device (room node)?
• What format should this data have, since inserting a decimal point is locale-specific?
• How to manage new values, given that previous ones can be useful to have around?
• Are there easy choices to make w.r.t. how to store the history of all this data?
• How to aggregate values, but more importantly perhaps: when to do this?

And that’s just incoming data. There will also need to be rules for automation and outgoing control data. Not to mention configuration settings, admin front-ends, live development, per-user settings, access rights, etc, etc, etc.

I’m not too interested yet in implementing things for real. Would rather first spend more time understanding the trade-offs – and learning JavaScript. By doodling as I’m doing now and by examining a lot of code written by others.

If you have any suggestions on what I should be looking into, let me know!

“Experience is what you get while looking for something else.” – Federico Fellini

I recently came across SocketStream, which describes itself as “A fast, modular Node.js web framework dedicated to building single-page realtime apps”.

And indeed, it took virtually no effort to get this self-updating page in a web browser:

The input comes from the serial port, I just added this code:

That’s not JavaScript, but CoffeeScript – a dialect with a concise functional notation (and significant white-space indentation), which gets turned into JavaScript on the fly.

The above does a lot more than collect serial data: the “try” block converts the text to a binary buffer in the form of a JavaScript DataView, ready for decoding and then publishes each packet on its corresponding channel. Just to try out some ideas…

I’m also using Jade here, a notation which gets transformed into HTML – on the fly:

And this is Stylus, a shorthand notation which generates CSS (yep, again on the fly):

All of these are completely gone once development is over: with one command, you generate a complete app which contains only pure JavaScript, HTML, and CSS files.

I’m slowly falling in love with all these notations – yeah, I know, very unprofessional!

Apart from installing SocketStream using “npm install -g SocketStream”, adding the SerialPort module to the dependencies, and scratching my head for a few hours to figure out how all this machinery works, that is virtually all I had to do.

Development is blindingly fast when it comes to client side editing: just save any file and the browser(s) will automatically reload. With a text editor that saves changes on focus loss, the process becomes instant: edit and switch to the browser. Boom – updated!

Server-side changes require nodemon (as described in the faq). After that, server reload becomes equally automatic during development. Pretty amazing stuff.

The trade-off here is learning to understand these libraries and tools and playing by their rules, versus having to write a lot more yourself. But from what I’ve seen so far, SocketStream with Express, CoffeeScript, Jade, Stylus, SocketIO, Node.js, SerialPort, Redis, etc. take a staggering amount of work off my shoulders – all 100% open source.

There’s a pubsub-over-WebSockets mechanism inside SocketStream. Using Redis.

Wow. Creating responsive real-time apps hasn’t been this much fun in a long time!

Home monitoring and home automation have some very obvious properties:

• a bunch of sensors around the house are sending out readings
• with actuators to control lights and appliances, driven by secure commands
• all of this within and around the home, i.e. in a fairly confined space
• we’d like to see past history, usually in the form of graphs
• we want to be able to control the actuators remotely, through control panels
• and lastly, we’d like to automate things a bit, using configurable rules

In information processing terms, this stuff is real-time, but only barely so: it’s enough if things happen within say a tenth of a second. The amount of information we have to deal with is also quite low: the entire state of a home at any point in time is probably no more than perhaps a kilobyte (although collected history will end up being a lot more).

The challenge is not the processing side of things, but the architecture: centralised or distributed, network topology for these readings and commands, how to deal with a plethora of physical interfaces and devices, and how to specify and manage the automation rules. Oh, and the user interface. The setup should also be organic, in that it allows us to grow and evolve all the aspects of our system over time.

It’s all about state and messages: the state of the home, current, sensed, and desired, and the events which change that state, in the form of incoming and outgoing messages.

What we need is MOM, i.e. Message-oriented middleware: a core which represents that state and interfaces through messages – both incoming and generated. One very clean model is to have a core process which allows some processes to “publish” messages to it and other to “subscribe” to specific changes. This mechanism is called pubsub.

Ideally, the core process should be launched once and then kept running forever, with all the features and functions added (at least initially) as separate processes, so that we can develop, add, fix, refine, and even tear down the different functions as needed without literally “bringing down the house” at every turn.

There are a couple of ways to do this, and as you may recall, I’ve been exploring the option of using ZeroMQ as the core foundation for all message exchanges. ZeroMQ bills itself as “the intelligent transport layer” and it supports pubsub as well as several other application interconnect topologies. Now, half a year later, I’m not so sure it’s really what I want. While ZeroMQ is definitely more than flexible and scalable enough, it also is fairly low-level in many ways. A lot will need to be built on top, even just to create that central core process.

Another contender which seems to be getting a lot of traction in home automation these days is MQTT, with an open source implementation of the central core called Mosquitto. In MOM terms, this is called a “broker”: a process which manages incoming message traffic from publishers by re-routing it to the proper subscribers. The model is very clean and simple: there are “channels” with hierarchical names such as perhaps “/kitchen/roomnode/temperature” to which a sensor publishes its temperature readings, and then others can subscribe to say “/+/+/temperature” to get notified of each temperature report around the house, the moment it comes in.

MQTT adds a lot of useful functionality, and optionally supports a quality-of-service (QoS) level as a way to handle messages that need reliable delivery (QoS level 0 messages use best-effort delivery, but may occasionally get dropped). The “retain” feature can hold on to the last message sent on each channel, so that when the system shuts down and comes back up or when a connection has been interrupted, a subscriber immediately learns about the last value. The “last will and testament” lets a publisher prepare a message to be sent out to a channel (not necessarily the same one) when it drops out for any reason.

All very useful, but I’m not convinced this is a good fit. In my perception, state is more central than messages in this context. State is what we model with a home monitoring and automation system, whereas messages come and go in various ways. When I look at the system, I’m first of all interested in the state of the house, and only in the second place interested in how things have changed until now or will change in the future. I’d much rather have a database as the centre of this universe. With excellent support for messages and pubsub, of course.

I’ve been looking at Redis lately, a “key-value store” which is not only small and efficient, but which also has explicit support for pubsub built in. So the model remains the same: publishers and subscribers can find each other through Redis, with wildcards to support the same concept of channels as in MQTT. But the key difference is that the central setup is now based on state: even without any publishers active, I can inspect the current temperature, switch setting, etc. – just like MQTT’s “retain”.

Furthermore, with a database-centric core, we automatically also have a place to store configuration settings and even logic, in the form of scripts, if needed. This approach can greatly simplify publishers and subscribers, as they no longer need local storage for configuration. Not a big deal when everything lives on a single machine, but with a central general-purpose store that is no longer a necessity. Logic can run anywhere, yet operate off the same central configuration.

The good news is that with any of the above three options, programming language choice is irrelevant: they all have numerous bindings and interfaces. In fact, because interconnections take place via sockets, there is not even a need to use C-based interface code: even the language itself can be used to handle properly-formatted packets.

I’ve set up a basic installation on the Mac, using Homebrew. The following steps are not 100% precise, but this is more or less all that’s needed on Windows, MacOSX, or Linux:

    brew install node redis
    npm install -g express
    redis-server               (starts the database server)
    express                    (creates a demo app)
    npm install connect-redis
    node app.js                (starts the demo app on port 3000)

There are many examples around the net on how to get started, such as this one, which is already a bit dated.

Let’s see where this leads to…