This post strays solidly into the realms of software and database architecture…

I’m having a lot of fun with JeeMon, but it’s also starting to look like a bone which is slightly too large to chew on…

As in any application, there is the issue of how to manage, represent, and store data structures. This is an area I’ve spent quite a few years on in a previous life. The nice bit is that for JeeMon I get to pick anything I like – there are no external requirements. Time will tell whether that freedom won’t become a curse.

So now I’m exploring the world of pure relational data structures again:

What you’re looking at is the definition of 4 drivers (I’m leaving off a bunch more to keep this example limited). This was rendered as nested HTML tables, btw – very convenient.

Each driver has a description of the type of values it can produce, and each set of values is grouped (as in the case of ookRelay2, which can generate data for three different types of packets it is relaying). Drivers can support more than one interface (the roomNode driver supports both remote packets and direct serial connection, for example). And drivers can expose a number of functions (in this case they all only expose a “decode” function).

In the “flat” SQL world, you’d need 5 tables to represent this sort of information. But since I’m supporting relations-as-values, it can all be represented as a single relation in JeeMon (I call ’em “views”, btw).

Note that this isn’t a hierarchical- or network-database. It’s still purely relational. There’s an “ungroup” operator to flatten this structure as many levels as needed, along with the usual “select”, “project”, “join”, etc. operators.

There’s also something else going on. Some of the information shown above is virtual, in the sense that the information is never actually stored into a table but gets extracted on-demand from the running application. The “values” relation, for example, is produced by scanning some simple lists stored (as dicts) in a Tcl variable:

And the “functions” relation is created on-the-fly by introspecting the code to find out which public functions have been exposed in each driver.

It’s a bit unusual to think of application data in relational terms. This lets you “join” a list with another one, and do strange things such as perform selection and projection on the list of all public functions defined in the application. But it’s also extremely powerful: one single conceptual mechanism to deal with information, whether present in application variables, i.e. run-time state, from a traditional database, or as the result of a remote request of some kind. Here’s another example: perform a join between a local (scalar) variable and a dict / map / hash (or whatever you want to call associative arrays), and you end up doing key lookup.

This approach turns (at least part of) an application on its head: you don’t need to register every fact that’s relevant into a database, you launch the app and it sets up “views” which will extract what they need when they need it, from the running app itself. Don’t call us, we’ll call you!

It might seem like a complicated way to do things (indeed, it is!), but there’s a reason for this madness. I’m looking for a design which can adapt to changes – even in the code – while running. If a driver is updated to add support for Ethernet, say, then it should reflect in these data structures without further explicit effort from the programmer. I’m not there yet by a long stretch, but this sort of experimentation is needed to get there. IMO, the magic lies in not copying facts but in seeking them out when needed, combined with smart caching.

My research project which pioneered this approach is called “Vlerq”, and I’m considering resuscitating it for use in JeeMon. Trouble is, it’s quite a bit too ambitious to implement fully (I’ve hit that wall before) – so my current experimentation is aimed at finding a workable path for the near future. Perfection is the enemy of done – and right now I’m seeing my “done” horizon racing further and further away. It’s like trying to exceed the speed of light – oh, wait, didn’t someone do that the other day?

Not only is there a glcdScope sketch which is useful for displaying voltage across the 0.1 Ω shunt in my 220V experiments, there’s also a little Spectrum Analyzer sketch to try out. Yummie!

Spectrum analysis tells you what frequencies are present in a given sample of a periodic signal. Lots of math there – just Google around for “Fast Fourier Transform” (FFT) and you’ll get it all on your plate if you’re curious.

But this stuff is a bit harder than a plain signal. Here’s what I’m seeing, with just a short wire connected, picking up some 50 Hz presumably (as with the scope test) – again with the “digital phosphor” persistence:

In principle, this graph is ok: lots of signal at lower frequencies and progressively less at higher frequencies. After all, with a perfect 50 Hz sine wave and no noise, we’d expect a single peak at the start of the scale.

The repeated peaks every few pixels also look promising. With a bit of luck they are in fact the harmonics, i.e. the 100 Hz, 150 Hz, … multiples – which is what you get when a signal is repetitive but not exactly a sine wave. Harmonics are what makes music special – the way to distinguish a note played on the violin and on the piano.

But I was hoping for something else: a bit more peaks at the right hand side of the graph. This would indicate that there are high frequencies in the signal, the computer’s switching power supply close to this setup, for example.

And worse: I’m seeing a completely flat line when hooking this up to the 220V current shunt. Looks like this signal is too weak to play FFT games with (should the data be auto-scaled?).

Anyway, here’s the glcdSpectrum50 sketch:

It relies on the fix_fft.cpp file by Tom Roberts which does the FFT heavy-lifting (original is here).

Also, note that I’m using a slightly different algorithm this time to determine the average signal value: the average of the last 256 samples is used to compute the center value subtracted from the next 256 samples. The outcome should be similar, as long as the signal is indeed symmetric around this value.

All in all a nice try, but it didn’t really provide much new insight (yet?).

Yeah, ok, this is probably a bit off topic…

I’m talking about installing Unix’y software on the Mac (which has a big fat GUI on top of a Posix-compliant OS).

There have been several projects in the past which have addressed the need to use a Mac as a Unix command-line system with all the packages coming out of the Linux, FreeBSD, etc. universes. The places where “tar”, “patch”, “make”, “autoconf”, “libtool”, and so on rule.

First there was Fink – implemented in Perl:

The Fink project wants to bring the full world of Unix Open Source software to Darwin and Mac OS X. We modify Unix software so that it compiles and runs on Mac OS X (“port” it) and make it available for download as a coherent distribution. Fink uses Debian tools like dpkg and apt-get to provide powerful binary package management. You can choose whether you want to download precompiled binary packages or build everything from source.

Then came MacPorts – implemented in Tcl:

The MacPorts Project is an open-source community initiative to design an easy-to-use system for compiling, installing, and upgrading either command-line, X11 or Aqua based open-source software on the Mac OS X operating system. To that end we provide the command-line driven MacPorts software package under a BSD License, and through it easy access to thousands of ports that greatly simplify the task of compiling and installing open-source software on your Mac.

And now there’s Homebrew – implemented in Ruby:

Homebrew is the easiest and most flexible way to install the UNIX tools Apple didn’t include with OS X. Packages are installed into their own isolated prefixes and then symlinked into /usr/local.

It’s interesting to note that all systems were written in a scripting language. Which makes perfect sense, given that they offer a huge jump in programmer productivity and no downsides to speak of in this context.

As for my pick: Homebrew gets it right.

Homebrew (i.e. the “brew” command) is installed with a one-liner:

    /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

This is a big deal, because the last thing you want is to use a package manager to overcome installation hassles, only to end up with… an extra step which introduces its own installation hassles!

The actual design of that one-liner is quite clever: it points to a script in a secure area, with full history of all changes made to that script, so you can make sure it won’t do weird (or bad) things with your system.

Homebrew doesn’t introduce new conventions (like /sw/ or /opt/, which need to be added to your exe search path). It’ll simply install the packages you ask for in /usr/local/bin etc (which means you don’t need to sudo all the time). And it does it intelligently, because it actually installs in /usr/local/Cellar and then puts symlinks in /usr/local/*. Which means there’s a way out of an installation, and there’s even a (drastic) way to get rid of all Homebrew-installed packages again: delete Cellar, and remove the dangling symlinks from /usr/local/*. Not that you’d ever need to – Homebrew supports uninstalls.

It’s succinct (and colorized):

No endless lists of compiler commands on my screen, telling me absolutely nothing, other than “CPU == busy!”.

It does the right thing, and works out all the dependencies:

    fairie:~ jcw$ brew search rrd
    rrdtool
    fairie:~ jcw$ brew install rrdtool
    ==> Installing rrdtool dependency: gettext
    ==> Downloading http://ftpmirror.gnu.org/gettext/gettext-0.18.1.1.tar.gz
    ######################################################################## 100.0%
    ==> Downloading patches
    ...
    ==> Summary
    /usr/local/Cellar/gettext/0.18.1.1: 368 files, 13M, built in 5.4 minutes
    ==> Installing rrdtool dependency: libiconv
    ==> Downloading http://ftpmirror.gnu.org/libiconv/libiconv-1.14.tar.gz
    ...
    ==> Summary
    /usr/local/Cellar/libiconv/1.14: 24 files, 1.4M, built in 70 seconds
    ==> Installing rrdtool dependency: glib
    ... etc, etc, etc ...
    ==> Installing rrdtool
    ==> Downloading http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.5.tar.gz
    ######################################################################## 100.0%
    ==> Patching
    patching file configure
    Hunk #1 succeeded at 31757 (offset 94 lines).
    Warning: Using system Ruby. RRD module will be installed to /Library/Ruby/...
    Warning: Using system Perl. RRD module will be installed to /Library/Perl/...
    ==> ./configure --prefix=/usr/local/Cellar/rrdtool/1.4.5 --mandir=/usr/local/Cel
    ==> make install
    /usr/local/Cellar/rrdtool/1.4.5: 148 files, 3.2M, built in 68 seconds
    fairie:~ jcw$ 

It does its work with “formulas”, i.e. Ruby scripts which describe how to fetch, build, and install a certain package. The formulas (formulae?) are managed on GitHub, which means that there is a massive level of collaboration going on: new packages, issue tracking, fixes, and discussion. Here’s what I got when self-updating it again after having done so one or two days ago:

    $ brew update
    remote: Counting objects: 294, done.
    remote: Compressing objects: 100% (103/103), done.
    remote: Total 250 (delta 174), reused 212 (delta 143)
    Receiving objects: 100% (250/250), 39.46 KiB, done.
    Resolving deltas: 100% (174/174), completed with 32 local objects.
    From http://github.com/mxcl/homebrew
       36f4400..99bc0b7  master     -> origin/master
    Updated Homebrew from 36f4400e to 99bc0b79.
    ==> New formulae
    apktool             hexedit             p11-kit             solid
    denyhosts           jbigkit             qi                  tinyfugue
    gearman-php         kbtin               shen
    graylog2-server     opencc              sisc-scheme
    ==> Updated formulae
    android-ndk         ffmpeg              libquicktime        python
    aqbanking           frink               libraw              python3
    audiofile           fuse4x              libvirt             sleepwatcher
    cassandra           fuse4x-kext         nasm                tomcat
    class-dump          gflags              nginx               transcode
    csshx               google-sparsehash   nss                 tsung
    dash                gpsbabel            pdfjam              xml-security-c
    dvtm                gwenhywfar          pixman*
    elasticsearch       libiconv*           pos
    $

But best of all IMO, the formulas use an almost declarative style. Here’s the one for re2c I just installed:

It’s not hard to guess what this script does. No wonder that so many people submit and tweak such formulas.

Note that there’s neither a description, nor explicit version handling in there (other than what can be gleaned from the download URL). Brew is an installer, not a catalog. Wanna know more? Visit the home pages. Brilliant.

It’s great to see the Ruby community adopt a concise, declarative, and Domain Specific Language style. Nothing new or revolutionary – has been done many times before – but nevertheless underappreciated, IMNSHO.

Here’s a web page I’ve been designing recently – one of many planned for the JeeMon system I’m working on:

A couple of notes:

• there are three “master” tables on this page: Devices, Interfaces, and Drivers
• devices are a 1:N mapping between drivers and interfaces
• … except that some devices are remote, and not tied to a local interface
• clicking on any row of these three tables displays details in the rightmost column
• drivers are listed but not managed on this page, that’s a software admin task

I’m describing this here for a number of reasons (other than showing off what I’ve been spending my days on).

First of all, you can see that this more or less covers what needs to be set up to in a home monitoring and/or home automation system. I like to see everything at a glance, i.e. on a single page and with as little navigation as possible. With this setup, I hope to keep it all essentially on one page. Whereby the right-side column may vary widely, depending on the device / interface / driver.

But it doesn’t really need to be about home automation at all. A very similar setup could be used to hook up to devices which I’m experimenting with in the lab, or devices which act like measuring instruments, or control some aspect of an experiment. Anything related to Physical Computing, really.

The other thing that interests me is the “degree of variety” needed to cover all the cases. Many devices will simply collect or send out one or more values, but not all devices are like that.

The RF12demo sketch is essentially a gateway from and to an entire group of RFM12B-based nodes. Each node can have its own driver. The ookRelay2 sketch is similar: it collects data from quite different devices, each of them sending out their specific packets with their specific types of measurements. It differs from RF12demo, in that it contains decoders for all the devices it knows about. There are no separate KS300 drivers, etc (although perhaps there should be, since some code is currently duplicated in the CUL driver).

The autoSketch driver is yet another beast. It listens on a serial interface for a line of the form “[…]” and when it sees one, tries to load a driver with the name in brackets and hand over control to it. This is the reason why just about all my sketches start off with the following boilerplate code:

    void setup() {
        Serial.begin(57600);
        Serial.println("\n[mySketch]");
        ...
    }

When associated with the autoSketch driver, this will automatically load a “mySketch” driver, if present. Which in turn means that all you have to do is plug in the JeeNode (JeeLink, Arduino, etc) and JeeMon will automatically detect it and start communicating with it. IOW, plug-and-play – with a simple implementation.

This is why there’s a “Use autoSketch for new interfaces” checkbox. I could have called it “Enable plug & play”.

But although this web page is functional in my test setup, it’s far from finished. The hardest part I want to get right, is to make the page completely dynamic. Right now, a click on a row will update the details via Ajax, but nothing will happen when there is no user interaction. What I want is that the page automatically adjusts when a JeeLink is plugged in (on the web server side). What I really want, is to generalize that mechanism to everything shown on any web page. The challenge is to do this without tedious periodic polling or complete table refreshes, just “pushing” changes in an event-driven manner. Events on the server side, that is.

The visual layout, styling, and behavior of a page like this has become very simple with today’s tools:

• jQuery is the core JavaScript library
• jQuery UI is used as basic style engine (the tabbed group on the right, for example)
• jQuery TableSorter adds support for (surprise!) table sorting via their headers
• Knockout is used to manage a lot of the user-facing behavior in the browser
• 960 grid regulates the page structure flow in CSS
• there is a small amount of custom CSS and custom Javascript (about 20 lines each)
• the entire page is implemented as a single Tcl source file (175 lines so far)

There are a few trivial utility functions in there, which are bound to migrate to a more general module once I better understand the trade-offs and benefits of re-use. And the rest is gobbledygook which looks like this:

This is the notation I came up with and described in my Playing with indentation post a short while back. This particular code describes the HTML text returned for drivers in the “details” column, via Ajax.

This notation gets expanded to HTML, but the real value of it is that describing the HTML on a page is fun, because it’s now trivial to insert grouping levels (i.e. mostly <div>’s) and rearrange stuff until it’s just right.

I wanted to learn a bit more about how to implement non-trivial forms in HTML + JavaScript. Without writing too much HTML, preferably…

There are a few versions of a tool floating around the web, which help calculate the constants needed to configure the RFM12B wireless module – such as this recent one by Steve (tankslappa).

Putting one and one together – it seemed like an excellent exercise to try and turn this into a web page:

If only I had known what I had gotten myself into… it took more than a full day to get all the pieces working!

In fairness, I was also using this as a way to explore idioms for writing this sort of stuff. There is a lot of repetitiveness in there, mixed with a frustrating level of variation in the weirdest places.

Due to some powerful JavaScript libraries, the result is extremely dynamic. There is no “refresh”. There is not even a “submit”. This is the sort of behavior I’m after for all of JeeMon, BTW.

The calculator has been placed on a new website, called http://jeelabs.org/tools/. The page is dynamically generated, but you can simply save a copy of it on your own computer because all the work happens in the browser. The code has been (lightly) tested with WebKit-based browsers (i.e. Safari and Chrome) and with FireFox. I’d be interested to hear how it does on others.

If you want to look inside, just view the HTML code, of course.

As I said, it’s all generated on-the-fly. From a page which is a pretty crazy mix of HTML, CSS, JavaScript, and Tcl. With a grid system and a view model thrown in.

You can view the source code here. I’ve explored a couple of ways of doing this, but for now the single-source-file-with-everything-in-it seems to work best. This must be the craziest software setup I’ve ever used, but so far it’s been pretty effective for me.

Why use a single programming language? Let’s mash ’em all together!

For yesterday’s setup, I had to write a bit of code. Here’s the main part:

(this “powerACS” sketch can be found here)

The idea is to keep reading the analog pin as often as possible (several thousand times per second), and then keep track of its moving average, using this little smoothing trick:

    avg = (499L * avg + value) / 500;

Each new measurement adds 1/500th weight to the average. I didn’t want to use floating point, so it takes some care to avoid long int overflow. Here’s how I scale the input (0 .. 1023) to microvolts (0 .. 3,300,000):

    uint32_t value = measure.anaRead() * (3300000L / 1023);

Note that 500 x 3300000 is under the max of a 32-bit int, so the above “avg” calculation is ok.

It’s easy to calculate the offset, once we know the average voltage. Keep in mind that we’re measuring 50 Hz AC, and that the result we’re after is the fluctuation – i.e. what you get when putting a multimeter in VAC measurement mode. The way to do that is to calculate the “average absolute difference from the average”:

IOW, what I want is the average height of those red bars. I flipped the lower half up because we’re interested in the absolute difference (otherwise the bars would just cancel out).

Here’s the calculation:

    if (value > avg)
      total += value - avg;
    else if (value < avg)
      total += avg - value;
    ++count;

When the time comes to report a “range” reading, we calculate it as follows and reset the other variables:

    uint32_t range = total / count;
    total = count = 0;

Here’s some sample debug output on the serial port:

    [powerACS]
    ...
    2560650 2569286 5916 31973363 5404
    2576775 2569256 5911 32175178 5443
    2563875 2569266 5917 32888792 5558
    2567100 2570205 5911 33200506 5616
    2573550 2569935 5917 32798060 5543
    2580000 2565675 5910 183251691 31007  <- 100 W light bulb
    2573550 2565108 5916 356840654 60317
    2570325 2565479 5916 355916529 60161
    2512275 2565467 5910 355676919 60182
    2525175 2565688 5916 356664359 60288
    2486475 2566547 5909 354854663 60053
    2480025 2566631 5916 355567509 60102
    2480025 2568658 5910 355608745 60170
    2583225 2569370 5916 178542547 30179  <- off
    2576775 2569122 5915 33193018 5611
    2570325 2569274 5911 33236267 5622
    2560650 2568988 5917 32873935 5555
    2576775 2569290 5911 32733630 5537
    2567100 2568564 5917 33390738 5643
    ...
    2560650 2569179 5917 32917499 5563
    2576775 2568640 5911 32354114 5473
    2570325 2569121 5916 32820006 5547
    2576775 2568319 5917 33896284 5728
    2570325 2569996 5911 33120608 5603
    2567100 2568514 5917 37697489 6371    <- 1.1 W power brick
    2550975 2568582 5911 36764137 6219
    2567100 2568767 5916 37434853 6327
    2567100 2568813 5911 36272763 6136
    2563875 2568659 5917 37738504 6377
    2563875 2568355 5911 37911395 6413
    2567100 2568586 5917 37444158 6328
    2563875 2568672 5916 37607002 6356
    2580000 2568879 5911 37293083 6309    <- off
    2570325 2568330 5917 33082874 5591
    2563875 2567713 5911 33007567 5584
    2567100 2568380 5917 32770050 5538
    2580000 2568846 5910 33002639 5584
    2570325 2568844 5917 33201661 5611

You can see the 60 mV signal when the power consumption is 100 W. Which is a bit low: I would have expected 100 W / 230 V * 0.185 V/A = 80 mV. But it turns out that the lamp really is an 87 W lamp, so that makes it 70 mV. I’ll attribute the remaining difference to my crude setup, for lack of a better explanation.

What’s worse though, is that there is always a 5.5 mV “hum” when no current is flowing (even when the whole circuit isn’t anywhere near a power line). One possible explanation could be ADC measurement noise: the 10-bit ADC resolution is 3.3 mV. Then again, note how multiple measurements do lead to a much finer resolution: the 60 mV readout fluctuates by only a fraction of one millivolt. Noise actually improves resolution, when averaged over multiple readings.

Still, the 5.5 mV baseline value makes it hard to reliably measure really low power levels. The 1.1 W test was from an unloaded power brick, i.e. an inductive load. It causes the output to jump from 5.5 mV to only 6.3 mV, which is not a very distinct result. Another 0.3 W test load didn’t even register.

Conclusion: this detects current fairly well, was very easy to set up, and offers galvanic isolation protection. But it does require a hookup to AC mains power and it’s not super effective at really low power levels.

As I mentioned in a recent post, the collectd approach fits right in with how wireless RF12 broadcast packets work.

Sounds like a good use for an EtherCard + JeeNode pair (or any other ENC28J60 + RFM12B combo out there):

The idea is to pass all incoming RF12 packets to Ethernet using UDP broadcasts. By using the collectd protocol, many different tools could be used to further process this information.

What I did was take the etherNode sketch in the new EtherCard library, and add a call to forwardToUDP() whenever a valid RF12 comes in. The main trick is to convert this into a UDP packet format which matches collectd’s binary protocol:

The sendUdp() code has been extended to recognize “multicast” destinations, which is the whole point of this: with multicast (a controlled form of broadcasting), you can send out packets without knowing the destination.

The remaining code can be found in the new JeeUdp.ino sketch (note that I’m now using Arduino IDE 1.0beta2 and GitHub for all this).

It took some work to get the protocol right. So before messing up my current collectd setup on the LAN here at JeeLabs, I used port 25827 instead of 25826 for testing. With JeeMon, it’s easy to create a small UDP listener:

And sure enough, eventually it all started to work, with RF12 packets getting re-routed to this listener:

Now the big test. Let’s switch the sketch to port 25826 and see what WireShark thinks of these packets:

Yeay – it’s workin’ !

The tricky part is the actual data. Since these are raw RF12 packets, with a varying number of bytes, there’s no interpretation of the data at this level. What I ended up doing, is sending the data bytes in as many collectd 64-bit unsigned int “counter values” as needed. In the above example, two such values were needed to represent the data bytes. It will be up to the receiver to get those values and convert them to meaningful readings. This decoding will depend on what the nodes are sending, and can be different for each sending JeeNode.

I’ve left the original web browser in as well. Here is the “JeeUdp” box, as seen through a web browser:

(please ignore the old name in the title bar, the name is now “JeeUdp”)

It’s not as automatic out of the box as I’d really like. For one, you have to figure out which IP address this unit gets from DHCP – one way to do so is to connect it to USB and open the serial console. The other bit is that you need to configure the unit to set its name, the UDP port, and the RF12 settings to use. There’s a “Configure” link of the web page to do this – a some point, I’d like to make JeeMon aware of this, so it can do the setup itself (via http). And the last missing piece of the puzzle is to hook this into the different drivers and decoders to interpret the data from these UDP packets in the same way as with a JeeLink on USB.

Ultimately, I’d like to make this work without any remote setup:

• attach Ethernet and power to this box (any number of them)
• each box starts reporting its status via UDP (including its IP address)
• a central JeeMon automatically recognizes these units
• you can now give a name to each box and fill in its RF12 configuration
• packets start coming in, so now you can specify the type of each node
• decoders kick in to pick up the raw data and generate meaningful “readings”
• that’s it – easy deployment of JeeNode-based networks is on the horizon!

Not there yet, but all the essential parts are working!

I’ve been making good progress since the recent post.

Some things work, some things don’t when you use “off-the-shelf Open Source”. For example, I started using Twitters Bootstrap as CSS grid framework for page layout. It looks great. But there was too much interference with other components, such as jQuery DataTables. Well, either that or I simply don’t get it. So I ended up removing it all, in favor of the 960 Grid System (note how all JavaScript development seems to be happening on GitHub).

So now I’ve got a first rough cut at the “Niner” layout, i.e. 3×3 pages accessed via tabs on the bottom/right. Here’s a screen shot from an iPad:

The tabs don’t look so great (I’m so inept at GUI design that I don’t even try), but the general design is shaping up nicely. The area at the bottom left shows the last two lines of the log on the server, and it’s updating in real time. So is the table itself, using Server-Sent events.

With this sort of dynamics, all sorts of details start to jump out. My first approach was to have one page per URL, i.e. “http://127.0.0.1:8181/1” for page 1, etc. But that causes a full page refresh, which is too disruptive, especially with that log area constantly updating.

Which is where JavaScript and Ajax come in, of course. So now navigation only updates the main area – not the bottom & right-side borders. I’m only testing with Safari/Chrome, and occasionally FireFox, BTW.

The indentation idea I described earlier is also working out nicely. Here’s the main page description which produces the HTML sent to the browser:

(as you can see, there are limits to syntax coloring when mixing several languages in a single source file!)

And here’s the description of the page with that table shown above:

Let me just say that I’m “over” HTML. Bye bye, <div> et al. Good riddance.

PS. Oh yes, and here’s the fun part – this defines the structure of the site:

Tabs and titles are automatically generated unless overridden in this layout.

I’m building a “home monitoring and automation” system. Not just the hardware “nodes”, but also the software side of things, i.e. a server running permanently. Web-based, so that it can be accessed from anywhere if needed.

I’ve got a lot of code lying around here, ready to be put to use for just that. It’s not that hard to create a website (although truly dynamic displays are perhaps a little harder).

One thing I’ve been struggling with for a long time, though, is the overall design. Not so much the look & feel, but the structure and behavior of it all.

I’m not too keen on systems with elaborate menu structures, tons of hoops dialogs to jump through, even scrolling messes with our innate sense of location. Been there, done that. Modes suck – they just create labyrinths.

What I’ve come up with is probably not for everyone (heh, nothing new here). An entire system, organized as nine pages. No more, no less. Instant navigation with a keyboard (digits, function keys, whatever), and trivial navigation with a mouse or touch screen:

Even a laptop screen with 3+3 buttons (or a joystick) could navigate to any page with just two clicks.

Here is a first exploration of what these pages could be for (vaporware, yeay!):

Each page is nearly the full screen, with only the bottom and right-side borders used. And each page can have any content, of course: tabs, sidebars, widget collections, it’s still all up in the air. There’s plenty of software to extend this system infinitely, using HTML, CSS, and JavaScript.

One of the things I’d like to do with this, is make the system automatically return to the main Status Overview page when idle for a minute or so. With the option to “pin” one of the other pages to prevent this from happening.

This might all seem very basic, inconsequential, and even arbitrary, but for me it’s a major step forward. I’ve been running around in circles for ages, because there was no structure in place to start assembling the different parts of the site. It’s all nice and well to develop stuff and tinker endlessly behind the scenes, but in the end this still needs to lead to a practical setup.

So now there’s a plan!

Bear with me – this isn’t directly related to Physical Computing, but I’ve been having fun with some code.

If you’ve ever written any web apps or designed a web site, chances are that you’ve also written some HTML, or at least HTML templates. Stuff like this, perhaps:

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset='utf-8' />
        <title>Flot graph demo</title>
        <script type='text/javascript' src='.../jquery.flot.js'></script>
        <script type='text/javascript'>
          $(function(){ ... });
        </script>
        <style type='text/css'>
          #placeholder { width: 600px; height: 300px; }
        </style>
      </head>
      <body>
        <div id='placeholder'></div>
      </body>
    </html>

Well, I’m getting a bit tired of all this nesting. The essence of it is buried far too deeply inside what I consider line noise. There are currently a number of developments related to HTML, CSS, and JavaScript which might change this sort of development considerably. Much of it seems to be driven by the Ruby and Rails community, BTW.

Some examples: for HTML, there is Haml. For CSS, there is Sass. And for JavaScript there is CoffeeScript.

All of these have in common (along with Python and YAML) that they treat indentation as a significant part of the notation. Before you raise an eyebrow: I know that the opinions on this are all over the map. So be it.

I wanted to try this with JeeMon, so after a bit of coding, I ended up with a “Significant Indentation Formatter” and then applied that to HTML. Here’s the above, re-written in a HAML-like notation:

    !html
      head
        meta/charset=utf-8
        title: Flot graph demo
        script/type=text/javascript/src=.../jquery.flot.js
        script/type=text/javascript
          $(function(){ ... });
        style/type=text/css
          #placeholder { width: 600px; height: 300px; }
      body>#placeholder

It generates the same HTML as above, but there’s less to write. Even less in fact, because I’ve also integrated it with JeeMon’s templating system:

    !html
      head
        meta/charset=utf-8
        title: Flot graph demo
        [JScript includes flot]
        [JScript wrap $js]
        [JScript style { #placeholder { width: 600px; height: 300px; } }]
      body>#placeholder

Here’s a slightly more elaborate KAKU send example which illustrates basic template looping:

    !html
      head
        meta/charset=utf-8
        title: KAKU send
        [JScript includes ui]
        [JScript wrap $js]
      body
        % foreach x {1 2 3 4}
          p.toggles
            button#on$x: On $x
            button#off$x: Off $x
        p#group
          label: Group:
          % foreach x {I II III IV}
            input#g$x/type=radio/name=g/value=$x
            label/for=g$x: $x
        p
          label: House Code:
          select
          % foreach x {A B C D E F G H I J K L M N O P}
            option/value=$x: $x

So far, this system doesn’t use standard conventions, is undocumented, hasn’t been used in anger or proven itself to handle more complex layouts, has almost no error handling, and has only been used by yours truly for a few hours. Still, I’ve adjusted all the examples in JeeRev to gain a bit more experience with this approach.

I like it. The above is pretty close to how I want to design the structures and sections of a web page.

As mentioned a few days ago, I’m in the process of moving all my software development to GitHub.

In a nutshell: Git is a Version Control system which lets you keep track of changes to groups of files, and do it in such a way that many people can make changes and nothing will ever get lost. You might feel lost if you really make a mess of it, but every change is tracked and can be recovered and compared against any other point in time. Based on git, GitHub takes it all to the next level, by fostering collaboration and combining tons of useful features – for free, via their website. GitHub is to git as JavaScript is to assembler (the analogy is a bit rough).

If you’re writing code, there is no excuse for not using a VCS. Even as a single developer, even if no-one else ever gets involved, and even if you’re the most organized person on the planet. An organized developer? Hah! :)

Life without a VCS looks like this:

• let’s write some code – great, it works!
• let’s try out something more difficult now
• drat, didn’t work, I hope my editor undo buffer lets me back out
• let’s make a copy of what worked before
• now let’s write some more code – great, it works!
• … time passes …
• time to write code again – hm, I messed up
• where was that copy I made earlier, again?
• drat, it doesn’t have my latest changes
• do I remember which part worked, and what I added?
• … time passes …
• I want to continue on that first attempt again
• hmm, guess I’ll start from scratch
• no wait, let’s make a copy before I mess up
• … time passes, the disk fills with copies, none of them work …

Now let’s revisit that with a VCS:

• write some code, check it in – add a brief comment
• try out something more difficult now
• check it in, let’s add a note that it’s still experimental
• hm, can’t seem to get it to work
• oh, well, I’ll go back to the previous version – easy!
• writing more code, having fun, we’re on a roll
• let’s check it in right away
• … disaster strikes, my laptop breaks down
• get it fixed or get a new one, check out from GitHub
• where was I? oh yes, got one good version and two bad ones
• let’s switch to that failed attempt again
• wait, let’s first merge the latest good changes in!
• cool, now I’ve got an up-to-date set of files to work on
• oh, darn, still can’t make it work, let’s save it with a note
• … time passes …
• ok, time to review the different attempts and take it from there

Note that there are no more copies of files. Only a repository (on your laptop and on GitHub – but you still need to back it up, like everything else). The repository contains the entire history of all the changes, including notes, backing out steps, merges, everything.

A lot of this is possible with any modern VCS, including the “subversion” setup I’ve been using until now. But GitHub makes it effortless, and throws in a lot of nice extras. You still need a “git” application on your laptop (there are several versions), and if you’re using a Mac, you can use GitHub’s own app to communicate with GitHub without having to go through a browser.

Here’s what a “diff” looks like, i.e. a comparison of the source code, showing only the differences:

One line was replaced, two more lines were added at the end. It’s totally obvious, right?

Here’s a history of the “commits” made, i.e. the times when a new version was sent to git and GitHub:

Almost everything is a link – you can compare versions, find out what else was changed at the same time, see what else any of the developers is working on, and of course download a full copy of any version you want.

The other mechanism I’d like to describe is branching and merging. That change shown above was not made by me but by “Vliegendehuiskat” (hello, Jasper :) – and the way it works is quite neat, if you’ve never seen collaboration via VCS in action.

What Jasper did, was create a “fork” of JeeLib, i.e. a personal copy which knows exactly where it came from, when, and from which version of which files. And then he made a small change (without even telling me!), and at some point decided that it would be a good idea to get this change back into the official JeeLib code. So he submitted a “pull request”, i.e. a request for me to pull his changed version back in:

This generated a new entry in GitHub’s issue tracker and an email to me. I liked his change and with one click, merged it back in. Only thing left to do was to “pull” those changes to my laptop copy as well.

GitHub has a graphical “network” view to display all commits, forks, branches, and merges:

Each dot is a commit. The black line is my code, the blue diversion shows Jasper’s fork, his commit, and my merge. Hovering over any of these dots will show exactly what happened.

This illustrates the simplest possible case of GitHub for collaboration. If you want to see how a entire team puts git and GitHub at the center of a large-scale collaborative software development effort, read this article by Scott Chacon (who works with and at GitHub, BTW).

I’ve been moving lots of stuff to GitHub these past few days, not all of it related to JeeLabs. Here are the three main projects, as far as physical computing goes:

Note another nice feature: you can instantly see how active a project is. JeeLib is the brand new combination of the Ports and RF12 libraries, while EtherCard and GLCDlib have been around a bit longer (and the full change history has been imported). It’s been over half a year since Steve and I worked on GLCDlib, as you can see – with a small blip before the summer, when I added font support.

There are a few features which I haven’t enabled at GitHub. One of them is their wiki – because having yet another spot on the web with pages related to JeeLabs would probably do more harm than good. The Café at https://jeelabs.net is as before the place to look for documentation. There are ways to merge this stuff, so maybe I’ll go there one day.

A warning: the “sketches” on GitHub have all been adjusted for the new upcoming Arduino 1.0 IDE, and do not work with Arduino 0022, which is still the main version today. So I’m slightly ahead of the game with this stuff.

I’m really looking forward to how the software will evolve – and you’re cordially invited to be part of it :)

PS. I’m updating to GitHub fairly often because I work on two different machines – and this is the easiest and safest way to keep stuff in sync.

Yesterday, the WordPress server dropped out. Looks like some joker decided to grab the whole site, no throttling, nothing. Oh well, I guess that’s what fat pipes lead to.

In principle, WordPress should be able to handle such things, but I think it caused the underlying MySQL to run out of memory and abort.

A restart of the WordPress VM fixed it, once I found out. But this gives me an excuse to describe some things I’ve been starting to set up here.

One of of things I did this summer, was set up a collectd process on most of the machines running here – collectd is a bit Linux-centric, but one of the things it can do is grab various system statistics and broadcast it on the LAN over UDP. There are loads of plugins, and you can interface to it with scripts to feed it anything you like.

The nice thing about UDP broadcasts, is that senders don’t need to know anything about receivers. As with the RF12 driver, you just kick stuff into the network and then it’s up to others to decide what to do with it, if anything.

Collectd is also extremely low overhead, both in terms of CPU & memory load and in terms of network load. I’ve enabled it permanently on five systems, sending out 50..100 system parameters each, once every 30 seconds.

Here’s a partial view of the information being collected:

This was generated in the browser using JeeMon and jstree, as an exercise (I’m looking for a more convenient JavaScript widget to adjust a live tree using JSON).

Broadcasting is very convenient, as the collectd tool and the RF12 driver have shown, because there’s no setup. In theory, packets can get lost without acknowledgements and re-transmissions. In practice, some do indeed get lost, but very rarely. No problem at all for periodic / repetitive data collection.

I created a small script to display system status on my development Mac, using a tool called GeekTool. It can put information right on the desktop, i.e. as unobtrusively as you want, really. Here’s what I see when there are no windows obscuring the desktop background:

On the left is a periodic “ps“, the output on the right is from “ssh myserver jeemon overview.tcl“. This is a remote call to the server using a small script to pull info out of a text file and format it. The output shows that there are 5 systems online (running collectd, that is), and the script last ran at 12:41 in 39 ms.

Perhaps more interesting is that the server also collects this data from the LAN. To do this, I just launch JeeMon next to this “main.tcl” script:

    History group * 1w/5m 1y/6h
    collectd listen sysinfo

The first line is optional – it defines the parameters for historical round-robin data storage. So what I end up with, is the last 7 days of information, with details every 5 minutes, as well as up to one year of information aggregated per 6 hours.

The second line starts listening for collectd packets on the standard UDP port, tagging the results with the “sysinfo” prefix. As side effect, JeeMon will update a “stored/state.map” text file once a minute with the current values of all variables plus some other details. This is what was used to generate the above GeekTool on-screen status. There’s a built-in web server to query and access the saved historical data.

There’s no further setup. It’s up to the clients to decide what to collect and how often to send it. Data storage on the server is fixed, since this is stored in round-robin fashion (like rrdtool). It’s now collecting nearly 600 800 variables and using about 100 125 Mb of disk space. Data collection adds 2% load on a 2.66 GHz CPU.

This could also be used to collect home sensor data.