The recently released (limited-edition) JeeNode Block has a couple of changes w.r.t. pin allocation, most notably the RFM12B’s “nIRQ” interrupt pin. This was moved from PD2 to PB1 (Arduino digital pin 2 to pin 9). The main reason for this change was to free the entire set op pins on PORTD, i.e. PD0..PD7 – in Arduino-speak: digital pin 0 through 7.

That means interrupts are no longer coming in over the INT0 line, and can no longer use the attachInterrupt() and detachInterrupt() functions in the Arduino run-time library. Instead, the RF12 driver needs to switch to pin-change interrupts, which I’ve now added to the code on GitHub.

To use the RF12 driver with the JeeNode Block, you need to make the following changes:

1. Get the latest version of JeeLib from GitHub

2. Enable pin-change interrupts by removing the comment so it defines PINCHG_IRQ:

       #define PINCHG_IRQ 1
   

3. Change the RFM_IRQ define (make sure you pick the ATmega328 case):

       // ATmega168, ATmega328, etc.
       #define RFM_IRQ 9
   

4. If you plan to use RF12demo.ino – don’t forget to change the LED pin to match the JeeNode Block:

       #define LED_PIN 8
   

5. Compile and upload, and things should work as on any other JeeNode.

Note that pin-change interrupts are a bit tricky. First of all, changes are not the same as responding to a level interrupt, but the new code should take care of that. More importantly, the RF12 driver assumes it’s the only one using pin changes on the corresponding “port” (in Atmel datasheet terminology). If you use more pins for port interrupts, you may have to adjust the RF12.cpp file.

These changes are not on by default, the latest RF12 driver comes configured for the original JeeNodes.

I’ve written before about source code respositories and Subversion / Git – about a year ago, in fact.

Suprisingly, lots of people working on software are still not using any “Version Control System” (VCS) !

So let me try a new angle today, and show you how you need only use 5% of those systems to get a huge benefit out of them when dealing with the Arduino sketches and libraries from JeeLabs. Since all the code from JeeLabs is now located at GitHub, the rest of this post will be about Git (and GitHub, a truly phenomenal free web resource).

Why bother with a VCS? Well… I once heard of a little anecdote about a sign at the dentist, which said:

“You don’t have to floss all your teeth, just the ones you want to keep.”

It’s a bit like that with source code. If you don’t write code, or don’t ever change it, then you don’t need to manage it. But if you do, then you know how much effort goes into a working sketch, library, script, program, etc. Things change – either your own code, libraries from others, tools like the Arduino IDE (1.0 broke lots of stuff), even core computer software or hardware. Things always change, and if you want to be able to deal with it, you need VCS.

Not everyone aspires to be a full-time software developer (what an odd world that would be!), but even in the simplest situations you can benefit from version control. Here’s a possible scenario, close to home JeeLabs:

• Suppose you want to use the EtherCard library, but not necessarily with the JeeLabs Ether Card hardware.
• The GitHub page includes a README at the bottom, with a link to the ZIP archive with all the code.
• So you download the ZIP, unpack it, and figure out where to put it so the Arduino IDE can find it.
• Say you want to try out the “backSoon” example, as demo of a tiny web server. You compile and upload.
• It doesn’t work – now what? After a lot of digging and searching, you figure out that the code was written for the SPI select signal tied to Arduino’s digital pin 8 (PB0) whereas your hardware uses pin 10 (PB2).
• After that it’s easy: change “#define SELECT_BIT 0” to “#define SELECT_BIT 2” in enc28j60.cpp.
• Fantastic, you’ve cut through the problem and made the EtherCard library work for you. Congratulations!

This is a typical case: code was written for a specific purpose, but a small change makes it more widely usable.

I picked this example, because it is likely that we’ll update the EtherCard library soon to support different pins at runtime. That’ll allow you to leave the EtherCard library alone and set things up in your sketch, i.e. at run time. Note that it’s always a trade-off – some things usually become more general, but only if there’s a need. You can’t write code to cover all the cases. Well, you could, but then you end up with a nightmare (I won’t give examples).

So sometime in the near future, there will be an update to the EtherCard library. That’s when trouble starts…

Because there are now three different versions of the EtherCard library, and you’re about to create a fourth:

• the original EtherCard library on GitHub
• your copy of that library, on your disk, with a tiny change made to it
• the updated! improved! whiter-than-white! new EtherCard library on GitHub
• your copy of the new library, which you’re now forced to update and adjust again

There’s more going on: in this particular case, that update made your small change superfluous. But now the API of the EtherCard library has changed, so you need to change each of your sketches using the EtherCard library.

It’s not hard to see how these little details can turn into a nightmare. You build something, and you end up having to keep track of all sorts of dependencies and changes on your disk. Worse: everyone is wasting their time on this!

Version control systems such as Git can do a lot more, but the key benefit they offer can be summarized as:

You can easily manage THREE different versions of all the source files: the OLD, the NEW, and YOURS.

So if all you want is to use open source code and stay up to date, then Git lets you do that – even if you make changes yourself in any of those files. And it gets even better: if your changes have nothing to do with the changes made to the original code, then all you need to do is to issue the “git pull” command, and the remote changes will be folded into your local copy of the source file – chances are that you can simply re-compile, upload, and keep on running with the latest updates!

If your local changes do interfere or overlap with the changes made to the original code, then “git pull” will detect this and avoid making the changes to prevent it from altering yours (which always take precedence). This is called a “conflict” – and Git offers several (maybe too many) ways to deal with it.

For JeeLabs, all the code is now maintained on a free and public web site called GitHub. You don’t have to have an account there to browse or download any source code. You can not only see all the latest code, you can also look at any previous changes, including the complete history from the moment the first version was placed on GitHub.

Here’s an example: https://github.com/jcw/jeelib/commit/6f8415607ff6eef3966363751975c8f807ef9814

You’ll see a comment about the change, the list of files affected, and for each file an exact summary of the changes and some nearby lines for context. This change has been recorded forever. If logged in, you could even discuss it.

Even if you never intend to share your own changes and code, you can see how this resource lets you examine the code I’m providing, the changes I’m making to it over time, and the affected files. You can view a source file as it was at any point in time (wouldn’t it be great if this also included the future?).

Here’s one more gem: say you are looking at the enc28j60.cpp file, and wondering when what change was made. There’s a way to see where each change came from (and the look at that specific change in a larger context, for example). This annotated source file is generated by “git blame” – follow this link as example.

So what VCS, Git, and GitHub offer is really no less than a Time Machine. But even if you don’t ever use that, Git gives you the ability to keep your code in sync with the changes happening in the original source code repository. And if something gets messed up, you can revert to an older version of the code – without having copies on your disk, which need to be given meaningful names so you can find them back. The “git diff” feature alone is worth it.

Wanna get started? Terrific. Cast your doubts aside, and keep in mind that you need a little time to get used to it.

How to start using Git (and GitHub) depends a bit on your operating system: on Windows, best option might be to install TortoiseGit (see also this intro, which goes a bit deeper). On the Mac, it’s included with the Xcode developer tools, but there’s actually a great app for it called GitHub for Mac, and on Linux it’s probably one of the best supported packages out there, since it came from the same guy who made Linux possible in the first place…

With Git, you can stay up to date, you’ll never lose changes, and you never have to guess who changed what, when, where, why, or how. Is there a new update? Try it! Doesn’t work as expected? Back out! It’s that simple.

After part 1 and part 2, I’d like to conclude with a description of how everything fits together.

The main functions in the public API are:

• uint8_t rf12_initialize (uint8_t id, uint8_t band, uint8_t g)
  Sets up the driver and hardware, with the RFM12B in idle mode and rxstate set to TXIDLE.

• uint8_t rf12_recvDone ()
  If in receive mode, check whether a complete packet has been received and is intended for us. If so, set rxstate to TXIDLE, and return 1 to indicate there is a fresh new packet. Otherwise, if we were in TXIDLE mode, enable the RFM12B receiver and set rxstate to TXRECV.

• uint8_t rf12_canSend ()
  This only returns true if we are in TXRECV mode, and no data has been received yet, and the RFM12B indicates that there is no signal in the air right now. If those are all true, set rxstate to TXIDLE and return 1.

• void rf12_sendStart (uint8_t hdr, ...)
  This function may only be called right after rf12_recvDone() or rf12_canSend() returned true, indicating that we are allowed to start transmitting. This turns the RFM12B transmitter on and set rxstate to TXPRE1.

• void rf12_sendWait (uint8_t mode)
  Can be called after rf12_sendStart() to wait for the completion of the transmission. The mode arg can be used to do this waiting in various low-power modes, to minimize the ATmega power consumption while the RFM12B module is drawing a relative large current (ca 25 mA at full power). This call is optional – either way, the RF12 driver will return to TXRECV mode after the transmission.

Note that this design places all time-critical code inside the RF12 driver. You don’t have to call rf12_recvDone() very often if you’re busy doing other things. Once a packet has been received, the RF12 driver will stop all further activity and prevent the current contents of the packet buffer from being overwritten.

Likewise, since you have to ask permission to send, the logic is such that the packet buffer will only be filled with outgoing data at a time when it is not being used for reception. Thus, a single packet buffer can be used for both.

The main thing to keep in mind, is that rf12_recvDone() needs to be called as main polling mechanism, even if you don’t care about incoming packets. You can’t simply call rf12_canSend() forever, hoping that it will return 1 at some point. Instead, use this logic:

    while (!rf12_candSend())
        rf12_recvDone();
    rf12_sendStart(...)

This code ignores all incoming packets, i.e. when rf12_recvDone() returns true, but the call is still needed to keep the finite state machine in the driver going.

So much for the general structure and flow through the RF12 driver. To find out more about the protocol and how the different header fields are used, see this post and this post. And lastly there’s the RF12 library‘s home page.

Yesterday, I described the big picture of the RF12 driver: a public polling-type API with a private interrupt-driven finite state machine (FSM). The FSM turns the RF12 driver into a background activity for the ATmega.

But first, let’s take a step back. This is how you would write code to send out a packet without FSM:

• set up the RFM12B for transmission
• feed it the first byte to send (i.e. 0xAA, the start of the preamble)
• wait for the RFM12B to request more data
• feed it more bytes, as long as there is more to send
• then terminate transmit mode and reset the RFM12B mode to idle

Here’s an old sketch which does exactly this, called rf12xmit.pde.

The problem with this code is that it keeps the ATmega occupied – you can’t do anything else while this is running. For sending, that wouldn’t even be such a big deal, but for reception it would be extremely limiting because you’d have to poll the RFM12B all the time to avoid missing packets. Even a simple delay() in your code would be enough to miss that first incoming byte – keep in mind that although packets don’t come in all the time, when they do there is only 160 µs time to deal with each incoming byte of data.

The solution is to use interrupts. It’s for this same reason that the millis() clock and the Serial handler in the Arduino runtime work with interrupts. You don’t want to constantly check them in your code. With interrupts, you can organize your own code however you want, and check whether a new RFM12B packet was received when you are ready for it. The simplest way is to add a call to rf12_recvDone() in the loop() body, and then simply make sure that the loop will be traversed “reasonably often”.

With interrupts, the RF12 driver is no longer in control of when things happen. It can’t wait for the next event. Instead, it gets activated when there is something to do – and the way it knows what the next step needs to be, is to track the previous step in a private state variable called rxstate.

As mentioned yesterday, the RF12 driver can be doing one of several different things – very roughly summarized as: waiting for reception of data, or waiting to send the next data byte. Let’s examine both in turn:

Waiting for reception: rxstate = TXRECV

The most common state for the RF12 driver is to wait for new data bytes, with the RFM12B in receive mode. Each interrupt at this point will be treated as a new data byte. Recall the packet format:

The good news, is that the RFM12B will internally take care of the preamble and SYN bytes. There won’t be any interrupts until these two have been correctly received and decoded by the RFM12B itself. So all we have to do is store incoming bytes, figure out whether we have received a full packet (by looking at the length byte in the packet), and verify the checksum at the end.

All of this happens as long as rxstate is set to TXRECV.

The driver doesn’t leave this mode when done, it merely shuts down the RFM12B receiver. Meanwhile, in each call to rf12_recvDone() we check whether the packet is complete. If so, the state is changed to TXIDLE and we return 1 to indicate that a complete packet has been received.

TXIDLE state is a subtle one: the driver could start sending data, but hasn’t yet done so. If the next API call is a call to rf12_recvDone(), then rxstate will be reset to TXRECV and we start waiting for incoming data again.

Sending bytes: rxstate = TXPRE1 .. TXDONE

I’ll skip the logic of how we enter the state TXPRE1 for now, and will just describe what happens next.

In these states, interrupts are used to keep things moving. Whenever an interrupt comes in, the driver decides what to do next, and adjusts the state. There’s a switch statement in rf12_interrupt() which looks like this:

It’s fairly compact, but the key is the “rxstate++” on the first line: the normal behavior is to do something and move to the next state. So the basic effect of this code is to proceed through each state in sequence:

    TXPRE1, TXPRE2, TXPRE3, TXSYN1, TXSYN2, ..., TXCRC1, TXCRC2, TXTAIL, TXDONE

If you look closely, you’ll see that it corresponds to the packet layout above. With one exception: after TXSYN2, the rxstate variable is set to a negative value. This is a special state where payload data is sent out from the buffer. In this context, the two header bytes are treated as payload, and indeed they “happen” to be placed in the data buffer right in front of the rest of the data, so the driver will send header bytes and payload data in the same way:

Ignore the details, just note that again there is the “rxstate++” in there to keep advancing the state.

At some point, incrementing these negative states will lead to state 0, which was cunningly defined to be the state TXCRC1. Now the switch statement takes over again, and appends the CRC etc to the outgoing data.

Finally, once state TXDONE has been reached, the interrupt code turns off the RFM12B’s transmit mode, which also means there will be no more interrupts coming in, and bumps the state one last time to TXIDLE.

This concludes today’s story. What you’re looking at is a fairly conventional way to write an interrupt-driven device driver. It may be full of trickery, but the logic is nevertheless quite clean: a public API to start things going and to pick up important state changes (such us having received a complete packet), while interrupts push through the states and “drive” the steps taken at each point.

Each interrupt does a little dance: where was I? oh yes, now do this and call me again when there is new work.

In a previous life I wrote a few Unix (not Linux) kernel drivers – this was for a PDP11 at the time. It’s amazing how the techniques and even the programming language are still the same (actually it’s more powerful now with C++). The difference is the price of the hardware – 10,000 x cheaper and affordable by anyone!

Tommorow, I’ll conclude with some more details about how everything fits together.

This is the first of 3 posts about the RF12 library which drives the RFM12B wireless modules on the JeeNode, etc.

The RF12 driver is a small but reasonably complex bit of software. The reason for this is that it has some stringent time constraints, which really require it to be driven through interrupts.

This is due to the fact that the packet data rate is set fairly high to keep the transmitter and receiver occupied as briefly as possible. Data rate, bandwidth, and wireless range are inter-related and based on trade-offs. Based on some experimentation long ago, I decided to use 49.2 kBaud as data rate and 134 KHz bandwidth setting. This means that the receiver will get one data byte per 162 µs, and that the transmitter must be fed a new new byte at that same rate.

With an ATmega running at 16 MHz, interrupt processing takes about 35 µs, i.e. roughly 20% of the time. It works down to 4 MHz in fact, with processor utilization nearing 100%.

Even with the ATtiny, which has limited SPI hardware support, it looks like 4 MHz is about the lower limit.

So how does one go about in creating an interrupt-driven driver?

The first thing to note is that interrupts are tricky. They can lead to hard-to-find bugs, which are not easy to reproduce and which happen only when you’re not looking – because interrupts won’t happen exactly the same way each time. And what’s worse: they mess with the way compiled code works, requiring the use of the “volatile” datatype to prevent compiler optimizations from caching too much. Just as with threads – a similar minefield – you need to prepare against all problems in advance and deal with weird things called “race conditions”.

The way the RF12 driver works, is that it creates a barrier between the high-level interface (the user callable API), and the lower-level interrupt code. The public calls can be used without having to think about the RFM12B’s interrupts. This means that as far as the public API is concerned, interrupt handling can be completely ignored:

Calling RF12 driver functions from inside other interrupt code is not a good idea. In fact, performing callbacks from inside interrupt code is not a good idea in general (for several reasons) – not just in the RF12 driver.

So the way the RF12 driver is used, is that you ask it to do things, and check to find out its current state. All the time-critical work will happen inside interrupt code, but once a packet has been fully received, for example, you can take as much time as you want before picking up that result and acting on it.

The central RF12 driver check is the call:

    if (rf12_recvDone()) ...

From the caller’s perspective, its task is to find out whether a new packet has been received since the last call. From the RF12′s perspective, however, its task is to keep going and track which state the driver is currently in.

The RF12 driver can be in one of several states at any point in time – these are, very roughly: idling, busy receiving a packet, packet pending in buffer, or busy transmitting. None of these can happen at the same time.

These states are implemented as a Finite State Machine (FSM). What this means, is that there is a (private) variable called “rxstate“, which stores the current state as an integer code. The possible states are defined as a (private) enum in rf12.cpp (but it can also have a negative value, this will be described later).

Note that rxstate is defined as a “volatile” 8-bit int. This is essential for all data which can be changed inside interrupt code. It prevents the compiler from applying certain optimizations. Without it, strange things happen!

So the “big picture” view of the RF12 driver is as follows:

• the public API does not know about interrupts and is not time-critical
• the interrupt code is only used inside the driver, for time-critical activities
• the RF12 driver is always in one of several well-defined “states”, as stored in rxstate
• the rf12_recvDone() call keeps the driver going w.r.t. non time-critical tasks
• hardware interrupts keep the driver going for everything that is time-critical
• “keeping things going” is another way of saying: adjusting the rxstate variable

In a way, the RF12 driver can be considered as a custom single-purpose background task. It’ll use interrupts to steal some time from the running sketch, whenever there is a need to do things quickly. This is similar to the milliseconds timer in the Arduino runtime library, which uses a hardware timer interrupt to keep track of elapsed time, regardless of what the sketch may be doing. Another example is the serial port driver.

Interrupts add some overhead (entering and exiting interrupt code is fairly tedious on a RISC architecture such as the ATmega), but they also make it possible to “hide” all sorts of urgent work in the background.

In the next post, I’ll describe the RF12 driver states in full detail.

PS. This is weblog post number 900 ! (with 3000 comments, wow)

Prompted by a question of the forum, I wanted to go bit into the way you can collect data from multiple JeeNodes as quickly as possible.

Warning: I’m completely disregarding the “1% rule” on 868 MHz, which says that a device should not be sending more than 1% of the time, so that other devices have a good chance of getting through as well (even if they are used for completely unrelated tasks). This rule is what keeps the 868 MHz band relatively clean – no one is allowed to “flood”. Which is exactly what I’m going to do in this test…

Ok, first of all note that all devices on the 868 MHz wireless ISM band have to share that frequency. It only works if at most one device is transmitting at a time. Many simple OOK transmitters, such as weather sensor nodes, don’t do that: they just send out their packet when they feel like it. Fortunately, most of them do so relatively infrequently, once every few minutes or so. And due to the 1% rule, most transmissions will be ok – since the 868 MHz band is available most of the time.

This changes when you start to try and push as much information across as you can. With one sender, it’s easy: just ignore the rule and send as much as you can. With the default RF12 settings, you should be able to get a few hundred small packets per second across. With occasional loss due to a collision with another sender.

But how do you get the maximum amount of data across from say three different nodes?

It won’t work to let them all send at will. It’s also a bit complicated to make them work in perfect sync, with each of them keeping accurate track of time and taking turns in the right order.

Here’s a simpler idea to “arbitrate media access”, as this is called: let the central node poll each of the remote nodes, and let each remote node then send out an ACK with the “requested” data only when asked.

I decided to give it a go with two simple sketches. One is the poller, which sits at the center and tries to obtain as many packets as it can:

It cycles over each of the remote node ID’s, sends them a packet, and waits briefly for a reply to come in. Note that the packet sent out is empty – it just needs to trigger the remote node to send an ACK with the actual payload.

The remote nodes each run a copy of the pollee sketch, which is even simpler:

They just wait for an empty incoming packet addressed to them, and reply with the data they want to get across. I just send the node ID and the current time in milliseconds.

Here is the result, with one poller and three pollee’s:

    1: 36974
    2: 269401
    3: 10128
    1: 36992
    2: 269417
    3: 10145
    1: 37009
    2: 269434
    3: 10163

As you can see, each node gets one packet across about once every 17 ms (this will slow down if more data needs to be sent). So that’s 6 short packets flying through the air every 17 ms, i.e. ≈ 350 packets per second.

There are ways to take this further, at the cost of extra complexity. One idea (called TDMA), is to send out one poll packet to line up the remote’s clocks, and then have them send their payload a specific amount of time later. IOW, each node gets its own “time slot”. This reduces the 6 packets to 4, in the case of 3 remote nodes.

No more collisions, but again: this will block every other transmission attempted on the 868 MHz band!

After the hardware to feed yesterday’s scrolling LED display, comes the software:

The code has also been added to GitHub as tickerLed.ino sketch.

Fairly basic stuff. I decided to simply pass incoming messages as is, but to do the checksum calculation and start/end marker wrapping locally in the sketch. The main trick is to use a large circular 1500-character FIFO buffer, so that we don’t bump into overrun issues. The other thing this sketch does is throttling, to give the unit time to collect a message and process it.

Here’s what the display shows after power-up, now that the JeeNode is feeding it directly:

(that’s the last part of the “Hello from JeeLabs” startup message built into the sketch)

So now, sending a 50-byte packet to node 17 group 4 over wireless with this text:

    <L1><PA><FE><MA><WC><FE>Yet another beautiful day!

… will update line 1 on page A and make the text appear. The unit rotates all the messages, and can do all sorts of funky stuff with transitions, pages, and time schedules – that’s what all the brackets and codes are about.

Time to put the end caps on and close the unit!

In the moving averages post a few days ago, I just picked whatever seemed reasonable for filtering – i.e. running a moving average of order 31 over 531 samples, sampling at roughly 7,000 samples/second (which is what the free-running ADC does with my clock choice). And indeed, it looks like it will work quite well.

The nice thing about moving averages done this way, is that the calculations are trivial: just add the new value in and omit the oldest one. All it takes is an N-stage sample memory, i.e. 31-int array in this case.

But diving a bit deeper into FIR filters, which includes such a moving average as simplest case, it’s clear that I was sampling more than needed. I really don’t need 531 samples to measure the peak-to-peak level of the underlying 50 Hz signal, I just need to measure for the duration of a few 50 Hz cycles (about 3 in the above case).

As it turns out, once you dive in there are many ways to improve things, and lots of online guides and tools.

There’s a lot more to FIR filter design, and there’s an amazing design technique by Parks and McClellan which lets you basically specify what cutoff frequency you want, and what attenuation you you want above a second (higher) frequency. Then I found this site with an easy to use on-line tool for visualizing it and doing all the calculations:

That’s attenuation on the Y axis vs frequency on the X axis. The gray lines are what I specified as requirement:

And this is the C code it generated for me:

    /*
    FIR filter designed with http://t-filter.appspot.com

    sampling frequency: 1000 Hz
    fixed point precision: 16 bits

    * 0 Hz - 50 Hz
      gain = 1
      desired ripple = 5 dB
      actual ripple = n/a
    * 100 Hz - 500 Hz
      gain = 0
      desired attenuation = -50 dB
      actual attenuation = n/a
    */

    #define FILTER_LENGTH 31

    int filter[FILTER_LENGTH] = {
      -226,
      -328,
      -486,
      -608,
      -628,
      -474,
      -81,
      595,
      1565,
      2793,
      4194,
      5646,
      7000,
      8103,
      8825,
      9076,
      8825,
      8103,
      7000,
      5646,
      4194,
      2793,
      1565,
      595,
      -81,
      -474,
      -628,
      -608,
      -486,
      -328,
      -226
    };

There’s a drawback in that this is no longer a moving average. Now each output of the filter is defined by applying these integer coefficients to each of the past 31 samples. So there’s more computation involved – a quick test tells me that each sample would take 100..200 µs extra (on an ATmega @ 16 MHz).

But the nice part of this is that it might support a lower-power implementation. Instead of running the ADC 531 times @ 7 KHz (with an unknown filter response), I could run the ADC 100 times, clocked on the 1 ms timer interrupt (and sleeping in between), and apply this modified FIR filter to extract a similar peak-to-peak results.

Why low power? Well, running this on batteries is probably not practical, but I might consider running this setup off a very low-power transformer-less supply. After all, the goal of these experiments is to create a simple low-cost sensor, which can then be used for all the major energy consumers in the house. I’m trying to reduce power consumption, not add yet more to it from all these extra AC current sensors!

Note that maybe none of this will be needed – a simple RC filter before the ADC pin, plus an order 7..31 moving average may well be more than sufficient after all.

But it’s good to know that this option is available if needed. And it’s pretty amazing to see how easily such DSP tasks can be solved, even for someone who has never done any digital signal processing before!

With the USB scope hookup from yesterday’s post it’s also quite easy to capture data for further experiments. It helps to have a fixed data set while comparing algorithms, so I used the DSO-2090 software again to capture over a million 8-bit samples.

The dump is binary data, but decoding the format is trivial. Some header bytes and then each value as 2-byte int. Just to show that JeeMon is also up to tasks like this, here’s the “rec2text.tcl” script I created for it:

And this is how it was used to generate a text file for the plots below:

(silly me – I’m still impressed when I see a script process over a million items in the blink of an eye…)

That’s a bit too much data for quick tests, but here are the first 10,000 values from the 75W-bulb-on-20-VAC:

There’s quite a bit of noise in there if you look more closely:

The measurements are all over the map, with values over almost the entire 0..255 range of the scope’s 8-bit ADC. The actual spikes are probably even larger, occasionally.

I hope that this is a decent dataset to test out filtering and DSP techniques. It’ll be much quicker to try things out on my Mac than on a lowly ATmega 8-bit MPU, and since both can run the same C & C++ code, it should be easy to bring things back over to the ATmega once it it does the right thing.

Hmmm… now where do I find suitable DSP filtering algorithms and coefficients?

Update – Found a great resource, thx Paul & Andreas. Stay tuned…

On to the next step in capturing samples from the direct 220V connection.

First, let me clarify why I called the first sketch “fatally flawed” – because it is, literally! The sketch required a button press to get started, a big no-no while hooked up to AC mains (isolated or not, I won’t touch it!).

The other problem I didn’t like after all, is that the sampling was taking place in burst, saving to the Memory Plug in between – which takes several milliseconds each time. That doesn’t produce a nice stream of equally-spaced ADC measurements.

So instead, I decided to redo the whole sketch and split it into two separate sketches in fact. One fills the EEPROM on the Memory plug, the other dumps it to the serial port. No more buttons or LEDs. Just a JeeNode with a Memory Plug. To make things work, a fairly peculiar series of steps has to be taken:

• upload the “saveToMem” sketch and plug in the Memory Plug
• disconnect, and hook up to the 0.1 Ω shunt etc (with power disconnected)
• insert AA battery to power the whole thing, it starts collecting
• turn on AC power
• let it run for a few seconds
• turn off AC power
• disconnect, remove the battery and Memory Plug, and reattach to USB
• upload the “saveFromMem” sketch
• open serial monitor and capture the dump to file (with copy & paste)

The key is to remove the Memory Plug as soon as it has been filled, so that the next power-up doesn’t start filling it all over again. There’s logic in the sketches to do nothing without Memory Plug.

Note that the internal ATmega EEPROM is also used to record how far the save has progressed (in units of 128 samples, i.e. 256 bytes).

It turns out that the writes to EEPROM via I2C take quite a bit of time. I settled on a 1 KHz sampling rate to avoid running into any timing issues. That’s 20 samples per 50 Hz cycle, which should be plenty to reliably identify that frequency even if its not a pure sine wave. The samples will tell.

Ok, first test run, nothing powered up. Result is mostly 511′s and a few 510′s, which is as expected – halfway the 0..1023 range:

    $ grep -n 510 unplugged.txt |wc
         115     115    1062
    $ grep -n 511 unplugged.txt |wc
       14605   14605  135044
    $ 

Next run is with the 60 W light bulb turned on a few seconds after sampling starts.

Whoops, not so good – I’m only getting 510, 511, and 512 readings, almost nothing else!

    $ grep -n 509 powered.txt |wc
           0       0       0
    $ grep -n 510 powered.txt |wc
         110     110    1012
    $ grep -n 511 powered.txt |wc
       13750   13750  126668
    $ grep -n 512 powered.txt |wc
         220     220    2026
    $ grep -n 513 powered.txt |wc
           0       0       0
    $ wc powered.txt 
       14084   14086   56366 powered.txt
    $

My conclusion so far is: can’t detect these small AC signals without additional amplification, it’s simply too weak.

It’s not really a setback, since I wasn’t planning on creating a directly-connected JeeNode setup as official solution, but it would have been nice to get a basic detection working with just a few resistors.

Maybe there’s an error in these sketches, but I’ve verified that the input senses ground as 0 and VCC as 1023, so that part at least is working as expected. I’ve placed the two sketches as a “gist” on GitHub, for reference (sampleToMem and sampleFromMem).

Back to the drawing board!

(Small change of plans, the continuation of yesterday’s post has been postponed to tomorrow)

I’ve been tipping my toes a bit in the waters of HTML5 + CSS3. Nothing spectacular, but here’s what came out:

Small “widgets” to display home readings on a web page (it’s in Dutch… but you can probably guess most of it).

There are no colors in here yet, but hey… just visit a site such as the Color Scheme Designer to fix that.

Here are the CSS styles I used:

And here is the funky code I used to generate the HTML for this in JeeMon:

It’s still a bit awkward to generate elements which need to be combined inline, i.e. <span> elements.

To see the actual generated HTML, use your browser to check the page source of the new Widgets page at http://tools.jeelabs.org/. To examine the entire file needed to generate this from JeeMon, click on this link.

For actual use, all I’ll need to change is: 1) generate the HTML with $blah Tcl variables, or 2) use JavaScript on the browser to fill in the values (driven by Server Sent Events, for example). The latter avoids page refreshes.

The separation of content, structure, and style has come a long way since the early days of HTML. It just feels right, the way this works out. Applications can be written with nearly complete independence between, eh… “looks” and “guts”. Even complete page make-overs which are well beyond the positioning capabilities of CSS can be taken care of with basic templating on the server side. It’s interesting that the elaborate and efficient templating “engines” which used to drive dynamic websites and which are still used in many frameworks are becoming irrelevant. With a bit of JavaScript smarts on the client side, the server now only needs to serve static files plus JSON data for REST-style Ajax calls. The good news for embedded home automation systems, is that very low-end servers may well turn out to be sufficient, even for sophisticated web presentations.

As I said, nothing spectacular, but I’m delighted to see how simple and cleanly-structured all this has become.

This post strays solidly into the realms of software and database architecture…

I’m having a lot of fun with JeeMon, but it’s also starting to look like a bone which is slightly too large to chew on…

As in any application, there is the issue of how to manage, represent, and store data structures. This is an area I’ve spent quite a few years on in a previous life. The nice bit is that for JeeMon I get to pick anything I like – there are no external requirements. Time will tell whether that freedom won’t become a curse.

So now I’m exploring the world of pure relational data structures again:

What you’re looking at is the definition of 4 drivers (I’m leaving off a bunch more to keep this example limited). This was rendered as nested HTML tables, btw – very convenient.

Each driver has a description of the type of values it can produce, and each set of values is grouped (as in the case of ookRelay2, which can generate data for three different types of packets it is relaying). Drivers can support more than one interface (the roomNode driver supports both remote packets and direct serial connection, for example). And drivers can expose a number of functions (in this case they all only expose a “decode” function).

In the “flat” SQL world, you’d need 5 tables to represent this sort of information. But since I’m supporting relations-as-values, it can all be represented as a single relation in JeeMon (I call ‘em “views”, btw).

Note that this isn’t a hierarchical- or network-database. It’s still purely relational. There’s an “ungroup” operator to flatten this structure as many levels as needed, along with the usual “select”, “project”, “join”, etc. operators.

There’s also something else going on. Some of the information shown above is virtual, in the sense that the information is never actually stored into a table but gets extracted on-demand from the running application. The “values” relation, for example, is produced by scanning some simple lists stored (as dicts) in a Tcl variable:

And the “functions” relation is created on-the-fly by introspecting the code to find out which public functions have been exposed in each driver.

It’s a bit unusual to think of application data in relational terms. This lets you “join” a list with another one, and do strange things such as perform selection and projection on the list of all public functions defined in the application. But it’s also extremely powerful: one single conceptual mechanism to deal with information, whether present in application variables, i.e. run-time state, from a traditional database, or as the result of a remote request of some kind. Here’s another example: perform a join between a local (scalar) variable and a dict / map / hash (or whatever you want to call associative arrays), and you end up doing key lookup.

This approach turns (at least part of) an application on its head: you don’t need to register every fact that’s relevant into a database, you launch the app and it sets up “views” which will extract what they need when they need it, from the running app itself. Don’t call us, we’ll call you!

It might seem like a complicated way to do things (indeed, it is!), but there’s a reason for this madness. I’m looking for a design which can adapt to changes – even in the code – while running. If a driver is updated to add support for Ethernet, say, then it should reflect in these data structures without further explicit effort from the programmer. I’m not there yet by a long stretch, but this sort of experimentation is needed to get there. IMO, the magic lies in not copying facts but in seeking them out when needed, combined with smart caching.

My research project which pioneered this approach is called “Vlerq”, and I’m considering resuscitating it for use in JeeMon. Trouble is, it’s quite a bit too ambitious to implement fully (I’ve hit that wall before) – so my current experimentation is aimed at finding a workable path for the near future. Perfection is the enemy of done – and right now I’m seeing my “done” horizon racing further and further away. It’s like trying to exceed the speed of light – oh, wait, didn’t someone do that the other day?

Not only is there a glcdScope sketch which is useful for displaying voltage across the 0.1 Ω shunt in my 220V experiments, there’s also a little Spectrum Analyzer sketch to try out. Yummie!

Spectrum analysis tells you what frequencies are present in a given sample of a periodic signal. Lots of math there – just Google around for “Fast Fourier Transform” (FFT) and you’ll get it all on your plate if you’re curious.

But this stuff is a bit harder than a plain signal. Here’s what I’m seeing, with just a short wire connected, picking up some 50 Hz presumably (as with the scope test) – again with the “digital phosphor” persistence:

In principle, this graph is ok: lots of signal at lower frequencies and progressively less at higher frequencies. After all, with a perfect 50 Hz sine wave and no noise, we’d expect a single peak at the start of the scale.

The repeated peaks every few pixels also look promising. With a bit of luck they are in fact the harmonics, i.e. the 100 Hz, 150 Hz, … multiples – which is what you get when a signal is repetitive but not exactly a sine wave. Harmonics are what makes music special – the way to distinguish a note played on the violin and on the piano.

But I was hoping for something else: a bit more peaks at the right hand side of the graph. This would indicate that there are high frequencies in the signal, the computer’s switching power supply close to this setup, for example.

And worse: I’m seeing a completely flat line when hooking this up to the 220V current shunt. Looks like this signal is too weak to play FFT games with (should the data be auto-scaled?).

Anyway, here’s the glcdSpectrum50 sketch:

It relies on the fix_fft.cpp file by Tom Roberts which does the FFT heavy-lifting (original is here).

Also, note that I’m using a slightly different algorithm this time to determine the average signal value: the average of the last 256 samples is used to compute the center value subtracted from the next 256 samples. The outcome should be similar, as long as the signal is indeed symmetric around this value.

All in all a nice try, but it didn’t really provide much new insight (yet?).

Yeah, ok, this is probably a bit off topic…

I’m talking about installing Unix’y software on the Mac (which has a big fat GUI on top of a Posix-compliant OS).

There have been several projects in the past which have addressed the need to use a Mac as a Unix command-line system with all the packages coming out of the Linux, FreeBSD, etc. universes. The places where “tar”, “patch”, “make”, “autoconf”, “libtool”, and so on rule.

First there was Fink – implemented in Perl:

The Fink project wants to bring the full world of Unix Open Source software to Darwin and Mac OS X. We modify Unix software so that it compiles and runs on Mac OS X (“port” it) and make it available for download as a coherent distribution. Fink uses Debian tools like dpkg and apt-get to provide powerful binary package management. You can choose whether you want to download precompiled binary packages or build everything from source.

Then came MacPorts – implemented in Tcl:

The MacPorts Project is an open-source community initiative to design an easy-to-use system for compiling, installing, and upgrading either command-line, X11 or Aqua based open-source software on the Mac OS X operating system. To that end we provide the command-line driven MacPorts software package under a BSD License, and through it easy access to thousands of ports that greatly simplify the task of compiling and installing open-source software on your Mac.

And now there’s Homebrew – implemented in Ruby:

Homebrew is the easiest and most flexible way to install the UNIX tools Apple didn’t include with OS X. Packages are installed into their own isolated prefixes and then symlinked into /usr/local.

It’s interesting to note that all systems were written in a scripting language. Which makes perfect sense, given that they offer a huge jump in programmer productivity and no downsides to speak of in this context.

As for my pick: Homebrew gets it right.

Homebrew (i.e. the “brew” command) is installed with a one-liner:

    /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

This is a big deal, because the last thing you want is to use a package manager to overcome installation hassles, only to end up with… an extra step which introduces its own installation hassles!

The actual design of that one-liner is quite clever: it points to a script in a secure area, with full history of all changes made to that script, so you can make sure it won’t do weird (or bad) things with your system.

Homebrew doesn’t introduce new conventions (like /sw/ or /opt/, which need to be added to your exe search path). It’ll simply install the packages you ask for in /usr/local/bin etc (which means you don’t need to sudo all the time). And it does it intelligently, because it actually installs in /usr/local/Cellar and then puts symlinks in /usr/local/*. Which means there’s a way out of an installation, and there’s even a (drastic) way to get rid of all Homebrew-installed packages again: delete Cellar, and remove the dangling symlinks from /usr/local/*. Not that you’d ever need to – Homebrew supports uninstalls.

It’s succinct (and colorized):

No endless lists of compiler commands on my screen, telling me absolutely nothing, other than “CPU == busy!”.

It does the right thing, and works out all the dependencies:

    fairie:~ jcw$ brew search rrd
    rrdtool
    fairie:~ jcw$ brew install rrdtool
    ==> Installing rrdtool dependency: gettext
    ==> Downloading http://ftpmirror.gnu.org/gettext/gettext-0.18.1.1.tar.gz
    ######################################################################## 100.0%
    ==> Downloading patches
    ...
    ==> Summary
    /usr/local/Cellar/gettext/0.18.1.1: 368 files, 13M, built in 5.4 minutes
    ==> Installing rrdtool dependency: libiconv
    ==> Downloading http://ftpmirror.gnu.org/libiconv/libiconv-1.14.tar.gz
    ...
    ==> Summary
    /usr/local/Cellar/libiconv/1.14: 24 files, 1.4M, built in 70 seconds
    ==> Installing rrdtool dependency: glib
    ... etc, etc, etc ...
    ==> Installing rrdtool
    ==> Downloading http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.5.tar.gz
    ######################################################################## 100.0%
    ==> Patching
    patching file configure
    Hunk #1 succeeded at 31757 (offset 94 lines).
    Warning: Using system Ruby. RRD module will be installed to /Library/Ruby/...
    Warning: Using system Perl. RRD module will be installed to /Library/Perl/...
    ==> ./configure --prefix=/usr/local/Cellar/rrdtool/1.4.5 --mandir=/usr/local/Cel
    ==> make install
    /usr/local/Cellar/rrdtool/1.4.5: 148 files, 3.2M, built in 68 seconds
    fairie:~ jcw$ 

It does its work with “formulas”, i.e. Ruby scripts which describe how to fetch, build, and install a certain package. The formulas (formulae?) are managed on GitHub, which means that there is a massive level of collaboration going on: new packages, issue tracking, fixes, and discussion. Here’s what I got when self-updating it again after having done so one or two days ago:

    $ brew update
    remote: Counting objects: 294, done.
    remote: Compressing objects: 100% (103/103), done.
    remote: Total 250 (delta 174), reused 212 (delta 143)
    Receiving objects: 100% (250/250), 39.46 KiB, done.
    Resolving deltas: 100% (174/174), completed with 32 local objects.
    From http://github.com/mxcl/homebrew
       36f4400..99bc0b7  master     -> origin/master
    Updated Homebrew from 36f4400e to 99bc0b79.
    ==> New formulae
    apktool             hexedit             p11-kit             solid
    denyhosts           jbigkit             qi                  tinyfugue
    gearman-php         kbtin               shen
    graylog2-server     opencc              sisc-scheme
    ==> Updated formulae
    android-ndk         ffmpeg              libquicktime        python
    aqbanking           frink               libraw              python3
    audiofile           fuse4x              libvirt             sleepwatcher
    cassandra           fuse4x-kext         nasm                tomcat
    class-dump          gflags              nginx               transcode
    csshx               google-sparsehash   nss                 tsung
    dash                gpsbabel            pdfjam              xml-security-c
    dvtm                gwenhywfar          pixman*
    elasticsearch       libiconv*           pos
    $

But best of all IMO, the formulas use an almost declarative style. Here’s the one for re2c I just installed:

It’s not hard to guess what this script does. No wonder that so many people submit and tweak such formulas.

Note that there’s neither a description, nor explicit version handling in there (other than what can be gleaned from the download URL). Brew is an installer, not a catalog. Wanna know more? Visit the home pages. Brilliant.

It’s great to see the Ruby community adopt a concise, declarative, and Domain Specific Language style. Nothing new or revolutionary – has been done many times before – but nevertheless underappreciated, IMNSHO.

Here’s a web page I’ve been designing recently – one of many planned for the JeeMon system I’m working on:

A couple of notes:

• there are three “master” tables on this page: Devices, Interfaces, and Drivers
• devices are a 1:N mapping between drivers and interfaces
• … except that some devices are remote, and not tied to a local interface
• clicking on any row of these three tables displays details in the rightmost column
• drivers are listed but not managed on this page, that’s a software admin task

I’m describing this here for a number of reasons (other than showing off what I’ve been spending my days on).

First of all, you can see that this more or less covers what needs to be set up to in a home monitoring and/or home automation system. I like to see everything at a glance, i.e. on a single page and with as little navigation as possible. With this setup, I hope to keep it all essentially on one page. Whereby the right-side column may vary widely, depending on the device / interface / driver.

But it doesn’t really need to be about home automation at all. A very similar setup could be used to hook up to devices which I’m experimenting with in the lab, or devices which act like measuring instruments, or control some aspect of an experiment. Anything related to Physical Computing, really.

The other thing that interests me is the “degree of variety” needed to cover all the cases. Many devices will simply collect or send out one or more values, but not all devices are like that.

The RF12demo sketch is essentially a gateway from and to an entire group of RFM12B-based nodes. Each node can have its own driver. The ookRelay2 sketch is similar: it collects data from quite different devices, each of them sending out their specific packets with their specific types of measurements. It differs from RF12demo, in that it contains decoders for all the devices it knows about. There are no separate KS300 drivers, etc (although perhaps there should be, since some code is currently duplicated in the CUL driver).

The autoSketch driver is yet another beast. It listens on a serial interface for a line of the form “[...]” and when it sees one, tries to load a driver with the name in brackets and hand over control to it. This is the reason why just about all my sketches start off with the following boilerplate code:

    void setup() {
        Serial.begin(57600);
        Serial.println("\n[mySketch]");
        ...
    }

When associated with the autoSketch driver, this will automatically load a “mySketch” driver, if present. Which in turn means that all you have to do is plug in the JeeNode (JeeLink, Arduino, etc) and JeeMon will automatically detect it and start communicating with it. IOW, plug-and-play – with a simple implementation.

This is why there’s a “Use autoSketch for new interfaces” checkbox. I could have called it “Enable plug & play”.

But although this web page is functional in my test setup, it’s far from finished. The hardest part I want to get right, is to make the page completely dynamic. Right now, a click on a row will update the details via Ajax, but nothing will happen when there is no user interaction. What I want is that the page automatically adjusts when a JeeLink is plugged in (on the web server side). What I really want, is to generalize that mechanism to everything shown on any web page. The challenge is to do this without tedious periodic polling or complete table refreshes, just “pushing” changes in an event-driven manner. Events on the server side, that is.

The visual layout, styling, and behavior of a page like this has become very simple with today’s tools:

• jQuery is the core JavaScript library
• jQuery UI is used as basic style engine (the tabbed group on the right, for example)
• jQuery TableSorter adds support for (surprise!) table sorting via their headers
• Knockout is used to manage a lot of the user-facing behavior in the browser
• 960 grid regulates the page structure flow in CSS
• there is a small amount of custom CSS and custom Javascript (about 20 lines each)
• the entire page is implemented as a single Tcl source file (175 lines so far)

There are a few trivial utility functions in there, which are bound to migrate to a more general module once I better understand the trade-offs and benefits of re-use. And the rest is gobbledygook which looks like this:

This is the notation I came up with and described in my Playing with indentation post a short while back. This particular code describes the HTML text returned for drivers in the “details” column, via Ajax.

This notation gets expanded to HTML, but the real value of it is that describing the HTML on a page is fun, because it’s now trivial to insert grouping levels (i.e. mostly <div>’s) and rearrange stuff until it’s just right.

I wanted to learn a bit more about how to implement non-trivial forms in HTML + JavaScript. Without writing too much HTML, preferably…

There are a few versions of a tool floating around the web, which help calculate the constants needed to configure the RFM12B wireless module – such as this recent one by Steve (tankslappa).

Putting one and one together – it seemed like an excellent exercise to try and turn this into a web page:

If only I had known what I had gotten myself into… it took more than a full day to get all the pieces working!

In fairness, I was also using this as a way to explore idioms for writing this sort of stuff. There is a lot of repetitiveness in there, mixed with a frustrating level of variation in the weirdest places.

Due to some powerful JavaScript libraries, the result is extremely dynamic. There is no “refresh”. There is not even a “submit”. This is the sort of behavior I’m after for all of JeeMon, BTW.

The calculator has been placed on a new website, called http://tools.jeelabs.org/. The page is dynamically generated, but you can simply save a copy of it on your own computer because all the work happens in the browser. The code has been (lightly) tested with WebKit-based browsers (i.e. Safari and Chrome) and with FireFox. I’d be interested to hear how it does on others.

If you want to look inside, just view the HTML code, of course.

As I said, it’s all generated on-the-fly. From a page which is a pretty crazy mix of HTML, CSS, JavaScript, and Tcl. With a grid system and a view model thrown in.

You can view the source code here. I’ve explored a couple of ways of doing this, but for now the single-source-file-with-everything-in-it seems to work best. This must be the craziest software setup I’ve ever used, but so far it’s been pretty effective for me.

Why use a single programming language? Let’s mash ‘em all together!

For yesterday’s setup, I had to write a bit of code. Here’s the main part:

(this “powerACS” sketch can be found here)

The idea is to keep reading the analog pin as often as possible (several thousand times per second), and then keep track of its moving average, using this little smoothing trick:

    avg = (499L * avg + value) / 500;

Each new measurement adds 1/500th weight to the average. I didn’t want to use floating point, so it takes some care to avoid long int overflow. Here’s how I scale the input (0 .. 1023) to microvolts (0 .. 3,300,000):

    uint32_t value = measure.anaRead() * (3300000L / 1023);

Note that 500 x 3300000 is under the max of a 32-bit int, so the above “avg” calculation is ok.

It’s easy to calculate the offset, once we know the average voltage. Keep in mind that we’re measuring 50 Hz AC, and that the result we’re after is the fluctuation – i.e. what you get when putting a multimeter in VAC measurement mode. The way to do that is to calculate the “average absolute difference from the average”:

IOW, what I want is the average height of those red bars. I flipped the lower half up because we’re interested in the absolute difference (otherwise the bars would just cancel out).

Here’s the calculation:

    if (value > avg)
      total += value - avg;
    else if (value < avg)
      total += avg - value;
    ++count;

When the time comes to report a “range” reading, we calculate it as follows and reset the other variables:

    uint32_t range = total / count;
    total = count = 0;

Here’s some sample debug output on the serial port:

    [powerACS]
    ...
    2560650 2569286 5916 31973363 5404
    2576775 2569256 5911 32175178 5443
    2563875 2569266 5917 32888792 5558
    2567100 2570205 5911 33200506 5616
    2573550 2569935 5917 32798060 5543
    2580000 2565675 5910 183251691 31007  <- 100 W light bulb
    2573550 2565108 5916 356840654 60317
    2570325 2565479 5916 355916529 60161
    2512275 2565467 5910 355676919 60182
    2525175 2565688 5916 356664359 60288
    2486475 2566547 5909 354854663 60053
    2480025 2566631 5916 355567509 60102
    2480025 2568658 5910 355608745 60170
    2583225 2569370 5916 178542547 30179  <- off
    2576775 2569122 5915 33193018 5611
    2570325 2569274 5911 33236267 5622
    2560650 2568988 5917 32873935 5555
    2576775 2569290 5911 32733630 5537
    2567100 2568564 5917 33390738 5643
    ...
    2560650 2569179 5917 32917499 5563
    2576775 2568640 5911 32354114 5473
    2570325 2569121 5916 32820006 5547
    2576775 2568319 5917 33896284 5728
    2570325 2569996 5911 33120608 5603
    2567100 2568514 5917 37697489 6371    <- 1.1 W power brick
    2550975 2568582 5911 36764137 6219
    2567100 2568767 5916 37434853 6327
    2567100 2568813 5911 36272763 6136
    2563875 2568659 5917 37738504 6377
    2563875 2568355 5911 37911395 6413
    2567100 2568586 5917 37444158 6328
    2563875 2568672 5916 37607002 6356
    2580000 2568879 5911 37293083 6309    <- off
    2570325 2568330 5917 33082874 5591
    2563875 2567713 5911 33007567 5584
    2567100 2568380 5917 32770050 5538
    2580000 2568846 5910 33002639 5584
    2570325 2568844 5917 33201661 5611

You can see the 60 mV signal when the power consumption is 100 W. Which is a bit low: I would have expected 100 W / 230 V * 0.185 V/A = 80 mV. But it turns out that the lamp really is an 87 W lamp, so that makes it 70 mV. I’ll attribute the remaining difference to my crude setup, for lack of a better explanation.

What’s worse though, is that there is always a 5.5 mV “hum” when no current is flowing (even when the whole circuit isn’t anywhere near a power line). One possible explanation could be ADC measurement noise: the 10-bit ADC resolution is 3.3 mV. Then again, note how multiple measurements do lead to a much finer resolution: the 60 mV readout fluctuates by only a fraction of one millivolt. Noise actually improves resolution, when averaged over multiple readings.

Still, the 5.5 mV baseline value makes it hard to reliably measure really low power levels. The 1.1 W test was from an unloaded power brick, i.e. an inductive load. It causes the output to jump from 5.5 mV to only 6.3 mV, which is not a very distinct result. Another 0.3 W test load didn’t even register.

Conclusion: this detects current fairly well, was very easy to set up, and offers galvanic isolation protection. But it does require a hookup to AC mains power and it’s not super effective at really low power levels.

As I mentioned in a recent post, the collectd approach fits right in with how wireless RF12 broadcast packets work.

Sounds like a good use for an EtherCard + JeeNode pair (or any other ENC28J60 + RFM12B combo out there):

The idea is to pass all incoming RF12 packets to Ethernet using UDP broadcasts. By using the collectd protocol, many different tools could be used to further process this information.

What I did was take the etherNode sketch in the new EtherCard library, and add a call to forwardToUDP() whenever a valid RF12 comes in. The main trick is to convert this into a UDP packet format which matches collectd’s binary protocol:

The sendUdp() code has been extended to recognize “multicast” destinations, which is the whole point of this: with multicast (a controlled form of broadcasting), you can send out packets without knowing the destination.

The remaining code can be found in the new JeeUdp.ino sketch (note that I’m now using Arduino IDE 1.0beta2 and GitHub for all this).

It took some work to get the protocol right. So before messing up my current collectd setup on the LAN here at JeeLabs, I used port 25827 instead of 25826 for testing. With JeeMon, it’s easy to create a small UDP listener:

And sure enough, eventually it all started to work, with RF12 packets getting re-routed to this listener:

Now the big test. Let’s switch the sketch to port 25826 and see what WireShark thinks of these packets:

Yeay – it’s workin’ !

The tricky part is the actual data. Since these are raw RF12 packets, with a varying number of bytes, there’s no interpretation of the data at this level. What I ended up doing, is sending the data bytes in as many collectd 64-bit unsigned int “counter values” as needed. In the above example, two such values were needed to represent the data bytes. It will be up to the receiver to get those values and convert them to meaningful readings. This decoding will depend on what the nodes are sending, and can be different for each sending JeeNode.

I’ve left the original web browser in as well. Here is the “JeeUdp” box, as seen through a web browser:

(please ignore the old name in the title bar, the name is now “JeeUdp”)

It’s not as automatic out of the box as I’d really like. For one, you have to figure out which IP address this unit gets from DHCP – one way to do so is to connect it to USB and open the serial console. The other bit is that you need to configure the unit to set its name, the UDP port, and the RF12 settings to use. There’s a “Configure” link of the web page to do this – a some point, I’d like to make JeeMon aware of this, so it can do the setup itself (via http). And the last missing piece of the puzzle is to hook this into the different drivers and decoders to interpret the data from these UDP packets in the same way as with a JeeLink on USB.

Ultimately, I’d like to make this work without any remote setup:

• attach Ethernet and power to this box (any number of them)
• each box starts reporting its status via UDP (including its IP address)
• a central JeeMon automatically recognizes these units
• you can now give a name to each box and fill in its RF12 configuration
• packets start coming in, so now you can specify the type of each node
• decoders kick in to pick up the raw data and generate meaningful “readings”
• that’s it – easy deployment of JeeNode-based networks is on the horizon!

Not there yet, but all the essential parts are working!

I’ve been making good progress since the recent post.

Some things work, some things don’t when you use “off-the-shelf Open Source”. For example, I started using Twitters Bootstrap as CSS grid framework for page layout. It looks great. But there was too much interference with other components, such as jQuery DataTables. Well, either that or I simply don’t get it. So I ended up removing it all, in favor of the 960 Grid System (note how all JavaScript development seems to be happening on GitHub).

So now I’ve got a first rough cut at the “Niner” layout, i.e. 3×3 pages accessed via tabs on the bottom/right. Here’s a screen shot from an iPad:

The tabs don’t look so great (I’m so inept at GUI design that I don’t even try), but the general design is shaping up nicely. The area at the bottom left shows the last two lines of the log on the server, and it’s updating in real time. So is the table itself, using Server-Sent events.

With this sort of dynamics, all sorts of details start to jump out. My first approach was to have one page per URL, i.e. “http://127.0.0.1:8181/1″ for page 1, etc. But that causes a full page refresh, which is too disruptive, especially with that log area constantly updating.

Which is where JavaScript and Ajax come in, of course. So now navigation only updates the main area – not the bottom & right-side borders. I’m only testing with Safari/Chrome, and occasionally FireFox, BTW.

The indentation idea I described earlier is also working out nicely. Here’s the main page description which produces the HTML sent to the browser:

(as you can see, there are limits to syntax coloring when mixing several languages in a single source file!)

And here’s the description of the page with that table shown above:

Let me just say that I’m “over” HTML. Bye bye, <div> et al. Good riddance.

PS. Oh yes, and here’s the fun part – this defines the structure of the site:

Tabs and titles are automatically generated unless overridden in this layout.

I’m building a “home monitoring and automation” system. Not just the hardware “nodes”, but also the software side of things, i.e. a server running permanently. Web-based, so that it can be accessed from anywhere if needed.

I’ve got a lot of code lying around here, ready to be put to use for just that. It’s not that hard to create a website (although truly dynamic displays are perhaps a little harder).

One thing I’ve been struggling with for a long time, though, is the overall design. Not so much the look & feel, but the structure and behavior of it all.

I’m not too keen on systems with elaborate menu structures, tons of hoops dialogs to jump through, even scrolling messes with our innate sense of location. Been there, done that. Modes suck – they just create labyrinths.

What I’ve come up with is probably not for everyone (heh, nothing new here). An entire system, organized as nine pages. No more, no less. Instant navigation with a keyboard (digits, function keys, whatever), and trivial navigation with a mouse or touch screen:

Even a laptop screen with 3+3 buttons (or a joystick) could navigate to any page with just two clicks.

Here is a first exploration of what these pages could be for (vaporware, yeay!):

Each page is nearly the full screen, with only the bottom and right-side borders used. And each page can have any content, of course: tabs, sidebars, widget collections, it’s still all up in the air. There’s plenty of software to extend this system infinitely, using HTML, CSS, and JavaScript.

One of the things I’d like to do with this, is make the system automatically return to the main Status Overview page when idle for a minute or so. With the option to “pin” one of the other pages to prevent this from happening.

This might all seem very basic, inconsequential, and even arbitrary, but for me it’s a major step forward. I’ve been running around in circles for ages, because there was no structure in place to start assembling the different parts of the site. It’s all nice and well to develop stuff and tinker endlessly behind the scenes, but in the end this still needs to lead to a practical setup.

So now there’s a plan!

Bear with me – this isn’t directly related to Physical Computing, but I’ve been having fun with some code.

If you’ve ever written any web apps or designed a web site, chances are that you’ve also written some HTML, or at least HTML templates. Stuff like this, perhaps:

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset='utf-8' />
        <title>Flot graph demo</title>
        <script type='text/javascript' src='.../jquery.flot.js'></script>
        <script type='text/javascript'>
          $(function(){ ... });
        </script>
        <style type='text/css'>
          #placeholder { width: 600px; height: 300px; }
        </style>
      </head>
      <body>
        <div id='placeholder'></div>
      </body>
    </html>

Well, I’m getting a bit tired of all this nesting. The essence of it is buried far too deeply inside what I consider line noise. There are currently a number of developments related to HTML, CSS, and JavaScript which might change this sort of development considerably. Much of it seems to be driven by the Ruby and Rails community, BTW.

Some examples: for HTML, there is Haml. For CSS, there is Sass. And for JavaScript there is CoffeeScript.

All of these have in common (along with Python and YAML) that they treat indentation as a significant part of the notation. Before you raise an eyebrow: I know that the opinions on this are all over the map. So be it.

I wanted to try this with JeeMon, so after a bit of coding, I ended up with a “Significant Indentation Formatter” and then applied that to HTML. Here’s the above, re-written in a HAML-like notation:

    !html
      head
        meta/charset=utf-8
        title: Flot graph demo
        script/type=text/javascript/src=.../jquery.flot.js
        script/type=text/javascript
          $(function(){ ... });
        style/type=text/css
          #placeholder { width: 600px; height: 300px; }
      body>#placeholder

It generates the same HTML as above, but there’s less to write. Even less in fact, because I’ve also integrated it with JeeMon’s templating system:

    !html
      head
        meta/charset=utf-8
        title: Flot graph demo
        [JScript includes flot]
        [JScript wrap $js]
        [JScript style { #placeholder { width: 600px; height: 300px; } }]
      body>#placeholder

Here’s a slightly more elaborate KAKU send example which illustrates basic template looping:

    !html
      head
        meta/charset=utf-8
        title: KAKU send
        [JScript includes ui]
        [JScript wrap $js]
      body
        % foreach x {1 2 3 4}
          p.toggles
            button#on$x: On $x
            button#off$x: Off $x
        p#group
          label: Group:
          % foreach x {I II III IV}
            input#g$x/type=radio/name=g/value=$x
            label/for=g$x: $x
        p
          label: House Code:
          select
          % foreach x {A B C D E F G H I J K L M N O P}
            option/value=$x: $x

So far, this system doesn’t use standard conventions, is undocumented, hasn’t been used in anger or proven itself to handle more complex layouts, has almost no error handling, and has only been used by yours truly for a few hours. Still, I’ve adjusted all the examples in JeeRev to gain a bit more experience with this approach.

I like it. The above is pretty close to how I want to design the structures and sections of a web page.

As mentioned a few days ago, I’m in the process of moving all my software development to GitHub.

In a nutshell: Git is a Version Control system which lets you keep track of changes to groups of files, and do it in such a way that many people can make changes and nothing will ever get lost. You might feel lost if you really make a mess of it, but every change is tracked and can be recovered and compared against any other point in time. Based on git, GitHub takes it all to the next level, by fostering collaboration and combining tons of useful features – for free, via their website. GitHub is to git as JavaScript is to assembler (the analogy is a bit rough).

If you’re writing code, there is no excuse for not using a VCS. Even as a single developer, even if no-one else ever gets involved, and even if you’re the most organized person on the planet. An organized developer? Hah! :)

Life without a VCS looks like this:

• let’s write some code – great, it works!
• let’s try out something more difficult now
• drat, didn’t work, I hope my editor undo buffer lets me back out
• let’s make a copy of what worked before
• now let’s write some more code – great, it works!
• … time passes …
• time to write code again – hm, I messed up
• where was that copy I made earlier, again?
• drat, it doesn’t have my latest changes
• do I remember which part worked, and what I added?
• … time passes …
• I want to continue on that first attempt again
• hmm, guess I’ll start from scratch
• no wait, let’s make a copy before I mess up
• … time passes, the disk fills with copies, none of them work …

Now let’s revisit that with a VCS:

• write some code, check it in – add a brief comment
• try out something more difficult now
• check it in, let’s add a note that it’s still experimental
• hm, can’t seem to get it to work
• oh, well, I’ll go back to the previous version – easy!
• writing more code, having fun, we’re on a roll
• let’s check it in right away
• … disaster strikes, my laptop breaks down
• get it fixed or get a new one, check out from GitHub
• where was I? oh yes, got one good version and two bad ones
• let’s switch to that failed attempt again
• wait, let’s first merge the latest good changes in!
• cool, now I’ve got an up-to-date set of files to work on
• oh, darn, still can’t make it work, let’s save it with a note
• … time passes …
• ok, time to review the different attempts and take it from there

Note that there are no more copies of files. Only a repository (on your laptop and on GitHub – but you still need to back it up, like everything else). The repository contains the entire history of all the changes, including notes, backing out steps, merges, everything.

A lot of this is possible with any modern VCS, including the “subversion” setup I’ve been using until now. But GitHub makes it effortless, and throws in a lot of nice extras. You still need a “git” application on your laptop (there are several versions), and if you’re using a Mac, you can use GitHub’s own app to communicate with GitHub without having to go through a browser.

Here’s what a “diff” looks like, i.e. a comparison of the source code, showing only the differences:

One line was replaced, two more lines were added at the end. It’s totally obvious, right?

Here’s a history of the “commits” made, i.e. the times when a new version was sent to git and GitHub:

Almost everything is a link – you can compare versions, find out what else was changed at the same time, see what else any of the developers is working on, and of course download a full copy of any version you want.

The other mechanism I’d like to describe is branching and merging. That change shown above was not made by me but by “Vliegendehuiskat” (hello, Jasper :) – and the way it works is quite neat, if you’ve never seen collaboration via VCS in action.

What Jasper did, was create a “fork” of JeeLib, i.e. a personal copy which knows exactly where it came from, when, and from which version of which files. And then he made a small change (without even telling me!), and at some point decided that it would be a good idea to get this change back into the official JeeLib code. So he submitted a “pull request”, i.e. a request for me to pull his changed version back in:

This generated a new entry in GitHub’s issue tracker and an email to me. I liked his change and with one click, merged it back in. Only thing left to do was to “pull” those changes to my laptop copy as well.

GitHub has a graphical “network” view to display all commits, forks, branches, and merges:

Each dot is a commit. The black line is my code, the blue diversion shows Jasper’s fork, his commit, and my merge. Hovering over any of these dots will show exactly what happened.

This illustrates the simplest possible case of GitHub for collaboration. If you want to see how a entire team puts git and GitHub at the center of a large-scale collaborative software development effort, read this article by Scott Chacon (who works with and at GitHub, BTW).

I’ve been moving lots of stuff to GitHub these past few days, not all of it related to JeeLabs. Here are the three main projects, as far as physical computing goes:

Note another nice feature: you can instantly see how active a project is. JeeLib is the brand new combination of the Ports and RF12 libraries, while EtherCard and GLCDlib have been around a bit longer (and the full change history has been imported). It’s been over half a year since Steve and I worked on GLCDlib, as you can see – with a small blip before the summer, when I added font support.

There are a few features which I haven’t enabled at GitHub. One of them is their wiki – because having yet another spot on the web with pages related to JeeLabs would probably do more harm than good. The Café at http://jeelabs.net is as before the place to look for documentation. There are ways to merge this stuff, so maybe I’ll go there one day.

A warning: the “sketches” on GitHub have all been adjusted for the new upcoming Arduino 1.0 IDE, and do not work with Arduino 0022, which is still the main version today. So I’m slightly ahead of the game with this stuff.

I’m really looking forward to how the software will evolve – and you’re cordially invited to be part of it :)

PS. I’m updating to GitHub fairly often because I work on two different machines – and this is the easiest and safest way to keep stuff in sync.

Yesterday, the WordPress server dropped out. Looks like some joker decided to grab the whole site, no throttling, nothing. Oh well, I guess that’s what fat pipes lead to.

In principle, WordPress should be able to handle such things, but I think it caused the underlying MySQL to run out of memory and abort.

A restart of the WordPress VM fixed it, once I found out. But this gives me an excuse to describe some things I’ve been starting to set up here.

One of of things I did this summer, was set up a collectd process on most of the machines running here – collectd is a bit Linux-centric, but one of the things it can do is grab various system statistics and broadcast it on the LAN over UDP. There are loads of plugins, and you can interface to it with scripts to feed it anything you like.

The nice thing about UDP broadcasts, is that senders don’t need to know anything about receivers. As with the RF12 driver, you just kick stuff into the network and then it’s up to others to decide what to do with it, if anything.

Collectd is also extremely low overhead, both in terms of CPU & memory load and in terms of network load. I’ve enabled it permanently on five systems, sending out 50..100 system parameters each, once every 30 seconds.

Here’s a partial view of the information being collected:

This was generated in the browser using JeeMon and jstree, as an exercise (I’m looking for a more convenient JavaScript widget to adjust a live tree using JSON).

Broadcasting is very convenient, as the collectd tool and the RF12 driver have shown, because there’s no setup. In theory, packets can get lost without acknowledgements and re-transmissions. In practice, some do indeed get lost, but very rarely. No problem at all for periodic / repetitive data collection.

I created a small script to display system status on my development Mac, using a tool called GeekTool. It can put information right on the desktop, i.e. as unobtrusively as you want, really. Here’s what I see when there are no windows obscuring the desktop background:

On the left is a periodic “ps“, the output on the right is from “ssh myserver jeemon overview.tcl“. This is a remote call to the server using a small script to pull info out of a text file and format it. The output shows that there are 5 systems online (running collectd, that is), and the script last ran at 12:41 in 39 ms.

Perhaps more interesting is that the server also collects this data from the LAN. To do this, I just launch JeeMon next to this “main.tcl” script:

    History group * 1w/5m 1y/6h
    collectd listen sysinfo

The first line is optional – it defines the parameters for historical round-robin data storage. So what I end up with, is the last 7 days of information, with details every 5 minutes, as well as up to one year of information aggregated per 6 hours.

The second line starts listening for collectd packets on the standard UDP port, tagging the results with the “sysinfo” prefix. As side effect, JeeMon will update a “stored/state.map” text file once a minute with the current values of all variables plus some other details. This is what was used to generate the above GeekTool on-screen status. There’s a built-in web server to query and access the saved historical data.

There’s no further setup. It’s up to the clients to decide what to collect and how often to send it. Data storage on the server is fixed, since this is stored in round-robin fashion (like rrdtool). It’s now collecting nearly 600 800 variables and using about 100 125 Mb of disk space. Data collection adds 2% load on a 2.66 GHz CPU.

This could also be used to collect home sensor data.