Computing stuff tied to the physical world

Archive for 2013

Wrapping up

In AVR, Hardware, News, Software, Musings on Oct 6, 2013 at 00:01

I’m writing this post while one of the test JeeNode Micro’s here at JeeLabs is nearing its eighth month of operation on a single coin cell:


It’s running the radioBlip2 sketch, sending out packets with an incrementing long integer packet count, roughly once every minute:

Screen Shot 2013-10-04 at 15.44.58

The battery voltage is also tracked, using a nice little trick which lets the ATtiny measure its own supply voltage. As you can see, the battery is getting weaker, dropping in voltage after each 25 mA transmission pulse, but still recovering very nicely before the next transmission:

Screen Shot 2013-10-04 at 15.45.45

Fascinating stuff. A bit like my energy levels, I think :)

But this post is not just about reporting ultra low-power consumption. It’s also my way of announcing that I’ve decided to wrap up this daily weblog and call it quits. There will be no new posts after this one. But this weblog will remain online, and so will the forum & shop.

I know from the many emails I’ve received over the years that many of you have been enjoying this weblog – some of you even from the very beginning, almost 5 years ago. Thank you. Unfortunately, I really need to find a new way to push myself forward.

This is post # 1400, with over 6000 comments to date. Your encouragement, thank-you’s, insightful comments, corrections and additions – I’m deeply grateful for each one of them. I hope that the passion which has always driven me to explore this computing stuff tied to the physical world technology and to write about these adventures, have helped you appreciate the creativity that comes with engineering and invention, and have maybe even tempted you to take steps to explore and learn beyond the things you already knew.

In fact, I sincerely hope that these pages will continue to encourage and inspire new visitors who stumble upon this weblog in the future. For those visitors, here’s a quick summary of the recent flashback posts, to help you find your way around on this weblog:

Please don’t ever stop exploring and pushing the boundaries of imagination and creativity – be it your own or that of others. There is infinite potential in each of us, and I’m certain that if we can tap even just a tiny fraction of it, the world will be a better place.

I’d like to think that I’ve played my part in this and wish you a lot of happy tinkering.

Take care,
Jean-Claude Wippler

PS. For a glimpse of of what I’m considering doing next, see this page. I can assure you that my interests and passions have not changed, and that I’ll remain as active as ever w.r.t. research and product development. The whole point of this change is to allow me to invest more focus and time, and to take the JeeLabs projects and products further, in fact.

PPS. Following the advice of some friends I highly respect, I’m making this last weblog post open-ended: it’ll be the last post for now. Maybe the new plans don’t work out as expected after all, or maybe I’ll want to reconsider after a while, knowing how much joy and energy this weblog has given me over the years. So let’s just call this a break, until further notice :)

Update Dec 2013 – Check out the forum at for the latest news about JeeLabs.

Flashback – Anatomy of a transmission

In Hardware on Oct 5, 2013 at 00:01

One of the really fun things I always like to work on, is to push the envelope on power savings in the JeeNode.

It all started long ago, but over the years, I did refine the measurement process as my insight and instruments both got better. Here’s the most primitive approach, inserting a multimeter in series with the power supply to measure the static current consumption:


The big problem with this is that a multimeter is far too slow to really see what’s going on. Only static idle current can be measured this way.

Another approach was to use a second JeeNode as power consumption tracker with some extra circuitry, to measure the voltage drop repeatedly:


This design can capture spikes by performing lots of repeated measurements, and in this particular case it even had a fairly large measurement range of 1 µA to 60 mA. This was done by using two separate sensing circuits and switching between them as needed.

But the real breakthrough came mid 2010, when Jörg Binkele sent me an oscilloscope snapshot of the voltage over a small series resistor. Here is one of the last ones, after lots of power saving techniques had already been applied:


As you can see from the annotations, there is an incredible amount of information one can glean from such oscilloscope traces.

This was one of the main reasons for me to learn more about oscilloscopes and eventually to buy a modern one myself – as described in a couple of posts, such as this one.

The capabilities of a modern digital storage oscilloscope (DSO) are truly phenomenal – and much of it is not even that much affected by the actual price range, although their cost does tend to keep them out of reach for occasional hobby use.

After many days of tinkering, it became possible to produce a very detailed map of the current consumption of a sensor acquisition + packet transmission cycle, as seen here:


The “haircomb” is the power consumption while sending individual bytes over SPI to the RFM12B module (and the received ACK after that), whereby the ATmega is even going to sleep between those bytes. For reference: the entire trace on this last screen is similar to the little blip marked “RFM12B transmission” on the previous screen. It’s not only considerably more detailed, but actually calibrated in 5-mA steps on the vertical axis.

Fascinating stuff, and no doubt a major reason why JeeNodes can perform so well nowadays in terms of ultra-low power consumption. If you had asked me this in 2008, I honestly would never have thought such a level of insight and performance to be attainable by anyone without access to a big research lab!

Flashback – Dive Into JeeNodes

In AVR, Hardware, Software, Linux on Oct 4, 2013 at 00:01

Dive Into JeeNodes (DIJN) is a twelve-part series, describing how to turn one or more remote JeeNodes, a central JeeLink, and a Raspberry Pi into a complete home monitoring setup. Well, ok, not quite: only a first remote setup is described with an LDR as light sensor, but all the steps to make the pieces work together are described.

More visually, DIJN describes how to get from here:

dijn01-essence.png   dijn01-diagram

.. to here:


This covers a huge range of technologies, from embedded Arduino stuff on an ATmega-based JeeNode, to setting up Node.js and the HouseMon software on a Raspberry Pi embedded Linux board. The total cost of a complete but minimal setup should be around €100. Less than an Xbox and far, far more educational and entertaining, if you ask me!

It’s all about two things really: 1) describing the whole range of technologies and getting things working, and 2) setting up a context which you can explore, learn, and hack on tinker with in numerous ways.

If you’re an experienced Linux developer but want to learn about embedded hardware, wireless sensors, physical computing and such, then this offers a way to hook up all sorts of things on the JeeNode / Arduino side of things.

If you’re familiar with hardware development or have some experience with the Arduino world, then this same setup lets you get familiar with setting up a self-contained low-power Linux server and try out the command line, and many shell commands and programming languages available on Linux.

If you’ve set up a home automation system for yourself in the past, with PHP as web server and MySQL as back end, then this same setup will give you an opportunity to try out rich client-side internet application development based on AngularJS and Node.js – or perhaps simply hook things together so you can take advantage of both approaches.

With the Dive Into JeeNode series, I wanted to single out a specific range of technologies as an example of what can be accomplished today with open source hardware and software, while still covering a huge range of the technology spectrum – from C/C++ running on a chip to fairly advanced client / server programming using JavaScript, HTML, and CSS (or actually: dialects of these, called CoffeeScript, Jade, and Stylus, respectively).

Note that this is all meant to be altered and ripped apart – it’s only a starting point!

Flashback – Easy Electrons

In Hardware on Oct 3, 2013 at 00:01

The Easy Electrons series is a set of weblog posts which were published end of 2010 and beginning 2011 to try and bring across a bit of intuition w.r.t. electronics.

Many many times, people not trained in electronics struggle with the concepts of voltage vs. current, wonder why some component suddenly overheats, what a capacitor does, why you need a resistor in a certain circuit, or how large that resistor should be.

Unlike software, hardware and electronics are bound by the laws of physics. Painfully so, in fact – you can damage stuff when hooking things up incorrectly, and you can even get hurt if the voltages or currents involved are high enough. And it’s not just AC mains that can hurt you: even a car battery can be dangerous if you don’t know what you’re doing.

Knowing why things happen is extremely important in electronics. And learning the basics is really just a matter of reading up on it and investing a bit of effort. There’s no magic – in fact, there is so much knowledge about electronics nowadays, that you really can predict and explain stuff, even before hooking things up.

Since I loved tinkering with electronics as a teenager, I was lucky to gain a good deal of very intuitive knowledge about electronics early on. So much so, that basic DC electricity knowledge became second nature to me. Ohm’s law is not some technical rule, it’s obvious – the linear relation between voltage and current, and the fact that resistance is simply the slope of that relationship, is so ingrained by now, that I don’t even think of Ohm’s law as something distinct from the essence of electricity. Same for Kirchhoff’s laws – and the water analogy which works so well in so many cases to match intuition to the real world.

So the Easy Electrons series was born – no math, no theory, just a set of posts which tackle a number of electronics concepts in what I hope can bring across some of that intuition. My drawing tools at the time were based on paper and a scanner, so the drawings were a bit crude and hard to read – here’s an example where I’m trying to explain how to drive LED strips which draw more power than the weak I/O pins of an ATmega can provide:


There are posts about voltage and current, about power and heat, about resistors and capacitors, about LEDS and diodes, and about transistors and MOSFETs, to name a few.

Don’t take my word for it, heck… don’t even read my posts if you don’t want to – all I can say is: if you ever hook up electronics circuits of your own imagination and invention: get to grips with the basics, so you’ll understand why hooking up a bunch of LEDs in parallel is usually not such a good idea, why using a power supply with a higher voltage is likely to lead to overheating and damage, and why hooking up relays and motors often requires extra protective circuitry.

It’s not hard, it’s not magic, it requires no math of any sophistication (and there’s always the brilliant Khan Academy if you want to brush up on it anyway). It can be exhilarating to be able to design and predict what a circuit will do before hooking anything together.

Electronics is a truly wonderful field to explore (and it’s huge!). You don’t need much to experiment with, and you will learn a lot – if only to become quite respectful of the laws of physics, which are merciless (still: most mishaps are harmless and nothing gets damaged).

You’ll run into everything from “nothing happens” to “it’s smoking”, and it’s terrific fun!

Flashback – Ports and I2C in JeeLib

In AVR, Software on Oct 2, 2013 at 00:01

One of the small innovations in the JeeNode is the use of “Ports”:


Most µC boards, including the Arduino, brought out all the I/O pins on one or more large headers, meaning that you had to pick and route individual pins for each case where some component or sensor was being connected. While flexible, it prevented adding stuff in a plug-an-play manner: hook up some sensors, use matching bits of C/C++ code, and done.

I also always kept running out of VCC and GND connection pins while experimenting. So instead of bundling all I/O pins, a decision was made in 2009 to bring out a couple of (nearly) identical “ports”, to any of which the same sensor could be attached. That meant you could connect up to at least four sensors independent of each other.

As it turns out, there are enough spare I/O pins on an ATmega to provide one digital and one analog pin to each port. Add GND, VCC (+3.3V) and PWR (the input voltage, could be up to about 13V), and you end up with a fairly general-purpose setup. One extra spare pin was provided as “IRQ” to allow connected sensors to draw attention to themselves via an interrupt, although this feature has admittedly so far not been used much on JeeNodes.

But the real improvement is the fact that the two I/O pins can be used as I2C bus. All of a sudden, each port is no longer limited to a single sensor. The I2C bus is so useful because many types of low-cost chips include an I2C interface in hardware, and it lets multiple chips operate on a single bus.

One of the first extensions added was the Expander Plug, an 8-bit general-purpose digital I/O interface. So now, with just two pins consumed on the ATmega, you could have up to 32 input/output pins (by daisy chaining 4 of these plugs and setting them each to a different I2C address). The Dimmer Plug takes this even further: up to 8 of them on one bus, each driving up to 16 LED’s – for a total of 128 dimmable LEDs on 2 I/O pins!

Officially, the I2C bus requires dedicated hardware with “slew control” to increase the accuracy of signals on the bus. And while the ATmega has such an interface, there is just one per ATmega. Fortunately, you can also “bit-bang” I2C in plain software, i.e. simulate the relatively slow pin changes with nothing but software on any 2 I/O pins. The speed is not quite as high, and the supported bus length is also limited to a few dozen centimeters, but for direct connection of a few plugs that still works out quite well.

And so the JeeNode has ended up having 4 “ports”, each of them individually capable of supporting either I2C, or one analog (AIO) and one digital (DIO) pin – or alternately 2 digital I/O’s, since each analog pin on the ATmega can also be used in digital mode.

Since then, lots of different types of “JeePlugs” have been created, some I2C, some not. Most of the plugs have a corresponding class and a demo sketch in the JeeLib library, making it very easy to hook them up and interface to them in software. With most plugs, you just have to define which port they are connected to – and in the case of I2C plugs, which bus address they have been set to.

The placement of these ports and the choice of very low-cost standard 6-pin headers took some experimentation, but I think it all turned out ok. Lots of expandability and flexibility.

Flashback – The first JeeNode PCB

In AVR, Hardware on Oct 1, 2013 at 00:01

The first JeeNode circuit board was very exciting to make, back in January 2009:


Initially, I just wanted to get the connections right and didn’t care for silkscreen labelling, so there was none. It was created with CadSoft’s EAGLE, a commercial package with a freeware version for small boards. The “JeeNode” name came a month later, by the way.

The learning curve of the EAGLE package is steep, but then again, drawing this sort of stuff is going to be complicated whatever you do, because it’s not about drawing a nice picture where only aesthetics matter, but a meticulously designed diagram of which all the paths and connections have to follow certain rules, such as wires not crossing (doh!) and wires all having a minimum width and separation.

Here’s the JeeNode v6 PCB in comparison, of which many thousands have been produced:


It’s still surprisingly similar in design to the first iteration, although the production details and fabrication process have greatly improved over the years. One aesthetic detail I’ve always been pleased with is the blue-with-gold color choice. The gold plating is easier to solder than the HASL process I used initially, and is much more resistant to corrosion. Don’t think too much of this gold layer, by the way: it’s chemically deposited (fairly nasty stuff, from what I understand) and only a few atoms thick. Still… enough to do its work, and visually attractive (all in the eye of the beholder, evidently).

The purpose of a PCB is three-fold: a physically strong base for all the components, a reliable interconnect between all the pins to implement the actual circuit design, and silkscreen labelling to ease assembly and use afterwards.

The result: a tangible product. Quite an exciting change for a software developer used to work with bits and bytes, and who could never before point to the “real” result of it all!

Flashback – Batteries came later

In AVR, Hardware, Software on Sep 30, 2013 at 00:01

During all this early experimentation in 2008 and 2009, I quickly zoomed in on the little ATmega + RFM12B combo as a way to collect data around the house. But I completely ignored the power issue…

The necessity to run on battery power was something I had completely missed in the beginning. Everyone was running Arduino’s off either a 5V USB adapter or – occasionally – off a battery pack, and never much more than a few days. Being “untethered” in many projects at that time, meant being able to do something for a few hours or a day, and swapping or recharging batteries at night was easy, right?

It took me a full year to realise that a wireless “node” tied to a wire to run for an extended period of time made no sense. Untethered operation also implies being self-powered:


Evidently, having lots of nodes around the house would not work if batteries had to be swapped every few weeks. So far, I just worked off the premise that these nodes needed to be plugged into a power adapter – but there are plenty of cases where that is extremely cumbersome. Not only do you need a power outlet nearby, you need fat power adapters, and you have to claim all those power outlets for permanent use. It really didn’t add up, in terms of cost, and especially since the data was already being exchanged wirelessly!

Thus started the long and fascinating journey of trying to run a JeeNode on as little power as possible – something most people probably know this weblog best for. Over the years, it led to some new (for me) insights, such as: transmission draws a “huge” 25 mA, but it’s still negligible because the duration is only a few milliseconds. By far the most important parameter to optimise for is sleep-mode power consumption of the entire circuit.

In September 2010, i.e. one year after starting on this low-power journey, the Sleepy class was added to JeeLib, as a way to make it easy to enter low-power mode:

class Sleepy {
    /// start the watchdog timer (or disable it if mode < 0)
    /// @param mode Enable watchdog trigger after "16 << mode" milliseconds 
    ///             (mode 0..9), or disable it (mode < 0).
    static void watchdogInterrupts (char mode);
    /// enter low-power mode, wake up with watchdog, INT0/1, or pin-change
    static void powerDown ();
    /// Spend some time in low-power mode, the timing is only approximate.
    /// @param msecs Number of milliseconds to sleep, in range 0..65535.
    /// @returns 1 if all went normally, or 0 if some other interrupt occurred
    static byte loseSomeTime (word msecs);

    /// This must be called from your watchdog interrupt code.
    static void watchdogEvent();

The main call was named loseSomeTime() to reflect the fact that the watchdog timer is not very accurate. Calling Sleepy::loseSomeTime(60000) gives you approximately one minute of ultra low-power sleep time, but it could be a few seconds more or less. To wait longer, you can call this code a few times, since 65,535 ms is the maximum value supported by the Sleepy class.

As a result of this little class, you can do things like put the RFM12B into sleep mode (and any other power-hungry peripherals you might have connected), go to sleep for a bit, and restore all the peripherals to their normal state. The effects can be quite dramatic, with a few orders of magnitude less power consumption. This extends a node’s battery lifetime from a few days to a few years – although you have to get all the details right to get there.

One important design decision in the JeeNode was to use a voltage regulator with a very low idle current (the MCP1700 draws 2 µA idle). As a result, when a JeeNode goes to sleep, it can be made to draw well under 10 µA.

Most nodes here at JeeLabs now keep on running for well over a year on a single battery charge. Everything has become more-or-less install and forget – very convenient!

Flashback – RFM12B wireless

In AVR, Hardware, Software on Sep 29, 2013 at 00:01

After the ATmega µC, the second fascinating discovery in 2008 was the availability of very low-cost wireless modules, powerful enough to get some information across the house:


It would take another few months until I settled on the RFM12B wireless module by HopeRF, but the uses were quickly falling into place – I had always wanted to track the energy consumption in the house, to try and identify the main energy consumers. That knowledge might then help reduce our yearly energy consumption – either by making changes to the house, or – as it turned out – by simply adjusting our behaviour a bit.

Here is the mouse trap which collected energy metering data at JeeLabs for several years:


This is also when I found Modern Devices’s Real Bare Bone Board Arduino clone by Paul Badger – all the good stuff of an Arduino, without the per-board FTDI interface, and with a much smaller form factor.

Yet another month would pass to get a decent interrupt-driven driver working, and some more tweaks to make transmission interrupt-based as well. The advantage of such as design is that you get the benefits of a multi-tasking setup without all the overhead: the RF12 driver does all its time-critical work in the background, while the main loop() can continue to use delay() calls and blocking I/O (including serial port transmission).

In February 2009, I started installing the RF12demo code on each ATmega, as a quick way to test and try out wireless. As it turned out, that sketch has become quite useful as central receiving node, even “in production” – and I still use it as interface for HouseMon.

In April 2009, a small but important change was made to the packet format, allowing more robust use of multiple netgroups. Without this change, a bit error in the netgroup byte will alter the packet in a way which is not caught by the CRC checksum, making it a valid packet in another netgroup. This is no big deal if you only use a single netgroup, but will make a difference when multiple netgroups are in use in the same area.

Apart from this change, the RF12 driver and the RFM12B modules have been remarkably reliable, with many nodes here communicating for years on end without a single hick-up.

I still find it pretty amazing that simple low-power wireless networking is available at such a low cost, with very limited software involvement, and suitable for so many low-speed data collection and signalling uses in and around the house. To me, wireless continues to feel like magic after all these years: things happening instantly and invisibly across a distance, using physical properties which we cannot sense or detect…

Flashback – Discovering the Arduino

In AVR, Hardware on Sep 28, 2013 at 00:01

It’s now just about 5 years ago when I started with JeeLabs, so I thought it might be a good idea to bring back some notes from the past. Get ready for a couple of flashback posts…

One of the first posts on this weblog was about the Arduino, or rather Atmel’s AVR ATmega chip I had just discovered (it was the ATmega168 back then):


It was the Arduino IDE which made it trivial to play with this chip, an open source multi-platform software package combining an editor, the avr-gcc compiler, and the avrdude uploader. And despite the use of very Arduino-ish names such as “sketches” (firmware) and “shields” (add-on hardware), it was all nearly-standard C and C++, with a couple of convenient libraries to easily access I/O pins, the ADC, the serial port, timers, and more.

Five years ago, a fascinating brand new world opened up for me. I knew all about C and C++ as well as digital I/O and boot loaders, but a lot of really interesting and powerful new technologies were new to me: I2C, SPI, embedded timer hardware, and above all: sleep mode. These chips were not only able to run at an amazing 16 MHz clock rate, they could actually go to sleep and use a watchdog timer to wake up again later, saving 3 orders of magnitude on energy consumption.

A small universe, controlled by standard software and able to interface to the real world.

Physical Computing. Low cost. Accessible to anyone.


LevelDB, MQTT, and Redis

In Software on Sep 27, 2013 at 00:01

To follow up on yesterday’s post, here are some more interesting properties of LevelDB, in the context of Node.js and HouseMon:

  • The levelup wrapper for Node supports streaming, i.e. traversing a set of entries for reading, as well as sending an entire set of PUTs and/or DELs as a stream of requests. This is a very nice match for Node.js and for interfaces which need to be sent across the network (including to/from browsers).

  • Events are generated for each change to the database. These can easily be translated into a “live stream” of changes. This is really nothing other than publish / subscribe, but again in a form which matches Node’s asynchronous design really well.

  • Standard encoding conventions can be defined for keys and or values. In the case of HouseMon, I’m using UTF-8 as key encoding default and JSON as value encoding. The latter means that the default “stuff” stored in the database is simply a JavaScript object, and this is also what you get back on access and traversal. Individual cases can still override these defaults.

Together, all these features and conventions end up playing together very nicely. One particularly interesting aspect is the “persistent pub-sub” model you can implement with this. This is an area where the “Message Queue Telemetry Transport” MQTT has been gaining a lot of attention for things like home automation and the “Internet of Things” IoT.

MQTT is a message-based system. There are channels, named in a hierarchical manner, e.g. “/location/device/sensor”, to which one can publish measurement values (“publish /kitchen/roomnode/temp 17”). Other devices or software can subscribe to such channels, and get notified each time a value is published.

One interesting aspect of MQTT is its “RETAIN” functionality: for each channel, the last value can be retained, meaning that anyone subscribing to a channel will immediately be sent the last retained value, as if it just happened. This turns an ephemeral message-based setup into a state-based mechanism with change propagation, because you can subscribe to a channel after devices have published new values on it, yet still see what the best known value for that channel is, currently. With RETAIN, the MQTT system becomes a little database, with one value per channel. This can be very convenient after power-up as well.

LevelDB comes from the other side, i.e. a database with change tracking, but the effect is exactly the same: you can subscribe to a set of keys, get all their latest values and be notified from that point on whenever any keys in this range changes. This makes LevelDB and MQTT functionally equivalent, although LevelDB can handle much larger key spaces (note also that MQTT has features which LevelDB lacks, such as QoS).

It’s also interesting to compare this with the Redis database: Redis is another key-value store, but with more data structure options. The main difference with LevelDB, is that Redis is memory-based, with periodic snapshots to disk, whereas LevelDB is disk based, with caching in memory (and thus able to handle datasets which far exceed available RAM). Redis, LevelDB, and MQTT all support pub-sub in some form.

One reason for me to select LevelDB for HouseMon, is that it’s an in-process embedded database. This means that there is no need for a separate process to run alongside Node.js – and hence to need to install anything, or keep a background daemon running.

By switching to LevelDB, the HouseMon installation procedure has been simplified to:

  • download a recent version of HouseMon from GitHub
  • install Node (which includes the “npm” package manager)
  • run “npm install” to get everything needed by HouseMon

And that’s it. At this point, you just start up Node.js and everything will work.

But it’s not just convenience. LevelDB is also so compact, performant, and scalable, that HouseMon can now store all raw data as well as a complete set of aggregated per-hour values without running into resource limits, even on a Raspberry Pi. According to my rough estimates, it should all end up using less than 1 GB per year, so everything will easily fit on an 8 or 16 GB SD card.

The LevelDB database

In Software on Sep 26, 2013 at 00:01

In recent posts, I’ve mentioned LevelDB a few times – a database with some quite interesting properties.

Conceptually, LevelDB is ridiculously simple: a “key-value” store which stores all entries in sorted key order. Keys and values can be anything (including empty). LevelDB is not a toy database: it can handle hundreds of millions of keys, and does so very efficiently.

It’s completely up to the application how to use this storage scheme, and how to define key structure and sub-keys. I’m using a simple prefix-based mechanism: some name, a “~” character, and the rest of the key, of which each field can be separated with more “~” characters. The implicit sorting then makes it easy to get all the keys for any prefix.

LevelDB is good at only a few things, but these work together quite effectively:

  • fast traversal in sorted and reverse-sorted order, over any range of keys
  • fast writing, regardless of which keys are being written, for add / replace / delete
  • compact key storage, by compressing common prefix bytes
  • compact data storage, by a periodic compaction using a fast compression algorithm
  • modifications can be batched, with a guaranteed all-or-nothing success on completion
  • traversal is done in an isolated manner, i.e. freezing the state w.r.t. further changes

Note that there are essentially just a few operations: GET, PUT, DEL, BATCH (of PUTs and DELs), and forward/backward traversal.

Perhaps the most limiting property of LevelDB is that it’s an in-process embedded library, and that therefore only a single process can have the database open. This is not a multi-user or multi-process solution, although one could be created on top by adding a server interface to it. This is the same as for Redis, which runs as one separate process.

To give you a rough idea of how the underlying log-structured merge-tree algorithm works, assume you have a datafile with a sorted set of key-value pairs, written sequentially to file. In the case of LevelDB, these files are normally capped at 2 MB each, so there may be a number of files which together constitute the entire dataset.

Here’s a trivial example with three key-value pairs:


It’s easy to see that one can do fast read traversals either way in this data, by locating the start and end position (e.g. via binary search).

The writing part is where things become interesting. Instead of modifying the files, we collect changes in memory for a bit, and then write a second-level file with changes – again sorted, but now not only containing the PUTs but also the DELs, i.e. indicators to ignore the previous entries with the same keys.

Here’s the same example, but with two new keys, one replacement, and one deletion:


A full traversal would return the entries with keys A, AB, B, and D in this last example.

Together, these two levels of data constitute the new state of the database. Traversal now has to keep track of two cursors, moving through each level at the appropriate rate.

For more changes, we can add yet another level, but evidently things will degrade quickly if we keep doing this. Old data would never get dropped, and the number of levels would rapidly grow. The cleverness comes from the occasional merge steps inserted by LevelDB: once in a while, it takes two levels, runs through all entries in both of them, and produces a new merged level to replace those two levels. This cleans up all the obsolete and duplicate entries, while still keeping everything in sorted order.

The consequence of this process, is that most of the data will end up getting copied, a few times even. It’s very much like a generational garbage collector with from- and to-spaces. So there’s a trade-off in the amount of writing this leads to. But there is also a compression mechanism involved, which reduces the file sizes, especially when there is a lot of repetition across neighbouring key-value pairs.

So why would this be a good match for HouseMon and time series data in general?

Well, first of all, in terms of I/O this is extremely efficient for time-series data collection: a bunch of values with different keys and a timestamp, getting inserted all over the place. With traditional round-robin storage and slots, you end up touching each file in some spot all the time – this leads to a few bytes written in a block, and requires a full block read-modify-write cycle for every single change. With the LevelDB aproach, all the changes will be written sequentially, with the merge process delayed (and batched!) to a different moment in time. I’m quite certain that storage of large amounts of real-time measurement data will scale substantially better this way than any traditional RRDB, while producing the same end results for access. The same argument applies to indexed B-Trees.

The second reason, is that data sorted on key and time is bound to be highly compressible: temperature, light levels, etc – they all tend to vary slowly in time. Even a fairly crude compression algorithm will work well when readings are laid out in that particular order.

And thirdly: when the keys have the time stamp at the end, we end up with the data laid out in an optimal way for graphing: any graph series will be a consecutive set of entries in the database, and that’s where LevelDB’s traversal works at its best. This will be the case for raw as well as aggregated data.

One drawback could be the delayed writing. This is the same issue with Redis, which only saves its data to file once in a while. But this is no big deal in HouseMon, since all the raw data is being appended to plain-text log files anyway – so on startup, all we have to do is replay the end of the last log file, to make sure everything ends up in LevelDB as well.

Stay tuned for a few other reasons on why LevelDB works well in Node.js …

Small Gravity Plug update

In Hardware on Sep 25, 2013 at 00:01

The Gravity Plug contains a BMA020 3-axis accelerometer from Bosch Sensortec, which is being replaced by a newer BMA150 chip. Luckily, it’s fully compatible in both hardware and software, so the GravityPlug implementation in JeeLib will continue to work just fine with either one.

But the new chip does have more features: you can override its calibration parameters and defaults, by changing values in a new EEPROM area added to the chip. That in itself is probably not too useful for simple setups, but the new chip also offers access to its built-in temperature sensor, so I’ve added some extra code and expanded the gravity_demo sketch to take advantage of that:

#include <JeeLib.h>

PortI2C myBus (1);
GravityPlug sensor (myBus);
MilliTimer measureTimer;

struct {
  int axes[3];
  int temp;
} payload;

void setup () {

    if (sensor.isPresent()) {
      Serial.print("sensor version ");
      Serial.println(, HEX);

    rf12_initialize(7, RF12_868MHZ, 42);

void loop () {
    if (measureTimer.poll(1000)) {
        memcpy(payload.axes, sensor.getAxes(), sizeof payload.axes);
        payload.temp = sensor.temperature();

        rf12_sendNow(0, &payload, sizeof payload);

        Serial.print("GRAV ");
        Serial.print(' ');
        Serial.print(' ');
        Serial.print(' ');

Here is some sample output:

GRAV 2 -10 130 41
GRAV 0 -9 133 41
GRAV 4 -14 127 41
GRAV 0 -12 134 41
GRAV 4 -15 130 41
GRAV 4 -13 129 42
GRAV 3 -8 128 41

The reported temperature is in units of 0.5°C, so the above is reporting 20.5 .. 21.0 °C.

Unfortunately, the old chip isn’t reporting anything consistent as temperature:

GRAV 51 -31 -225 2
GRAV 50 -29 -225 0
GRAV 51 -30 -224 1
GRAV 52 -30 -227 1
GRAV 51 -30 -229 1
GRAV 52 -31 -223 2
GRAV 51 -31 -225 1

I have not found a way to distinguish these two chips at runtime – both report 0x02 in register 0 and 0x12 in register 1. Makes you wonder what the point is of having a chip “ID” register in the first place…

Anyway – in the near future, the Gravity Plug will be shipped with the new BMA150 chips. It should have no impact on your projects if you use these accelerometers, but at least you’ll be able to “sort of” tell which is which from the temperature readout. Note that the boards will probably still be marked as “BMA020” on the silkscreen for a while – there’s little point in replacing them, given that the change is so totally compatible either way…

In search of a good graph

In Software on Sep 24, 2013 at 00:01

I’ve been using the DyGraphs package for some time now in HouseMon, but it’s not perfect. Here is a generated graph for this morning, comparing two different ways to measure the same solar power generation:

Screen Shot 2013-09-23 at 09.20.52

The green series comes from the a pulse counter which generates a pulse every 0.5 Wh. These values then get sent to HouseMon every 3 seconds. A fairly detailed set of readings, as you can see – this is a typical morning in autumn on a cloudy day.

The blue series comes from the SMA inverter, which I’m reading every 5..6 minutes. It might be more accurate, but there is far less data coming in.

The key is that the area (i.e. Wh) under these two lines should be identical, if both devices are accurately calibrated. But this doesn’t quite seem to be the case here. (see update)

One known problem is that DyGraphs is plotting these “step charts” incorrectly: the value of the step should be at the end of the step, but DyGraphs can only plot steps given a starting value. So the way to read these blue steps, I think, is to keep the points, but to think of the step lines as being drawn horizontally preceding that point, not trailing it. I’m not fully convinced that this explains all the differences in the above examples, but it would seem to be a better match.

I’ve been looking at Highcharts and Highstocks as an alternative graphing library to use. They seem to support steps-with-end-values. But this sort of use really has quite a few other requirements as well, such as: being able to draw large datasets efficiently, support for multiple independent time series (DyGraphs was a bit limited here as well), interactive and intuitive zooming, good automatic choice of axis scales and fonts.

It would also be nice to be able to efficiently redraw a graph when new data points come in. Graphs used to be live in HouseMon, but at some point I ran into a snag and never bothered to fix it. For me, right now, live updating is less important than being able to generate accurate graphs. What’s the point of showing results on screen after all, if those results are not really represented truthfully…

I’ll stay on the lookout for better graphing solution. All suggestions welcome!

Update – See @ward’s comments below for the proper explanation of what’s going on here. It’s not a graphing problem after all (or rather: a different one…).

Animation in the browser

In Software on Sep 23, 2013 at 00:01

If you’ve programmed for Mac OS X (my actual experience in that area is fairly minimal), then you may know that it has a feature called Core Animation – as a way to implement all sorts of flashy smooth UI transitions – sliding, fading, colour changes, etc.

The basic idea of animation in this context is quite simple: go from state A to state B by varying one or more values gradually over a specified amount of time.

It looks like web browsers are latching onto this trend as well, with CSS Animations. It has been possible to implement all sorts of animations for some time, using timed JavaScript loops running in the background to simulation this sort of stuff, as jQuery has been doing. Then again, getting such flashy features into a whole set of independently-developed (and competing) web browsers is bound to take more time, but yeah… it’s happening. Here’s an illustration of what text can do in today’s browsers.

In Angular, browser-side animation has recently become quite a bit simpler.

I’m no big fan of attention-drawing screen effects. There was a time when HTML’s BLINK tag got so much, ehm, attention that every page tried to look like a neon sign. Oh, look, a new toy tag, and it’s BLINKING… neat, let’s use it everywhere!

Even slide transitions (will!) get boring after a while.

But with dynamic web pages updating in real time, it seems to me that there is a case to be made for subtle hints that something has just happened. As trial, I used a fade transition (from jQuery) in the previous version of HouseMon to briefly change the background of a changed row to very light yellow, which then fades back to white:

Screen Shot 2013-09-22 at 20.26.32

The effect seems to work well – for me anyway: it’s easy to spot the latest changes in a table, without getting too distracted when looking at some specific data.

In the latest version of HouseMon, I wanted to try the same effect using the new Angular animation feature, and I must say: the basic mechanism is really simple – you make animations happen by defining the CSS start settings, the finish settings, and the time period plus transition style to use. In the case of adding a row to a table or deleting a row, it’s all built-in and done by just adding some CSS definitions, as described here.

In this particular case, things are slightly more involved because it’s not the addition or deletion of a row I wanted to animate, but the change of a timestamp on a row. This requires defining a custom Angular “directive”. Directives are the cornerstone of Angular: they can (re-) define the behaviour of HTML elements or attributes, and let you invent new ones to do just about anything possible in the DOM.

Unfortunately, directives can also be a bit obscure at times. Here is the one I ended up defining for HouseMon:

ng.directive 'highlightOnChange', ($animate) ->                                  
  (scope, elem, attrs) ->                                                        
    scope.$watch attrs.highlightOnChange, ->                                     
      $animate.addClass elem, 'highlight', ->                                    
        attrs.$removeClass 'highlight'   

With it, we can now use a new custom highlightOnChange attribute to an element to implement highlighting when a value on the page changes.

Here is how it gets used (in Jade) on the status table rows shown in yesterday’s post:


In prose: when row.time changes, start the highlight transition, which adds a custom highlight class and removes it again at the end of the animation sequence.

The CSS then defines the actual animation (using Stylus notation):

  transition 1s linear all
  background lighten(yellow, 75%)
  background white

I’m leaving out some finicky details, but that’s the essence of it all: set up an $animate call to trigger on the proper conditions, and then define what is going go happen in CSS.

Note that Angular picks name suffixes for the CSS classes during the different phases of animation, but then it can all be used for numerous types of animation: you pick the CSS setting(s) at start and finish, and the browser’s CSS engine will automatically “run” the selected animation – covering motions, colour changes, and lots more (could fonts be emboldened gradually or even be morphed in more sophisticated ways? I haven’t tried it).

The result of all this is a table with subtle row highlights wherever a new reading comes in. Triggered by some Angular magic, but the actual highlighting is a (not-yet-100%-standardised) browser effect.

Working with a CSS grid

In Software on Sep 22, 2013 at 00:01

As promised yesterday, here is some information on how you can get a page layouts off the ground real quick with the Zurb Foundation 4 CSS framework. If you’re more into Twitter Bootstrap, see this excellent comparison and overview. They are very similar.

It took a little while to sink in, until I read this article about how it’s supposed to be used, and skimmed through a 10-part series on webdesign tuts+. Then the coin dropped:

  • you create rows of content, using a div with class row
  • each row has 12 columns to put stuff in
  • you put the sections of your page content in divs with class large-6 and columns
  • each section goes next to the preceding one, so 6 and 6 puts the split in the middle
  • any number from 1 to 12 goes, as long as you use at most 12 columns per row

So far so good, and pretty trivial. But here’s where it gets interesting:

  • each column can have more rows, and each row again offers 12 columns to lay out
  • for mobile use, you can add small-N classes as well, with other column allocations

I haven’t done much with this yet, but this page was trivial to generate:

Screen Shot 2013-09-20 at 00.57.16

The complete definition of that inner layout is in app/status/view.jade:

        h3 Status
            th Driver
            th Origin
            th Name
            th Title
            th Value
            th Unit
            th Time
          tr(ng-repeat='row in status | filter:statusQuery | orderBy:"key"')
            td {{row.type}}
            td {{row.tag}}
            td {{}}
            td {{row.title}}
            td(align='right') {{row.value}}
            td {{row.unit}}
            td {{row.time | date:"MM-dd &nbsp; HH:mm:ss"}}

In prose:

  • the information shown uses the first 9 columns, (the remaining 3 are unused)
  • in those 9 columns, we have a header row and then the actual data table as row
  • the header is a h1 header on the left and an input on the right, split as 2:1
  • the table takes the full width (of those 9 columns in which it resides)
  • the rest is standard HTML in Jade notation, and an Angular ng-repeat loop
  • there’s also live row filtering in there, using some Angular conventions
  • the alternating row backgrounds and general table style are Foundation’s defaults

Actually, I left out one thing: the entries automatically get highlighted whenever the time stamp changes. This animation is a nice way to draw attention to updates – stay tuned.

Responsive CSS layouts

In Software on Sep 21, 2013 at 00:01

One of the things today’s web adds to the mix, is the need to address devices with widely varying screen sizes. Seeing a web site made for an “old” 1024×768 screen layout squished onto a mobile phone screen is not really great if everything just ends up smaller on-screen.

Enter grid-based CSS frameworks and responsive CSS.

There’s a nice site to quickly try designs at different sizes, at Here’s how my current HouseMon screen looks on two different simulated screens:

Screen Shot 2013-09-19 at 23.45.02

Screen Shot 2013-09-20 at 00.17.54

It’s not just reflowing CSS elements (the bottom line stacks vertically on a small screen), it also completely alters the behaviour of the navigation bar at the top. In a way which works well on small screens: click in the top right corner to fold out the different menu choices.

Both Zurb Foundation and Twitter Bootstrap are “CSS frameworks” which provide a huge set of professionally designed UI elements – and now also do this responsive thing.

The key point here is that you don’t need to do much to get all this for free. It takes some fiddling and googling to set all the HTML sections up just right, but that’s it. In my case, I’m using Angular’s “ng-repeat” to generate the navigation bar menu entries on-the-fly:'NavCtrl')
          | &nbsp; {{}}
      li.myNav(ng-repeat='nav in navbar')
        a(ng-href='{{nav.route}}') {{nav.title}}
        a(href='/admin') Admin

Yeah, ok, pretty tricky stuff. But it’s all documented, and nowhere near the effort it’d take to do this sort of thing from scratch. Let alone get the style and visual layout consistent and working across all the different browsers and versions.

Tomorrow, I’ll describe the way grid layouts work with Zurb Foundation 4 – it’s sooo easy!

In praise of AngularJS

In Software on Sep 20, 2013 at 00:01

This is how AngularJS announces itself on its home page:

HTML enhanced for web apps!

Hm, ok, but not terribly informative. Here’s the next paragraph:

HTML is great for declaring static documents, but it falters when we try to use it for declaring dynamic views in web-applications. AngularJS lets you extend HTML vocabulary for your application. The resulting environment is extraordinarily expressive, readable, and quick to develop.

That’s better. Let me try to ease you into this “framework” on which HouseMon is based:

  • Angular (“NG”) is client-side only. It doesn’t exist on the server. It’s all browser stuff.
  • It doesn’t take over. You could use NG for parts of a page. HouseMon is 100% NG.
  • It comes with many concepts, conventions, and terms you must get used to. They rock.

I can think of two main stumbling blocks: “Dependency Injection” and all that “service”, “factory”, and “controller” stuff (not to mention “directives”). And Model-View-Something.

It’s not hard. It wasn’t designed to be hard. The terminology is pretty clever. I’d say that it was invented to produce a mental model which can be used and extended. And like a beautiful clock, it doesn’t start ticking until all the little pieces are positioned just right. Then it will run (and evolve!) forever.

Dependency injection is nothing new, it’s essentially another name for “require” (Node.js), “import” (Python), or “#include” (C/C++). Let’s take a fictional piece of CoffeeScript code:

value = 123
myCode = (foo, bar) -> foo * value + bar

console.log myCode(100, 45)

The output will be “12345”. But with dependency injection, we can do something else:

value = 123
myCode = (foo, bar) -> foo * value + bar

define 'foo', 100
define 'bar', 45
console.log inject(myCode)

This is not working code, but it illustrates how we could define things in some more global context, then call a function and give it access to that context. Note that the “inject” call does something very magical: it looks at the names of myCode’s arguments, and then somehow looks them up and finds the values to pass to myCode.

That’s dependency injection. And Angular uses it all over the place. It will often call functions through a special “injector” and get hold of all args expected by the function. One reason this works is because closures can be used to get local scope information into the function. Note how myCode had access to “value” via a normal JavaScript closure.

In short: dependency injection is a way for functions to import all the pieces they need. This makes them extremely modular (and it’s great for test suites, because you can inject special versions and mock objects to test each piece of code outside its normal context).

The other big hurdle I had to overcome when starting out with Angular, is all those services, providers, factories, controllers, and directives. As it turns out, that too is much simpler than it might seem:

  • Angular code is packaged as “modules” (I actually only use a single one in HouseMon)
  • in these modules, you define services, etc – more on this in a moment
  • the whole startup is orchestrated in a specific way: boot, config, run, fly
  • ok, well, not “fly” – that’s just my name for it…

The startup sequence is key – to understand what goes where, and what gets called when:

  • the browser starts by loading an HTML page with <script> tags in it (doh)
  • all the code needed on the browser side has to be loaded up front
  • what this code does is define modules, services, and so on
  • nothing is really “running” at this point, these are all function definitions
  • the last step is to call angular.bootstrap(document, ['myApp']);
  • at this point, all the config sections defined in the modules are run
  • once that is over, all the run sections are run, this is the real application start
  • when that is done, we in essence enter The Big Event Loop in the browser: lift off!

So on the client side, i.e. the browser, the entire application needs to be structured as a set of definitions. A few things need to be done early on (config), a few more once everything has been defined and Angular’s “$rootScope” has been put in place (read: the main page context is ready, a bit like the DOM’s “document.onload”), and then it all starts doing real work through the controllers tied to various HTML elements in the DOM.

Providers, factories, services, and even constants and values, are merely variations on a theme. There is an excellent page by Matt Haggard describing these differences. It’s all smoke and mirrors, really. Everything except directives is a provider in Angular.

Angular is like the casing of a watch, with a powerful pre-loaded spring already in place. You “just” have to figure out the role of the different pieces, put a few of them in place, and it’ll run all by itself from then on. It’s one of the most elegant frameworks I’ve ever seen.

With HouseMon, I’ve put the main pieces in place to show a web page in the browser with a navigation top bar, and the hooks to tie into Primus and RPC. The rest is all up to the modules, i.e. the subfolders added to the app/ folder. Each file is an Angular module. HouseMon will automatically load them all into the browser (via a /primus/primus.js file generated on-the-fly by Primus). This has turned out to be extremely modular and flexible – and it couldn’t have been done without Angular.

If you’re looking for a “way in” for the HouseMon client-side code, then the place to start is app/index.jade, which uses Jade’s extend mechanism to tie into main/layout.jade, so that would be the second place to start. After that, it’s all Angular-structured code, i.e. views, view templates with controllers, and the services they call.

I’m quite excited by all this, because I’ve never before been able to develop software in such a truly modular fashion. The interfaces are high-level, clean (and clear) and all future functionality can now be added, extended, replaced, or even removed again – step by step.

Ground control to… browser

In Software on Sep 19, 2013 at 00:01

After yesterday’s RPC example, which makes the server an easy-to-use service in client code, let’s see how to do things in the other direction.

I should note that there is usually much less reason to do this, since at any point in time there can be zero, one, or occasionally more clients connected to the server in the first place. Still, it could be useful for a server to collect client status info, or to periodically save some state. The benefit of server-initiated requests being that it turns authentication around, assuming we trust the server.

Ok, so here’s some silly code on the client which I’d like to be able to call from the server:

ng = angular.module 'myApp'

ng.config ($stateProvider, navbarProvider, primus) ->
  $stateProvider.state 'view1',
    url: '/'
    templateUrl: 'view1/view.html'
    controller: 'View1Ctrl'
  navbarProvider.add '/', 'View1', 11
  primus.client.view1_twice = (x) ->
    2 * x

ng.controller 'View1Ctrl', ->

This is the “View 1” module, similar to the Admin module, exposing code for server use:

  primus.client.view1_twice = (x) ->
    2 * x

On the server side, the first thing to note is that we can’t just call this at any point in time. It’s only available when a client is connected. In fact, if more than one client is connected, then there will be multiple instances of this code. Here is app/view1/

module.exports = (app, plugin) ->
  app.on 'running', (primus) ->
    primus.on 'connection', (spark) ->
      spark.client('view1_twice', 123).then (res) ->
        console.log 'double', res

The output on the server console, whenever a client connects, is – rather unsurprisingly:

    double 246

What’s more interesting here, is the logic of that server-side code:

  • the app/view1/ exports one function, accepting two args
  • when loaded on startup, it will be called with the app and Primus plugin object
  • we’re in setup mode at this point, so this is a good time to define event handlers
  • the app.on 'running' handler gets called when all server setup is complete
  • now we can set up a handler to capture all new client connections
  • when such a connection comes in, Primus will hand us a “spark” to talk to
  • a spark is essentially a WebSocket session, it supports a few calls and streaming
  • now we can call the ‘view1_twice’ code on the client via RPC
  • this creates and returns a promise, since the RPC will take a little bit of time
  • we set it up so that when the promise is fulfilled, console.log ... gets called

And that’s it. As with yesterday’s setup, all the communication and handling of asynchronous callbacks is taken care of behind the scenes.

There is some asymmetry in naming conventions and the uses of promises, but here’s the summary:

  • to define a function ‘foo’ on the host, define it as = …
  • to call that function on the client, use host ‘foo’, …
  • the result is an Angular promise, use “.then …” to pick it up if needed
  • to define a function ‘bar’ on the client, define it as primus.client = …
  • to call that function from the server, use spark.client ‘bar’, …
  • the result is a q promise, use “.then …” to pick it up

Ok, so there are some slight differences between the two promise API’s. But the main benefit of this all is that the passing of time and networking involved is completely abstracted away. Maybe I’ll be able to unify this further, some day.

Promise-based RPC

In Software on Sep 18, 2013 at 00:01

Or maybe this post should have been titled: “Taming asynchronous RPC”.

Programming with asynchronous I/O is tricky. The proper solution IMO, is to have full coroutine or generator support in the browser and in the server. This has been in the works from some time now (see ES6 Harmony). In short: coroutines and generators can “yield”, which means something that takes time can suspend the current stack state until it gets resumed. A bit like green (non-preemptive) threads, and with very little overhead.

But we ain’t there yet, and waiting for all the major browsers to reach that point might be a bit of a stretch (see the “Generators (yield)” entry on this page). The latest Node.js v0.11.7 development release does seem to support it, but that’s only half the story.

So promises it is for now. On the server side, this is available via Kris Kowal’s q package. On the client side AngularJS includes promises as $q. And as mentioned before, I’m also using q-connection as promise-based Remote Procedure Call RPC mechanism.

The result is really neat, IMO. First, RPC from the client, calling a function on the server. This is app/admin/ for the client side:

ng = angular.module 'myApp'

ng.config ($stateProvider) ->
  $stateProvider.state 'admin',
    url: '/admin'
    templateUrl: 'admin/view.html'
    controller: 'AdminCtrl'

ng.controller 'AdminCtrl', ($scope, host) ->
  $scope.hello = host 'admin_dbinfo'

This Angular boilerplate defines an Admin page with app/admin/view.jade template:

h3 Storage use
    th Group
    th Size
  tr(ng-repeat='(group,size) in hello')
    td {{group}}
    td(align='right') ≈ {{size}} b

The key is this one line:

$scope.hello = host 'admin_dbinfo'

That’s all there is to doing RPC with a round-trip over the network. And here’s the result, as shown in the browser (formatted with the Zurb Foundation CSS framework):

Screen Shot 2013-09-17 at 13.37.33

There’s more to it than meets the eye, but it’s all nicely tucked away: the host call does the request, and returns a promise. Angular knows how to deal with promises, so it will fill in the “hello” field when the reply arrives, asynchronously. IOW, the above code looks like synchronous code, but does lots of things at another point in time than you might think.

On the server, this is the app/admin/ code:

Q = require 'q'

module.exports = (app, plugin) ->
  app.on 'setup', -> = ->
      q = Q.defer()
      app.db.getPrefixDetails q.resolve

It’s slightly more involved because promises are more exposed. First, the RPC call is defined on application setup. When called, it creates a new promise, calls getPrefixDetails with a callback, and immediately returns this promise to fill in the result later.

At some point, getPrefixDetails will call the q.resolve callback with the result as argument, and the promise becomes fulfilled. The reply is then sent to the client by the q-connection package.

Note that there are three asynchronous calls involved in total:

    rpc call -> network send -> database action -> network reply -> show
                ============    ===============    =============

Each of the underlined steps is asynchronous and based on callbacks. Yet the only step we had to deal with explicitly on the server was that custom database action.

Tomorrow, I’ll show RPC in the other direction, i.e. the server calling client code.

Flow – the inside perspective

In Software on Sep 17, 2013 at 00:01

This is the last of a four-part series on designing big apps (“big” as in not embedded, not necessarily many lines of code – on the contrary, in fact).

The current 0.8 version of HouseMon is my first foray into the world of streams and promises, on the host server as well as in the browser client.

First a note about source code structure: HouseMon consists of a set of subfolders in the app folder, and differs from the previous SocketStream-based design in that dependency info, client-side code, client-side layout files + images, and host-side code are now all grouped per module, instead of strewn across client, static, and server folders.

The “main” module starts the ball rolling, the other modules mostly just register themselves in a global app “registry”, which is simply a tree of nested key/value mappings. Here’s a somewhat stripped down version of app/main/

module.exports = (app, plugin) ->
  app.register 'nodemap.rf12-868,42,2', 'testnode'
  app.register 'nodemap.rf12-2', 'roomnode'
  app.register 'nodemap.rf12-3', 'radioblip'
  # etc...

  app.on 'running', ->
    Serial = @registry.interface.serial
    Logger = @registry.sink.logger
    Parser = @registry.pipe.parser
    Dispatcher = @registry.pipe.dispatcher
    ReadingLog = @registry.sink.readinglog

    app.db.on 'put', (key, val) ->
      console.log 'db:', key, '=', val

    jeelink = new Serial('usb-A900ad5m').on 'open', ->

      jeelink # log raw data to file, as timestamped lines of text
        .pipe(new Logger) # sink, can't chain this further

      jeelink # log decoded readings as entries in the database
        .pipe(new Parser)
        .pipe(new Dispatcher)
        .pipe(new ReadingLog app.db)

This is all server-side stuff and many details of the API are still in flux. It’s reading data from a JeeLink via USB serial, at which point we get a stream of messages of the form:

{ dev: 'usb-A900ad5m', msg: 'OK 2 116 254 1 0 121 15' }

(this can be seen by inserting “.on('data', console.log)” in the above pipeline)

These messages get sent to a “Logger” stream, which saves things to file in raw form:

L 17:18:49.618 usb-A900ad5m OK 2 116 254 1 0 121 15

One nice property of Node streams is that you can connect them to multiple outputs, and they’ll each receive these messages. So in the code above, the messages are also sent to a pipeline of “transform” streams.

The “Parser” stream understands RF12demo’s text output lines, and transforms it into this:

{ dev: 'usb-A900ad5m',
  msg: <Buffer 02 74 fe 01 00 79 0f>,
  type: 'rf12-868,42,2' }

Now each message has a type and a binary payload. The next step takes this through a “Dispatcher”, which looks up the type in the registry to tag it as a “testnode” and then processes this data using the definition stored in app/drivers/

module.exports = (app) ->

  app.register 'driver.testnode',
    in: 'Buffer'
        title: 'Battery status', unit: 'V', scale: 3, min: 0, max: 5

    decode: (data) ->
      { batt: data.msg.readUInt16LE 5 }

A lot of this is meta information about the results decoded by this particular driver. The result of the dispatch process is a message like this:

{ dev: 'usb-A900ad5m',
  msg: { batt: 3961 },
  type: 'testnode',
  tag: 'rf12-868,42,2' }

The data has now been decoded. The result is one parameter in this case. At the end of the pipeline is the “ReadingLog” write stream which timestamps the data and saves it into the database. To see what’s going on, I’ve added an “on ‘put'” handler, which shows all changes to the database, such as:

db: reading~testnode~rf12-868,42,2~1379353286905 = { batt: 3961 }

The ~ separator is a convention with LevelDB to partition the key namespace. Since the timestamp is included in the key, this storage will store each entry in the database, i.e. as historical storage.

This was just a first example. The “StatusTable” stream takes a reading such as this:

db: reading~roomnode~rf12-868,5,24~1379344595028 = {
  "light": 20,
  "humi": 68,
  "moved": 1,
  "temp": 143

… and stores each value separately:

db: status~roomnode/rf12-868,5,24/light = {
// and 3 more...

Here, the information is placed inside the message, including the time stamp. Keys must be unique, so in this case only the last value will be kept in the database. In other words: this fills a “status” table with the latest value of each measurement value.

As last example, here is a pipeline which replays a saved log file, decodes it, and treats it as new readings (great for testing, but useless in production, clearly):

createLogStream = @registry.source.logstream

  # .pipe(new Replayer)
  .pipe(new Parser)
  .pipe(new Dispatcher)
  .pipe(new StatusTable app.db)

Unlike the previous HouseMon design, this now properly deals with back-pressure.

The “Replayer” stream is a fun one: it takes each message and delays passing it through (with an updated timestamp), so that this stream will simulate a real-time feed with new messages trickling in. Without it, the above pipeline processes the file as fast as it can.

The next step will be to connect a change stream through the WebSocket to the client, so that live status information can be displayed on-screen, using exactly the same techniques.

As you can see, it’s streams all the way down. Onwards!

Flow – the application perspective

In Software on Sep 16, 2013 at 00:01

This is the third of a four-part series on designing big apps (“big” as in not embedded, not necessarily many lines of code – on the contrary, in fact).

Because I couldn’t fit it in three parts after all.

Let’s talk about the big picture, in terms of technologies. What goes where, and such.

A decade or so ago, this would have been a decent model of how web apps work, I think:


All the pages, styles, images, and some code fetched as HTTP requests, rendered on the server, which sits between the persistent state on the server (files and databases), and the network-facing end.

A RESTful design would then include a clean structure w.r.t how that state on the server is accessed and altered. Then we throw in some behind-the-secnes logic with Ajax to make pages more responsive. This evolved to general-purpose client-side JavaScript libraries like jQuery to get lots of the repetitiveness out of the developer’s workflow. Great stuff.

The assumption here is that servers are big and fast, and that they work reliably, whereas clients vary in capabilities, versions, and performance, and that network connection stability needs to stay out of the “essence” of the application logic.

In a way, it works well. This is the context in which database-backed websites became all the rage: not just a few files and templating to tweak the website, but the essence of the application is stored in a database, with “widgets”, “blocks”, “groups”, and “panes” sewn together by an ever-more elaborate server-side framework – Ruby on Rails, Django, Drupal, etc. WordPress and Redmine too, which I gratefully rely on for the JeeLabs sites.

But there’s something odd going on: a few days ago, the WordPress server VM which runs this daily weblog here at JeeLabs crashed on an out-of-memory error. I used to reserve 512 MB RAM for it, but had scaled it back to 256 MB before the summer break. So 256 MB is not enough apparently, to present a couple of weblog pages and images, and handle some text searches and a simple commenting system.

(We just passed 6000 comments the other day. It’s great to see y’all involved – thanks!)

Ok, so what’s a measly 512 MB, eh?

Well, to me it just doesn’t add up. Ok, so there’s a few hundred MB of images by now. Total. The server is running off an SSD disk, so we should be able to easily serve images with say 25 MB of RAM. But why does WP “need” hundreds of MB’s of RAM to serve a few dozen MB’s of daily weblog posts? Total.

It doesn’t add up. Self-inflicted overhead, for what is 95% a trivial task: serving the same texts and images over and over again to visitors of this website (and automated spiders).

The Redmine project setup is even weirder: currently consuming nearly 700 MB of RAM, yet this ought to be a trivial task, which could probably be served entirely out of say 50 MB of RAM. A tenfold resource consumption difference.

In the context of the home monitoring and automation I’d like to take a bit further, this sort of resource waste is no longer a luxury I can afford, since my aim is to run HouseMon on a very low-end Linux board (because of its low power consumption, and because it really doesn’t do much in terms of computation). Ok, so maybe we need a bit more than 640 KB, but hey… three orders of magnitude?

In this context, I think we can do better. Instead of a large server-side byte-shuffling engine, we can now do this – courtesy of modern browsers, Node.js, and AngularJS:


The server side has been reduced to its bare minimum: every “asset” that doesn’t change gets served as is (from files, a database, whatever). This includes HTML, CSS, JavaScript, images, plain text, and any other “document” type blob of information. Nginx is a great example of how a tiny web server based on async I/O and written in C can take care of any load we down-to-earth netizens are likely to encounter.

Let me stress this point: there is no dynamic “templating” or “page generation” on the server side. This takes place in the browser – abstracted away by AngularJS and the DOM.

In parallel, there’s a “Real-Time Server” running, which handles the application logic and everything that needs to be dynamic: configuration! status! measurements! commands! I’m using Node.js, and I’ve started using the fascinating LevelDB database system for all data persistence. The clever bit of LevelDB is that it doesn’t just fetch and store data, it actually lets you keep flows running with changes streamed in and out of it (see the level-livefeed and multilevel projects).

So instead of copying data in and out of a persistent data store, this also becomes a way of staying up to date with all changes on that data. The fusion of a database and pubsub.

On the client side, AngularJS takes care of the bi-directional flow between state (model) and display (view), and Primus acts as generic connection library for getting all this activity across the net – with connection keep-alive and automatic reconnect. Lastly, I’m thinking of incorporating the q-connection library for managing all asynchronicity. It’s fascinating to see network round trip delays being woven into the fabric of an (async / event-driven) JavaScript application. With promises, client/server apps are starting to feel like a single-system application again. It all looks quite, ehm… promising :)

Lots of bits that need to work together. So far, I’m really delighted by the flexibility this offers, and the fact that the server side is getting simpler: just the autonomous data acquisition, a scalable database, a responsive WebSocket server to make the clients (i.e. browsers) come alive, and everything else served as static data. State is confined to the essence of the app. The rest doesn’t change (other than during development of course), needs no big backup solution, and the whole kaboodle should easily fit on a Raspberry Pi – maybe something even leaner than that, one day.

The last instalment tomorrow will be about how HouseMon is structured internally.

PS. I’m not saying this is the only way to go. Just glad to have found a working concoction.

Flow – the developer perspective

In Software on Sep 15, 2013 at 00:01

This is the second of a three-part four-part series on designing big apps (“big” as in not embedded, not necessarily many lines of code – on the contrary, in fact).

Software development is all about “flow”, in two very different ways:

I’ve been looking at lots of options on how to address that first bit, i.e. how to set up an environment which can help me through the edit-run-debug cycle as smoothly and quickly as possible. I didn’t find anything that really fits my needs, so as any good ol’ software developer would do, I scratched this itch recently and ended up creating a new tool.

It’s all about flow. Iterating through the coding process without pushing more buttons or keys than needed. Here’s how it works – there are three kinds of files I need to deal with:

  • HTML – app pages and docs, in the form of Jade and Markdown text files
  • CSS – layout and visual design, in the form of Stylus text files
  • CODE – the logic of it all, host- and client-side, in the form of CoffeeScript text files

Each of these are not the real thing, in the sense that they are all dialects which need to be translated to HTML, CSS, and JavaScript, respectively. But I don’t want a solution which introduces lots of extra steps into that oh-so treasured edit-run-debug cycle of software development. No temporary files, please – let’s generate everything as needed on-the-fly, and let’s set up a single task to take care of it all.

The big issue here is when to do all that pre- (and re-) processing. Again, since staying in the flow is the goal, that really should happen whenever needed and as quickly as possible. The workflow I’ve found to suit me well is as follows:

  • place all the development source files into one nested area (doh!)
  • have the system watch for changes to any files in this area
  • if it’s a HTML file change: reload the browser
  • if it’s a CSS file change: update the browser (no need to reload)
  • if it’s a CODE file change: restart the server and reload the browser

The term for this is “live reload”. It’s been in use for ages by web developers. You save a file, and BAM … the browser updates itself. But that’s only half the story with a client + server setup, as you also may have to restart the server. In the past, I’ve done so using the nodemon utility. It works, but along with other tools too watch for changes and pre-compile to the desired format, it all got a bit clunky.

In the new HouseMon project, it’s all done behind the scenes in a single process. Well, two actually: when in development mode, HouseMon forks itself into a supervisor and a worker child. The supervisor just restarts the worker whenever it exits. And the worker watches for files changes and either reloads or adjusts the browser, or exits. The latter becomes a way for the worker to quickly restart itself from scratch due to a code change.

The result works phenomenally well: I edit in MacVim, multiple files even, and nothing happens. I can bring up the Dash reference guides to look up something. Still nothing happens. But when I switch away from MacVim, it’s set up to save all changes, and everything then kicks in on auto-pilot. Instantly or a few seconds later (depending on the change), the latest changes will be running. The browser may lose a little of its context, but the URL stays the same, so I’m still looking at the same page. The console is open, and I can see what’s going on there. Without ever switching to the browser (ok, I need to switch away from MacVim to something else, but often that will be the command line).

It’s not yet the same as live in-app development (as was possible with Tcl), but it’s getting mighty close, because the app returns to where it was before without pushing any buttons.

There’s one other trick in this setup which is a huge time-saver: whenever the supervisor is launched (“node .“), it scans through all the folders inside the main ./app folder, and collects npm and bower package files as well as Makefile files. All dependencies and requirements are then processed as needed, making the use of 3rd party packages virtually effortless – on the server and in the client. Total automation. Look ma, no hands!

None of this machinery needs to be present in production mode. All this reloading and package dependency handling stuff can then be ignored. Deployment just needs a bunch of static files, served through Node.js, Nginx, Apache, whatever – plus a light-weight Node.js server for the host-side logic, using WebSockets or some fallback mechanism.

As for the big picture on how HouseMon itself is structured… more coming, tomorrow.

Flow – the user perspective

In Software on Sep 14, 2013 at 00:01

This is the first of a three-part four-part series on designing big apps (“big” as in not embedded, not necessarily many lines of code – on the contrary, in fact).

Ok, so maybe what follows is not the user perspective, but my user perspective?

I’m not a fan of pushing clicking buttons. It’s a computer invention, and it sucks.

Interfaces (I’m not keen on the term user interfaces either, but it’s more to the point than just “interfaces”) – anyway, interfaces need interaction to indicate what information you want to see, and to perform a real action, i.e. something that has effect on some permanent state. From setting up preferences or a configuration, to turning on a light.

But apart from that, interaction to fetch or update something is tedious, unwanted, silly, and a waste of time. Pushing a button to see the time is 2,718,281 times more inconvenient than looking at the time. Just like asking for something involves a lot more interaction (and social subtleties) than having the liberty to just do it.

When I look outside the window, I expect to see the actual weather. And when I see information on a screen, I expect it to be up-to-date. I don’t care if the browser does constant refreshes to pull changes from a server or whether it’s set up to respond to push notifications coming in behind the scenes. When I see “22.7°C”, it needs to be a real reading, with a known and acceptable lag and accuracy – unless it’s historical data.

This goes a lot further than numbers and graphs. When I see a button, then to me that is a promise (and invitation) that some action will be taken when I push it. Touch on mobile devices has the advantage here, although clicking via the mouse or typing a power-user keyboard shortcut is often perfectly acceptable – even if slightly more indirect.

What I don’t want is a button which does nothing, or generates an error message. If that button isn’t intended to be used, then it should be disabled or it shouldn’t be there at all:

Screen Shot 2013-09-13 at 11.56.54

Screen Shot 2013-09-13 at 11.57.17

That’s the “Graphs” install page in HouseMon (the wording could have been better).

When a list or graph of readings is shown on-screen, this needs to auto-update in real time. That’s the whole point of HouseMon – access to specific information (that choice is a preference, and will require interaction) to see what’s going on. Now – or within the last few minutes, in the case of sensors which are periodically sending out their readings. A value without an indication of its “up-to-date-ness” is useless. That could be either implicit by displaying it in some special way if the information is stale (slowly turning red?), or explicit, by displaying a time stamp next to it (a bit ugly and tedious).

Would you want it any other way when doing online banking? Would you accept seeing your account balance without guarantee that it is recent?nah, me neither :) – so why do we put up with web pages with copies of some information, at some point in time, getting obsolete the very moment that page is shown?

When “on the web” – as a user – I want to deal with accurate information. When I see something tagged as “updated 2 minutes ago”, then I want to see that “2” change into a “3” within the next minute. See GitHub for an example of this. It works, it makes sense, and it makes the whole technology get out of the way: the web should be an intermediary between some information and me. All the stuff in between is secondary.

Information needs to update – we need to stop copying it into a web page, and send that copy off to the browser. All the lipstick in the world won’t help if what we see is a copy of what we’re looking for. As developers, we need to stop making web sites fancy – we should make them live first and then make them gorgeous. That’s what RIA‘s and SPA‘s are about.

One more thing – not only does information need to flow, these flows should be visualised:

Screen Shot 2013-09-13 at 11.02.15

Such a user interface can be used during development to manage data flows and to design software which ties each result shown on-screen back to its origin. In tomorrow’s apps, every change should propagate – all the way to the pixels shown on-screen by the browser.

Now let’s go back to static web servers to make it happen. Stay tuned…

Gearing up for the new web

In Software on Sep 13, 2013 at 00:01

Well… new for me, anyway. What a fascinating world in motion we live in:

  • first there was the “pre-web”, with email BBS’es and very clunky modem links
  • then came Nestcape Navigator – and the world would never be the same again
  • hey, we can make the results dynamic, let’s generate pages through CGI
  • and not just outgoing, we can have forms and page refreshes to interact!
  • we need lots of ways to generate page variations, let’s add modules to the server
  • nah, we can do better than that: let’s add PHP / Python / Ruby inside the server!
  • and so the first major web explosion of the internet was born…

It took a decade, and another decade to make fast internet access mainstream, which is where we are today. There are lots and lots and LOTS of “web frameworks” in existence now, for any programming language you care about.

The second major wave of evolution came with a whole bunch of new acronyms, such as RIA and SPA. This is very much Google’s turf, and this is the technology which powers Gmail, for example. Rich and responsive interactivity. It’s going everywhere by now.

It was made possible by the super-trio of HTML, CSS, and JS (JavaScript). They’ve all evolved to define the structure, style, and logic of everything you see in the browser.

It’s hard to keep up, but as you know, I’ve picked two very active technologies for doing all my new software development: AngularJS and Node.js. Both by Google, by the way – but also both freely available as wide-open source (here and here). They’ve changed my life :)

It’s been a steep learning curve. I don’t mean just getting something going. I mean getting comfortable with it all. Understanding the idioms and quirks (there always are), and figuring out how to debug stuff (async is nasty, especially when not using promises). Can’t say I’m in the clear yet, but the fog is lifting and the fun level is rising!

I started coding HouseMon many months ago and it’s been running here for a long time now, giving me a chance to let it all sink in. The reality is: I’ve been doing it all wrong.

That’s not necessarily a bad thing, by the way: ya gotta learn stuff by doin’, right?

Ok, so I’ve been using Events as the main decoupling mechanism in HouseMon, i.e. publish and subscribe inside the app. The beauty of this is that it lets you keep the functionality of the application highly modular. Each subsystem subscribes to what it is interested in, and publishes results when they are available. No need to consider who needs this, or even how many subscribers there will be for what gets published.

It sounds great, and in a way it is, but it is a bit like a loose canon: events fire all over the place, with little insight in what is going on and when. For some apps, this is probably great, but with fairly stable continuous processing flows as in a home monitoring setup, it’s more chaotic than need be. It all works fine “in the small”, i.e. when writing little modules (“Briqs” in HouseMon-speak) and adding minor drivers / decoders, but I constantly felt lost with respect to the big picture. And then there’s this nasty back pressure dilemma.

But now the pieces are starting to fall into place. I’ve figured out the “big” structure of the HouseMon application as well as the “little” inner organisation of it all.

It’s all about flow. Three Four stories coming up – stay tuned…

My development setup – utilities

In Software on Sep 12, 2013 at 00:01

As promised, one final post about my development setup – these are all utilities for Mac OSX. There are no doubt a number of equivalents for other platforms. Here’s my top 10:

  • Alfred – Keyboard commands for what normally requires the mouse. I have shortcuts such as cmd-opt-F1 for iTerm, cmd-opt-F2 for MacVim. It launches the app if needed, and then either brings it to the front or hides it. Highly configurable, although I don’t really use that many features – just things like “f blah” to see the list of all folder names matching “blah”. Apart from that, it’s really just an alternative for Spotlight (local disk searches) + Google (or DuckDuckGo) + app launcher. Has earned the #1 keyboard shortcut spot: cmd+space. Top time-saver.

  • HomeBrew – This is the package manager which handles (almost) anything from the Unix world. Installing node.js (w/ npm) is brew install node. Installing MacVim is brew install macvim. Updates too. The big win is that all these installs are compartmentalised, so they can be uninstalled individually as well. Empowering.

  • Dash – Technical reference guides at your fingertips. New guides and updates added a few times a week. Looking for the Date API in JavaScript is a matter of typing cmd-opt-F8 to bring up Dash, and then “js:date”. My developer-pedia.

  • GitHub – Actually, it’s called “GitHub for Mac” – a tool to let you manage all the daily git tasks without ever having to learn the intricacies of git. It handles the 90% most common actions, and it does those well and safely. Push early, push often!

  • SourceTree – Since I’m no git guru, I use SourceTree to help me out with more advanced git actions, with the GUI helping me along and (hopefully) preventing silly mistakes. Every step is trackable, and undoable, but when you don’t know what you’re doing with git, it can become a real mess in the project history. This holds my hand.

  • Dropbox – I’m not into iCloud (too much of a walled garden for me), but for certain things I still need a way to keep things in sync between different machines, and with a few people around the world. Dropbox does that well (though BitTorrent Sync may well replace it one day). Mighty convenient.

  • 1Password – One of the many things that get synced through Dropbox are all my passwords, logins, bank details, and other little secrets. Securely protected by a strong password of course. Main benefit of 1P is that it integrates well with all the browsers, so logging in anywhere, or even coming up with new arbitrary passwords is a piece of cake. Also syncs with mobile. Indispensable.

  • DevonThink – This is my heavyweight external brain. I dump everything in there which needs to be retained. Although plain local search is getting so good and fast that DT is no longer an absolute must, it partitions and organises my files in separate databases (all plain files on disk, so no extra risk of data loss). Intelligent storage.

  • CrashPlan – Nobody should be without (automated, offline) backups. CrashPlan takes care of all the machines around the house. I sleep much better at night because of it. Its only big drawback is that there’s always a 500 MB background process to make this happen (fat Java worker, apparently), and that it burns some serious CPU cycles when checking, or catching up, or cleaning up – whatever. Still, essential.

  • Textual – Yeah, well, ok, this one’s an infrequent member on this list: IRC chat, on the #jeelabs channel for example. Not used much, but when it’s needed, it’s there to get some collaboration done. I’m thinking of hooking this into some sort of personal agents, maybe even have a channel where my house chats about the more public readings coming in. Social, on demand.

The point of all the things I’ve been mentioning in these past few posts, is that it’s all about forming habits, so that a different part of the brain starts doing some of the work for you. There is a perfect French term for getting tools to work for you that way:

  • prolongements du corps (a great example of this is driving a car)

This is my main motivation for adopting touch typing and in particular for using the command-line shell and a keyboard-command based text editor. It (still) doesn’t come easy, but every day I notice some little improvement. As always: Your Mileage May Vary.

Ok, that wraps it up for now. Any other tools you consider really useful? Let me know…

My development setup – hardware

In Hardware on Sep 11, 2013 at 00:01

After yesterday’s notes about my development software, some comments about hardware.

As you may have noticed, I use Apple’s hardware with Mac OS/X. Have gone from big (clunky!) setups to the svelte 1 kg 11″ MacBook Air, and now I’m back on a 15″ model. It’s unlikely that I’ll ever go back to a desktop-only setup, and even the 1920 x 1200 pixel 24″ screen on my desk has been sitting idle for many months now (apart from its use in the FanBot project). One reason is that screen switching with multiple physical screens has not been convenient so far (the next OS revision this fall promises to fix that), but even as is, I find the constant switching between different pixel sizes disruptive. Nowadays, I’m often in house-nomad mode: switching places several times a day around the house – from the desk, to the couch, to a big comfy rotating chair, and back. Heck, even outside, at times!

So one setup it is for me. And these days, that’s a 15″ Retina MacBook Pro (“RMBP15”):

Screen Shot 2013-09-04 at 14.48.48

There’s a lot to say about this, but the essence boils down to: 1920 x 1200 with VM’s.

To me it’s not essential which operating runs on the host: pick one which you feel really at home with, and go for a laptop size and build quality that suits you and your budget.

Now the crazy thing about the MBPR15 is that its screen is not 1920 x 1200, but 2880 x 1800 pixels. And out of the box, the machine comes set to a 1440 x 900 “logical” screen size, i.e. doubling up all the pixels. Which, in my view, is too small as main development environment – at least by today’s measures (hey, we’ve all been there – but it really is worth stepping up whenever you can).

So there’s this curious 1.5x magnification setting on this Mac laptop – does this mean that a 1-pixel thick line will end up getting drawn as “one pixel and a half”?

Obviously not. It’ll all be anti-aliased, as you can see in this close-up:

Screen Shot 2013-09-04 at 15.04.40

If you click on the image above, you’ll see a larger version. There is some interesting stuff going on behind the scenes when using a RMBP15 in 1920 x 1200 “interpolated” mode:

  • to the applications, the screen is reported as being 1920 x 1200
  • so that’s the way apps deal with for screen area and placement
  • for lines, the Mac OSX graphics engine will do its anti-aliasing thing
  • for text, the graphics engine will draw the fonts at their optimal resolution

That last one does the trick: when drawing a 12-point text, the graphics engine will actually draw an 18-point version, using the full screen resolution. As a result, text comes out as sharp as the LCD display will allow, without the application having to do a thing.

I tend to go for the smallest font sizes in editors and command-line shells which my glass-assisted (no, not that one) eyes can still read well (Menlo 10, for monospaced fonts). This gives me a 100 line window height in MacVim, which is perfect. But on a real 1920 x 1200 display, that would actually be pushing it. However, due to the rendering going on inside a Retina Mac, what you actually get to see is text drawn in a 15-point font on a display with over 200 dpi resolution. The result is excellent.

These choices for screen, window, and font sizes are really hitting the sweet spot for me. An 100 x 80 character editing window with splits and tabs as needed, a decent area to see the browser’s console log (and command line), and a main HMTL display area which is still almost exactly the 1366 x 768 size of a common small laptop screen. I rarely use the Mac’s multiple-desktop feature (called “spaces” in OSX), because I don’t have the patience to wait through its sweep-left-and-right animations. And because there’s no need: one “mode” with carefully-positioned windows, and other windows which can be moved to the front and back – a bit disruptively, but that’s because IMO that’s exactly what they are!

The second part of my laptop story is that since everything is running on an Intel 64-bit chip, virtualisation comes easy. This means both Windows and Linux can be run at the same time on the same machine (assuming you have enough RAM). I regularly fire up a Linux VM, and then ssh into it from the command line. For editing, I don’t even have to leave MacVim: just opening a file as scp://debianvm/housemon/ will open a directory window in MacVim and let me navigate from there. With the entire editor environment intact for fast file selection, tags, folding, etc. With Windows, it’s a matter of “mounting” my home directory on Windows, and all the local command line tools can be used for editing, git, diff, and so on (there’s a lot more possible, but I don’t use Windows much).

I’ll finish off tomorrow with some other handy software utilities I’ve come to rely on.

PS. No, this wasn’t intentionally made to coincide with Apple’s latest publicity blitz :)

My development setup – software

In Software on Sep 10, 2013 at 00:01

To follow up on yesterday’s post, here are the different software packages in my laptop-based development setup:

  • MacVim is my editor of choice these days. Now that I’ve switched to touch-typing mode, and given that I’ve used vi(m) for as long as I can remember, this is now growing on me every day. It’s all about muscle memory, and off-loading activities to have more spare brain cycles left for the task at hand.

    I chose MacVim over pure vim in the command line for one reason: it can be set to save all files on focus loss, i.e. when switching another app to the foreground. Tried a dark background for a while, but settled on a light one in the end. Dark backgrounds caused trouble with screen reflections when sitting with my back to a window.

  • TextMate is the second editor I use, but exclusively in browse mode. It can be fired up from the command line (e.g. cd housemon && mate .) and makes it very easy to skim and search across large source collections. Having two entirely different contexts means I can keep vim and its history focused on the code I’m actually working on. I only started using this dual-editor approach a few months ago, but by now it’s become a habit to start them both up – on different areas of the screen.

  • Google Chrome is my main development web browser. I settled on it (over Safari) for mostly one reason: it supports having the developer console pane open to the side of the main screen. There are plenty of dev tools included, from browsing the DOM structure to various performance graphs. Using a dedicated browser for development lets me flush the cache and muck with settings without disrupting other web activities.

  • Safari is my secondary browser. This is the one which keeps history, knows how to log into sites, and ends up having lots of tabs and windows open to keep track of all the distractions on the web. Reference docs, searching for information, and all the other time sinks the web has to offer. It drops nicely to the background when I want to get back to the real thing, but it’s there alongside MacVim whenever I need it.

  • iTerm 2 is what I use as command line. The only reason I’m using this instead of the built-in Terminal application, is that it supports split panes (I don’t use screen or tmux much). It’s set up to start up with two windows side-by-side, because tabs are not always convenient: sometimes you really need to see a few things at the same time.

    I do use tabs in all these apps, dragging them around between windows when needed. Sometimes, I’ll open a new iTerm window and make it much larger (all from the keyboard), to see all the output of a ps or grep command, for example.

  • nvAlt is a fork of Notational Velocity – a note-taking app which stores each note as a plain text file. Its main feature is that it starts up and searches everything instantly. Really. In the blink of an eye, I can find all the notes I wrote about anything. I use it for logging how I installed some tricky software and for lots and lots of lists with design ideas. Plain text. Instant. Clickable URLs. Shared through Dropbox. Perfect.

There’s a lot more to it, but these are the six tools I now use day in day out for software development. Even for embedded work, I’ll use the Arduino IDE only as compile + upload step, editing code in MacVim nowadays – because it has so much more functionality.

All of the above are either included in Mac OSX or open source software. I’m inclined to use OSS as a way to keep my learning investments safe. If ever one of these tools ends up going nowhere, or in an unwanted direction, then chances are that some developer(s) will fork the project to keep things on track (as has already happened with TextMate, iTerm, and nvAlt). In the long term, these core tools are simply too important…

My development setup

In Software on Sep 9, 2013 at 00:01

The tools I use for software development have evolved greatly over the years. Some have been replaced, other have simply gotten better, much better, over time. I’d like to describe the setup a bit in the next few posts – and why / how I made those choices. “YMMV”, as they say, but I think it could be of interest to other developers, whether soon-to-be or seasoned-switchers. Context: I develop in (for?) JavaScript (server- and browser-side).

Ok, here goes… the main tools I use are:

  • 2 editors – yes, two…
  • 2 browser – yes, two…
  • 2 command-line windows, yes, two…
  • a note-taking app (sorry, just one :)

Here is a snapshot of all the windows, taken at a fairly arbitrary moment in time:


And this is the setup in actual use:


Tomorrow, I’ll go over the different bits and pieces, and how they work out for me.

3 years on one set of batteries

In AVR, Hardware on Sep 8, 2013 at 00:01

Ok, so maybe it’s getting a bit boring to report these results, but one of the JeeNodes I installed long ago has just reached a milestone:

Screen Shot 2013-09-06 at 23.30.55

That “buro JC” node has been running on a single battery charge for 3 years now:

And it’s not even close to empty: this is a JeeNode USB with a 1300 mAh LiPo battery tied to its back, and (as I just measured) it’s still running at 3.74 V, go figure.

Let’s do the math on what’s going on here:

  • the battery is specified as 1300 mAh, i.e. 1300 mA for one hour and then it’s empty
  • in this case, it has been running for some 1096 x 24 = 26,304 hours total
  • so the average current consumption must have been under 1300 / 26304 = 50 µA
  • well… in that case, the battery should be empty by now, but it isn’t
  • in fact, I suspect that the average power consumption is more like 10..25 µA

Two things to note: 1) LiPo batteries pack a lot of energy, and 2) they have a really low self-discharge rate, so they are able to store that energy for a long time.

The other statistic worth working out, is the amount of energy consumed by a single packet transmission. Again, first assuming that the battery would be dead by now, and that the microcontroller and the rest of the circuit are not drawing any current:

  • 1,479,643 packets have been sent out, i.e. ≈ 1300 / 1500000 = under 1 µAh per packet
  • since 60 packets are sent out per hour: about 60 µAh per hour, i.e. 60 µA continuous
  • energy can also be expressed in coulombs, i.e. 60 µC gets used per packet transmission (3,600 seconds to the hour, but there were 60 packets sent out during that period)
  • so despite the fact that the RFM12B draws a substantial 25 mA of current during transmission, it does it so briefly that overall it’s still extremely low-power (a few ms every 1s, so that’s a truly minute duty cycle)

The conclusion here is: for these types of uses, with occasional brief wireless sensor data transmissions, the power consumption of the wireless module is not the main issue. It’s far more important to keep the idle (i.e. sleep mode) of the entire circuit under control.

The 2nd result is also a record, a JeeNode Micro, running over 6 months on a coin cell:

This one is running the newer radioBlip2 sketch, which also measures and reports the battery voltage before and after packet transmission. As you can see, the coin cell is struggling a bit, but its voltage level is still fine: it drops to 2.74 V right after sending out a packet (drawing 25 mA), and then recovers the rest of the time to a fairly high 2.94 V. This battery sure isn’t empty yet. Let’s see how many more months it can keep this up.

The 3rd result (penlight test), is this setup, based on the latest JNµ v3:

The timing values are way off though: it has also been running for over 6 months, but I accidentally caused it to reset when moving things around earlier this summer. This one is running with a switching boost regulator. The Eneloop battery started out at 1.3 V and has now dropped slightly to 1.28 V – it should be fine for quite some time, as these batteries tend to run down gradually to 1.2V before they start getting depleted. This is a rechargeable battery, but Eneloop is known to hold on to its charge for a surprisingly long time (losing 20% over 2 years due to self-discharge, if I remember correctly).

You can see the boost regulator doing its thing, as the output voltage sent to the ATtiny is the same 3.04 V as it was on startup. That’s the whole idea: it regulates to a fixed level, while sucking the battery dry along the way…

Note that all these nodes are not sensing anything. They’re just bleeping once a minute.

Anyway… so much for the progress report on a pretty long-running experiment :)

My software

In Software on Sep 7, 2013 at 00:01

Most of the software I write, and most of what I’ve written in the past, has gradually been making its way into GitHub. It’s a fantastic free resource for managing code (with no risk of lock-in – ever – since all git history gets copied), and it’s becoming more and more effective at collaboration as well, with a superb issue tracker and excellent searching and statistics facilities. Over the years, each developer home page has evolved into an excellent intro on what he/she has been up to lately:

Screen Shot 2013-09-03 at 14.58.03

As of recently, I’ve turned things up to 11 by giving everyone who submits a pull request full developer access (it’s a manual process, but effortless). As a result, if you submit a pull request, then whenever I merge it into the official repository, I’ll also add you as developer so that you no longer have to fork to get changes in. Just clone the original repo, and push changes back in.

It may sound crazy, but it really isn’t (and it’s not my idea, although I forgot where I first read about it). The point being that contributors are scarce, and good contributors who actually help take a project forward are scarce as gold. Not because there aren’t any, but because few people take the time to actually contribute and participate in a project created by someone else (shame on me for often being like that!). Time is the new scarce resource, so helping others become more productive by not having to go through the pull-request tango to add and tweak stuff really makes sense to me. Let alone giving real credit to people, by not only accepting their gifts but also giving back visibility and responsibility.

I don’t think there is a risk. Every serious developer will check with the team members before committing a contentious or disruptive change, and even then, all it takes is to create a branch and do the work there (while still letting others see what it’s about).

The other change is that I recently discovered the DocumentUp site, which can generate a very nice front page for any project on GitHub, from just its README page – like this one.

This is so convenient that I’ve set up a website with pointers for most of my code:

Screen Shot 2013-09-03 at 13.45.33

There! That URL ought to be fairly easy to remember :)

Another site (service?) which presents (remixes?) GitHub issues in a neat way is

Screen Shot 2013-09-06 at 00.16.16

There’s a lot more going on with GitHub. Did you know that all issue trackers support live chat? Just keep an issue page open, and you’ll see new comments come in – in real time! Perfect for a quick back-and-forth discussion about some intricate problem. Adding images like screen shots is equally easy: just drag and drop the image into the discussion page.

Did you know that each GitHub project can manage releases, either as is or with detailed information per release? Either way, you get a list of versioned ZIP and TAR archives, simply by adding public tags in Git. And even without tagged releases, there’s always a “Download ZIP” link on the right side of each project page, so you don’t even have to install or use git at all to obtain the latest version of that code.

Screen Shot 2013-09-03 at 15.23.37

There are some 4 million public projects on GitHub. Not all exciting, but tons of them are!

Decoding bit fields – part 2

In Software on Sep 6, 2013 at 00:01

To follow up on yesterday’s post, here is some code to extract a bit field. First, a refresher:


The field marked in blue is the value we’d like to extract. One way to do this is to “manually” pick each of the pieces and assemble them into the desired form:


Here is a little CoffeeScript test version:

buf = new Buffer [23, 125, 22, 2]

for i in [0..3]
  console.log "buf[#{i}] = #{buf[i]} = #{buf[i].toString(2)} b"

x = buf.readUInt32LE 0
console.log "read as 32-bit LE = #{x.toString 2} b"

b1 = buf[1] >> 7
b2 = buf[2]
b3 = buf[3] & 0b111
w = b1 + (b2 << 1) + (b3 << 9)
console.log "extracted from buf: #{w} = #{w.toString 2} b"

v = (x >> 15) & 0b111111111111
console.log "extracted from int: #{v} = #{v.toString 2} b"

Or, if you prefer JavaScript, the same thing:

var buf = new Buffer([23, 125, 22, 2]);

for (var i = 0; i <= 3; ++i) {
  console.log("buf["+i+"] = "+buf[i]+" = "+buf[i].toString(2)+" b");

var x = buf.readUInt32LE(0);
console.log("read as 32-bit LE = "+x.toString(2)+" b");

var b1 = buf[1] >> 7;
var b2 = buf[2];
var b3 = buf[3] & 0x7;
var w = b1 + (b2 << 1) + (b3 << 9);
console.log("extracted from buf: "+w+" = "+w.toString(2)+" b");

var v = (x >> 15) & 0xfff;
console.log("extracted from int: "+v+" = "+v.toString(2)+" b");

The output is:

    buf[0] = 23 = 10111 b
    buf[1] = 125 = 1111101 b
    buf[2] = 22 = 10110 b
    buf[3] = 2 = 10 b
    read as 32-bit LE = 10000101100111110100010111 b
    extracted from buf: 1068 = 10000101100 b
    extracted from int: 1068 = 10000101100 b

This illustrates two ways to extract the data: “w” was extracted as described above, but “v” used a trick: first use built-in logic to extract an integer field which is too big (but with all the bits in the right order), then use bit shifting and masking to pull out the required field. The latter is much simpler, more concise, and usually also a little faster.

Here’s an example in Python, illustrating that second approach via “unpack”:

import struct

buf = struct.pack('BBBB', 23, 125, 22, 2)
(x,) = struct.unpack('<l', buf)

v = (x >> 15) & 0xFFF

And lastly, the same in Lua, another popular language:

require 'struct'
require 'bit'

buf = struct.pack('BBBB', 23, 125, 22, 2)
x = struct.unpack('<I4', buf)

v =, 15), 0xFFF)

So there you go, lots of ways to extract useful data from the RF12demo sketch output!

Decoding bit fields

In Software on Sep 5, 2013 at 00:01

Prompted by a recent discussion on the forum, and because it’s a recurring theme, I thought I’d go a bit (heh…) into how bit fields work.

One common use-case in Jee-land, is with packets coming in from a roomNode sketch, which are reported by the RF12demo sketch on USB serial as something like:

OK 3 23 125 22 2

The roomNode sketch tightly packs its results into a packet using this C structure:

struct {
    byte light;     // light sensor: 0..255
    byte moved :1;  // motion detector: 0..1
    byte humi  :7;  // humidity: 0..100
    int temp   :10; // temperature: -500..+500 (tenths)
    byte lobat :1;  // supply voltage dropped under 3.1V: 0..1
} payload;

So how do you get the values back out, given just those 4 bytes (plus a header byte)?

To illustrate, I’ll use a slightly more complicated example. But first, some diagrams:



That’s 4 consecutive bytes in memory, and two ways a 32-bit long int can be mapped to it:

  • little-endian (LE) means that the first byte (at #00) points to the little end of the int
  • big-endian (BE) means that the first byte (at #00) points to the big end of the int

Obvious, right? On x86 machines, and also the ATmega, everything is stored in LE format, so that’s what I’ll focus on from now on (PIC and ARM are also LE, but MIPS is BE).

Let’s use this example, which is a little more interesting (and harder) to figure out:

struct {
    unsigned long before :15;
    unsigned long value :12;  // the value we're interested in
    unsigned long after :3;
} payload;

That’s 30 bits of payload. On an ATmega the compiler will use the layout shown on the left:


But here’s where things start getting hairy. If you had used the definition below instead, then you’d have gotten a different packed structure, as shown above on the right:

struct {
    unsigned int before :15;
    unsigned int value :12;  // the value we're interested in
    unsigned int after :3;
} payload;

Subtle difference, eh? The reason for this is that the declared size (long vs int) determines the unit of memory in which the compiler tries to squeeze the bit field. Since an int is 16 bits, and 15 bits were already assigned to the “before” field, the next “value” field won’t fit so the compiler will restart on a fresh byte boundary! IOW, be sure to understand what the compiler does before trying to decode stuff. One way to figure out what’s going on, is to set the values to some distinctive bit patterns and see what comes out as bytes:

payload.before = 0b1;  // bit 0 set
payload.value = 0b11;  // bits 0 and 1 set
payload.after = 0b111; // bits 0, 1, and 2 set

Tomorrow, I’ll show a couple of ways to extract this bit field.

Back pressure

In Software on Sep 4, 2013 at 00:01

No, not the medical term, and not this carambolage either (though it’s related):


What I’m talking about is the software kind: back pressure is what you need in some cases to avoid running into resource limits. There are many scenario’s where this can be an issue:

  • In the case of highway pile-ups, the solution is to add road signalling, so drivers can be warned to slow down ahead of time – before that decision is taken from them…
  • Sending out more data than a receiver can handle – this is why traditional serial links had “handshake” mechanisms, either software (XON/XOFF) or hardware (CTS/RTS).
  • On the I2C bus, hardware-based clock stretching is often supported as mechanism for a slave to slow down the master, i.e. an explicit form of back pressure.
  • Sending out more packets than (some node in) a network can handle – which is why backpressure routing was invented.
  • Writing out more data to the disk than the driver/controller can handle – in this case, the OS kernel will step in and suspend your process until things calm down again.
  • Bringing a web server to its knees when it gets more requests than it can handle – crummy sites often suffer from this, unless the front end is clever enough to reject requests at some point, instead of trying to queue more and more work which it can’t possibly ever deal with.
  • Filling up memory in some dynamic programming languages, where the garbage collector can’t keep up and fails to release unused memory fast enough (assuming there is any memory to release, that is).

That last one is the one that bit me recently, as I was trying to reprocess my 5 years of data from JeeMon and HouseMon, to feed it into the new LevelDB storage system. The problem arises, because so much in Node.js is asynchronous, i.e. you can send off a value to another part of the app, and the call will return immediately. In a heavy loop, it’s easy to send off so much data that the callee never gets a chance to process it all.

I knew that this sort of processing would be hard in HouseMon, even for a modern laptop with oodles of CPU power and gigabytes of RAM. And even though it should all run on a Raspberry Pi eventually, I didn’t mind if reprocessing one year of log files would take, say, an entire day. The idea being that you only need to do this once, and perhaps repeat it when there is a major change in the main code.

But it went much worse than I expected: after force-feeding about 4 months of logs (a few hundred thousand converted data readings), the Node.js process RAM consumption was about 1.5 GB, and Node.js was frantically running its garbage collector to try and deal with the situation. At that point, all processing stopped with a single CPU thread stuck at 100%, and things locked up so hard that Node.js didn’t even respond to a CTRL-C interrupt.

Now 1.5 GB is a known limit in the V8 engine used in Node.js, and to be honest it really is more than enough for the purposes and contexts for which I’m using it in HouseMon. The problem is not more memory, the problem is that it’s filling up. I haven’t solved this problem yet, but it’s clear that some sort of back pressure mechanism is needed here – well… either that, or there’s some nasty memory leak in my code (not unlikely, actually).

Note that there are elegant solutions to this problem. One of them is to stop having a producer push data and calls down a processing pipeline, and switch to a design where the consumer pulls data when it is ready for it. This was in fact one of the recent big changes in Node.js 0.10, with its streams2 redesign.

Even on an embedded system, back pressure may cause trouble in software. This is why there is an rf12_canSend() call in the RF12 driver: because of that, you cannot ever feed it more packets than the (relatively slow) wireless RFM12B module can handle.

Soooo… in theory, back pressure is always needed when you have some constraint further down the processing pipeline. In practice, this issue can be ignored most of the time due to the slack present in most systems: if we send out at most a few messages per minute, as is common with home monitoring and automation, then it is extremely unlikely that any part of the system will ever get into any sort of overload. Here, back pressure can be ignored.

Electricity usage patterns

In Hardware, Software on Sep 3, 2013 at 00:01

Given that electricity usage here is monitored with a smart meter which periodically phones home to the electricity company over GPRS, this is the sort of information they get to see:

Screen Shot 2013-09-02 at 11.20.38

Consumption in blue, production in green. Since these are the final meter readings, those two data series will never overlap – ya can’t consume and produce at the same time!

I’m reading out the P1 data and transmitting it wirelessly to my HouseMon monitoring setup (be sure to check the develop branch, which is where all new code is getting added).

There’s a lot of information to be gleaned from that. The recurring 2000+ W peaks are from a 7-liter kitchen boiler (3 min every 2..3 hours). Went out for dinner on Aug 31st, so no (inductive) home cooking, and yours truly burning lots of midnight oil into Sep 1st. Also, some heavy-duty cooking on the evening of the 1st (oven dish + stove).

During the day, it’s hard to tell if anyone is at home, but evenings and nights are fairly obvious (if only by looking at the lights lit in the house!). Here’s Sep 2nd in more detail:

Screen Shot 2013-09-02 at 11.23.50

This one may look a bit odd, but that double high-power blip is the dish washer with its characteristic two heating cycles (whoops, colours reversed: consumption is green now).

Note that whenever there is more sun, there would be fewer consumption cycles, and hence less information to glean from this single graph. But by matching this up with other households nearby, you’d still get the same sort of information out, i.e. from known solar power vs. returned power from this household. Cloudy patterns will still match up across a small area (you can even determine the direction of the clouds!).

I don’t think there’s a concern for (what little) privacy (we have left), but it’s quite intriguing how much can be deduced from this.

Here’s yet more detail, now including the true house usage and solar production values, as obtained from some pulse counters, this hardware and the homePower driver:

Screen Shot 2013-09-02 at 11.52.21

There is a slight lag in smart meter reporting (a value on the P1 port every 10s). This is not an issue of the smart meter though: the DyGraphs package is only able to plot step lines with values at the start of the step, even though these values pertain to the past 10 seconds.

Speaking of which – there was a problem with the way data got stored in Redis. This is no longer an issue in this latest version of HouseMon, because I’m switching over to LevelDB, a fascinating time- and space-efficient database engine.

Making software choices

In Musings on Sep 2, 2013 at 00:01

If there were just one video in the field of software and technology which I’ve watched this summer and would really like to recommend, then it’s this one from about a year ago:

Brian Ford: Is Node.js Better?

It’s 40 minutes long. It’s a presentation by Brian Ford, who has earned his marks in the Ruby world (not to be confused with another Brian Ford from the Angular.js community). He gets on stage at JSConf US 2012, a major conference on JavaScript and Node.js, and spends almost half an hour talking about everything but JavaScript.

At the end, he voices some serious concerns about Node.js in the high-end networking arena w.r.t. its single event-loop without threading, and how the Ruby community hit that wall long ago and made different choices. Interesting, also on a technical level.

But this is not really about language X vs language Y (yawn) – it’s about how we make choices in technology. No doubt your time is limited, and you may not even be interested in Node.js or Ruby, but all I can say is: it made me re-evaluate my positions and convictions, taught me about the power of honest argumentation, and got me to read that brilliant book by Daniel Kahneman, titled Thinking ,Fast and Slow (heavy read – I’m halfway).

Elevating stuff…

It’s been a while…

In Musings on Sep 1, 2013 at 00:01

… since that last blog post. Time to get back into the fun – and I can’t wait!

This summer was spent in comfort and relaxation, a lot of it in and around the house, as everyone had left Houten, leaving behind a totally calm and quiet village, with lots of really nice summer days. As usual, when things are well, time passes quickly…

From a nice bike-visit to around Zwolle, to a very pleasant stay in Copenhagen with a trip to the splendid Louisiana Museum, we had a delightfully “slow” summer for a change.


I spent days on end reading (eReader/iPad) and also had a great time at the OHM2013 hacker event. Made me feel old and young at the same time… I suppose that’s good :)

With a fantastic new discovery: a presentation and workshop on Molecular Cooking. Some pretty amazing stuff with food, such as spherification – we made “apple-juice caviar”, and a soup which makes one side of the mouth warm and the other side cold! (using fluid gels)

Here are some of the other things you can do with this (from the Molécule-R site):

Screen Shot 2013-08-31 at 16.29.46

Lots of fun and ideas to try out. It’s a charming mix between exploring original new tastes and playing tricks with the senses. It’s also called food hacking, which could explain why this topic came up at OHM2013 (alongside activism, banking, and fablabs).

A summer of contrasts… from a Hanseatic city to modern gastronomic creativity!

Summer Break

In News on Jul 1, 2013 at 00:01

It’s that time of year again: 1st of July, and time for me to take a break.

Just as last year, this is going to be my last post on this JeeLabs weblog post for a while – two months to be exact. This isn’t so much about going on holiday (nothing planned yet, other than a few short trips) as it is about performing an internal reset.

The past year has been mostly about taking a new direction in software (by which I mean Node.js and the Dive Into JeeNodes series) – not the embedded Arduino stuff), and about exploring numerous electronics and software topics – a.k.a. Physical Computing.

Speaking of which, yesterday’s post was a good example of what I hope to keep up for a long time to come: introducing electronics and mixing it with embedded microcontrollers to show just how easy it is to tie the two fields together, and to keep on enticing everyone who comes across this weblog to explore, learn, tinker, and play with this stuff. It’s all very low-cost, it’s wide open to make tons of new ideas happen, and there is an immense body of knowledge, experience, and open source software + hardware available to anyone with some spare time, a healthy dose of curiosity, and an internet connection.


There’s a lot to be done beyond what I’ve been dabbling in here at JeeLabs. The field of wireless communication has only just started for the home and the hobbyist. There’s an explosion going on w.r.t. small affordable Linux boards, which take Physical Computing to totally new levels of capability. And there is a huge need to find easy and enticing entry paths into all this, if you ask me. The more there is to learn, the more we need to come up with ways to help people “find their way in”. The RoboCup 2013 event (which will be over by the time you read this) has shown that there is a great opportunity to expose kids of all ages to technology. From hundreds of cheering cardboard FanBots all the way to amazing self-organizing teams of autonomous “Middle Size League” football-playing robots.

The future has only just begun. What an amazing times we live in!

Second bit of news is that it’s time to kick off a summer sale in the web shop once again. And as in previous years, the following discount is for existing JeeLabs supporters, i.e. it can only be requested if you have ordered products from JeeLabs in the past:


And while I’m at it, let me list all the key dates for this period, here at JeeLabs:

  • July 1st – weblog paused and summer sale kicks off
  • July 31st – sale ends at midnight, 0:00 CEST time
  • September 1st – daily weblog resumes

Due to the fine efforts by Martyn Judd & Co., the shop will remain open throughout the summer break, with fulfilment continuing as before from the UK Center in Cambridge. The staff level will be somewhat reduced during August, but we will nevertheless attempt to keep up the regular flow of sending your packages promptly.

There is also a summer sale at Modern Device, but note that since the shops are separate and independent entities, we cannot extend the same offer across the shops. Please check the shop you have purchased from in the past to see what’s on sale.

If you’re looking for geek stuff to do this summer: check out the chronological index of this weblog, maybe there is something that interests you, somewhere in those past 1,364 posts?

Anyway: thank you for all the interest, comments, discussions, tips, and appreciative emails – I’m honoured by everything that’s happening around JeeLabs and looking forward to lots of new ideas, developments, sharing, and products – starting again on September 1st.

Have a nice summer (or winter, on the southern hemisphere)!

From PWM to voltage

In Hardware on Jun 30, 2013 at 00:01

The Arduino library has always had an “analogWrite()” function, even though the ATmega doesn’t have any way to generate a varying voltage. So why the name?

Well, what most microcontrollers can do is generate a pulse-width modulated signal, also known as PWM. That’s just a fancy way of saying that the microcontroller periodically generates a pulse, of which the width can be varied under software control.

Here’s a diagram from the Arduino PWM tutorial page:


It’s very easy to generate an approximate voltage between 0 and VCC by simply adding a resistor and capacitor to act as low-pass filter:


That image came from an excellent article by Scott Daniels about this same topic on the website. Check it out for much more background information.

So how do these two little components turn PWM into a voltage?

To see what’s going on, we can tie the resistor to a PWM pin, call “analogWrite(N,128)” to set a 50% duty cycle, i.e. halfway, and then watch the result on the oscilloscope:


(the above was generated with a 10 kΩ resistor and 0.1 µF capacitor – see this calculator)

What you’d want to see, is a 1.65 V output. As you can see, the real thing is indeed centered around that value, but with a lot of “ripple” (yellow = PWM, blue = Vout).

The reason for this is that the PWM is charging and discharging the capacitor. The average value will be the ratio of on-to-off time against the digital voltage, i.e. 3.3V, so a 50% duty cycle will lead to half VCC, i.e. 1.65V. But it’s still charging & discharging.

Can we get a cleaner signal out, without that up-and-down wavering? Yes: by increasing the repetition rate, i.e. the PWM frequency (while keeping the on-off ratio the same). There’s a page on the Arduino site about this. Here’s the result, with a 64x higher rate:


The purple line is a third probe on the same output signal, but set to AC coupling and much higher sensitivity, to show that the ripple is still there, but much much lower.

Now I can get to the point of this post: my intention was to see if I could generate a clean sawtooth wave, to use as supply voltage (after amplification) for testing circuits at various VCC levels – as used a few times in recent posts. Here’s the code I came up with:

void setup () {}

void loop () {
  for (int i = 0; i < 256; ++i) {
    analogWrite(5, i);

The result is this:


And now there’s indeed a sawtooth, but as you can see, sudden changes like the down edge take some time to pass through the RC filter, somewhat messing up the signal.

No problem, if we increase the delay time in the above loop by a factor of 50:


Voilá – things are fine now. A pretty smooth voltage ramp from 0 to VCC every 2 seconds, all from a single digital output pin on the ATmega – we’ve created a DAC!

FanBot Registration System

In Software, Linux on Jun 29, 2013 at 00:01

Nearly 400 FanBots have already been built – see this page and the KekBot weblog.

To finish this little series, I wanted to describe the little setup I built and how it was done. Nothing spectacular, more an example of how you can create a software cocktail / mashup to get a one-off short-lived project like this going fast.

First the basic task at hand: allow kids to register their FanBot and print out a “passport” with a picture of their creation and some information about when and how to get their FanBot back after the RoboCup 2013 event. At the same time, it would be nice to have a monitor display the current count and show images of the last few FanBot pictures taken.


I decided to use 3 Raspberry Pi’s as “slave”, each with a webcam, an LCD screen (thanks David, for the laser-cut frame!), and Ethernet + power. The code running on them is on github. It would be nice if each slave could continue to accept registrations even with the network connection temporarily down, so I went for the following approach:

  • collect registrations locally, using rsync to push new ones to the master
  • everything is based on a few simple shell scripts for quick adjustment
  • two C/C++ programs were needed: to read the FanBot data and to drive the LCD
  • do a git pull on power-up and re-compile as needed before starting operation
  • periodic webcam readout and periodic rsync, using loops and sleep calls

For the master, I thought about using the Odroid/U2, but decided against it as there would be too much setup involved in the last minute. Next idea was to use an Acer Inspire One Linux netbook, but it turned out to be way too slow. For some reason, Node.js was consuming 100% CPU (I suspect a bad nodemon disk change polling error on the slow SSD). No time to figure that out, so I ended up using my second Mac laptop instead.

The master logic started out as follows:

  • an rsync server was set up to collect files from all the slaves in one spot
  • the display is presented via a browser in full-screen mode and Node.js
  • the original plan was to also allow outside access, but we never got to that
  • for live updating, I decided to use SocketStream (same as in HouseMon)
  • a PDF was generated for each image by convert, which is part of ImageMagick
  • an AppleScript then prints each PDF to the USB-attached laser-printer

These choices turned out to be very convenient: by having the rsync folder inside the Node.js application area, and using nodemon, refresh became automatic (nodemon + socketstream will force a refresh on each client whenever a file system change is detected). So all I had to do was write a static Node.js app to display the count and the latest images.

So far so good. A couple of glitches, such as an erratic printer (new firmware fixed it), an overheating RPi (fixed by clocking at 700 MHz again), and all the slave rsyncs fighting when the same registration was present on more than one (fixed by adding the “-u” flag).

During installation, we found out that there was no wired ethernet w/ DHCP, as I had assumed. Oops. So the Mac laptop was turned into a WiFi -> wired bridge, since WiFi was available most of the time. Whoops: “most” is not good enough, the RPi’s have no clock, they really need an NTP server – especially since the whole system was based on rsync and file modification times! We hacked a fixed DNS – but internet still has to be there.

And then the fun started – various little wishes came up after the fact:

  • can we get images + passports off-site during the event? sure: rsync them to Dropbox
  • could we have a page with all images? sure: on my server at JeeLabs, I made an rsync loop from one Dropbox area to another, and made a little photo-album using Dropbox
  • could we show a live count, even though the monitor website was not accessible? sure: I simply added another shell loop on my server again, to generate an image with the count in it, and made it show up as first one in the photo album.

Here is the shell script for that last feature:

cd ~/Dropbox/FanBots
while :
  N=$(echo `ls *.jpg | wc -l`)
  if [ "$N" != "$P" ]
    echo $N - `date`
    convert -background black -fill red -size 480x640 \
      -gravity center "label:$N\n\nFanBots" 0-teller.png
  sleep 10

This can run anywhere, since Dropbox is taking care of all updates in both directions!

The point of all this is not which languages I used (your choice will definitely be superior), nor what hardware was selected for this (it changed several times). The point is that:

  1. it all got done in a very limited amount of time
  2. the basic rsync approach made things resilient and self-recovering
  3. github pulls made it possible to apply last-minute tweaks
  4. some nice changes were possible very late in the game
  5. dropbox integration made it possible to add features remotely

Happy ending: all the kids are having a good time … including me :)

PS. Tomorrow (Sunday) is the RoboCup 2013 finals. See you there?

Status of the RFM12B

In Hardware on Jun 28, 2013 at 00:01

The RFM12B wireless radio module has been around for quite some time. When I found out about it at the time, I really liked the mix of features it provided – far more capable than the morse-code like OOK modules in cheap sensors, very low power, and available at a considerably lower cost than the XBee and other ZigBee solutions out there:


There was little software for it at the time, and writing an interrupt-driven driver for it was quite a challenge when this all started, but nowadays that’s no longer an issue – the RF12 driver which is now part of JeeLib has turned out to work quite well.

There have been some rumours recently, and understandably also some worries, that the RFM12B might be phased out, but if I may paraphrase Mark Twain on this: the reports of the RFM12B’s death have been greatly exaggerated…

After some email exchanges between HopeRF and Martyn Judd, I’m now happy to report that we have received a formal statement from HopeRF to clarify the issue:

“The popular RFM12B S1/S2 modules remain in production with a large volume of long standing and long term orders. There is NO plan to discontinue the series and NO plan to end production in the foreseeable future. It is correct that we no longer recommend this series to customers developing new projects, since we have developed more recent designs, for example the RFM6X series. These incorporate functionality improvements that can provide a better price/performance for specific applications.”

— Derek Zhu, Marketing Manager, HopeRF

As I see it, this means that the RFM12B will be around for quite some time and that we don’t have to worry about supplies for keeping all current networks based on these modules working. I definitely intend to keep using RFM12B’s at JeeLabs and in products designed and produced by JeeLabs.

Having said that, I’m also evaluating alternatives and looking for a convenient option to act as next step forward. My focus is on low-power consumption and good range for use within the house, both of which I hope to take even further. I’d also like to make sure that over-the-air flashing with JeeBoot will work well with the current as well as future choices.

In my opinion the RFM12B continues to be a simple & excellent low-cost and low-power foundation, so I’m taking my time to very carefully review any new options out there.

Meet the FanBot

In Hardware, Musings on Jun 27, 2013 at 00:01

The FanBot is a very simple robot based on a small PCB with microcontroller by Peter Brier, some LEDs as eyes and mouth, and a servo to allow the robot to wave its arms:


Over a thousand boards have been produced, along with accessories to let children create a little cardboard robot with their own name and a little program to store a personalised sequence of LED blinks and servo movements. The µC is an ARM LPC11U24 chip, donated by NXP – which has plenty of power, but more importantly: can be programmed by powering it up as a USB memory stick.


Wednesday was the kick-off / trial day, with 120 kids dropping by and creating their own FanBot. The FanBots will all be used for the main event to cheer on the main RoboCup contestants. Here’s a quick impression of the first 80 “fans” (it’s a huge task to get them all up there, checked, stapled, and connected – not to mention the power/network setup!):


It’s a really wonderful project, IMO: lots of young kids get exposed to new technology, learning about robots by building their own, and creating a huge collection of truly individual and personal robots, all cheering together!

For more info, see Peter and Marieke’s KekBot weblog – there’s also a progress page.

The RoboCup championship itself uses more sophisticated robots, such as these:

BvOF RoboCup2013

Many more pictures of this event can already be found on the RoboCup 2013 website and on their Flickr group. The event has only just started so if you’re in the neighbourhood: it’s free, and bound to be oodles of fun for kids of any age!

Myra and I had a wonderful time, and I even had a chance to see Asimo in action, live!

And JeeLabs ended up getting a spot on the sponsor page – not bad, eh?

Update – Forgot to mention that one of the requirements of RoboCup is that everything must be made open source after the event. This means that any advances made can be used by anyone else in the next year. What a fantastic way to stimulate shared progress!

Packaged but not virtualised

In Hardware, Software, Linux on Jun 26, 2013 at 00:01

Since VMware started virtualising x86 hardware many years ago, many uses have been found for this approach of creating fully self-contained software environments. I’ve used VMware many years, and was pleased when Apple switched the Macs to “Intel inside”.

Nowadays, I can run numerous variants of Windows as well as Linux on a single laptop. I switched to Parallels as VM core, but the principle remains the same: a large file is used as virtual hard disk, with an operating system installed, and from then on the whole environment simply “runs” inside the Mac host (in my case).

Nowadays, we all take this for granted, and use virtual environments as ultra-low cost web servers without even giving it any thought. I have a number of virtual machines running on a Mac Mini for all the JeeLabs sites in fact (and some more). One piece of hardware, able to suspend and resume N completely independent systems at the click of a mouse.

But there’s a drawback: you end up duplicating a lot of the operating system and main services in each VM, and as a result this stuff not only consumes disk space but also leads to a lot of redundant backup data. I keep the VM’s very lean and mean here, but they still all eat up between 3 and 10 gigbytes of disk space. Doesn’t sound like much, but I’m pretty sure my own data uses less than 10% of that space…

Both Parallels and VMware (and probably also VirtualBox) support snapshotting, which means that in principle you could set up a full Linux environment with a LAMP stack, and then use snapshots to only store what’s different for each VM. But that’s not useful in the long run, because as soon as you get system updates, they’ll end up being duplicated again.

It’s a bit like git “forks”: set up a state, create a fork to create a variant, and then someone decides to add some features (or fix some bugs) to the original which you’d also like to adopt. For git, the solution is to merge the original changes into your fork, and use “fast-forwarding” (if I understand it correctly – it’s all still pretty confusing stuff).

So all in all, I think VM’s are really brilliant… but not always convenient. Before you know it, you end up maintaining and upgrading N virtual machines every once in a while.

I’ve been using TurnkeyLinux as basis of all my VM’s because they not only maintain a terrific set of ready-to-run VM images, but because they also make them self-updating out of the box, and because they’ve added a very convenient incremental daily backup mechanism which supports restarting from the back-up. You can even restart in the cloud, using Amazon’s EC2 VM “instances”.

And then someone recently told me about Docker – “The Linux container engine”.

This is a very interesting new development. Instead of full virtualisation, it uses “chroot” and overlay file systems to completely isolate code from the rest of the 64-bit Linux O/S it is running on. This lets you create a “box” which keeps every change inside.

My view on this, from what I understand so far, is that Docker lets you play git at the operating system level: you take a system state, make changes to it, and then commit those changes. Not once, but repeatedly, and nested to any degree.

The key is the “making changes” bit: you could take a standard Linux setup as starting point (Docker provides several base boxes), install the LAMP stack (if someone hasn’t done this already), commit it, and then launch that “box” as web server. Whereby launching could include copying some custom settings in, running a few commands, and then starting all the services.

The way to talk to these Docker boxes is via TCP/IP connections. There’s also a mechanism to hook into stdin/stdout, so that you can see logs and such, or talk to an interactive shell inside the box, for example.

The difference with a VM, is that these boxes are very lightweight. You can start and stop them as quickly as any other process. Lots of ’em could be running at the same time, all within a single Linux O/S environment (which may well be a VM, by the way…).

Another example would be to set up a cross compiler for AVR’s/Arduino’s (or ARMs) as a box in Docker. Then a compile would be: start the box, feed it the source files, get the results out, and then throw the box away. Sounds a bit drastic, but it’s really the same as quitting an application – which throws all the state in RAM away. The point is that now you’re also throwing away all the uninteresting side-effects accumulated inside the box!

I haven’t tried out much so far. Docker is very new, and I need to wrap my mind around it to really understand the implications, but my hunch is that it’s going to be a game changer. With the same impact on computation and deployment as VMware had at the time.

As with VM’s, Docker boxes can be saved, shared, moved around, and launched on different machines and even different environments.

For more info, see this page which draws an analogy with freight containers, and how one standard container design has revolutionised the world of shipping and freight / cargo handling. I have this feeling that the analogy is a good one… a fascinating concept!

Update – Here’s a nice example of running a Node.js app in Docker.

FanBots at RoboCup 2013

In Hardware, Software, Linux, ARM on Jun 25, 2013 at 00:01

There’s an event coming up in a few days in Eindhoven, called the RoboCup 2013:


This is the world championship of robotic football. For all ages, and with various levels of sophistication. Some robots are tiny cars, trying to work together to play a match, some robots are little human-like machines consisting of lots of servos to make them walk on two feet, and some robots are very sophisticated little “towers” on three wheels, zipping along at impressive speeds and able to catch, hold, and kick a standard-size football.

I’m not directly involved in this, but I saw some of the preparations and it really promises to be a fantastic event. Lots of geeks with laptops, soldering irons, and mechanical stuff. The kick-off is next Wednesday and the event lasts for five days, with the finals on Sunday.

On the side, there is a gorgeous (not-so) little project going on, whereby kids of ages 8 to 12 can visit a workshop to build their own FanBot (for free). The FanBot can blink its eyes and mouth, and wave its arms – part of the exercise is to create a little “program” of things it does at the press of a button, so this will also be a first encounter with programming!

There’s a Dutch page about the project and a general page in English mentioning FanBots.

But the really exciting goal will be to create a huge stand for all the FanBots and make them cheer the football-playing robots in the main event hall. The aim is to get up to 1000 of them, all blinking and waving – and all orchestrated and controlled from a central system. It’s all being designed and built by Peter Brier and by Marieke Peelen of KekkeTek.

One task in the project is to register all the FanBots and print out a little “passport” as last step – which the kids can then use to pick up their own FanBot once the RoboCup 2013 event is over. This is the part I volunteered to help with.

I’ll describe the setup in more detail soon, but that’s where the Raspberry Pi’s w/ LCD screen and webcams will be used. It’ll be fun to see whether we can get 1000 kids to build and sign up their little robots during three days – there’s quite a bit of logistics involved!

Admission is free, so if you’re in the neighbourhood: don’t miss it – it’s going to be fun!

Using eMMC as root disk

In Hardware, Linux on Jun 24, 2013 at 00:01

The Odroid/U2 mentioned yesterday also has the option to add an eMMC card, right next to the µSD card slot – I went for a whopping 64 GB “SSD” in this case:


The reason to use this, is that it’s indeed nearly three times as fast:

# hdparm -tT /dev/mmcblk0

 Timing cached reads:   1604 MB in  2.00 seconds = 802.41 MB/sec
 Timing buffered disk reads: 146 MB in  3.02 seconds =  48.40 MB/sec

That’s faster than the maximum transfer rate of USB 2.0, by the way.

But how do you set these things up? The bottom view shows yet another connector, yuck:


It turns out that you can get an adapter from the Odroid/U2 shop:


That turns it into a µSD card, but unfortunately it wasn’t recognised in my Mac laptop. Luckily, this concoction did seem to do the trick:


Through this, I was able to burn the Debian disk image onto it, and through some clever logic in the bootstrap loader, the eMMC card simply takes precedence when present and starts up fine with the same device name as the initial µSD card (which now ends up on /dev/mmcblk1 if also present).

One last step is needed to resize the root partition to take full advantage of the entire eMMC card. Google found this article which explains the whole (nasty) procedure.

Final result:

# free
             total       used       free     shared    buffers     cached
Mem:       2031632      77660    1953972          0       9540      37420
-/+ buffers/cache:      30700    2000932
Swap:            0          0          0

And… tada!

# df -H
Filesystem      Size  Used Avail Use% Mounted on
rootfs           62G  1.2G   58G   2% /
udev            1.1G     0  1.1G   0% /dev
tmpfs           209M  222k  208M   1% /run
/dev/mmcblk0p2   62G  1.2G   58G   2% /
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs           417M     0  417M   0% /run/shm
/dev/mmcblk0p1  133M  8.5M  124M   7% /boot

Neat, neat, neat – although the total system cost is close to $200 at this point, due to the expensive eMMC memory card – five times the cost of a Raspberry Pi, but with four times the RAM, a comfy 60 GB of fast SSD storage, and probably 5..10 times the performance.

It’s not for everyone and it has no GPIO pins to hook anything up, but I like it!

More about the Odroid/U2

In Hardware, Linux, ARM on Jun 23, 2013 at 00:01

As mentioned recently, there are many alternatives to the Raspberry Pi. I’m looking mostly at ARM-based systems these days, because Node.js support for MIPS is not really there (yet?), and you really want a JIT-type system to get the most out of a Node.js based setup.

Well, having received the Odroid/U2, first thing I had to do is get a micro-HDMI to HDMI adapter (always some new connector/cable type, yawn…). Also had to get a 16 GB µSD card, because the Debian image I wanted to try was for cards of 8 GB or more (yawn!).

But now that everything is in, I’m finally able to take that little black cube for a spin:


First impression: it’s fast, much snappier than a Raspberry Pi. Try launching aptitude, for example – which takes a long time on RPi’s and other low-end ARM & MIPS systems.

Both are running class-10 SD cards, as far as I can tell. The RPi is running at 700 MHz:

# cat /proc/cpuinfo 
Processor   : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS    : 697.95
Features    : swp half thumb fastmult vfp edsp java tls 
Hardware    : BCM2708

The OU2 has four cores running at 1.7 GHz:

# cat /proc/cpuinfo 
Processor   : ARMv7 Processor rev 0 (v7l)
processor   : 0
BogoMIPS    : 1992.29
processor   : 1
BogoMIPS    : 1992.29
processor   : 2
BogoMIPS    : 1992.29
processor   : 3
BogoMIPS    : 1992.29
Features    : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls 
Hardware    : ODROIDU2

Disk speeds are interesting, first RPi then OU2:

# hdparm -tT /dev/mmcblk0
 Timing cached reads:   340 MB in  2.00 seconds = 169.91 MB/sec
 Timing buffered disk reads:  62 MB in  3.00 seconds =  20.66 MB/sec

# hdparm -tT /dev/mmcblk0
 Timing cached reads:   1616 MB in  2.00 seconds = 808.23 MB/sec
 Timing buffered disk reads:  52 MB in  3.05 seconds =  17.03 MB/sec

I haven’t set up the eMMC card yet, which is said to be several times faster than µSD.

Power consumption of the OU2 is 2.9 W on the AC mains side of a little 5V adapter I’m using. Not bad. I’m tempted to try and set this up as my only always-on server here at JeeLabs, including all the web server stuff even. There’s 2 GB of RAM in this thing, should be enough to run some serious stuff. Perhaps even the (Java-based) CrashPlan backup system I’ve been using for almost a year now.

I don’t really care that much about file storage – a 64 GB eMMC disk would be plenty for everything that needs to be online here at all times, perhaps with an external 2.5″ USB hard disk for secondary / archival backup.

Storage – let’s do the math

In Musings on Jun 22, 2013 at 00:01

Doesn’t it ever end? Storage density, I mean:


That’s a 16 GB µSD card. There are also 64 GB cards, but they are a bit more expensive.

Ok, let’s see – that’s 0.7 x 11 x 15 mm = 115.5 mm3 in volume.

Now let’s take a 2.5″ hard disk. The dimensions are 9.5 x 70 x 100 mm = 66,500 mm3.

So an already-small 2.5″ disk drive (or cell phone, if you prefer) takes as much space as 575 µSD cards, if you don’t count the card slot or other mounting options.

Let’s go for the 64 GB cards, that’s a whopping 36.8 terabytes of solid-state data. At €50 each, that’d require a €28K investment for about €0.80 per gigabyte. I’ll pass for now :)

But how much is 36 TB? Well, you’d be able to store over 5 Kb of data for each living person on this planet, for example. And carry it in your pocket.

Hello sir. One millisec… let me check your name, travel history, family tree, and CV.

What a terrifying thought.

Cheap webcam

In Hardware on Jun 21, 2013 at 00:01

For a project I’m involved in (yeah, yeah, it’ll all be posted in due time), I needed a couple of webcams to take simple snapshots. Started out with a cheap unit from eBay, but that one really wasn’t up to its task. Then I found this limited-stock offer at Conrad:


It’s €10 and it even includes a headset which I don’t need. Go figure…

Can’t be much, eh? Well, how about 1280×1024 with auto-focus and auto-white balance?


(you can click-through to see the full-size image)

Amazing stuff. Amazing bargains. Amazing internet. Amazing mail-order. Wow…

The Knapsack problem

In Musings on Jun 20, 2013 at 00:01

This is related to the power consumption puzzle I ran into the other day.

The Knapsack problem is a mathematical problem, which is perhaps most easily described using this image from Wikipedia:


Given a known sum, but without information about which items were used to reach that sum (weight in this example), can we deduce what went in, or at least reason about it and perhaps exclude certain items?

This is a one-dimensional (constraint) knapsack problem, i.e. we’re only dealing with one dimension (weight). Now you might be wondering what this has to do with anything on this weblog… did I lose my marbles?

The relevance here, is that reasoning about this is very much like reasoning about which power consumers are involved when you measure the total house consumption: which devices and appliances are involved at any point in time, given a power consumption graph such as this one?

Screen Shot 2013-06-17 at 09.35.49

By now, I know for example that the blip around 7:00 is the kitchen fridge:

Screen Shot 2013-06-17 at 09.35.49

A sharp start pulse as the compressor motor powers up, followed by a relatively constant period of about 80 W power consumption. Knowing this means I could subtract the pattern from the total, leaving me with a cleaned up graph which is hopefully easier to reason about w.r.t. other power consumers. Such as that ≈ 100 W power segment from 6:35 to 7:45. A lamp? Unlikely, since it’s already light outside this time of year.

Figuring out which power consumers are active would be very useful. It would let us put a price on each appliance, and it would let us track potential degradation over time, or things like a fridge or freezer needing to be cleaned due to accumulated ice, or how its average power consumption relates to room temperature, for example. It would also let me figure out which lamps should be replaced – not all lamps are energy efficient around here, but I don’t want to discard lamps which are only used a few minutes a day anyway.

Obviously, putting an energy meter next to each appliance would solve it all. But that’s not very economical and also a bit wasteful (those meters are going to draw some current too, you know). Besides, not all consumers are easily isolated from the rest of the house.

My thinking was that perhaps we could use other sensors as circumstantial evidence. If a lamp goes on, the light level must go up as well, right? And if a fridge or washing machine turns on, it’ll start humming and generating heat in the back.

The other helpful bit of information, is that some devices have clear usage patterns. I can recognise our dishwasher from its very distinctive double power usage pulse. And the kitchen boiler from it’s known 3-minute 2000 W power drain. Over time, one could accumulate the variance and shape of several of these bigger consumers. Even normal lamps have a plain rectangular shape with fairly precise power consumption pattern.

Maybe prolonged data collection plus a few well thought-out sensors around the house can give just enough information to help solve the knapsack problem?

What if the signal lines are very long?

In Hardware on Jun 19, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

This question came from figuring out how to drive an SPI chip over a few meters distance. Normally, you’re not supposed to do that: I2C and SPI in particular are designed for use on a PCB, i.e. distances in the order of dozens of centimeters at best. Fortunately, in this case speed was not an issue, so what if we simply take the clock signal way down, to say 10 KHz (50 µs on + 50 µs off). Can we create a solid SPI setup across a few meters of wiring?

To try this out, I decided to take a reel with some 70 to 80 meters of multi-core wire:


Being untwisted and unshielded, this is probably as bad as it gets for signalling purposes. Then I started with a clean 10 KHz square wave of 0..5V from the signal generator:


Here’s what happens to the signal on the signal generator side (i.e. same probe placement), when simply connecting the open wires of the above cable to signal and ground:


This is caused by the load and the reflection of the signal on the end of that long cable. Think of it as an echo of that voltage change, propagating back and forth and colliding with the signal – causing this fairly substantial change in signal rise time and shape..

But to really see the effects, we have to also look at the signal at the other end, where it’s going to be picked up and used. I made the following setup, with three different probes:

JC's Grid, page 81

(sorry for the confusion: “B” and “M” were mixed up in this drawing)

  • YELLOW probe on the incoming signal generator end
  • MAGENTA probe on the outgoing signal
  • BLUE probe on the outgoing ground connection
  • RED is the MAGENTA minus the BLUE level (as math function)

All the vertical scales are set to the same 1V per division from now on:


Note that there are some really strange effects in there: the magenta output signal is “ringing”, as is to be expected with such a long coil of wire and lots of stray capacitance, inductance, and resistance. But the blue “ground” level is also ringing, almost as much in fact! And the yellow input is funny, too. Let’s zoom in a bit on that rising edge:


What does this mean for signal propagation? Well, as you can see, everything rattles and shakes in situations like these. It really makes a difference for example as to how the end is connected to power. If you use the (blue) output ground as reference, then the signal will appear as the difference (i.e. the red line), which has less extreme ringing than if you use the (magenta) output referenced to the incoming ground pin.

None of these effects are problematic at this 10 KHz signalling rate. If you need to pass signals like these, it definitely pays to A) keep the signalling rate low, B) reduce the steepness of the edges, and C) add a fairly substantial load resistance at the end of the wire (between signal and ground, say 330..1000 Ω). The reasons and details of all this will have to wait for some future posts.

What I do want to mention, is that you can actually see the speed of light in action here. Let’s zoom even further into that rising edge (sorry, blue and magenta are reversed now):


It takes over 400 ns for the yellow rising edge to reach the other end. In vacuum, light travels some 120 meter in that amount of time – not so very different from the 80 meter or so of cable the electricity has to traverse. Pretty amazing what you can do these days, with nothing more than a signal generator, a modern oscilloscope, and a coil of wire!


In Hardware on Jun 18, 2013 at 00:01

After yesterday’s puzzle, I thought that would be the end of it. But sensitised to this, I did check again this morning. And guess what… the problem came back!

Screen Shot 2013-06-17 at 08.32.41

This time, I was determined to find the culprit. Starting around my office with lots of powered devices, I narrowed it down to the power adapter of an Acer Inpire One netbook:

Screen Shot 2013-06-17 at 09.03.52

Seems conclusive, right? Extra evidence: I took that netbook apart a while ago to install more RAM. All went well, except that the battery charge circuit never seems to have recovered – it’s pretty tiny stuff in there, I must have damaged or disconnected something. Luckily, I was using this netbook only in the lab, i.e. plugged in, so no big deal for me.

Except… this netbook was not plugged in before yesterday. This is a different puzzle!

And digging a little deeper, the pattern is not even exactly tied to when it’s plugged in.

Oh well – this gives me an excuse to talk about knapsacks… stay tuned.

Power consumption puzzle

In Musings on Jun 17, 2013 at 00:01

Recently I had to go abroad for a few days, and given a new access path I just added to my HouseMon setup, it let me track things while away from home. Is that good? Eh, sort of…

The nice thing is that I can now check the solar production in the evening while away from home. And obviously I can then also check on the energy consumption – here’s what I saw:

Screen Shot 2013-06-15 at 19.03.26

Very odd. Something was turning on and off all the time! Here’s a close-up:

Screen Shot 2013-06-15 at 19.00.26

As you can see, I did figure it out in the end – but only after this had been going on for two days, on the phone with Liesbeth to try and identify the source of this behaviour.

It was a bit worrying – that’s a 300..400 watt power draw, every 45 seconds. Going on for days. This is enough energy to cause serious heat problems and my worry was that it was some sort of serious malfunction, with a thermal safety constantly kicking in.

Yikes. Especially when you’re away, knowing there is no one at home during the day…

The answer to this riddle isn’t 100% conclusive, because I did find a way to stop the pulsing, but turning power on again did not cause the pulsing to resume:

Screen Shot 2013-06-15 at 19.29.32

This is crazy. The only explanation I found, was the laser printer: I had printed a few pages before going away (and as far as I can remember, I did turn the printer off after use). The printer was behaving erratically, and I had to turn it on and off a few times.

So my conclusion is: the laser printer (which has a toner heater built-in that can indeed easily draw a few hundred watts) got stuck in some weird mode. And kept me worried for a few days – all just because of me being a geek and checking my graphs!

I don’t know what lesson to take home from all this: stop monitoring? stop checking? stop traveling? – or perhaps just stop worrying when there’s nothing you can do about it ;)

Oscilloscope sampling rate

In Hardware, Linux on Jun 16, 2013 at 00:01

Just to finish the series on Raspberry Pi I/O – here’s proof of its GPIO pulsing speed:


The trouble is the irregularity caused by Linux’s multi-tasking properties, and I wanted to show you some examples of that. After a few dozen captures, I came up with this one:


Looks pretty bad, right? Tons of hickups in what was meant to be a 5.2 MHz pulse train.

Not so fast…

The sample rate used by the oscilloscope for this capture was 10 MHz, so it’s sampling the signal at 100 ns intervals. That’s extremely risky, because when the signal is also pulsing at this rate, you might end up missing each peak for a substantial amount of time. This would make it appear as if the signal was low or high all the time, despite its rapid changes.

This effect is called “aliasing”, and the solution is simple:

Do Not Sample At A Rate Close To The Frequencies In Your Signal!

I’m inclined to discard the above snapshot – it’s probably a measurement artefact.

Here’s another screen, this time at 50 MHz sample rate, i.e. 20 ns between captures:


At that rate, it’s impossible to miss pulses which are all in the range of 100 ns or longer. As you can see, there are still glitches (almost on every snapshot there is one), but it’s not as bad as the previous screen shot was suggesting.

Anyway. Enjoy your RPi and all the Linux high-level stuff it can do (like, ehm, run the Arduino IDE and cross-compile stuff for ATmega’s), but beware of treating it like a CPU which is exclusively at your service.

A Raspberry Pi is fast most of the time, but it’s (occasionally) a bit occupied with itself…

Fast I/O on a Raspberry Pi

In Hardware, Software, Linux on Jun 15, 2013 at 00:01

To follow up on yesterday’s post about WiringPi and how it was used to display an image on a colour LCD screen attached to the RPi via its GPIO pins.

A quick calculation indicated that the WiringPi library is able to toggle the GPIO pins millions of times per second. Now this might not seem so special if you are used to an ATmega328 on a JeeNode or other Arduino-like board, but actually it is

Keep in mind that the RPi code is running under Linux, a multi-tasking 32-bit operating system which not only does all sorts of things at (nearly) the same time, but which also has very solid memory and application protection mechanisms in place:

  • each application runs in its own address space
  • the time each application runs is controlled by Linux, not the app itself
  • applications can crash, but they cannot cause the entire system to crash
  • dozens, hundreds, even thousands of applications can run next to each other
  • when memory is low, Linux can swap some parts to disk (not recommended with SD’s)

So while the LCD code works well, and much faster than one might expect for all the GPIO toggling it’s doing, this isn’t such a trivial outcome. The simplest way to deal with the RPi’s GPIO pins, is to manipulate the files and directories under the /sys/class/gpio/ “device tree”. This is incredibly useful because you can even manipulate it via plain shell script, using nothing but the “echo”, “cat”, and “ls” commands. Part of the convenience is the fact that these manipulations can take place entirely in ASCII, e.g. writing the string “1” or “0” to set/reset GPIO pins.

But the convenience of the /sys/class/gpio/ virtual file system access comes at a price: it’s not very fast. There is too much involved to deal with individual GPIO pins as files!

WiringPi uses a different approach, called “memory mapped files”.

Is it fast? You bet. I timed the processing time of this snippet of C code:

int i;
for (i = 0; i < 100000000; ++i) {
  digitalWrite(1, 1);
  digitalWrite(1, 0);
return 0;

Here’s the result:

    real    0m19.592s
    user    0m19.490s
    sys     0m0.030s

That’s over 5 million pulses (two toggles) per second.

This magic is possible because the I/O address space (which is normally completely inaccessible to user programs) has been mapped into a section of the user program address space. This means that there is no operating system call overhead involved in toggling I/O bits. The mapping is probably virtualised, i.e. the kernel will kick in on each access, but this is an interrupt straight into protected kernel code, so overhead is minimal.

How minimal? Well, it takes less than 90 ns per call to digitalWrite(), so even when running at the RPi’s maximum 1 GHz rate, that’s less than 90 machine cycles.

Note how the RPi can almost toggle an I/O pin as fast as an ATmega running at 16 MHz. But the devil is in the details: “can” is the keyword here. Being a multi-tasking operating system, there is no guarantee whatsoever that the GPIO pin will always toggle at this rate. You may see occasional hick-ups in the order of milliseconds, in fact.

Is the RPi fast? Definitely! Is it guaranteed to be fast at all times? Nope!

What it took to generate that LCD image

In Hardware, Software, Linux on Jun 14, 2013 at 00:01

As shown yesterday, it’s relatively easy to get a bitmap onto an LCD screen connected to a few I/O pins of the Raspberry Pi.

But there were some gotcha’s:

  • the image I wanted to use was 640×480, but the LCD is only 320×240 pixels
  • the WiringPi library has code to put a bitmap on the screen, but not JPEGs
  • this is easy to fix using the ImageMagick “convert” command
  • but the result has 24-bit colour depth, whereas the LCD wants 16-bit (5-6-5)

The “convert” command makes things very easy, but first I had to install ImageMagick:

    sudo apt-get install ImageMagick

Then we can run “convert” from the command-line (long live the Unix toolkit approach!):

    convert snap.jpg -geometry 320x240 -normalize snap.rgb

This handles both the rescaling and the transformation to a file with (R,G,B) types, each colour is one byte. As expected, the resulting image file is 320 x 240 x 3 = 230,400 bytes.

I didn’t see a quick way to convert the format to the desired 5-6-5-bit coding needed for the LCD, and since the code to write to the LCD is written in C anyway (to link to WiringPi), I ended up doing it all in a straightforward loop:

#define PIXELS (320*240)
uint8_t rgbIn24 [PIXELS][3];
unsigned rgbOut15 [PIXELS];


for (x = 0; x < PIXELS; ++x) {
  uint8_t r = rgbIn24[x][0] >> 3;
  uint8_t g = rgbIn24[x][1] >> 2;
  uint8_t b = rgbIn24[x][2] >> 3;
  rgbOut15[x] = (r << 11) | (g << 5) | b;

Note that this is a C program compiled and running under Linux on the RPi, and that it can therefore easily allocate some huge arrays. This isn’t some tiny embedded 8-bit µC with a few kilobytes – it’s a full-blown O/S running on a 32-bit CPU with virtual memory!

The other gotcha was that the bitmap to be supplied to the LCD library on top of WiringPi was expected to store each pixel in a 32-bit int. Total waste of space, and probably the result of a rushed port from the UTFT code (written to assume 16-bit ints) to the RPi. Which is why rgbOut15 is declared as “unsigned” (int, i.e. uint32_t) array, although the image really only needs an uint16_t. Oh well, what’s an extra 150 kB, eh?

Anyway, it works. Perfection can wait.

Note that this sends out all the data pixel by pixel to the display, and that each pixel takes the bit-banging of 2 bytes (the interface to the LCD is 8-bit wide), so each pixel takes at least 20 calls to digitalWrite(): 8 to set the data bits, and 2 two to toggle a clock pin – times two for the 2 bytes per pixel. That’s at least 4.6 million GPIO pin settings in half a second!

Tomorrow, I’ll show how WiringPi can make GPIO pins change in under 100 nanoseconds.

Getting an image on the LCD

In Hardware, Linux on Jun 13, 2013 at 00:01

To follow up on yesterday’s setup, I’ve managed to drive the display from Linux, using the WiringPi library by Gordon Henderson.

It supports driving the Raspberry Pi’s GPIO pins from C using Arduino- / Wiring-like calls to digitalWrite(). That, in combination with the UTFT library by Henning Karlsen, and some work done by iTead to combine everything was enough to get this self-portrait:


The display is a bit shiny, and the contrast and brightness are a bit low, unfortunately :(

Drawing the image takes about half a second, but considering that it’s all done through bit-banging of a few I/O pins from a “user space” program, that’s pretty good actually!

There are a couple of interesting (and sneaky) details, which I’ll go into in the next post.

Hurray for open source and the remixing / sharing it enables!

LCD screen for the RPi

In Hardware, Linux on Jun 12, 2013 at 00:01

For a little side-project I’m involved in (more on that in a future post), I wanted to try adding a little LCD screen with touch sensor to a Linux setup. Found this one at iTead:


There’s a little ribbon cable and adapter (just re-wiring) available for it, which allows plugging this thing into a Raspberry Pi. The nice thing about this screen, apart from being 320×240 pixels and 16-bit color TFT, is its physical size – just right for a stacked setup:


Stay tuned, as I explore how to hook this thing up and make it show stuff…

Accurate timing with jittery packets

In Hardware on Jun 11, 2013 at 00:01

A few days back I mentioned the SYNC message mechanism in CANopen, and how it can have about 150 µs jitter when sent out on a CAN bus running at 1 Mbit/s.

Here’s the basic idea of sending out SYNC packets at a pre-defined fixed rate:

JC's Grid, page 78

Once you have such a mechanism, you can do things like send out commands which all devices need to perform on the next sync for example. Nice for controlling the X, Y, and Z axis of of a CNC machine with the accuracy needed to make them all move exactly in sync.

The SYNC message has a very very high priority on the CAN bus, but even so, it can be delayed if it wants to send while some other transmission is taking place. So even though it will probably be first one to go out, it still has to wait for the bus to become available:

JC's Grid, page 78 copy

As a result, that SYNC pulse we wanted to arrive on time T ended up coming in at T + X. With X being under 200 µs when the bus is running at 1 MHz, but still.

Can we do better?

Yes, and no. We can’t avoid the jitter in that SYNC packet. But while the master sending out that SYNC packet cannot time it more precisely, it does know when the packet actually went out. So it does in fact know what T + X ended up being for that transmission.

And here’s the clever bit: the master can now send out a second (slightly lower priority) packet with a time stamp in it, i.e. containing that “T + X” value, down to the microsecond in fact. The arrival time of that time stamp packet is not so important. What matters is that each slave tracks exactly when the sync came in, and then uses the time stamp to adjust for any difference or drift that may be taking place.

This won’t help make the SYNC pulses come in more accurately, but it will let each slave know exactly what the time of that pulse was, and hence how accurately their own clocks are running. With this mechanism, all clocks can safely be assumed to run within 1..2 µs of each other, and therefore all future commands can now be tied to specific times which are known to be matched up between all slaves with two orders of magnitude better accuracy.

Ingenious, that CAN bus, eh?

Another new kid on the block

In Linux, ARM on Jun 10, 2013 at 00:01

Besides the Odroid U2 quad-core Linux board mentioned a few days ago, there’s another option in the same price range as the Raspberry Pi – the Beaglebone Black, a.k.a. “BBB”:

Screen Shot 2013-06-06 at 23.15.47

Yeah, ok, it’s slightly more expensive, but it comes with 2 GB of speedy eMMC memory built-in, whereas you need to add an SD card to make the RPi start up.

This is a bigger deal than you might think, because the BBB comes with Angström Linux pre-installed, and you can get going by simply plugging the BBB into a USB port:

  • the moment you do, a memory disk appears, and it gets auto-mounted
  • there’s a “start.htm” file you can double-click to launch your web browser
  • then just follow instructions and you’ll be up and running

This is very similar to the way the MBED ARM module works, and it shows that streamlining the first encounter really can help people to get started – fast!

From a general point of view, the Beaglebone black is actually quite different from the RPi, despite their obvious similarities of both being low-cost, low-power, yet fairly performant little Linux stand-alone boards with with keyboard-monitor-and-mouse capability and built-in Ethernet. Here’s where they differ, at least as I see it:

  • The RPi is aimed at the educational market, i.e. for learning about Linux and doing cool stuff with it once you’ve mastered that first step. Get a board, hook into a huge community, get your hands dirty, and start learning and hacking!

  • The BBB is aimed more at the embedded Linux market, i.e. for building all sorts of advanced projects with Linux and a powerful computing platform inside. Plug it in, put your Linux and physical computing skills to work, and make neat things happen!

The difference? The RPi is the most affordable solution out there, and does really well at being a complete Linux system with fairly good video and audio capabilities, but somewhat limited I/O expandability (even just thinking about how to hook up and size an expansion board on top is a puzzle). The BBB is packed with faster and more powerful hardware (but less focused on video and audio) and is loaded with expansion connectors and pins, ready to add a board, or even a whole stack of them. Even without boards on top (called capes), you have tons of I/O pins to play with – whether you need digital, analog, PWM, or all sorts of bussed interfaces, it’s often all there.

Some details about the BBB running at 300 MHz idle (it scales to 1000 MHz on demand):

# cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 2 (v7l)
BogoMIPS        : 297.40
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x3
CPU part        : 0xc08
CPU revision    : 2

Hardware        : Generic AM33XX (Flattened Device Tree)
Revision        : 0000
Serial          : 0000000000000000

For a nice 20-min story about a project where RPi’s and BBB’s were used, see this video.

There’s a project called BoneScript, which aims to make the Beaglebone’s I/O capabilities as easy to use from JavaScript (i.e. Node.js) as what the Arduino IDE did for C/C++ and the ATmega’s I/O pins. Sounds like an excellent step in the right direction to me.

Either way, I see a bright future ahead for Node.js running on these little boards!

What’s inside that chip?

In AVR, Hardware on Jun 9, 2013 at 00:01

Came across an interesting post the other day, from a Russian site which looks “under the hood” of some sophisticated chips – some well-known in the West, some less so.

First, the technique of using sulfuric and nitric acid is described and illustrated:


But the really bit interesting bit is what comes out. Here’s the ATmega8:


(both pictures were copied from that site, i.e. – I hope that’s ok)

The size of this chip is 2855 x 2795 µm, i.e less than 3 x 3 mm.

Fascinating! I wonder what all the different bits and pieces are…

Trying out CANopen

In Hardware, ARM on Jun 8, 2013 at 00:01

Time for some electronics doodling… here are two LPCxpresso boards, with LPC11C24 ARM chips on each. One of them has been cut in half and hacked, as you can see:


The reason for using this chip is that they have CANopen drivers built into them, in ROM. That makes it a lot easier to get started, with so many things being unclear anyway – from the compiler toolchain (I ended up with a mix of ARM’s Linaro toolchain and NXP’s LPCxpresso IDE), to figuring out how to get a basic CAN packet on the bus and off again.

Anyway, couple of days of hardware + firmware tinkering, and here’s the result:


I’m getting a lot of mileage out of the Hameg HMO2024 scope as you can see, including its hardware CAN bus decoding (a paid option, but free at the time I got the intro bundle).

You’re looking at 4 packets, i.e. 2 request/response pairs going over the bus.

Note how the voltages are jumping up and down a bit. For CAN, this is irrelevant, because it works based on the difference between the CAN-L and CAN-H levels. It’s caused by the master node running its CAN interface at a slightly lower voltage (USB + diode drop). Then again, it’s rather useful for debugging because you can see who sends what! As is clearly visible above, the ACK bit comes from the other node in each exchange.

In terms of speed, there’s one packet every 145 µs, which translates to 55 Kbyte/sec for the 8-byte payload, i.e. 440 Kbit/s out of 1000 Kbit/s actual throughput (more, since there is also an address, the ACK bit, etc). It might not seem like much, but keep in mind that this is also the worst-case timing. The CAN bus does not start thrashing or losing packets when pushed to its limit – all that can happen on a maximally loaded-bus is that the lowest-priority messages will hardly ever get a chance to pass through.

On the software side, a CAN bus is really quite simple to use, since the built-in hardware does all the hard work of acquiring and releasing the bus, resending on packet damage (a noise spike, for example), adding and checking CRCs, buffering a few incoming packets, and even filtered reception – so that only specific ranges of message IDs get picked up.

Not much of this is available for Atmel chips, alas. And that ARM world sure is a lot less physical computing friendly w.r.t. open-source and platform-agnostic toolchains.

All those ARM Cortex chip manufacturers still seem to be living in the dark ages…

CANopen sits on top of CAN

In Hardware on Jun 7, 2013 at 00:01

The CANopen protocol sits on top of the low-level CAN bus protocol. Due to the way CAN bus packets are tagged with a message ID, but not specifically routed from or to a specific device (i.e. node), and due to the maximum packet payload size of 8 bytes, you really need something like CANopen to make it usable. For reference, here’s a 2-node CAN bus setup:


The interesting bit about CANopen, is that it’s all about data acquisition and process control (that’s what you do when you put this stuff in a car, after all). And what intrigues me about that, is the similarity with monitoring and controlling stuff around the house.

I’ve been reading myself silly lately, to wrap my head around all the CANopen terminology and to try and distill the essence of… ehm, CANopen, life, and a few other things :)

But instead of just describing CANopen as it is, and pouring lots of its concepts and new terminology on you, let me try to describe this from an outsider’s perspective:

  • CANopen works in terms of accessing and saving objects. It’s not necessarily 100% centralised, but that’s how it often gets used: a central CANopen node to control, inquire, and manage a bunch of CANopen slave nodes, all on the same CAN bus.

  • There are a lot of conventions w.r.t. these objects, because the goal of CANopen is to allow devices from different manufacturers to inter-operate (that’s probably why it’s called “open”). So there’s a standard way to find out which devices are present and what their basic capabilties are, to enable/disable each one of them, to configure their communication settings, to start and stop data acquisition streams – all with lots of room for manufacturer- and device-specific details to be managed as well.

  • The central concept here is the “Object Dictionary” in each device, which is indexed by a 16-bit number, plus a numeric 8-bit subindex for entries which are more structured. Objects can be 1..4 bytes, in which case they can be exchanged in one packet, or larger, in which case CANopen takes care of all the packing and unpacking.

  • There is a simple yet effective heartbeat mechanism, so that each device (and of course also the master) can figure out which devices are currently present, which are active, and – with timeouts – which devices seem to have dropped out for some reason.

  • Each device announces itself when powering up, so auto-discovery is possible, even for devices which the master has never seen before – although it’ll be more effective when the master has a detailed description of each type of device, so that it can optimally manage and control it.

  • There are mechanisms for urgent error reporting, dropping out gracefully, and throttling to avoid bus congestion.

  • Everything is prioritised (the CAN bus does that), with top priority (msg ID 000) given to essential (but infrequent) master messages, such as starting and stopping device activity on the bus. Note that, due to the top priority, these will always get through.

  • Next in priority is a time synchronisation message, which can be used to make all device clocks “sing in sync”. With a 1 Mbit/s bus rate, these sync messages will never have more than about 150 µs jitter (and there’s a clever optional scheme to increase the time resolution in all slaves to 1 µs).

There is much more which I can’t all describe in this post, but the one design idea I’d like to highlight is the distinction between managing a large set of objects (via SDO’s, i.e. Service Data Objects) and maximally efficient data streams (via PDO’s, i.e. Process Data Objects). Where PDO’s can be set up to be sent across in a regular time-synchronised manner or in a more ad-hoc / event-based / asynchronous fashion.

I really like this distinction: there is a large heterogenous mix of data objects, which can be accessed and changed via individual SDO requests, and there are streams of PDO data which don’t waste time and bandwidth on describing themselves – just the raw bytes. The heterogenous data objects can fully describe those streams, their exact byte (even bit) structure, the conditions under which data gets sent, their frequency, and their priority.

Lots of interesting connections with the way the RF12 driver sends packets, the announcer packet idea, and the way HouseMon distinguishes between incoming data and the meta-data inside each driver describing these incoming packets.

A thousand days!

In AVR, Hardware on Jun 6, 2013 at 00:01

It’s getting a bit repetitive, but I just noticed this:

Screen Shot 2013-06-05 at 17.27.15

That’s over 1000 days of sending out one packet a minute on this thing:


… and over 100 days on the recent JeeNode Micro v3:


The battery boost version runs off a single Eneloop AA battery, and is doing fine too:



What if you’re out of wireless range?

In Hardware on Jun 5, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

Ok, so you’ve got some JeeNodes up and running, all talking to each other or to a central node via the wireless RFM12B module. Or… maybe not: the signal is too weak! Now what?

There are several approaches you can try to improve wireless range:

  • optimise your existing antenna(s)
  • lower the data rate and reduce the bandwidth
  • use a more advanced type of antenna
  • use a directional antenna
  • install a repeater of some kind

Let’s go through each of these in turn.

First thing to try is to optimise the little wire “whip” antenna’s that come standard with a JeeNode. Make sure the antenna wire is 82 mm long (that’s for 868 MHz), is sticking up (or sideways) perpendicular to the board, and check that both antenna’s are pointing more or less in the same direction (but not in the direction of the other node: the RF field is circular around the wire, not on top or below).

One thing to keep in mind with these weak signals, is that salty bags of water (us people, that is) tend to absorb RF energy, so these radios work better with us out of the way. Be sure to take a step back while tweaking and hunting for the best orientation!

If that doesn’t help enough, you can do one more thing without messing with electronics or hardware: reduce the datarate of the transmitter and receiver (they have to match). See the RFM12B Command Calculator for settings you can change. To reduce the data rate by two thirds, call rf12_control(0xC614) after the call to rf12_initialize(), for example. The bad news is that you have to do this in all the nodes which communicate with each other – all the data rates have to match!

This in itself won’t extend the range by that much, but with lower data rates you can also reduce the bandwidth in the receiver (with rf12_control(0x94C2)). You can think of this approach as: speaking more slowly and listening more closely. The effects should be quite noticeable. Radio amateurs have been using this technique to get halfway around the world on mere milliwats, using a system called QRSS.

If that doesn’t give you the desired range – here are a few more tricks, but they all require extra hardware: improve the antenna, use “directional” antennas, or use a repeater.

Here’s an example of an improved omni-directional antenna design, as seen on eBay:


And here’s a directional “Yagi” antenna, which needs to be aimed fairly accurately:


I haven’t tried either of these (you can build them yourself), but the omni-directional one was mentioned and described in Frank Benschop’s presentation on JeeDay. He reported getting quite good results, once all the antenna + cabling quirks were resolved.

If neither of these are an option, then the last trick you can try is to add a relay / repeater node to your network, as described in this weblog post some time ago. This will double the range if you place that node in the middle of the two nodes which can’t reach each other, but it adds some complexity to the packet addressing mechanism.

Move over, raspberry

In Hardware, Linux on Jun 4, 2013 at 00:01

The Raspberry_Pi is a great little board at an amazing little price. But as mentioned yesterday, it’s not very fast as a server. I suspect that a lot of it has to do with the SD card interface and the SD card itself, so I started looking around for alternatives.

And boy, there sure are lots of ’em – still well below €100. I’ll single out one system, the Odroid U2 – knowing full well that there must be over a dozen others out there:


It’s smaller than a Raspberry Pi, but it comes mounted on a “big” (6x6x6 cm) heat sink.

The specs are pretty impressive:

Screen Shot 2013-06-01 at 11.45.13

And the board is neat – the result of a huge mobile phone market driving size down:



Could this be used as central web / file / home automation server?

Speedy Raspberry

In Hardware, Linux on Jun 3, 2013 at 00:01

The Raspberry Pi is an amazing little board, with an amazing amount of power and functionality at an incredible price – it’s probably fair to say that our technology geek’s world will never be the same again, now that we have the RPi.

But it’s no speed monster…

Sometimes during development, it really gets in the way – especially when you’re used to working on a modern fast laptop, which is some 20 times faster (and that’s per-core).

So I decided to overclock one of my RPi’s, and see what it does. Overclocking in Raspbian is trivial with the standard raspi-config utility:

Screen Shot 2013-05-28 at 16.52.58

Just go to the Overclock section, and pick one:

Screen Shot 2013-05-28 at 16.46.48

I went for the fastest there is, but decided to run a 2-hour test to make sure it really works. Turns out that someone has already done all the work, and even created such a stress test:

root@raspberrypi:~# ./ 
Testing overclock stability...
reading: 1
945+1 records in
945+1 records out
3965190144 bytes (4.0 GB) copied, 352.455 s, 11.3 MB/s
reading: 2
reading: 10
945+1 records in
945+1 records out
3965190144 bytes (4.0 GB) copied, 358.522 s, 11.1 MB/s
writing: 1
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 95.2065 s, 5.6 MB/s
writing: 2
writing: 10
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 83.4848 s, 6.4 MB/s
./ line 18:  2414 Terminated
              nice yes > /dev/null
CPU freq: 1000000
CPU temp: 58376
[    5.217175] Registered led device: led0
[    8.943589] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[    9.418819] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   19.181046] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
[   19.431053] bcm2835-cpufreq: switching to governor ondemand
[   19.431082] bcm2835-cpufreq: switching to governor ondemand
[   21.736829] Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SS
[   22.134893] ip_tables: (C) 2000-2006 Netfilter Core Team
[   22.174573] nf_conntrack version 0.5.0 (7774 buckets, 31096 max)
[  750.951153] smsc95xx 1-1.1:1.0: eth0: kevent 2 may have been dropped
Not crashed yet, probably stable.

Ya gotta love that last message, nicely semi re-assuring ;)

I did fit a set of cooling fins from Watterott (some nice IR heat pictures there):


As checked with vcgencmd measure_temp, the temperature never rose above 58.4°C.


Solar PV production

In Hardware on Jun 2, 2013 at 00:01

This is the sort of graph I’ve been waiting for – now that there is more sun, at last:

Screen Shot 2013-05-29 at 00.28.30

(the steep increases and decreases are probably due to shadows from the other roofs)

All values were obtained from the SB5000TL inverter as 5-minute readouts over Bluetooth:

  • total delivered power on AC mains – 29 kWh for the entire day
  • total incoming power on the east-facing roof (12 panels)
  • total incoming power on the west-facing roof (10 panels)

The east-facing panels are actually slight to the north, and the west-facing ones slightly to the south. Which might explain the similar peak production levels despite the different number of panels, i.e. different “insolation” angles.

The peak values are not the highest I’ve seen so far. A cold day with light clouds recently pushed the maximum to over 4700 watts. But that was a rare event. These values are fine with me – I find such levels of solar PV output in a location well up north on this planet (52°N, to be precise) pretty amazing, in fact.

It takes two to CAN

In Hardware, ARM on Jun 1, 2013 at 00:01

The CAN bus described yesterday keeps drawing my interest. It feels right on many levels…

Anyway, for a real CAN bus test, I’ve set up two nodes, to be able to see a full handshake:


That’s one node sending out a 0x101 packet, and another node responding to it with a 0x202 packet with the same payload. There are lots of interesting things going on here:

  • The CAN protocol requires any receiving node to send out an ACK bit the moment it has received a valid packet – this doesn’t tell the sender who got it – only that some node was listening and got it. The only thing you can conclude at this stage really, is that there is actual communication – it’s up to higher level layers to verify things.

  • The sender is a node connected to a 3.3V-based CAN transceiver, so the voltages it puts out on the bus are lower than the receiving / echoing node (which is an ARM-based LPCxpresso 11C24 board with a 5V transceiver, running a standard CAN example in the LPCxpresso IDE). As you can see, voltages are hopping a bit, but all that matters is the voltage difference between the two lines, so everything is fine.

  • There is no one listening to the reply, so the LPC11C24 doesn’t get a good ACK on that, and keeps re-sending the packet. It does this a number of times and will then give up.

  • The Hameg HMO2024 scope with its added serial decoding options is very effective in this setup – I made it trigger on a 0x202 packet, and it shows a nice list of the data it has captured. Since I’m using the analog channels, the levels are also clearly visible.

  • The bus is running at 500 KHz, so the basic bit timing is in multiples of 2 µs. Zooming in, I saw that the gap between each packet is 22 µs, which matches exactly the CAN bus specs requiring nodes to wait for at least 11 idle bits before they start. Speaking of which: the CAN names for “0” and 1″ are “dominant” and “recessive”, respectively – because a dominant bit always wins and forces the bus to a “0” (voltage applied).

  • The packet times here are ≈ 250 µs, so there’s some 10% of overhead for the gaps between packets, and the total “overhead” for getting 8 bytes of payload across (i.e. 128 µs) is about 100% – but that includes things like bit-stuffing, a message ID, a CRC check, and an ACK. Half the theoretical bit rate to be able to send real-time data with little latency, good reliability, and precisely tagged is actually very good.

Note that the CAN bus requires special transceiver chips. The level conversions are minimal (from “0” and “1” to pulling the two lines apart, in terms of voltage), but what they also add is a substantial amount of protection against common-mode signals, which is when both signals move up or down by the same voltage. This is extremely important in a car, where large currents can lead to huge spikes and voltage changes, but it really doesn’t hurt in other applications either: the CAN bus is a very nice way to tie pieces together – and when there is an issue with ground levels, the whole design even allows for opto-couplers to be used for complete isolation between nodes.

The “standard” way to hook up CAN nodes is through three pins (CAN-H, CAN-L, and GND), but it’s not uncommon to include a +5V supply pin, making this a neat 4-wire bus.

Yes, we CAN bus

In Hardware on May 31, 2013 at 00:01

The CAN Bus is a very interesting wired bus design, coming from the automobile industry (and probably built into every European car made today). It’s a bus with an ingenious design, avoiding bus collisions and supporting a good level of real-time responsiveness.

I’ve been intrigued by this for quite some time, and decided to dive in a bit.

There are several interesting design choices in CAN bus:

  • it’s all low-voltage, just 0..5V (even 0..3.3V) is all it takes on each connected node
  • the bus is linear, reaching from 40 m @ 1 Mbit/s to 500 m @ 125 kbit/s, or even longer
  • signalling is based on voltage between two wires, and terminated by 120 Ω on each end
  • signals are self-clocked, with bit-stuffing to insert bit-transitions when needed

But the three most surprising aspects of the CAN bus design are probably the following:

  • the design is such that collisions cannot happen: one of the two senders always wins
  • each CAN bus packet can have at most 8 bytes of data (and is CRC-checked)
  • as described recently, messages have no destination, but only a message ID (type)

What’s also interesting is that – like I2C – this protocol tends to be fully implemented in hardware, and is included in all sorts of (usually ARM-based) microcontrollers. So unlike UARTs, RS485, I2C, and SPI, you simply get complete and valid packets in and out of the peripheral. No need to deal with framing, CRC checking, or timing decisions.

You can almost feel the car-like real-time nature of these design trade-offs:

  • short packets – always! – so the bus is released very quickly, and very often
  • no collisions, i.e. no degradation in bus use and wasted retransmits as it gets busier
  • built-in prioritisation, so specific streams can be sent across with controlled latencies
  • with a 16-bit CRC on each 0..8 byte packet, chances of an undetected error are slim

Since my scope includes hardware CAN bus decoding, I decided to try it out:


The message has an ID of 0x101 (message ID’s are either 11 or 29 bits), eight bytes of data (0xAA55AA55FF00FF00), and a CRC checksum 0x1E32. I’m using a 500 KHz bit clock.

If you look closely, you can see that there are never more than 5 identical bits in a row. That’s what bit-stuffing does: insert an opposite bit to avoid longer stretches of identical bits, as this greatly helps deduce exact timings from an incoming bit-stream.

It seems crazy to limit packets to just 8 bytes – what could possibly be done with that, without wasting it all on counters and offsets to send perhaps 4 bytes of real data in each packet? As it turns out, it really isn’t so limiting – it just takes a somewhat different mindset. And the big gain is that multiple information streams end up getting interleaved very naturally. As long as each of them is reasonable, that is: don’t expect to get more than 2 or 3 data streams across a 1 Mb/s bus, each perhaps no more than 100 kb/s. Then again, you can expect these to arrive within a very consistent and predictable time, regardless of what other lower-priority burst traffic is going on.

Neat stuff…

Energy Hack June 15 in Berlin

In News on May 30, 2013 at 00:01

For the German readers… got this announcement in which might be of interest to you:

[…] It will be a day of hacking the open energy data by Berlin’s main energy provider and exploring hardware hacks to manage energy consumption in one’s own household. You can find all the necessary info on

The day will be mostly in German, but since the topic is global and all Germans speak English, we hope to make everyone feel at home.


Some more details, now “auf Deutsch”:

Am Samstag, 15. Juni, veranstaltet die Open Knowledge Foundation Deutschland e.V. in Berlin einen Hackday zum Thema „Energie der Zukunft“, zu dem wir euch alle herzlich einladen.

Ziel des Energie-Hackdays ist es, die abstrakten Themen Stromversorgung und -verbrauch besser greifbar zu machen und Verbraucher zur effizienteren Nutzung von Energie anzuregen. Gemeinsam mit Programmiererinnen, Softwareentwicklern und Open Data-Enthusiasten wollen wir mit den Daten des Berliner Stromnetzes experimentieren und Anwendungen verschiedenster Art bauen. Besonders freuen wir uns über Hardwarehacks, die zum Stromsparen anregen oder sich auf andere Weise mit dem Thema Energie auseinander setzen.

Zur Verfügung stehen unter anderem Echtzeitdaten zur Last und Erzeugung in Berlin sowie durchschnittliche Verbrauchsdaten von Haushalten, die ihr als Open Data in den Datenkatalogen des Netzbetreibers und der Stadt Berlin findet. Außerdem haben wir elektronische Zähler zum „Selbstablesen“. Falls ihr Anregungen oder Ideen für weitere interessante Datensätze habt oder Material zum Hardware-Basteln braucht, das ihr nicht selbst mitbringen könnt, kontaktiert uns!

I won’t have time for it, alas, but if you’re interested, go to

What if you’re lost on this site?

In Musings on May 29, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

With over 1300 posts on this weblog, it’s easy to get lost. Maybe you stumbled onto one of the posts after a web search, and then kept reading. Some people told me they just started reading it all from end to finish (gulp!).

It’s not always easy to follow the brain dump of a quirky Franco-Dutch maverick :)

Let me start by listing the resources related to all this JeeStuff:

  • This daily weblog is my train-of-thought, year-in, year-out. Some projects get started and never end (low-power tweaking?), others get started and never get finished (sad, but true), yet others are me going off on some tangent, and finally there are the occasional series and mini-series – diving a bit deeper into some topic, or trying to explain something (electronics, usually).
  • There’s a chronological index, which I update from time to time using a little script. It lists just the titles and the tags. It’s a quick way to see what sort of topics get covered.
  • Most post are tagged, see the “tag cloud” at the bottom of each page. Clicking on a term leads to the corresponding posts, as one (large) page. This is probably a good way to read about certain topics when you come to this web site for the first time.
  • At the bottom of each weblog page is a list of posts, grouped by month. Frankly, I don’t think it’s that useful – it’s mostly there because WordPress makes it easy to add.
  • And there’s the search field, again at the bottom of each page. It works quite well, but if your search term is too vague, you’ll get a page with a huge list of weblog posts.

Apart from this weblog, which is at, there is also the community site at, and the shop at It’s a bit unfortunate that they all look different, and that they all use different software packages, but that’s the way it is.

The community site contains a number of areas:

  • The café is a publik wiki, with reference materials, projects, and pointers to other related pages and sites. Note that although it’s a wiki, it is not open for editing without signing up – that’s just to keep out spammers. Everyone is welcome, cordially invited even, to participate.

  • The software I write – with the help and contributions of others – ends up on GitHub so that anyone can browse the code, see what is being added / changed / fixed over time, and also create a fork to hack on. My code tends to be MIT-licensed wherever possible, so it’s all yours to look at, learn from, re-use, whatever.

  • There is documentation at for several of the more important packages and libraries on GitHub. Updating is a manual step here, so it can lag, occasionally. These pages are generated by Doxygen.

  • The hardware area lists all the products which have escaped from JeeLabs, and are ending up all over the world. It’s a reference area, which should be the final word on what each product is and isn’t.

  • There are several forums for discussion, making suggestions, asking questions, and posting notes about anything somehow related (or at least relevant) to JeeLabs.

  • For real-time discussion, there’s a #jeelabs IRC channel, though I rarely leave my IRC client running very long. Doesn’t seem to be used much, but it’s there to be used.

If you’re new to electronics, you could go through the series called Easy electrons. For a write-up about setting up a sensor network at home, see the Dive Into JeeNodes series.

What else? Let me know, please. I find it very hard to get in the mindset of someone reaching this site for the first time. If you are lost, chances are that others will be too – so any tips and suggestions on how to improve this site for new visitors would be a big help.

You can always reach me via the info listed on the “About” page.

Idling in low-power mode

In AVR, Software on May 28, 2013 at 00:01

With a real-time operating system, going into low-power mode is easy. Continuing the recent ChibiOS example, here is a powerUse.ino sketch which illustrates the mechanism:

#include <ChibiOS_AVR.h>
#include <JeeLib.h>

const bool LOWPOWER = true; // set to true to enable low-power sleeping

// must be defined in case we're using the watchdog for low-power waiting
ISR(WDT_vect) { Sleepy::watchdogEvent(); }

static WORKING_AREA(waThread1, 50);

void Thread1 () {
  while (true)

void setup () {
  rf12_initialize(1, RF12_868MHZ);


void mainThread () {
  chThdCreateStatic(waThread1, sizeof (waThread1),
                    NORMALPRIO + 2, (tfunc_t) Thread1, 0);

  while (true)

void loop () {
    Sleepy::loseSomeTime(16); // minimum watchdog granularity is 16 ms

There’s a separate thread which runs at slightly higher priority than the main thread (NORMALPRIO + 2), but is idle most of the time, and there’s the main thread, which in this case takes the role of the idling thread.

When LOWPOWER is set to false, this sketch runs at full power all the time, drawing about 9 mA. With LOWPOWER set to true, the power consumption drops dramatically, with just an occasional short blip – as seen in this current-consumption scope capture:


Once every 16..17 ms, the watchdog wakes the ATmega out of its power-down mode, and a brief amount of activity takes place. As you can see, most of these “blips” take just 18 µs, with a few excursions to 24 and 30 µs. I’ve left the setup running for over 15 minutes with the scope background persistence turned on, and there are no other glitches – ever. Those 6 µs extensions are probably the milliseconds clock timer.

For real-world uses, the idea is that you put all your own code in threads, such as Thread1() above, and call chThdSleepMilliseconds() to wait and re-schedule as needed. There can be a number of these threads, each with their own timing. The lowest-priority thread (the main thread in the example above) then goes into a low-power sleep mode – briefly and repeatedly, thus “soaking” up all unused µC processor cycles in the most energy-efficient manner, yet able to re-activate pending threads quickly.

What I don’t quite understand yet in the above scope capture is the repetition frequency of these pulses. Many pulses are 17 µs apart, i.e. the time Sleepy::loseSomeTime() goes to sleep, but there are also more frequent pulses, spread only 4..9 ms apart at times. I can only guess that this has something to do with the ChibiOS scheduler. That’s the thing with an RTOS: reasoning about the repetitive behavior of such code becomes a lot trickier.

Still… not bad: just a little code on idle and we get low-power behaviour almost for free!

Wireless, the CAN bus, and enzymes

In Musings on May 27, 2013 at 00:01

How’s that for a title to get your attention, eh?

There’s an interesting mechanism in communication, which has kept me intrigued for quite some time now:

  • with JeeNode and RF12-based sensors, wireless packets are often broadcast with only a sender node ID (most network protocols use both a source and a destination)
  • CAN bus is a wired bus protocol from the car industry, its messages do not contain a destination address, there is just a 11- or 29-bit “message ID”

What both these systems do (most of the time, but not exclusively), is to tag transmitted packets with where they came from (or what their “meaning” is) and then just send this out to whoever happens to be interested. No acknowledgements: in the case of wireless, some messages might get lost – with CAN bus, the reliability is considerably higher.

It’s a bit like hormones and other chemicals in our blood stream, added for a specific purpose, but not really addressed to an area in the body. That’s up to various enzymes and other receptors to pick up (I know next to nothing about biology, pardon my ignorance).

Couple of points to note about this:

  • Communicating 1-to-N (i.e. broadcasting) is just as easy as communicating 1-to-1, in fact there is no such thing as privacy in this context – anyone / anything can listen-in on any conversation. The senders won’t know.
  • There is no guaranteed delivery, since the intended targets may not even be around or listening. The best you can do, is look for the effects of the communication, which could be an echo from the receiving end, or some observable side-effect.
  • You can still set up focused interactions, by agreeing on a code / channel to use for a specific purpose: A can say “let’s discuss X”, and B can say “I’ll be listening to topic X on channel C”. Then both A and B could agree to tag all their messages with “C”, and they’ll be off on their own (public) discussion.
  • This mode of communicating via “channels” or “topics” is quite common, once you start looking for it. The MQTT messaging system uses “channels” to support generic data exchange. Or take the human-centric IRC, for example. Or UDP’s multicast.
  • Note that everything which has to do with discovery on a network also must rely on such a “sender-id-centric” approach, since by definition it will be about finding a path to some sender which doesn’t know about us.

Having no one-to-one communication might seem limiting, but it’s not. First of all, the nature of both wireless and busses is such that everything reaches everyone anyway. It’s more about filtering out what we’re not interested in. The transmissions are the same, it’s just the receivers which apply different filtering rules.

But perhaps far more importantly, is that this intrinsic broadcasting behaviour leads to a different way of designing systems. I can add a new wireless sensor node to my setup without having to decide what to do with the measurements yet. Also, I will often set up a second listen-only node for testing, and it just picks up all the packets without affecting my “production” setup. For tests which might interfere, I pick a different net group, since the RF12 driver (and the RFM12B hardware itself) has implicit “origin-id-filtering” built in. When initialised for a certain net group, all other packets automatically get ignored.

Even N-to-1 communication is possible by having multiple nodes send out messages with the same ID (and their distinguishing details elsewhere in the payload). This is not allowed on the CAN bus, btw – there, each sender has to stick to unique IDs.

The approach changes from “hey YOU, let me tell you THIS”, to “I am saying THIS”. If no one is listening, then so be it. If we need to make sure it was received, we could extend the conventions so that B nods by saying “got THIS” and then we just wait for that message (with timeouts and retries, it’s very similar to a traditional ACK mechanism).

It’s a flexible and natural model – normal speech works the same, if you think about it…

PS. The reason this is coming up, is that I’m looking for a robust way way to implement JeeBoot auto-discovery.

90 days on a coin cell

In Hardware on May 26, 2013 at 00:01

Just saw that my JeeNode Micro test setup has been running and “blipping” for 90 days:

Screen Shot 2013-05-25 at 11.08.00

The voltage is starting to drop a bit, and the voltage drop before and after using the radio has increased from 0.08 to 0.16 V (reported with a granularity of 0.02 V), but everything seems to be fine. It has pumped out over 120,000 packets so far.

The other test is a JeeNode Micro with boost regulator, running of one Eneloop AA battery. That battery voltage has also dropped a bit, but as you can see, the boost regulator is doing its thing and still providing the exact same 3.04 V as it did when the test was started.

It’ll be interesting to see how long each of these setups holds out. I have no idea, really. It’s not just a matter of capacity – with the coin cell, it’ll also depend on how long the battery can continue to provide these brief 20..20 mA power bursts for each transmission.


ChibiOS for the Arduino IDE

In AVR, Software on May 25, 2013 at 00:01

A real-time operating system is a fairly tricky piece of software, even with a small RTOS – because of the way it messes with several low-level details of the running code, such as stacks and interrupts. It’s therefore no small feat when everything can be done as a standard add-on library for the Arduino IDE.

But that’s exactly what has been done by Bill Greiman with ChibiOS, in the form of a library called “ChibiOS_AVR” (there’s also an ARM version for the Due & Teensy).

So let’s continue where I left off yesterday and install this thing for use with JeeNodes, eh?

  • download a copy of from this page on Google Code
  • unpack it and inside you’ll find a folder called ChibiOS_AVR
  • move it inside the libraries folder in your IDE sketches folder (next to JeeLib, etc)
  • you might also want to move ChibiOS_ARM and SdFat next to it, for use later
  • other things in that ZIP file are a README file and the HTML documentation
  • that’s it, now re-launch the Arduino IDE to make it recognise the new libraries

That’s really all there is to it. The ChibiOS_AVR folder also contains a dozen examples, each of which is worth looking into and trying out. Keep in mind that there is no LED on a standard JeeNode, and that the blue LED on the JeeNode SMD and JeeNode USB is on pin 9 and has a reverse polarity (“0” will turn it on, “1” will turn it off).

Note: I’m using this with Arduino IDE 1.5.2, but it should also work with IDE 1.0.x

Simple things are still relatively simple with a RTOS, but be prepared to face a whole slew of new concepts and techniques when you really start to dive in. Lots of ways to make tasks and interrupts work together – mutexes, semaphores, events, queues, mailboxes…

Luckily, ChibiOS comes with a lot of documentation, including some general guides and how-to’s. The AVR-specific documentation can be found here (as well as in that ZIP file you just downloaded).

Not sure this is the best place for it, but I’ve put yesterday’s example in JeeLib for now.

I’d like to go into RTOS’s and ChibiOS some more in the weeks ahead, if only to see how wireless communication and low-power sleep modes can be fitted in there.

Just one statistic for now: the context switch latency of ChibiOS on an ATmega328 @ 16 MHz appears to be around 15 µs. Or to put it differently: you can switch between multiple tasks over sixty thousand times a second. Gulp.

Blinking in real-time

In AVR, Software on May 24, 2013 at 00:01

As promised yesterday, here’s an example sketch which uses the ChibiOS RTOS to create a separate task for keeping an LED blinking at 2 Hz, no matter what else the code is doing:

#include <ChibiOS_AVR.h>

static WORKING_AREA(waThread1, 50);

void Thread1 () {
  const uint8_t LED_PIN = 9;
  pinMode(LED_PIN, OUTPUT);
  while (1) {
    digitalWrite(LED_PIN, LOW);
    digitalWrite(LED_PIN, HIGH);

void setup () {

void mainThread () {
  chThdCreateStatic(waThread1, sizeof (waThread1),
                    NORMALPRIO + 2, (tfunc_t) Thread1, 0);
  while (true)

void loop () {

There are several things to note about this approach:

  • there’s now a “Thread1” task, which does all the LED blinking, even the LED pin setup
  • each task needs a working area for its stack, this will consume a bit of memory
  • calls to delay() are forbidden inside threads, they need to play nice and go to sleep
  • only a few changes are needed, compared to the original setup() and loop() code
  • chBegin() is what starts the RTOS going, and mainThread() takes over control
  • to keep things similar to what Arduino does, I decided to call loop() when idling

Note that inside loop() there is a call to delay(), but that’s ok: at some point, the RTOS runs out of other things to do, so we might as well make the main thread similar to what the Arduino does. There is also an idle task – it runs (but does nothing) whenever no other tasks are asking for processor time.

Note that despite the delay call, the LED still blinks in the proper rate. You’re looking at a real multitasking “kernel” running inside the ATmega328 here, and it’s preemptive, which simply means that the RTOS can (and will) decide to break off any current activity, if there is something more important that needs to be done first. This includes suddenly disrupting that delay() call, and letting Thread1 run to keep the LEDs blinking.

In case you’re wondering: this compiles to 3,120 bytes of code – ChibiOS is really tiny.

Stay tuned for details on how to get this working in your projects… it’s very easy!

It’s time for real-time

In Software on May 23, 2013 at 00:01

For some time, I’ve been doodling around with various open-source Real-time operating system (RTOS) options out there. There are quite a few out there to get lost in…

But first, what is an RTOS, and why would you want one?

The RTOS is code which can manage multiple tasks in a computer. You can see what it does by considering what sort of code you’d write if you wanted to periodically read out some sensors, not necessarily all at the same time or equally often. Then, perhaps you want to respond to external events such as a button press of a PIR sensor firing, and let’s also try and report this on the serial port and throw in a command-line configuration interface on that same serial port…

Oh, and in between, let’s go into a low-power mode to save energy.

Such code can be written without RTOS, in fact that’s what I did with a (simpler) example for the roomNode sketch. But it gets tricky, and everything can become a huge tangle of variables, loops, conditions, and before you know it … you end up with spaghetti!

In short, the problem is blocking code – when you write something like this, for example:

void setup () {}

void loop () {
  digitalWrite(LED, HIGH);
  digitalWrite(LED, LOW);

The delay() calls will put the processor into a busy loop for as long as needed to make the requested number of milliseconds pass. And while this is the case, nothing else can be done by the processor, other than handling hardware interrupts (such as timer ticks).

What if you wanted to respond to button presses? Or make a second LED blink at a different rate at the same time? Or respond to commands on the serial port?

This is why I added a MilliTimer class to JeeLib early on. Let’s rewrite the code:

MilliTimer ledTimer;
bool ledOn;;

void setup () {
  ledTimer.set(1); // start the timer

void loop () {
  if (ledTimer.poll()) {
    if (ledOn) {
      digitalWrite(LED, LOW);
    } else {
      digitalWrite(LED, HIGH);
    ledOn = ! ledOn;
  // anything ...

It’s a bit more code, but the point is that this implementation is no longer blocking: instead of stopping on a delay() call, we now track the progress of time through the MilliTimer, we keep track of the LED state, and we adjust the time to wait for the next change.

As a result, the comment line at the end gets “executed” all the time, and this is where we can now perform other tasks while the LED is blinking in the background, so to speak.

You can get a lot done this way, but things do tend to become more complicated. The simple flow of each separate activity starts to become a mix of convoluted flows.

With a RTOS, you can create several tasks which appear to all run in parallel. You don’t call delay(), but you tell the RTOS to suspend your task for a certain amount of time (or until a certain event happens, which is the real magic sauce of RTOS’es, actually).

So in pseudo code, we can now write our app as:

  TASK 1:
    turn LED on
    wait 100 ms
    turn LED off
    wait 400 ms

    start task 1
    do other stuff (including starting more tasks)

All the logic related to making the LED blink has been moved “out of the way”.

Tomorrow I’ll expand on this, using an RTOS which works fine in the Arduino IDE.

What if the sun doesn’t shine?

In Musings on May 22, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

Slightly different question this time – not so much about investigating, but about coming up with some ideas. Because, now that solar energy is being collected here at JeeLabs and winter is over, there’s a fairly obvious pattern appearing:

Screen Shot 2013-05-14 at 12.47.42

Surplus solar energy during the day, but none in the evenings and at night for cooking + lighting (it looks like the heater is still kicking in at the end of the day, BTW).

This particular example shows that the amount of surplus energy would be more or less what’s needed in the evening – if only there were a way to store this energy for 6 hours…

Looking at some counters over that same period, I can see that the amount of energy is about 2.5 kWh. The challenge is to store this amount of energy locally. Some thoughts:

  • A 12 V lead-acid battery could be used, with 2.5 kWh corresponding to some 208 Ah.
  • But that’s a lower bound: let’s assume 90% conversion efficiency in both directions, i.e. 81% for charge + discharge (i.e. 19% losses) – we’ll now need a 257 Ah battery.
  • But the lifetime of lead-acid batteries is only good if you don’t discharge them too far. So-called deep cycle batteries are designed specifically for cases like these, where the charge/discharge is going to happen day in day out. To use them optimally, you shouldn’t discharge them over 50%, so we’ll need a battery twice as large: 514 Ah.

Let’s see… three of these 12V 230 Ah units could easily do the job:

Screen Shot 2013-05-14 at 13.14.23

Note that the cost of the batteries alone will be €2,000 and their total weight 200 kg!

There’s an interesting article about the energy shortage after the Fukushima disaster, including a good diagram about a somewhat similar issue (lowering evening peak use):


Although driven by a much harsher reality in that article, I wouldn’t be surprised to see new “one-day storage” solutions come out of all this, usable in the rest of the world as well.

For winter-time, I suppose one could heat up a large water tank, and then re-use it for heating in the evening. Except, ehm, that there’s a lot less surplus energy in winter.

Are there any other viable “semi off-grid” options out there? A flywheel in the basement?

PS. New milestone reached yesterday: total solar production so far has caught up with the consumption here at JeeLabs during that same period (since end October, that is).

MPPT hunting

In Hardware on May 21, 2013 at 00:01

Solar panels are funny power sources: for each panel, if you draw no power, the voltage will rise to 15..40 V (depending on the type of panel), and when you short them out, a current of 5..12 A will flow (again, depending on type). My panels will do 30V @ 8A.

Note that in both cases just described, the power output will be zero: power = volts x amps, so when either one is zero, there’s no energy transfer! – to get power out of a solar panel, you have to adjust those parameters somewhere in between. And what’s even worse, that optimal point depends on the amount of sunlight hitting the panels…

That’s where MPPT comes in, which stands for Maximum Power Point Tracking. Here’s a graph, copied from, with thanks:


As you draw more current, there’s a “knee” at which the predominantly voltage-controlled output drops, until the panel is asked to supply more than it has, after which the output voltage drops very rapidly.

Power is the product of V and A, which is equivalent to the surface of the area left of and under the current output point on the curve.

But how do you adjust the power demand to match that optimal point in the first place?

The trick is to vary the demand a bit (i.e. the current drawn) and then to closely watch what the voltage is doing. What we’re after is the slope of the line – or in mathematical terms, its derivative. If it’s too flat, we should increase the load current, if it’s too steep, we should back off a bit. By oscillating, we can estimate the slope – and that’s exactly what my inverter seems to be doing here (but only on down-slopes, as far as I can tell):

Screen Shot 2013-05-14 at 15.35.03

As the PV output changes due to the sun intensity and incidence angle changing, the SMA SB5000TL inverter adjusts the load it places on the panels to get the most juice out of ’em.

Neat, eh?

Update – I just came across a related post at Dangerous Prototypessynchronicity!

Tricky graphs – part 2

In Software on May 20, 2013 at 00:01

As noted in a recent post, graphs can be very tricky – “intuitive” plots are a funny thing!

Screen Shot 2013-05-14 at 10.35.43

I’ve started implementing the display of multiple series in one graph (which is surprisingly complex with Dygraphs, because of the way it wants the data points to be organised). And as you can see, it works – here with solar power generation measured two different ways:

  • using the pulse counter downstairs, reported up to every 3 seconds (green)
  • using the values read out from the SB5000TL inverter via Bluetooth (blue)

In principle, these represent the same information, and they do match more-or-less, but the problem mentioned before makes the rectangles show up after each point in time:

Screen Shot 2013-05-14 at 10.40.11

Using a tip from a recent comment, I hacked in a way to shift each rectangle to the left:

Screen Shot 2013-05-14 at 10.40.34

Now the intuition should be that the area in the coarser green rectangles should be the same as the blue area they overlap. But as you can see, that’s not the case. Note that the latter is the proper way to represent what was measured: a measurement corresponds to the area preceding it, i.e. that hack is an improvement.

So what’s going on, eh?

Nice puzzle. The explanation is that the second set of readings is incomplete: the smaRelay sketch only obtains a reading from the inverter ever 5 minutes or so, but that’s merely an instantaneous power estimate, not an averaged reading since the previous readout!

So there’s not really enough data in the second series to produce an estimate over each 5-minute period. All we have is a sample of the power generation, taken once every so often.

Conclusion: there’s no way to draw the green rectangles to match our intuition – not with the data as it is being collected right now anyway. In fact, it’ probably more appropriate to draw an interpolated line graph for that second series, or better still: dots

Update – Oops, the graphs are coloured differently, my apologies for the extra confusion.

Supply noise sensitivity

In Hardware on May 19, 2013 at 00:01

Yesterday’s post showed how with 3 resistors, one capacitor, and a P-MOSFET, you can set up a circuit to measure battery voltage with a voltage divider, even for voltages above VCC.

The whole point of this is that it can be switched off completely, drawing no current between measurements.

While trying this out, I started with a 1 MΩ pull-up on the P-MOSFET gate, and got this:


A very odd switch-off pattern, looked like an oscillation of some kind. Even with with the 100x faster switch-off using a 10 kΩ pull-up instead, the problem persisted:


This turned out to be a problem with the power supply. I was using a little USB plug with a switching regulator. These tend to work fine, but they do create a bit of “ripple voltage”, i.e. the 5V output is not exactly 5V DC. Here are the fluctuations, typical of units like these:


In other words: that little ripple was greatly amplified near the point where the P-MOSFET was starting to turn off, thus creating a regular but highly exaggerated turn-off pattern. Because – in a certain range – MOSFETs act like amplifiers, just like regular transistors.

It all went away when I switched to the lab supply, but it sure took some head-scratching…

Anyway, in real use this won’t matter, since the whole point is to use this with batteries.

Zero-power measurement – part 2

In Hardware on May 18, 2013 at 00:01

After a great suggestion by Max, on yesterday’s post, here’s a another circuit to try:

JC's Grid, page 73

It adds a capacitor and a resistor, but it allows using a P-MOSFET and a divider ratio which can now use the entire ADC range, not just 1 V or so as in yesterday’s circuit. Note however that if VCC is not fixed to the same value under all conditions, then the ADC’s reference voltage can float, and use of the 1.1V bandgap may still be needed.

Here’s the voltage at the top of the divider, showing how it switches on and off:


That’s with the pull-up resistor value R set to 1 MΩ, which takes 208 ms to turn the MOSFET back off. We don’t need that long, a 10 kΩ resistor for R will do fine:


That still gives us 2 ms to measure the supply level. Note that turn-off is automatic. DIO needs to be turned high again, but that can happen later. In my test code, I left it low for 1s to, then high for 7s.

Here’s a neat set of superimposed measurements (using persistence), while varying the high voltage from 3.5 to 12.0 V in 0.5 V steps:


Warning: for 12V, the divider ratio must be changed so the centre tap stays under VCC.

Note that with higher voltages, the MOSFET will turn off sooner – this is because there is now more current flowing through the pull-up resistor. But still plenty of time left to measure: 1 ms is more than enough for an ADC.

Tomorrow, an example of how these measurements can sometimes go awry…

Zero-power battery measurement

In Hardware on May 17, 2013 at 00:01

As promised, here’s a circuit which can be used to measure a voltage higher than VCC without drawing any current while not measuring:

Screen Shot 2013-05-15 at 13.40.54

Besides the fact that this needs an N-FET + I/O pin, there are several finicky details.

First of all, note that the following circuit will not drop the power consumption to zero:

Screen Shot 2013-05-15 at 14.38.42

The idea in itself is great: set DIO to logic “0” before performing a measurement, acting as GND level for the resistor divider (10 + 10 kΩ would be fine here). Then, to switch it off, set DIO to an input, so that the pin becomes high-impedance.

The problem is that the pin divider is still connected and that the AIO pin cannot float any higher than VCC + 0.6 (the drop over the internal ESD protection diode). The top resistor remains connected between PWR and VCC + 0.6, therefore it’s still leaking some current.

That also explains why the first circuit does better: the MOSFET disconnects all I/O pins from that PWR line, so that there is just a resistor from AIO to ground (which is harmless).

But there’s a catch: we need to be able to turn the N-channel MOSFET on and off, which means we need to be able to apply a voltage to its gate which is a few volts above the drain pin (the bottom one, attached to AIO). With a resistive divider of 10 + 10 kΩ on a 6V PWR line, that voltage will immediately rise to 3V, and there’s no way the DIO pin can keep the MOSFET on (it can only go up to logic “1”, i.e. 3.3V).

The solution is to use a different divider ratio: say 50 + 10 kΩ. Then, a 6V PWR level leads to a 1V level on the AIO pin, i.e. on the drain of the MOSFET. With DIO set to “1”, that means the MOSFETs gate will be 2.3V above the drain – enough to keep it turned on.

BTW, all this tinkering over the past few days has left me with a bunch a funky headers :)


Anyway, to summarise the zero-power battery monitor:

  • to work with 6V PWR, use a 50 (or 47) kΩ top resistor and 10 kΩ for the bottom one
  • use an N-channel MOSFET with low turn-on voltage (called a “logic level MOSFET”)
  • to measure the voltage, set DIO to “1”
  • measure the voltage on the AIO pin, where 0..1V will correspond to 0..6V on PWR
  • to turn off the divider, set DIO to “0”

As you can see, this approach requires an active component to switch things and an extra I/O pin, but then you do end up with a circuit which can completely switch off.

For simple uses, I’d just use yesterday’s setup – sub-microamp is usually good enough!

Measuring the battery without draining it

In Hardware on May 16, 2013 at 00:01

In yesterday’s post, a resistive voltage divider was used to measure the battery voltage – any voltage for that matter, as long as the divider resistor values are chosen properly.

With a 6V battery, a 10 + 10 kΩ divider draws 0.3 ma, i.e. 300 µA. Can we do better?

Sure: 100+100 kΩ draws 30 µA, 1+1 MΩ draws 3 µA, and 10+10 MΩ draws just 0.3 µA.

Unfortunately there are limits, preventing the use of really high resistor divider values.

The ATmega328 datasheet recommends that the output impedance of the circuit connected to the ADC input pin be 10 kΩ or less for good results. With higher values, there is less current available to charge the ADC’s sample-and-hold capacitor, meaning that it will take longer for the ADC to report a stable value (reading it out more than once may be needed). And then there’s the leakage current which every pin has – it’s specified in the datasheet as ± 1 µA max in or out of any I/O pin. This means that a 1+1 MΩ divider may not only take longer to read out, but also that the actual value read may not be accurate – no matter how long we wait or how often we repeat the measurement.

So let’s find out!

The divider I’m going to use is the same as yesterday, but with higher resistor values.

Let’s go all out and try 10 + 10 MΩ. I’ll use the following sketch, which reads out AIO1..4, and sends out a 4-byte packet with the top 8 bits of each ADC value every 8 seconds:

#include <JeeLib.h>

byte payload[4];

void setup () {
  rf12_initialize(22, RF12_868MHZ, 5);
  DIDR0 = 0x0F; // disable the digital inputs on analog 0..3

void loop () {
  for (byte i = 0; i < 4; ++i) {
    analogRead(i);                    // ignore first reading
    payload[i] = analogRead(i) >> 2;  // report upper 8 bits

  rf12_sendNow(0, payload, sizeof payload);

This means that a reported value N corresponds to N / 255 * 3.3V.

With 5V as supply, this is what comes out:

L 10:18:14.311 usb-A40117UK OK 22 193 220 206 196
L 10:18:22.675 usb-A40117UK OK 22 193 189 186 187
L 10:18:31.026 usb-A40117UK OK 22 193 141 149 162
L 10:18:39.382 usb-A40117UK OK 22 193 174 167 164
L 10:18:47.741 usb-A40117UK OK 22 193 209 185 175

The 193 comes from AIO1, which has the 10 + 10 kΩ divider, and reports 2.50V – spot on.

But as you can see, the second value is all over the map (ignore the 3rd and 4th, they are floating). The reason for this is that the 10 MΩ resistors are so high that all sorts of noise gets picked up and “measured”.

With a 1 + 1 MΩ divider, things do improve, but the current draw increases to 2.5 µA:

L 09:21:25.557 usb-A40117UK OK 22 198 200 192 186
L 09:21:33.907 usb-A40117UK OK 22 198 192 182 177
L 09:21:42.256 usb-A40117UK OK 22 197 199 188 183
L 09:21:50.606 usb-A40117UK OK 22 197 195 187 183
L 09:21:58.965 usb-A40117UK OK 22 197 197 186 181
L 09:22:07.315 usb-A40117UK OK 22 198 198 190 184

Can we do better? Sure. The trick is to add a small capacitor in parallel with the lower resistor. Here’s a test using 10 + 10 MΩ again, with a 0.1 µF cap between AIO2 and GND:


Results – at 5V we get 196, i.e. 2.54V:

L 10:30:27.768 usb-A40117UK OK 22 198 196 189 186
L 10:30:36.118 usb-A40117UK OK 22 198 196 188 183
L 10:30:44.478 usb-A40117UK OK 22 198 196 186 182
L 10:30:52.842 usb-A40117UK OK 22 198 196 189 185
L 10:31:01.186 usb-A40117UK OK 22 197 196 186 181

At 4V we get 157, i.e. 2.03V:

L 10:33:31.552 usb-A40117UK OK 22 158 157 158 161
L 10:33:39.902 usb-A40117UK OK 22 158 157 156 157
L 10:33:48.246 usb-A40117UK OK 22 158 157 159 161
L 10:33:56.611 usb-A40117UK OK 22 158 157 157 159
L 10:34:04.959 usb-A40117UK OK 22 159 157 158 161

At 6V we get 235, i.e. 3.04V:

L 10:47:26.658 usb-A40117UK OK 22 237 235 222 210
L 10:47:35.023 usb-A40117UK OK 22 237 235 210 199
L 10:47:43.373 usb-A40117UK OK 22 236 235 222 210
L 10:47:51.755 usb-A40117UK OK 22 237 235 208 194
L 10:48:00.080 usb-A40117UK OK 22 236 235 220 209


Note how the floating AIO3 and AIO4 pins tend to follow the levels on AIO1 and AIO2. My hunch is that the ADC’s sample-and-hold circuit is now working in reverse: when AIO3 is read, the S&H switches on, and levels the charge on the unconnected pin (which still has a tiny amount of parasitic capacitance) and the internal capacitance.

The current draw through this permanently-connected resistor divider with charge cap will be very low indeed: 0.3 µA at 6V (Ohm’s law: 6V / 20 MΩ). This sort of leakage current is probably fine in most cases, and gives us the ability to check the battery level in a wireless node, even with battery voltages above VCC.

Tomorrow I’ll explore a setup which draws no current in sleep mode. Just for kicks…

What if we want to know the battery state?

In Hardware on May 15, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

One useful task for wireless sensor nodes, is to be able to determine the state of the battery: is it full? is it nearly depleted? how much life is there left in them?

With a boost converter such as the AA Power Board, things are fairly easy because the battery voltage is below the supply voltage – just hook it up to an analog input pin, and use the built-in ADC with a call such as:

word millivolts = map(analogRead(0), 0, 1023, 0, 3300);

This assumes that the ATmega is running on a stable 3.3V supply, which acts as reference for the ADC.

If that isn’t the case, i.e. if the ATmega is running directly off 2 AA batteries or a coin cell, then the ADC cannot use the supply voltage as reference. Reading out VCC through the ADC will always return 1023, i.e. the maximum value, since its reference is also VCC – so this can not tell us anything about the absolute voltage level.

There’s a trick around this, as described in a previous post: measure a known voltage with the ADC and then deduce the reference voltage from it. As it so happens, the ATmega has a 1.1V “bandgap” voltage which is accurate enough for this purpose.

The third scenario is that we’re running off a voltage higher than 3.3V, and that the ATmega is powered by it through a voltage regulator, providing a stable 3.3V. So now, the ADC has a stable reference voltage, but we end up with a new problem: the voltage we want to measure is higher than 3.3V!

Let’s say we have a rechargeable 6V lead-acid battery and we want to get a warning before it runs down completely (which is very bad for battery life). So let’s assume we want to measure the voltage and trigger on that voltage dropping to 5.4V.

We can’t just hook up the battery voltage to an analog input pin, but we could use a voltage divider made up of two equal resistors. I used two 10 kΩ resistors and mounted them on a 6-pin header – very convenient for use with a JeeNode:


Now, only half the battery voltage will be present on the analog input pin (because both resistor values are the same in this example). So the battery voltage calculation now becomes a variant of the previous formula:

word millivolts = map(analogRead(0), 0, 1023, 0, 3300) * 2;

But there is a drawback with this approach: it draws some current, and it draws it all the time. In the case of 2x 10 kΩ resistors on a 6V battery, the current draw is (Ohm’s law kicking in!): 6 V / 20,000 Ω = 0.0003 A = 0.3 mA. On a lead-acid battery, that’s probably no problem at all, but on smaller batteries and when you’re trying to conserve as much energy as possible, 0.3 mA is huge!

Can we raise the resistor values and lower the current consumption of this voltage divider that way? Yes, but not indefinitely – more on that tomorrow…

Energy, power, current, charge

In Hardware on May 14, 2013 at 00:01

The International System of Units, or SI from the French Système International is a wonderfully clever refinement of the original metric system.

Took me a while to get all this clear, but it really helps to understand electrical “units”:

  • power says something about intensity: volts times amperes, the unit is watt
  • energy says something about effort: power times duration, the unit is watt-second
  • current says something about rate: charge per time unit, the unit is ampere
  • charge says something about pressure: more charge raises volts, the unit is coulomb

Of course, some units get expressed differently – that’s just to scale things for practical use:

  • a kilowatt (kW) is 1000 watts
  • a watt-hour (Wh) is 3600 watt-seconds
  • a kilowatt-hour (kWh) is 1000 watt-hour
  • a milli-ampere (mA) is 1/1000 of an ampere
  • a micro-coulomb (µC) is 1/1000000 of a coulomb

But there are several more useful equivalences:

  • When a 1.5 V battery is specified as 2000 mAh (i.e. 2 Ah), then it can deliver 1.5 x 2 = 3 Wh of energy – why? because you can multiply and divide units just like you can with their quantities, so V x Ah = V x A x h = W x h = Wh
  • Another unit of energy is the “joule” – which is just another name for watt-second. Or to put it differently: a watt is one joule per second, which shows that a watt is a rate.
  • A joule is also tied to mechanical energy: one joule is one newton-meter, where the “newton” is the unit of force. A newton is what it takes to accelerate 1 kg of mass by 1 m/s2 (i.e. increase the velocity by 1 m/s in one second – are you still with me?).
  • So the watt also represents a mechanical intensity (i.e. strength). Just like one horsepower, which is defined as 746 W, presumably the strength of a single horse…
  • Got a car with a 100 Hp engine? It can generate 74.6 kW of power, i.e. accelerate a 1000 kg weight by 74.6 m/s2, which is ≈ 20 km/h speed increase every second, or in more popular terms: 0..100 km/h in 5 seconds (assuming no losses). But I digress…

The point is that all those SI units really make life easy. And they’re 100% logical…

Measurement intervals and graphs

In Software on May 13, 2013 at 00:01

One more post about graphs in this little mini-series … have a look at this example:

Screen Shot 2013-05-10 at 19.33.51

Apart from some minor glitches, it’s in fact an accurate representation of the measurement data. Yet there’s something odd: some parts of the graph are more granular than others!

The reason for this is actually completely unrelated to the issues described yesterday.

What is happening is that the measurement is based on measuring the time between pulses in my pulse counters, which generate 2000 pulses per kWh. A single pulse represents 0.5 Wh of energy. So if we get one pulse per hour, then we know that the average consumption during that period was 0.5 W. And when we get one pulse per second, then the average (but near-instantaneous) power consumption over that second must have been 1800 W.

So the proper way to calculate the actual power consumption, is to use this formula:

    power = 1,800,000 / time-since-last-pulse-in-milliseconds

Which is exactly how I’ve been measuring power for the past few years here at JeeLabs. The pulses represent energy consumption (i.e. kWh), whereas the time between pulses represents estimated actual power use (i.e. W).

This type of measurement has the nice benefit of being more accurate at lower power levels (because we then divide by a larger and relatively more accurate number of milliseconds).

But this is at the same time also somewhat of a drawback: at low power levels, pulses are not coming in very often. In fact, at 100 W, we can expect one pulse every 18 seconds. And that’s exactly what the above graph is showing: less frequent pulses at low power levels.

Still, the graph is absolutely correct: the shaded area corresponds exactly to the energy consumption (within the counter’s measurement tolerances, evidently). And line drawn as boundary at the top of the area represents the best estimate we have of instantaneous power consumption across the entire time period.

Odd, but accurate. This effect goes away once aggregated over longer period of time.

Tricky graphs

In Software on May 12, 2013 at 00:01

Yesterday’s post introduced the new graph style I’ve added to HouseMon. As mentioned, I am using step-style graphs for the power (i.e. Watt) based displays. Zooming in a bit on the one shown yesterday, we get:

Screen Shot 2013-05-10 at 19.21.18

There’s something really odd going on here, as can be seen in the blocky lines up until about 15:00. The blocks indicate that there are many missing data points in the measurement data (normally there should be several measurements per minute). But in itself these rectangular shapes are in fact very accurate: the area is Watt times duration, which is the amount of energy (usually expressed in Joules, Wh, or kWh).

So rectangles are in fact exactly right for graphing power levels: we want to represent each measurement as an area which matches the amount of energy used. If there are fewer measurements, the rectangles will be wider, but the area will still represent our best estimate of the energy consumption.

But not all is well – here’s a more detailed example:

Screen Shot 2013-05-10 at 19.17.03

The cursor in the screenshot indicates that the power usage is 100.1 W at 19:16:18. The problem is that the rectangle drawn is wrong: it should precede the cursor position, not follow it. Because the measurement is the result of sampling power and returning the cumulative power level at the end of that measurement period. So the rectangle which is 100.1 W high should really be drawn from 19:18:00 to 19:18:16, i.e. from the time of the previous measurement.

I have yet to find a graphing system which gets this right.

Tomorrow: another oddity, not in the graph, but related to how power is measured…

New (Dy)graphs in HouseMon

In Software on May 11, 2013 at 00:01

I’ve switched to the Dygraphs package in HouseMon (see the development branch). Here’s what it now looks like:

Screen Shot 2013-05-10 at 19.15.08

(click for the full-scale resolution images)

Several reasons to move away from Flotr2, actually – it supports more line-chart modes, it’s totally geared towards time-series data (no need to adjust/format anything), and it has splendid cursor and zooming support out of the box. Oh, and swipe support on tablets!

Zooming in both X and Y directions can be done by dragging the mouse inside the graph:

Screen Shot 2013-05-10 at 19.16.19

You’re looking at step-style line drawing, by the way. There are some quirks in this dataset which I need to figure out, but this is a much better representation for energy consumption than connecting dots by straight lines (or what would be even worse: bezier curves!).

Temperatures and other “non-rate” entities are better represented by interpolation:

Screen Shot 2013-05-10 at 19.17.40

BTW, it’s surprisingly tricky to graphically present measurement data – more tomorrow…

Maxing out the Hameg scope

In AVR, Hardware on May 10, 2013 at 00:01

Yesterday’s post was about how test equipment can differ not only in terms of hardware, but also the software/firmware that comes with it (anyone hacking on the Owons or Rigols yet to make the software more feature-full?).

Here’s another example, where I’m using just about all the bells and whistles of the Hameg HMO series scopes – not for the heck of it, but because it really can help gain more insight about the circuit being examined.

This is my second attempt at understanding what sort of start up currents need to be available for the new JeeNode Micro v3 to properly power up:


I’m applying a 0..2V power up ramp (yellow line) as power supply, using a 1 Hz sawtooth signal. This again simulates an energy harvesting setup where the power supply slowly ramps up (the real thing would actually rise far more slowly, i.e. when using a solar cell + supercap, for example). The current consumed by the JNµ v3 (blue line) is measured by measuring the voltage drop across a 10 Ω resistor – as usual.

The current consumption starts at about 0.85V and rises until the power supply reaches about 1.4V. At that point, the current consumption is about 77 µA. Then the ATtiny84A comes out of reset, enters a very brief high-current mode (much higher than the peak shown, but this is averaged out), and then goes into ultra low-power sleep mode. The sketch running on the JNµ is the latest power_down.ino, here in simplified form:

#include <JeeLib.h>

void setup () {

void loop () {}

Note that since this is the new JNµ v3, the RFM12B module never even gets powered up, so there’s no need to initialise the radio and then put it in sleep mode.

The red line uses the Hameg’s advanced math features to perform digital filtering on top of the averaging already performed during acquisition: the averaging keeps the power-up spike visible (albeit distorted), at the cost of leaving some residual noise in the blue trace, while the IIR digital low-pass filter applied to that result then makes it possible to estimate the 77 µA current draw just before the ATtiny84A starts running.

Here’s the zoomed-in view, showing the interesting segment in even more detail:


The IIR filtering seen here is slightly different, with a little spike due to the following power-up spike, so the 86 µA reported here is slightly on the high side.

Note how the Hameg’s storage, high sensitivity, averaging, adjustable units display, variable vertical scale, math functions, on-screen measurements, on-screen cursors, and zooming all come together to produce a pretty informative screen shot of what is going on. Frankly, I wouldn’t know how to obtain this level of info in any other way.

So what’s all this fuss about measuring that 77 µA level?

Well, this is how much current the JNµ draws before it starts running its code. There’s nothing we can do to reduce this power consumption further until it reaches this point. In the case of energy harvesting, the supply – no matter how it’s generated – will have to be able to briefly deliver at least 77 µA to overcome the startup requirements. If it doesn’t, then the supply voltage (presumably a supercap or rechargeable battery) will never rise further, even though a JNµ can easily draw less than a tenth of that current once it has started up and goes into ultra-low power with brief occasional wake-ups to do the real work.

What I’m taking away from this, is that a solar energy setup will need to provide at least 0.1 mA somewhere during the day to get a JNµ started up. Once that happens, and as long as there is enough power to supply the average current needed, the node can then run at lower power levels. But whenever the supercap or battery runs out, another period with enough light is needed to generate that 0.1 mA again.

It all sounds ridiculously low, and in a way it is: 0.1 mA could be supplied for over two years by 3 AA batteries. The reason for going through all this trouble, is that I’d really like to find a way to run off indoor solar energy light levels. Even if it only works near a window, it would completely remove the need for batteries. It would allow me to just sprinkle such nodes where needed and collect data … forever.

Oscilloscope firmware

In Hardware on May 9, 2013 at 00:01

Oscilloscopes are very complex instruments. The “front end” is all about being able to capture a huge range of signals at a huge rate of speeds. This is what lets you hook up the same probe to AC mains one day, and pick up millivolt signals another day, and to collect many minutes of data on a single screen vs displaying the shape of a multi-MHz wave. This isn’t just about capture, at least as important is the triggering part: how to decide what to pick up for display on the screen.

For low sampling rates, it’s very easy to use an ADC and just collect some data points – as shown in this older weblog post, even AC mains, although triggering can be an issue.

With the Xminilab presented recently, a lot of this has been solved in software, supporting a pretty impressive range of options, even for triggering. The Xminilab is particularly interesting because the full source code is available.

But for serious work, you’ll need an Owon or Rigol scope. These can sample at up to 1 Gsa/s, i.e. one billion samples per second. Truly, truly capable front-ends, able to handle very wide voltage and acquisition speed ranges.

The Hameg HMO2024 is more expensive, and many of its specs are not much better than the Owon (worse even, in some cases: a smaller display size and less sample memory).

The devil is in the details. Here’s a recent screen from the HMO2024 (borders cropped):


And here’s my first cut at acquiring the same info on the Owon (click for full size):


Let me add that I now have lots of experience with the Hameg, and only just started using the Owon, so there might be relevant features I’ve failed to set up in an optimal fashion.

A couple of quick observations:

  • This is not a “typical” measurement setup: a very slow, low-amplitude signal is nothing like the usual measurements one would come across, with higher signal levels, and faster sampling rates. Then again, that’s part of the whole point of an oscilloscope: it’s so versatile that you end up using it in lots of situations!
  • As you can see, the Owon has a lot more pixels to display a signal on, so I was able to increase the voltage sensitivity one notch to get more detail, and capture a bit longer.
  • Some differences are obvious but not that important: the Owon provides less information on-screen about the current settings, and it does not use anti-aliasing for the traces (i.e. intensity variations to produce a fake “sharpening” effect of steep lines).

The two major differences are that: 1) the Hameg lets me apply additional digital signal processing to effectively reduce the random variations and smooth out the signal (this is done after capture, but before drawing things on-screen, i.e. all in software/firmware), and 2) the Hameg includes support for a “reference trace”, i.e. storing a previous trace in its built-in memory, and displaying it in white next to a new capture – to compare power consumption with and without WiFi, in this case.

Note that the Owon capture depth was set to 1,000 samples instead of the maximum 10 Msa, otherwise the screen would have shown a very wide red trace, almost completely swamping out the signal shown on screen. With this reduced setting, the current consumption is still fairly easy to estimate, despite the lack of low-pass filtering.

Is this a show-stopper for the Owon? Not really. It still gives a pretty good impression of the current consumption pattern during starup of the Carambola 2. If you really wanted to improve on this, you could insert an analog filter (a trivial RC filter with just 2 passive components would do). With a bit of extra work, I’m sure you can get at least as good a current consumption graph on the Owon.

The trade-off is (recurring) convenience and setup time vs. (up-front) equipment cost.

PS. The Rigol DS1052E does have a low-pass filter – every scope has different trade-offs!

PPS. For a great view into oscilloscope development over the past 5 years, see Dave Jones’ comparison video of the Rigol DS1052E and the new – phenomenal!DS2000 series.

What if I turn the chip around?

In Hardware on May 8, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

Ok, you’re all excited, you’ve built some electronic circuit – either by assembling a kit, or all on a breadboard, or perhaps you’ve even go so far as to design and create a custom PCB.

Any non-trivial circuit will have polarised components on it, whether capacitors, diodes, transistors, regulators, or… the most common one in oh so many varieties: a “chip”, with 6..40 pins, or sometimes even more.

Mr. Murphy loves chips. Because sooner or later, you’ll connect one the wrong way around. Even if you know what you’re doing, sometimes the orientation marker on a chip is fairly hard to see, especially on the smaller SMD types.

So what happens if we put things in the wrong way around?

Obvious answer: it depends (on the chip).

Comforting answer: more often than not, nothing will happen, the thing will get hot, and it’ll still work fine once you fix the problem, i.e. turn the chip around and reconnect it.

The good news is that it’s not so easy to really damage most chips, with a few precautions:

  • use a “weak” power supply, i.e. one which can’t put out to much current, as current leads to heat, and heat is usually the cause of component damage – a lab power supply with adjustable “current limiting” set to a low value is a very good idea
  • keep your hands near the ON/OFF switch when powering up a circuit for the first time, keep your eyes open, and … use your nose: bad stuff due to heat often shows itself as smoke (by then, it’s often too late), and as components getting far too hot, and starting to smell a bit
  • for low-voltage circuits, and this includes almost all digital circuits: place your fingers on several of the key components right after turning power on: if you sense anything getting hot, turn off the power – NOW!
  • sensing heat is an excellent way to save a project from serious damage: we can easily tell if something heats up to 50 °C or more, yet most silicon-based chips will be able to heat up way beyond that before actually getting damaged (125..175 °C) – so as long as you turn the power off quickly enough, chances are that nothing really will break down, and chips and resistors will often start to smell – a useful warning sign!

Note that analog circuits tend to get damaged much more easily. Put a transistor the wrong way around, and it’ll probably go to never-never land the moment power is applied.

One reason digital chips are so resilient, is the fact that they are full of ESD diodes. These tend to be on each of the I/O pins of a chip, as protection against Electrostatic Discharge. Here’s what a typical I/O pin circuit on a digital chip looks like:

JC's Grid, page 72

Nothing happens under normal conditions, since the diodes are all in blocking mode. When the I/O pin voltage rises above VCC or drops below GND, however, the diodes start to conduct, while trying to remove the charge, so that the voltage levels never reach values which might damage the sensitive oxide isolation (that’s the “O” in CMOS and MOSFET).

Now have a look at what happens when a chip gets powered up with bad voltages on two of the I/O pins (the light-blue parts are not conducting and can be ignored):

JC's Grid, page 72 copy

The way to look at this is that the pin(s) with the highest voltage will start feeding into the (internal) VCC connections, and the pins with the lowest voltage will start drawing current from the (internal) GND connections. Or, to put it a different way – some I/O pins will act as VCC and GND supply lines, albeit with some internal ESD diodes in between:

JC's Grid, page 72 copy 2

In this diagram, VCC and GND are fed from pins which were not intended as such!

As you can see, the diodes now start conducting as well, drawing a certain amount of current. If these currents are not higher than the diodes can handle (usually at least a few mA per diode), then the chip will act more or less like a short to the rest of the circuit. With a bit of luck, your power supply will decide to lower its output voltage and enter “current limiting” mode. The result: nothing works, but nothing truly dramatic happens either. It just gets hot and all the voltages end up being completely wrong.

Sooo… next time you power up your new project for the first time: stay alert, use your fingers, be ready to cut power, and… relax. If it doesn’t work right away (it hardly ever does!), you’ll usually have time to figure out the problems, fix them, and get going after all.

Note that there are no guarantees (things do occasionally break), but usually it’s fixable.

Carambola 2 power consumption

In Hardware, Linux on May 7, 2013 at 00:01

The Carambola 2 mentioned yesterday is based on a SoC design which uses amazingly little power – considering that it’s running a full Linux-based OpenWrt setup.

There are a couple of ways to measure power consumption. If all you’re after is the average power on idle, then all you need to do is insert a multimeter in the power supply line and set it in the appropriate milliamp range. Wait a minute or so for the system to start up, and you’ll see that the Carambola 2 draws about 72 mA @ 5V, i.e. roughly a third of a watt.

If you have a lab power supply, you can simply read the power consumption on its display.

But given an oscilloscope, it’s actually much more informative to see what the power consumption graph is, i.e. over time. This will show the startup power use and also allows seeing more detail, since these systems often periodically cycle through different activities.

The setup for “seeing” power consumption is always the same: just insert a small resistor in series with the “Device Under Test”, and measure the voltage drop over that resistor:

JC's Grid, page 51

Except that in this case, we need to use a smaller resistor to keep the voltage drop within bounds. Given that the expected currents will be over 100 mA, a 100 Ω resistor would completely mess up the setup. I found a 0.1 Ω SMD resistor in my lab supplies, so that’s what I used – mounting it on a 2-pin header for convenience:


With 0.1 Ω, a 100 mA current produces a voltage of 10 mV. This should have a negligible effect on the power supplied to the Carambola 2 (a 1 Ω resistor should also work fine).

Here’s the result on the scope – white is the default setup, yellow is with WiFi enabled:


Sure takes all the guesswork out of what the power consumption is doing on startup, eh?

Embedded Linux – Carambola 2

In Hardware, Linux on May 6, 2013 at 00:01

This has got to be one of the lowest-cost and simplest embedded Linux boards out there:


It’s the Carambola 2 by

Screen Shot 2013-05-04 at 11.02.06

The 28 x 38 mm (!) bare board is €19 excl VAT and shipping, and the development bundle (as shown above) is €33. The latter has a Carambola2 permanently soldered onto it, with 2 Ethernet ports, a slave USB / console / power port, a USB host port, a WiFi chip antenna (which is no longer on the base board, unlike the original Carambola), and a switching power supply to generate 3.3V from the USB’s 5V.

The processor is a MIPS-based Atheros chip, and with 64 MB ram and 11 MB of available flash space, there is ample room to pre-populate this board with a lot of files and software.

The convenience of the development setup, is that it includes an FTDI chip, so it comes up as a USB serial connection – you just need to find out what port it’s on, connect to it at 115200 baud via a terminal utility such as “screen” on Mac or Linux, and you’ll be hacking around in OpenWrt Linux in no time.

Note that this setup is very different from a Raspberry Pi: MIPS ≠ ARM, for one. The RPi has a lot more performance and RAM, has hardware floating point, and is more like a complete (portable) computer with its HDMI video out port. The benefit of the Carambola 2 is its built-in WiFi, built-in flash, and its low power – more on that tomorrow.

Meet the Owon SDS 7102V – part 2

In Hardware on May 5, 2013 at 00:01

Today’s post continues where we left off yesterday. Here are the front-panel controls:


Nice and tidy. Absolutely effective, as far as I could establish in my first impressions. As with all modern scopes, there are lots of features behind all those buttons, and many of them lead to “soft menu’s”, i.e. menu’s shown on screen (on three sides sometimes: right, left, and bottom). That’s what the right and bottom buttons next to the screen are for. There’s one “multipurpose” rotary encoder knob, which is used when selecting from the occasional menu popping up on the left.

The only downside is that you can end up moving your hands around a lot while setting things up and while making adjustments. Coming from a different brand, I had some trouble remembering where averaging, FFT, trigger settings, etc. were, but that’s bound to get easier over time as muscle memory sets in. Because operating any complex instrument with lots of knobs and features really is about motions and muscle memory. It just takes a bit of time and practice.

One remarkable feature of this scope is its very deep 10 megasamples acquisition depth (it’s adjustable, from 1,000 samples up). This makes it very easy to take a single snapshot of an event, and then to zoom in to see specific events in full detail.

One use would be to decode serial communication signals such as UARTs and I2C data packets. There is no built-in decoding, so this needs to be done manually. Then again, you can save all 10 million samples to a USB stick so with some software it would be possible to perform such decoding automatically on a standard PC or Mac, albeit after-the-fact.

Power consumption is very low: 0.77W standby, 18W when turned on.

You might be wondering how this oscilloscope compares to the Xminilab and the Hameg HMO series – which are about a fifth and five times as expensive, respectively. But with such an extreme price range, it’s impossible to answer this question other than: the more you pay, the more you get. Pretty obvious, and also pretty useless as guideline, I’m afraid.

Would I buy the Xminilab if I had no more than $100 to spend? Yes. While it’s limited and does require a lot more ingenuity and patience, it can still help to understand what’s going on, and to address problems that couldn’t be solved without a scope.

Would I recommend the Owon for serious electronics use? Definitely. It lets you capture all the info you need, and “see” what’s going on – both analog and digitally – for frequencies up to dozens of MHz. With much larger display & more memory than the Rigol DS1052E.

Would I purchase a Hameg HMO series again, even though it’s so darn expensive? Yes. The software, the math features, the logic analyser, and the serial decoding – it all adds up, yet it’s still half the price of the “low end” Agilent models. And, not to be ignored: its (cropped but informative) screenshots are perfect for the 604 pixel width of this weblog!

I’ll explore the capabilities of the Owon SDS 7102V scope in more practical scenarios in the weeks to come. Stay tuned…

Meet the Owon SDS 7102V

In Hardware on May 4, 2013 at 00:01

Here’s another “loaner” from David Menting, this time it’s his scope, the Owon SDS 7102V – which is sales-speak for a dual-channel 100 MHz digital storage oscilloscope:


This unit is available in the Netherlands from EleShop, for € 450 incl VAT, which makes it only marginally more expensive than the ubiquitous Rigol DS1052E with 320×240 display and 50 MHz bandwidth.

This thing is amazingly thin (total size is 7 x 34 x 16 cm), yet packs an 800 x 600 pixel color LCD screen to present a really detailed display (click to see the full size image):


In a way, more is better. But keep in mind that the 8-bit ADC’s typically used in modern “affordable” scopes can only measure 256 different voltage levels full-scale. To really benefit from 512 or more vertical pixels resolution, you either need a 9-bit ADC or some sort of oversampling and averaging. Having said that, I would definitely consider 320×240 as low end nowadays – this screen is a huge improvement, in displaying much finer detail as well as in helping estimate voltage levels at a glance.

Here’s an example of just how much screen real-estate this scope has:


You might recognise the two waveforms above as the 10 MHz and 25 MHz signals generated by my signal generator – same as used in this recent weblog post.

Tomorrow, I’ll show you the front panel and I’ll add some comparative notes…

Instrument limits

In Hardware on May 3, 2013 at 00:01

Last week’s post illustrated some limitations of electronic measuring equipment. In this case, I was using the TTi TG2511 Arbitrary Waveform Generator (which I have yet to use for “arbitrary” waveforms) and the Hameg HMO2024 Digital Storage Oscilloscope.

The TG2511’s rise and fall times are specified in the neighbourhood of 10 ns, which has a fairly atrocious effect on a 25 MHz “square wave” signal:


(the scope’s own rise time is under 2 ns)

Both are excellent instruments, but already fairly high-end for hobbyist use. To put it in perspective: the total cost of this sort of equipment is more than a hundred JeeNodes with sensors! Add to that the fact that you only need the higher specs of these instruments once in a while (how often depends of course on your level and depth of interest), and it’s pretty obvious that it can be very hard to justify such expenses.

I’ve always been annoyed by this. And I’ve always been on the lookout for alternatives:



That’s the Xminilab, mentioned recently, and a sine-wave generator from eBay. The total cost for both is around €100.

Unfortunately, lower-end equipment really does have lower-end specifications. The measurements made yesterday could not have been done with the above, for example: sine waves are not square waves, and the 2 megasamples/second of the Xminilab scope is not fast enough to analyse rise times at 1 MHz, let alone 10 MHz.

Tomorrow, I’ll explore (“review” is too big a word for it) a more affordable modern oscilloscope, to show what can and cannot be done with it.


In Hardware on May 2, 2013 at 00:01

The other day, someone gave me an autotransformer – a hefty 10 kg of metal and wires:


Made by Philips, probably well over half a century ago (even before Philips had a logo?):


AC mains did not include grounding at the time, just 2 banana jacks spaced 19 mm apart:


So what does it do? Well, an autotransformer (a.k.a. Variac) allows you to generate an adjustable AC voltage from the fixed AC mains voltage. At the time, AC mains was 220V – nowadays, it’s 230V in Europe, so the output should now reach 260/220*230 ≈ 272 VAC.

Here’s the schematic, similar to the one printed on the side of this device:


(this isn’t fully variable, like the unit above, but the taps are a first approximation)

One way to explain what’s going on – at least as first approximation – is that it works like a transformer, but with a variable number of turns on the secondary side. Think of the incoming voltage as generating an alternating magnetic field of a certain strength, with X Volts per turn. The “tap” (which is a mechanical wiper) makes contact with one of the turns of all the turns laid out in a circular fashion, creating a circuit with a variable number of turns. The more turns, the higher the output voltage.

The intriguing bit is that the output voltage can actually exceed the input voltage, by adding a few more spare turns at the top – or equivalently: by placing the input voltage on a tap and not entirely at the end of the coil.

Note that the output of such an autotransformer is not isolated from the input, unlike regular transformers with separate primary and secondary coils.

The other difference is that part of the energy is not transferred as magnetic flux, but directly through the shared windings. It merely acts “more or less” like a regular transformer, in practical use.

I’m very pleased with this gift, which will allow me to explore the effects of a varying AC mains voltage on all sorts of appliances, power supplies, etc. – from very low voltages to somewhat over the normal 230 VAC.

What if the supply is under 3.3V?

In AVR, Hardware on May 1, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

To follow up on a great suggestion from Martyn, here’s a post about the different trade-offs and implications of running an ATmega at lower voltages.

The standard Arduino Uno, and all models before it, have always operated the ATmega at 5.0V – which used to be the standard TTL levels used in the 7400 series of chips used in the 1960’s and 1970’s. The key benefit of a single standardised voltage level being that it made it possible to combine different chips from different vendors.

To this day, even though most semiconductor logic has evolved from bipolar junction transistor to CMOS, the voltage level has often been kept at 5V, with slightly adjusted – but compatible – voltage levels for “0” and “1”, respectively.

Nowadays, chips operate at lower voltages because it leads to lower power consumption and because it is a better fit for batteries and LiPo cells. In fact, lots of new chips operate at 3.3V and will not even tolerate 5.0V.

The ATmega328p is specified to run at a very wide range, from 1.8V all the way up to 5.5V. Which is great for ultra low-power use, supporting different battery options, and even for energy harvesting scenario’ such as a solar panel charging a supercap, for example.

But there are still many trade-offs to be aware of!

The first one is the system clock rate, which is limited (see also this older post):

Screen Shot 2013-04-30 at 21.59.07

If you look closely, you’ll see that 16 MHz is out of spec for 3.3V, the way the JeeNode is running, that is. In practice, this has never caused any known problems, but lowering the voltage further might just be too much.

The good news is that it’s not really the crystal oscillator which is causing problems, but the main circuitry of the ATmega, and that there’s a very easy fix for it: when running at voltages below 3.3V, you should set the ATmega’s pre-scaler to 2, causing the system clock to run at 8 MHz. When running even lower, perhaps under 2.4 V or so, set the pre-scaler to 4, i.e. run the system clock at 4 MHz. This also explains the presence of the divide-by-8 fuse bit: when you need to start up at low voltages, you can force the ATmega to always power up with a clock pre-scaler of 8, and then adjust the pre-scaler under software control after power-up, once the voltage has been verified to be sufficient. Without this setting, an ATmega would not be able to reliably start up at 1.8V, even if it’s meant to run much faster most of the time.

Note that the RF12 driver will still work at 4 MHz, but not less: the interrupt service time will be too slow for proper operation at any slower rate.

Another important issue to be aware of when running a JeeNode at voltages under 3.3V, is that the MCP1702 voltage regulator will no longer be able to regulate the incoming voltage. It can only reduce the input voltage for regulation, so when there is no “headroom” left, the regulator will just pass whatever is left, minus a small “dropout” voltage difference. Hence its name LDO: Low-DropOut.

The problem is that all LDO regulators start consuming (i.e. wasting) more idle current in this situation. See this weblog post for the measured values – which can be substantial!

To avoid this, you should really disconnect the LDO altogether, or at least its ground pin.

A third aspect of running at lower voltages, is that you need to verify that all the parts of the circuit continue to work. This applies to sensors as well as to the RFM12B radio – which should only be operated between 2.2V and 3.8V.

Actually, some experiments a while back showed that the radio could work down to 1.85V, but I suspect that things like transmit power will be greatly reduced at such supply levels.

Lastly, when the supply voltage is lowered, you need to keep some secondary effects in mind: the ADC will operate against a lower reference voltage as well, so its scale will change. One ADC step will be just under 3.3 mV when the supply is 3.3V, but this drops to under 2.0 mV per step with a 2.0V supply.

To summarise: yes, an ATmega/ATtiny can run at voltages below 3.3V, and even an entire JeeNode can, but you need to reduce the system clock by switching the pre-scaler to 2 or 4, and you need to make sure that all parts of your setup can handle these lower voltages.

Meet the Manson NSP 3630

In Hardware on Apr 30, 2013 at 00:01

It’s always nice to explore more equipment, to see how it behaves in the lab. I already have a very nice dual-voltage lab power supply, but this one is interesting due to its relatively low cost (€90 + shipping), fairly high power, and very convenient small size:


It’s the Manson NSP 3630, available from Reichelt in Germany by mail-order. At 27 x 15 x 7 cm, it really should easily fit in most home labs. With many thanks to David Menting for letting me play with it a bit and report some first impressions.

The display is quite simple, yet still nicely readable:


Power consumption, without any load is 0.07W when turned off with the power switch on the back of the unit, and about 2.0W when turned on, without load.

There are two (optical encoder) rotary knobs to adjust the maximum voltage and current, respecively, over a range of 0.7 .. 36V and 0 .. 3A – i.e. plenty for most situations.

A drawback with this particular supply, is that it’s a bit of a hack to pre-set the current limit. You can turn it down during use to reduce the limit to the point where the voltage starts dropping, but if you want to set it up ahead of time, then the way to do it is to short out the power supply while adjusting the knob, to see the value it displays.

This is a switched-mode power supply, which explains why it is so small and requires only a small fan, but it does lead to some residual noise on the output. Hooking it up to the scope, still under no-load conditions, you can see that the output varies between about +10 mV and -20 mV of “ripple” above and under the preset value, respectively (5V in this case):


To produce this image, the scope was set to “peak-detect” mode, which captures the high and low value at each point in time, and the trace is drawn in “envelope” mode, which is a variation of persistent display showing the most extreme values ever reached as long as the scope is kept on. The last trace is the one shown in the middle, while the top and bottom lines are the largest variations ever reached – I think I left it running for a minute or so.

It’s a pretty good result, actually, for a switching power supply. Variations of this kind should not cause any problems for most digital circuits, which usually can tolerate fairly large variations on the power supply lines.

Three laws – part 3

In Hardware on Apr 29, 2013 at 00:01

Yesterday’s post was about Kirchoff’s Current Law.

Now let’s look at the second version of that law, Kirchhoff’s Voltage law:

JC's Grid, page 72

This law says something about the voltages in a circuit: voltages cancel out, regardless of which way you follow a path in the circuit, i.e.:

V1 - V2 - V3 = 0    (hence V1 = V2 + v3)

But also:

V3 - V4 - V5 = 0    (hence V3 = V4 + V5)


V1 - V2 - V4 - V5 = 0    (hence V1 = V2 + V4 + V5)

The signs are based on how the arrows are drawn: voltages are always relative, so it all depends on the direction of each of those arrows. Since V2 and V3 are opposite to the direction of “following” a single path through the circuit, they end up negative in the above equation. Same for V4 and V5.

Now this version of Kichhoff’s law is really useful. If each of the resistors has the same value (it doesn’t matter what the value is), then we can reason about the voltages in different parts of the circuit, even when only V1 is known at the start.

Let’s look at this circuit again in terms of voltages, with V1 set to 3V:

JC's Grid, page 72

Since V1 = V2 + V3, and all resistors have the same value, it’s a safe guess that the voltage over both will be the same. A more exact reasoning is to include the currents, and since they are the same for the two resistors on the left, we know that the voltages will be the same (Ohm’s law, regardless of the value of these resistors – as long as they are the same).

Perhaps a bit more surprising is that V4 is 0V. This is again a consequence of Ohm’s law: since the pins on the right are not connected, no current will flow. With no current, the voltage over a resistor is always 0V (Ohm’s law, again).

Since V4 is 0V, and V2 = V4 + V5 as we saw above, V5 must be 1.5V, the same as V2.

Note how a voltage over a resistor can be zero, and over two open pins can be non-zero.

If we use a multi-meter with a really high input resistance (any modern multi-meter will do, usually they have 10 MΩ or more), to make sure no current starts flowing, then we can indeed measure the voltage to be half of the 3V between top and bottom, i.e. 1.5V.

The main point to take away from this, is that the three laws are intimately inter-connected. Given known resistors, which is often the case, Ohm’s law dictates the relationship between voltage and current, and with the two variants of Kirchhoff’s law, it is usually possible to “reason your way into” the different points in a circuit, given an externally applied power supply.

One last example: what is the equivalent resistance of two resistors of 1000 Ω in parallel? Well, say we apply voltage V1, then the current through one resistor will be V1/1000. The same current will go through the other resistor, since it too has V1 on both pins. So the total current of the two in parallel will be 2 x V1 / 1000 = V1 / 500. What is the resistance when we apply V1 and see a current V1 / 500 flowing? Again, Ohm’s law: R = U / I, i.o.w. R = V1 / (V1 / 500) = 500. So the general answer is: the equivalent resistance of two resistors in parallel is half their resistance. A direct consequence of Ohm’s and Kirchhoff’s laws.

Get used to the three laws, and with a bit of practice, the rest should fall into place!

Three laws – part 2

In Hardware on Apr 28, 2013 at 00:01

After Mr. Ohm yesterday, let’s see what Mr. Kirchhoff has to say.

Kirchhoff’s First Law is about current. In a nutshell – what goes in, must come out:

JC's Grid, page 72

I’ve drawn resistors, but that’s really irrelevant here – each component could be anything.

The current I1 must be the same as the current I2, It does not matter what I3 is in this case (assuming the two pins next to it are not feeding or drawing any outside current).

Also, if I3 is zero, i.e. if the pins are not connected to anything at all, then the current through the leftmost resistors is identical. In this case, they are essentially in series, with the resistor on the right not doing anything at all (no current = no voltage drop = Ohm’s Law).

This makes it possible to reason about that point in the middle, where the three resistors meet. The currents at that point must cancel out: that’s what Kirchhoff’s Current Law says.

Suppose all three resistors are 1 kΩ, and the current I1 is 1 mA:

JC's Grid, page 72

If the two pins on the right are left open, no current will flow there. So the same current I1 (which is also the same as I2) will flow through both resistors on the left. Total voltage drop from top to bottom will be 1 mA x 1 kΩ = 1V on the top resistor and another 1V on the bottom one, for a total voltage of 2V across the left two pins.

Or to put it more practically: if you place a 2V supply across those left two pins, then 1 mA will flow. The voltage in the center point will be halfway, i.e. at 1V.

What will happen when we short the pins on the right?

JC's Grid, page 72

Again, there’s 1 mA flowing in from the top, so there will be 1 mA coming out the bottom. The bottom-left and right resistor will together see a current of 1 mA going through them. Since they are both the same 1 kΩ, it should not come as a surprise that each resistor will get half the current, i.e. 0.5 mA each. Total voltage from top to bottom will be 1.5V.

This isn’t such a great example in terms of practical use, since normally the reasoning goes the other way around: what current will flow when I apply voltage X to the entire circuit?

That’s the other version of this law, described tomorrow.

Three laws – part 1

In Hardware on Apr 27, 2013 at 00:01

It’s always a puzzle to predict just what will happen when you hook up some components.

But as mentioned in the what-if series, it’s really useful to be able to do so, to avoid surprises and damage. Also for questions such as: Why does a higher voltage cause a higher risk of damage in one case and less so in the other? Why do I need a resistor in series, and of what value? What if I don’t have the right resistor or a different voltage?

Lots of complex issues, but the simplest and most important case usually is static analysis and DC (direct current) voltages, i.e. when only one or two states are involved, and not so much the switching and AC (alternating current) behaviours.

You just need to get familiar with three “laws” of electricity:

  • Ohm’s Law – given two of: voltage, current, and resistance, we can predict the third.
  • Kirchhoff’s Current Law – current always adds up: what goes in, must come out.
  • Kirchhoff’s Voltage Law – how voltages “spread” across interconnected components.

That and learning what the basic behaviour is of resistors, diodes, capacitors, etc, and you’ve got all the knowledge you need to “see” what a circuit does, before even building it and trying it out. And by this I really mean literally “visualising” what is going on – it only takes a little practice to turn this into a very intuitive skill.

You just have to grasp the essence of those three laws. So let’s get on with it, eh?

Ohm’s Law

This is by far the most important one. It says everything about resistance. The unit of resistance is – not surprisingly – the “Ohm”: a resistor of 1 Ohm will let 1 Ampère of current flow when you apply 1 Volt of electric potential difference over it. The formula is:

U = I x R

(U = voltage, I = current, R = resistance)

Same law, different uses, by simple algebraic manipulation: I = U / R, and R = U / I. If you know two of the units, you can calculate the 3rd.

  • What happens when I touch both poles of a car battery?
    My skin resistance will be some 100 kΩ, so I = U / R = 12 / 100,000 = 0.000,12 A = 120 µA. A tiny current, I wouldn’t sense a thing, so the answer is: “nothing happens”.

  • What happens when I place a metallic nail across that same 12V car battery?
    Let’s say the resistance of that nail is 0.1 Ω, i.e. almost a short, so I = U / R = 12 / 0.1 = 120 A, a huge amount of current. The nail will heat up like crazy, it might even melt!

Same battery, very different outcomes.

And it’s not just useful to predict such extreme cases. It also helps understand why some hookups are inherently safe: if my power supply delivers no more than 5V, and I am playing with resistors of 1 kΩ or more, then no matter how I hook things up (apart from shorting things out), there will never be more than I = U / R = 5 / 1,000 = 0.005 A = 5 mA of current through my circuit. A 1 kΩ resistor in series with just about any component will “limit” the current to 5 mA, which virtually prevents damage to just about any component.

Another example: suppose I am using a 12 V power supply, and want to turn on an LED with it. Most LED’s glow nicely at 20 mA. So if I put a R = U / I = 12 / 0.020 = 600 Ω resistor in series with the LED, I can be certain that the LED won’t get damaged. Better still, if I only have a 1 kΩ = 1000 Ω resistor, I can predict that it’ll probably work just fine in this same circuit as the current will be at most I = U / R = 12 / 1,000 = 12 mA. Using a 100 Ω resistor would be a bad idea (max current 120 mA), and using a 10 kΩ resistor probably also wouldn’t work (1.2 mA might be too little to make the LED light up).

These are all approximations, but they are extremely useful – even as such.

Some consequences of the simple ” U = I x R ” formula Georg Simon Ohm gave us:

  • twice the voltage => twice the current
  • twice the resistance => half the current
  • twice the current => twice the voltage drop

One thing to take away from all this, is that it’s not a bad idea to buy some 100 Ω, 1 kΩ, and 10 kΩ resistors. Having a bunch of spare resistors can often come in handy, as a way to limit the current (and hence avoid damage), and these three values are often all you need to try out a few things in circuits running at 1.5 .. 12V.

Tomorrow, we’ll give the stage to Gustav Kirchhoff!

What if I mix 3.3/5.0V – part 3

In Hardware on Apr 26, 2013 at 00:01

Welcome to the weekly What-If series, also available via the Café wiki.

There’s still much more to say about all this – as can be expected. One of the suggestions made in the comments was to use a few diodes in forward direction. Since these each have about 0.65V drop, three of them ought to bring down the voltage from 5.0V to 3.3V.

Time to hook up the signal generator and scope again. Be prepared for some surprises!

As I mentioned in my original post, the “official” way to handle this, is to use a “level converter” chip, which is based on some active circuitry, i.e. some transistors or MOSFETs. The resistor solutions described yesterday are less accurate, as you’ll see…

Here is a 1 MHz square wave @ 5V (yellow) feeding yesterday’s 4.7 kΩ + 10 kΩ resistive voltage divider to produce a 3.3V signal (purple):


That’s quite a different signal coming out! A typical capacitive charge / low-pass effect.

Note that the signal generator has a ≈ 10 nS rise-time, i.e. the edges are not completely vertical, and that the Hameg scope probe has a 14 pF loading capacitance, according to the specs. So some of these effects are artefacts of this measurement setup.

Let’s raise the frequency to 10 MHz (the horizontal scale is now a very fast 10 ns/div):


Hardly a square wave, and no well-defined 0-to-3.3V transitions either, as you can see.

Now let’s try this circuit:

JC's Grid, page 71

The reasoning being that the diodes will “drop” the voltage from its high 5V level to 3.3V:


Quite an asymmetric effect, although lowering the resistor to 1 kΩ ought to improve it.

The above is again a 1 MHz square wave input, and here’s the same at 10 MHz:


This final setup is so far off the desired voltage levels that it probably won’t work.

Given these outcomes, I’m inclined to stick with a single 10 kΩ resistor in series. Or perhaps drop it to 1 kΩ to get better rise and fall times. Active MOSFET-based level-shifting circuits are starting to make a lot more sense now – they exist for a reason!

What if I mix 3.3/5.0V – part 2

In Hardware on Apr 25, 2013 at 00:01

Welcome to the new weekly What-If series, also available via the Café wiki.

Yesterday’s post was about mixing 3.3V logic levels with 5V logic levels.

The real trouble, apart from things not working, is damaging some component, of course. And in electronics, almost all damage comes from overheating. At some point (well over 100 °C, usually), chips really do get “fried” and irrevocably damaged occasionally.

The trick is to avoid overheating. This in often easy to do by limiting the current that can flow. How do you limit the current? Simple: make sure there are some resistors in series!

But first, let’s examine what’s really going on when we connect a 5V logic “1” to an input which handles only 3.3V logic levels:

JC's Grid, page 71

On the left, the typical circuit inside a digital chip at each I/O pin. The essential ingredients are two ESD protection diodes to VCC and GND, respectively, and the MOSFET input gate.

The diodes are placed in the “blocking” direction, i.e. they normally do nothing but block the current. The MOSFET gate is a very high impedance input, essentially it acts just like a capacitor, in fact. The rest of the input circuitry can be ignored for our purposes.

So normally, input signals on such a pin just “float” and follow whatever voltage is applied. Until the voltage is too high, or too low, that is.

On the right, I’ve redrawn the exact same circuit, but “lifting” the input well above VCC level. To make this clear, I’ve drawn the input pin above the VCC pin, as an intuitive way to represent voltage as height in the diagram.

Now you can see what happens above VCC + 0.65 or so: the top diode will start to conduct in this situation. So if the input is set to 5V, and VCC is 3.3V, then the diode will become a conductor and try to pull either 5V down or 3.3V up, to reach equilibrium again.

This is a danger sign. The amount of current flowing will rise as long as these voltages are more than about 0.65 V apart. If both power supplies are very strong, there could be several amps of current – the 5V supply could even start powering the 3.3V circuit!

And that’s where the damage-through-heat comes in: a 1 A current over a diode with a 0.65 V difference leads to 0.65 W of heat being dissipated. Far more than these tiny on-chip diodes can handle. In fact, they really are only designed to carry up to a milliamp or so. The result: a “blown” on-chip diode, and since the voltage difference continues to exist, the damage will probably propagate to other diodes and components on the chip.

There are a few ways to prevent this. One is to use special “level-converter” chips, designed to take one voltage on the input, and then generate a different voltage on the output.

But there are much simpler ways to deal with such small differences, especially when the pins are all close together and on the same board:

JC's Grid, page 71

On the left, a traditional 4.7 kΩ + 10 kΩ resistive “voltage divider”, which takes one voltage and divides it down to a lower voltage. This works, but in this case we can even get by with a single resistor, as shown on the right.

The reason the single resistor works, is that current will start to flow through the diode as before, but now also through the 10 kΩ resistor. Since the circuit seeks equilibrium, in this case, there will be about 5.0 – 3.3 = 1.7 V across resistor + diode, i.e. ≈ 1 V across the resistor. With a 10 kΩ resistor, it only takes 0.1 mA of current to generate a corresponding voltage drop, so that’s when equilibrium will be reached. These current levels are completely harmless and can easily be sustained by the on-chip diode.

So there you have it: we’ve explained why direct connections might lead to overheating, and how a 1..10 kΩ resistor in series can prevent it, while still allowing the circuit to work.

All this required, was some theory and a basic understanding of the internal circuitry.

PS. These resistor solutions are sensitive to noise and capacitive loading (they act as low-pass filters), so this only works well when signal lines are short, a few cm or so. For reliable high-speed signaling over longer distances, a level-shifter chip would be a better way to go.

What if I mix 3.3V and 5V?

In Hardware on Apr 24, 2013 at 00:01

Welcome to the new weekly What-If series, also available via the Café wiki.

As you probably know, mixing 3.3V logic with 5V logic is usually a bad idea, but why and what if you need to do it anyway?

This topic is too complex for a single post, but let’s just start and see where it leads to:

  • VCC in these examples is the supply voltage, i.e. either 3.3V or 5.0V, depending on which chip we’re looking at. The goal being to connect a mix of these.

  • Digital I/O pins work by treating everything below a certain voltage as “0” and everything above another voltage as “1”. In between, the levels are undefined and could be interpreted either way, depending on temperature, stray capacitances, moon phases, who knows…

  • For the ATmega328, for example, everything between -0.5V and 0.3 x VCC is treated as “0” and everything between 0.6 x VCC and VCC+0.5V as “1”.

  • For 5V signals, that translates to: under 1.5V is “0” and over 3.0V is “1”, respectively.

  • Also relevant, is that unloaded output pins tend to be very close to the 0V and VCC ground and power supply levels, respectively.

So mixing is in fact not a problem at all for the following scenario: 3.3V levels for output signals, tied to 5V levels for the input signals. If all you need, is to read logic levels on say a 5V Arduino Uno from a 3.3V JeeNode, then just tie the signals together and you’re all set.

Connecting a 3.3V level output pin to a 5V level input pin works fine.

The problem occurs in the other direction: a “1” output on a 5V logic level is about 5V, whereas the maximum allowed input level for a “1” on a chip powered by 3.3V is 3.3+0.5, i.e. at most 3.8V.

Hooking these together without taking care of the problem will lead to problems, such as overheating chips and even smoke or fire (although this is almost impossible with simple battery- or USB-powered hookups).

Tomorrow I’ll describe the cause of these problems, along with some simple solutions.

New series – What If?

In Hardware, News, Software on Apr 23, 2013 at 00:01

Questions are very useful: “what would happen if…” is the foundation of science, after all.

Conjectures and Refutations is a famous book by the late philosopher Sir Karl Popper. I could not possibly summarise it (heck, I haven’t even read it), but what I take away from what I’ve read and heard about it, is that theories can be judged on their predictive value. A theory in itself is no more than an intellectual exercise, but its real value lies in being able to apply it to what-if questions. The stronger a theory, the better it should predict outcomes. The way to “refute” a theory, is to come up with an example where it fails. Rinse and repeat, and you’ve captured the essence of science.

Want to predict what will happen when you place a 100 Ω resistor across a 9V battery? That’s easy, given the proper theory: take Ohm’s Law (i.e. a theory which has stood the test of time), and apply it – a current of ≈ 11 90 mA will flow. Actually a bit less due to the internal resistance of the battery, which goes to show how strong theories can be refined further, leading to even more accurate predictions.

The what-if question is a great way to experiment, especially in electronics and electro-mechanics, because it lets you be prepared and avoid silly (and sometimes catastrophic) outcomes, such as a damaged component, a harmful burn, or even an explosion.

This approach lends itself to all sorts of practical questions:

  • What if I short out a 3x AA battery pack?
  • What if I connect my chip the wrong way around?
  • What if I have to use a 12V power supply instead of 5V?

But also issues as varied as:

  • What if I omit a certain component from my circuit?
  • What if I unplug the Raspberry Pi without shutting it down?
  • What if I wanted to use HouseMon in combination with MySQL?

Properly phrased, what-if questions are essential for practical experiments, and – by extension – also the key to building useful circuits and automated installations.

A useful variation of the what-if question is to help predict “bad” outcomes and estimate the risk of an experiment, such as: can shorting out my power supply cause real damage?

Starting tomorrow, I’m launching a new series on this weblog, titled “What-If Wednesday”. As far as I’m concerned, it can run as long as there are interesting questions I can answer, so please feel free to suggest lots of topics in the comments below. These weekly posts will be tagged What-If, and I’m also setting up a new wiki page to collect them all.

Sooo… please help me out folks, and send in some nice what-if questions!

Winding down

In News, Musings on Apr 22, 2013 at 00:01

The JeeDay 2013-04 event is over.

I would like to warmly thank the 40 or so people who attended on Friday and Saturday. It is clear to me from the kind follow-up emails that the event was appreciated by many of you and I really hope that everyone got something useful and stimulating out of this.

Allow me to also thank the “anonymous sponsor” at this point for funding the venue, the coffee and drinks, and Saturday’s lunch. I’ve passed on your and my appreciation, and it has gratefully been accepted. As several people have pointed out, this whole concept of an anonymous sponsor is really a contradiction in terms, so let’s all just cherish the fact that philanthropy (and mystery) still exists, even in today’s western societies.

This is probably the point where I’m expected to write sentences full of superlatives, self-congratulatory remarks, let’s-conquer-the-world type of pep-talk, congratulations for the speakers and their choice of interesting topics, all sorts of grandiose plans, and where I’d also describe how stimulating all the discussions on the side turned out to be.

I could, and it’d be true. But I won’t…

Instead, I’d like to give this a somewhat different (personal / philosophical) twist.

We’re focused on success. We crave rewards. We seek recognition. So when something good (for some definition of “good”) happens, we want to take it further.

Again. Better. More.

Yet to me, that’s not what JeeDay was about. Sure, we could do it again. In fact, I’d love to and I’ve even sort-of committed to organising another JeeDay a year from now. We’ll see.

But to me, JeeDay is not about the next step or some future trend. It’s about this event we just had. Some 10 talks from people describing what they like to do in their free time. That’s quite a special situation, when you stop and think about it: here we all are, a few dozen geeks with a common techie interest, and this what we choose to spend our time, our creative energies, and our money on. We could do anything, yet this is what we want to do. In. Our. Free. Time.

Now of course, everyone’s reasons will differ. But to me, it’s pretty amazing: there’s rarely a financial reward (heck, it usually costs money!). There’s often not much recognition. These are not TED talks, we’re not working on some high-visibility successful project and showing the world. We just tinker in private, we come up with stuff, we learn, and we like doing it.

In my view, this is about the top two tiers of Maslow’s hierarchy of needs:


The basic idea being that you can’t really get to focus on the levels above before the levels underneath have more or less been covered.

This is – again, in my perception – not about success, and probably not even about peer recognition, but about the intrinsic fun of discovery, invention, creation, and problem-solving. And about finding out how others deal with this. It’s no accident that most of it happens as open source, either: open source (hardware + software) and sharing is what floats to the top when the intrinsic puzzles and their solutions dominate.

In a world where so much is about ownership, money, and time, I think that’s precious.

I hope JeeDay has helped you find and follow your passion. Everything else is secondary.

PS. The mystery topic in my presentation was JeeBoot – more to follow soon.

Cheap power analysis

In Hardware on Apr 21, 2013 at 00:01

Remember this screen shot?


It was a carefully captured analysis of the power consumption of a JeeNode, running the RoomNode sketch, and sending / receiving wireless RFM12B packets. There’s a fantastic amount of info in there, to help understand which part of the code and which activity is drawing the most power. It was a great help at the time to reduce power consumption, allowing these nodes to run well over a year on a couple of AA batteries.

Trouble with this, is that you need an expensive piece of equipment, called an oscilloscope. Long-time readers might remember that I’ve written extensively about this. These things cost anywhere from a few hundred Euro, to thousands, or even tens of thousands for high-end units. I ended up settling for a Hameg HMO2024, which is a great instrument, but with a pretty hefty price of well over €1000.

So how would you go about analysing the power consumption of your sketch without plunking down this sort of cash? Well, there really are not that many alternatives, you have to see the current-vs-time graph to be able to understand what’s going on.

Luckily, there is a fairly capable little unit from Gabotronics, called the Xminilab. It pushes an ATXmega (note the “X”) to its limits, allowing it to capture quite a bit of information, just like its bigger brothers. It even includes things like FFT analysis, an 8-channel Logic Analyser, and an AWG Signal Generator! Last but not least, the software is open source.

Interested in how capable this $69 device is? Well, check this out:


Do you recognise the waveform? The Xminilab has captured a packet transmission, a bit like the one shown at the start of this post (it’s a different sketch, i.e. radioBlip2, hence a different pattern). It may not look like much, but it should be sufficient to see the effect of changes in the code and to optimise power consumption with it.

So, do you need a scope? IMO, anyone wishing to explore electronics should have one. Whether second-hand or the above-mentioned Xminilab, it really helps to be able to see things in a way where our human senses fall short – such as these brief events. It’s the most versatile instrument in the lab, if you ask me – even with a 128×64 pixel LCD screen.

PS. I don’t recommend the even lower-cost $49 Xprotolab (which I also have). It has the same functionality, but with its tiny OLED display it really is too hard to read, IMO.

Automated + manual 230V

In Hardware on Apr 20, 2013 at 00:01

(This is based on an idea by Ard Jonker, who planted a seed and watched the coin drop, if you excuse the mixed metaphors…)

The recent experiment with direct relay switching suggests that it is possible to switch a latching relay with just two I/O pins tied together plus a 100 µF cap.

The thing about that circuit, is that it draws no current in either ON or OFF state – it only needs a little energy to change the state.

That means it could in principle be powered by a very low-power source, such as this other fun projects I had to shelve a while back. The reasons for this remain as valid as ever: I can’t realistically turn this into a safe kit, given the direct connection to AC mains. So while the thought of having 80 of these sprinkled around the house and consuming under 1 Watt total is a tempting thought, it just isn’t going to happen.

Which doesn’t prevent ME from using it anyway, of course…

Ok, now let’s bring a couple of components together:

  • a JeeNode Micro
  • a 12 mW AC mains supply
  • a directly-powered latching relay
  • a toggle switch
  • power cabling

Here they are, with a nice plastic case (whoops, forgot to include the JNµ, oh well):


The toggle switch is the small but essential ingredient here. Let me explain:

JC's Grid, page 70

This is a switch which can be operated manually and remotely. Flipping the switch or the relay has the same effect: toggling power, regardless of the state of the other component!

This means it can be operated even when the automated system is off or disconnected, or has crashed. And likewise, the power remains under remote control regardless of the state of the manual toggle switch. This solves a key problem with all those cheap remote power switches out there: the necessity to find the remote, because there is no local switch anymore. And the fact that it breaks down when the home automation system fails.

Given the relay used, I doubt that this solution will be able to control more than 30..50 W, but there are plenty of such devices around the house these days, even LED lighting.

I think I’m gonna have to start messing with AC mains again… with caution, of course.

See you in Houten!

In News on Apr 19, 2013 at 00:01

Psst, did you know? We’re getting together…

Here’s the location, right next to the railway station:


If you prefer to come by car, set your driving directions for Kamillehof in Houten to find a parking spot (or use the parking garage on the other side of the railway). Note that there is essentially only one way to get there:


Any attempt to drive differently will probably lead you to a bicycle path or one-way street…

JeeDay flier

Bring your pet projects (not the pets, please), bring your curiosity, and bring your energy. Let’s have some techie fun, eh?

(9)50 days and counting

In AVR, Hardware, Software on Apr 18, 2013 at 00:01

The new JeeNode Micro has joined the ranks of the test nodes running here to determine battery life in real time. The oldest one has been running over 2 and a half years now:


That LiPo battery has a capacity of 1300 mAh, and since it’s still running, this implies that the average current draw must be under 1300/24/950 = 57 µA, including self-discharge.

The other two battery tests now running here are based on the new JeeNode Micro v3, i.e. this one – using a boost converter:


… and another one using a coin cell. Here’s a picture of a slightly older version:


The boost version is running off a rechargeable AA battery, of type Eneloop, which I use all over the house now. These batteries hold 1900 mAh, but there’s a substantial penalty for running off one AA cell with a boost converter:

  • energy is not free, i.e. drawing 10 µA at 3.3V means we will need to draw 30 µA at 1.1V, even if we could use a 100% efficient boost converter
  • real-world efficiency is more like 70..80% at these levels, so make that 40..45 µA
  • the boost converter has some idle current consumption, probably 10..20 µA, so this means we’ll draw 50..65 µA even if the JeeNode Micro only uses up 10 µA at 3.3V (actually, it’s 3.0V in the latest JNµ)

This would translate to 1900/.065/24/365, i.e. ≈ 3-year life. Or perhaps 2, if we account for the Eneloop’s 85% per year self-discharge.

The coin cell option runs off a CR2032 battery, which is rated at about 225 mAh, i.e. considerably less than the above options. Still, since there are no boost converter losses, this translates to 225/0.010/24/365, i.e. ≈ 2.5 years of life if the JNµ draws only 10 µA.

These figures look excellent, but keep in mind that 10 µA average power consumption is a very very optimistic value, particularly when there are sensors hooked up and packets are being sent out. I’d be quite happy with a 6-month battery life for a single AA or a coin cell.

For reference, here is an earlier post about all these power calculations.

Here are the current reports I’m getting via HouseMon:

Screen Shot 2013-04-17 at 11.12.08

That’s just about 50 days off a coin cell. Let’s see how it holds up. The nice bit in these tests is that the new nodes now report several different voltage level measurements as well (this also consumes some energy!). They haven’t dropped much yet, but when they do, I hope that we’ll be able to use the drop as a predictor.


Electro:camp 13.04

In News on Apr 17, 2013 at 00:01

As if one event is not enough – here’s a reminder for another one coming up soon:

This the next one in a series of bi-annual meetings, from people in Germany, Belgium, the Netherlands, the UK, and beyond – this time in Kaiserslautern @ the Fraunhofer institute.

See the wiki for further details. Don’t forget to register if you wish to attend.

As I see it, Electro:camp is more focused on electricity metering and monitoring, whereas JeeDay – which has yet to define itself, clearly – will be more of a “maker/hacker” style event with focus on DIY home projects, low-power wireless, and electronics as a hobby.

JeeDay next Fri+Sat

In News on Apr 16, 2013 at 00:01

(whoops, this went out one day earlier than planned… before it was ready)

Just as reminder, here’s the announcement (as you can see, I’m no graphics designer):

JeeDay flier

And this is the draft time schedule for the entire event:

JeeDay 2013-04

For more information, see the JeeDay wiki page.

PS – The is no pre-registration, everyone is welcome. The venue is large enough to accommodate some 25 people on Friday and about 35 on Saturday, which matches more or less the interest on the original list.

Energy savings…

In News, Musings on Mar 31, 2013 at 00:01

Here’s a view of the solar energy production earlier this week here at JeeLabs:

Screen Shot 2013-03-26 at 18.49.36

Best day so far … 19.3 kWh in one day: that’s some 2.5 x our average daily consumption!

This graph was made with HouseMon, which is still in the early stages but I’m viewing it on a daily basis – the Status and Graphs pages are already quite practical. Then again, it’s a constant reminder that the progress of this project is considerably slower than I had hoped when I started out. One reason for that is that I’m still hesitant to make some major design decisions – mostly because I don’t have enough experience and don’t feel confident enough with Node.js and CoffeeScript yet. So many things still feel awkward :(

Speaking of insufficient progress… it’s time to switch off:

wallpaper power symbol green

(image by TheBigDaveC, as found on this site)

I’m going to take a brief break, and interrupt this daily flow of weblog posts for a while.

It has happened before, and it will happen again: I want to clear my head and focus on some projects which take a bit more concentration than I seem to be finding these days.

But no worries: this daily weblog will resume before JeeDay (April 19 + 20), so there will still be enough time to get the latest info and news out to you.

Soooo… see you then, and more importantly: see you there !

(Pssst… in case you haven’t seen this… let that nano stuff inspire you… pretty amazing!)

Sharing Node ID’s

In Software on Mar 30, 2013 at 00:01

The RF12 driver has a 5-bit slot in its header byte to identify either the sender or the destination of a packet. That translates to 32 distinct values, of which node ID “0” and “31” are special (for OOK and catch-all use, respectively).

So within a netgroup (of which there can be over 250), you get at most 30 different nodes to use. Multiple netgroups are available to extend this, of course, but then you need to set up one central node “listener” for each netgroup, as you can’t listen to multiple netgroups with a single radio.

I’m using JeeNodes all over the place here at JeeLabs, and several of them have been in use for several years now, running on their own node ID. In fact, many have never been reflashed / updated since the day they were put into operation. So node ID’s are starting to become a bit scarce here…

For testing purposes, I use a different netgroup, which nicely keeps the packets out of the central HouseMon setup used for, eh… production. Very convenient.

But with the new JeeNode Micro, I actually would like to get lots more nodes going permanently. Especially if the coin cell version can be made to run for a long time, then it would be great to use them as door sensors, lots more room nodes, and also stuff outside the house, such as the mailbox, humidity sensors, and who knows what else…

Now the nice thing about send-only nodes such as door sensors, is that it is actually possible to re-use the same node ID. All we need is a “secondary ID” inside the packet.

I’ve started doing this with the radioBlip2 nodes I’m using to run my battery lifetime tests:

Screen Shot 2013-03-26 at 10.11.14

(BATT-0 is this unit, BATT-1 is a JNµ w/ coin cell, BATT-2 is a JNµ w/ booster on 1x AA)

This is very easy to do: add the extra unique secondary ID in the payload and have the decoder pick up that byte as a way to name the node as “BATT-<N>”. In this case, it’s even compatible with the original radioBlip node which does not have the ID + battery levels.

Here is the payload structure from radioBlip2, with secondary ID and two battery levels:

struct {
  long ping;      // 32-bit counter
  byte id :7;     // identity, should be different for each node
  byte boost :1;  // whether compiled for boost chip or not
  byte vcc1;      // VCC before transmit, 1.0V = 0 .. 6.0V = 250
  byte vcc2;      // battery voltage (BOOST=1), or VCC after transmit (BOOST=0)
} payload;

And here’s the relevant code snippet from the decoder in HouseMon:

  decode: (raw, cb) ->
    count = raw.readUInt32LE(1)
    result =
      tag: 'BATT-0'
      ping: count
      age: count / (86400 / 64) | 0
    if raw.length >= 8
      result.tag = "BATT-#{raw[5]&0x7F}"
      result.vpre = 50 + raw[6]
      if raw[5] & 0x80
        # if high bit of id is set, this is a boost node reporting its battery
        # this is ratiometric (proportional) w.r.t. the "vpre" just measured
        result.vbatt = result.vpre * raw[7] / 255 | 0
        # in the non-boost case, the second value is vcc after last transmit
        # this is always set, except in the first transmission after power-up
        result.vpost = 50 + raw[7] if raw[7]
    cb result

In case you’re wondering: the battery voltage is encoded as 20 mV steps above 1.0V, allowing a single byte to represent values from 1.0 to 6.1V (drat, it’ll wrap when the boost version drops under 1.0V, oh well…).

Note that this trick works fine for send-only nodes, but it would need some additional logic in nodes which expect an ACK, since the ACK will be addressed to all nodes with the same ID. In many cases, that won’t matter, since presumably all the other nodes will be sound asleep, but for precise single-node ACKs, the reply will also need to include the extra secondary ID, and all the nodes will then have to check and compare it.

So now it’s possible to re-use a node ID for an entire “family” of nodes:

  • over 250 different netgroups are available for use
  • each netgroup can have a maximum of 30 different node ID’s
  • each node ID can be shared with over 250 send-only nodes

That’s over 1.8 million nodes supported by the RF12 driver. More than enough, I’d say :)

JeeNode Micro start-up power

In AVR, Hardware on Mar 29, 2013 at 00:01

The JeeNode Micro v3 includes a P-channel MOSFET to control power to the RFM12B radio. This isn’t just a new gimmick – the goal was to “fix” the RFM12B wireless radio’s startup power consumption, which can prevent an ultra-low power source from ever building up a high enough supply voltage for a JeeNode to start up.

Now that the JNµ is in production, it’s time to measure how well such an approach works. Get ready for a bunch of scope screenshots, all based on the same circuit as before:

JC's Grid, page 51

… except that now the entire JeeNode Micro is in there, and I’m using a 10 Ω resistor.

I’ll be applying a 1 Hz ramp signal going from 0.0 to 3.0V using the power booster behind a signal generator, to see exactly what amount of current is being drawn. In all the images below, the yellow trace is the input voltage (i.e. a simulated power supply), and the blue trace is the voltage over the 10 Ω resistor – that means 1 vertical division on the blue trace corresponds to 0.5 mA when the display shows 5 mV/div:


The above image is just a baseline: a simple blink sketch which never enables the radio, and which then toggles some I/O pins every 500 ms. As you can see, the ATtiny84 comes out of power-on reset at about 1.4V and ends up drawing about 3.5 mA at 3.0V.

Fuses are set to low=C2 high=D7 ext=FF, i.e. BOD disabled, startup asap on RC @ 8Mhz.

Now let’s look at the same setup with the JNµ running radioBlip2.ino:


This time, the sketch enables the MOSFET to power up the radio, measures the battery voltage, tries to send out a packet (this will fail at 1.4V), and goes into deep sleep. A short but very quick (and high!) blip before power consumption drops to almost zero.

The third measurement is with a sketch doing nothing but powering down right away:

#include <JeeLib.h>

void setup () {
    cli(); // disable all interrupts

void loop () {}

Which produces this result:


I’ve bumped the scope sensitivity up to its maximum of 1 mV/div (i.e. 100 µA/div) and am now adding a lot of averaging to try and keep the displayed noise levels low. The “blip” is the ATtiny getting out of reset and powering down completely.

As last test, I repeated the above, but now using a sweep of 10 s (0.1 Hz), and filtering the signal through the lowest low-pass setting available, i.e. 5 Hz. This loses the important spike at 1.4V, which is of course still there, but improves the readout of the baseline:


As you can see, the power consumption now never rises above ≈ 60 µA – that’s a ten-fold improvement over what we get with the RFM12B connected to power in the standard way.

The shape of this curve is quite interesting: it’s essentially resistive (since it’s more or less linear), but the current starts at 1.2V, i.e. after overcoming two extra diode drops.

This is the power-up “hump” which any ultra-low power supply based on solar cells or other energy harvesting techniques will need to overcome, so that the ATtiny can switch itself into power down mode and let the supply voltage rise further.

I think that’s an excellent result, and am looking forward to trying a few things out!

Software development

In Software, Musings on Mar 28, 2013 at 00:01

As you probably know, I’m now doing all my software development in just two languages – C++ in the embedded world and JavaScript in the rest. Well, sort of, anyway:

  • The C++ conventions I use are very much watered down from where C++ has been going for some time (with all the STL machinery). I am deliberately not using RTTI, exceptions, namespaces, or templates, because they add far too much complexity.
  • Ehm… C++ templates are actually extremely powerful and would in fact be a big help in reducing code overhead on embedded processors, but: 1) they require a very advanced understanding of C++, 2) that would make my code even more impenetrable to others trying to understand it all, and 3) it can be very hard to understand and fix templating-related compiler error messages.
  • I also use C++ virtual functions sparingly, because they tend to generate more code, and what’s worse: VTables use up RAM, a scarce resource on the ATmega / ATtiny!
  • As for programming in JavaScript: I don’t, really. I write code in a dialect called CoffeeScript, which then gets compiled to JavaScript on-the-fly. The main reason is that it fixes some warts in the JavaScript syntax, and keeps the good parts. It’s also delightfully concise, although I admit that you have to learn to read it before you can appreciate the effect it has on making the essence of the logic stand out better.
  • There is an incredible book called CoffeeScript Ristretto by Reginald Braithwaite, which IMO is a must read. There is also a site called which appears to have the entire content of that book online (although I find the PDF version more readable). Written in the same playful style as The Little Schemer, so be prepared to get your mind twisted by the (deep and valuable) concepts presented.
  • To those who dismiss today’s JavaScript, and hence CoffeeScript, on the basis of its syntax or semantics, I can only point to an article by Paul Graham, in which he describes The Blub Paradox. There is (much) more to it than meets the eye.
  • In my opinion, CoffeeScript + Node.js bring together the best ideas from Scheme (functions), Ruby (notation), Python (indentation), and Tcl (events).
  • If you’re craving for more background, check out How I Learned To Enjoy JavaScript and some articles it links to, such as JS: The Right Way and Idiomatic JavaScript.

I’m quite happy with the above choices now, even though I still feel frustratingly inept at writing CoffeeScript and working with the asynchronous nature and libraries of Node.js – but at every turn, the concepts do “click” – this really feels like the right way to do things, staying away from all the silliness of statically compiled languages and datatypes, threads, and blocking system calls. The Node.js community members have taken some very bold steps, adopted what people found worthwhile in Ruby on Rails and other innovations, and lived through the pain of the all-async nature by coming up with libraries such as Async, as well as other great ones like Underscore, Connect cq. Express, Mocha, and Marked.

I just came across a very nice site about JavaScript, called SuperHero, via this weblog post. Will be going through it soon, to try and improve my understanding and (hopefully) skills. If you like video’s, check out PeepCode, i.e. this one on Node.js + Express + CoffeeScript.

As for the client-side JavaScript framework, I’m using AngularJS. There’s a nice little music player example here, with the JavaScript here, illustrating the general style quite well, IMO.

Isn’t it amazing how much knowledge and tools we have at our fingertips nowadays?

It seems to me that for software technologies and languages to gain solid traction and momentum, it will all come down to how “learnable” things are. Because we all start from scratch at some point. And we all only have so many hours in the day to learn things well. There is this never-ending struggle between picking the tools which are instantly obvious (but perhaps limited in the long rung) vs. picking very advanced tools (which may take too much time to become proficient in). Think BASIC vs. Common Lisp, for example. It gets even worse as the world is moving on – how can anyone decide to “dive in” and go for the deep stuff, when it might all be obsolete by the time the paybacks really set in?

In my case, I want to not only build stuff (could have stayed with Tcl + JeeMon for that), but also take advantage of what others are doing, and – with a bit of luck – come up with something which some of you find attractive enough to build upon and extend. Time will tell whether the choices made will lead there…

One other very interesting new development I’d like to mention in this context, is the Markdown-based Literate Programming now possible with CoffeeScript: see this weblog post by Jeremy Ashkenas, the inventor & implementor of CoffeeScript. I’m currently looking into this, alongside more traditional documentation tools such as Docco and Groc.

Would it make sense to write code for HouseMon as a story? I have no idea, yet…

Avrdude in CoffeeScript

In AVR, Software on Mar 27, 2013 at 00:01

Here’s a fun project – rewriting avrdude in CoffeesCript, as I did a while back with Tcl:

{SerialPort} = require 'serialport'

pageBytes = 128

avrUploader = (bytes, tty, cb) ->
  serial = new SerialPort tty, baudrate: 115200

  done = (err) ->
    serial.close ->
      cb err

  timer = null
  state = offset = 0
  reply = ''

  states = [ # Finite State Machine, one function per state
      ['0 ']
      buf = new Buffer(20)
      buf.fill 0
      buf.writeInt16BE pageBytes, 12
      ['B', buf, ' ']
      ['P ']
      state += 1  if offset >= bytes.length
      buf = new Buffer(2)
      buf.writeInt16LE offset >> 1, 0
      ['U', buf, ' ']
      state -= 2
      count = Math.min bytes.length - offset, pageBytes
      buf = new Buffer(2)
      buf.writeInt16BE count, 0
      pos = offset
      offset += count
      ['d', buf, 'F', bytes.slice(pos, offset), ' ']
      ['Q ']

  next = ->
    if state < states.length
      serial.write x  for x in states[state++]()
      reply = ''
      timer = setTimeout (-> done state), 300

  serial.on 'open', next

  serial.on 'error', done

  serial.on 'data', (data) ->
    reply += data
    if reply.slice(-2) is '\x14\x10'
      clearTimeout timer

And here’s a little test which uploads the RF12demo sketch into a JeeNode over serial:

fs = require 'fs'
hex = fs.readFileSync '/Users/jcw/Desktop/RF12demo.cpp.hex', 'ascii'

hexToBin = (code) ->
  data = ''
  for line in code.split '\n'
    count = parseInt line.slice(1, 3), 16
    if count and line.slice(7, 9) is '00'
      data += line.slice 9, 9 + 2 * count
  new Buffer(data, 'hex')

avrUploader hexToBin(hex), '/dev/tty.usbserial-AH01A0GD', (err) ->
  console.error 'err', err  if err
  console.log hexToBin(hex).length

Just copy it all into a file and run it as: coffee

It’s fairly dense code, but as you can see, the stk500v1 protocol requires very little logic!

It’s a software goose (again)

In AVR, Software on Mar 26, 2013 at 00:01

Yesterday’s weblog post documented my attempts to debug a problem while programming the JeeNode Micro with a Flash Board. I really thought it was a hardware bug.

Given that the results where slightly changing as I tried different things, but nevertheless seemed really consistent and repeatable when retrying exactly the same steps over and over again should have caused an alarm bell to ring, but alas it took me quite some time to figure it out: repeatable is a hint that the problem might be a software issue, not hardware.

So the next step was to use the data dump feature from the scope to collect the data written and well as the data being read back. These can be saved as CSV files, and with a bit of scripting, I ended up with this comparison:

Screen Shot 2013-03-19 at 10.47.35

This shows three editor windows super-imposed and lined up with, from left to right:

  • the data read back
  • the data written
  • the hex file with the compiled sketch

As you can see, the writes are exactly what they should be – it’s the read which gives junk. And it’s not even that far off – just a few bits!

Looking closer, it is clear that some bits are “0” where they should have been “1”.

Uh, oh – I think I know what’s going on…

It looks like the chip hasn’t gone through an erase cycle! Flash programming works as follows: you “erase” the memory to set all the bits to “1”, and then you “program” it to set some of them to “0”. Programming can only flip bits from “1” to “0”!

And indeed… that was the problem. The Arduino boot loaders are set up to auto-erase on request, so avrdude is called from the IDE with “bulk erase” disabled. That way, the IDE just sends the pages it wants to program, and the boot loader will make sure the erase is performed just before writing the page.

In this case, however, there is no boot loader: we are using the normal ISP conventions, whereby erasing and programming must be explicitly requested.

It was a matter of dropping the “-d” “-D” flag from the avrdude command in platform.txt:


And now it works, flawlessly! Doh – this “bug” sure has kept me busy for a while…

Wild goose chase

In AVR, Hardware, Software on Mar 25, 2013 at 00:01

Sometimes, when messing with stuff, you end up in a blind alley which makes no sense…

This happened to me a few days ago, as I was trying to get a good setup going for programming the new JeeNode Micro v3.

The code in the IDE was working, I was able to program the JNµ via avrdude on the command line, but for some crazy reason, the same thing just kept failing when launched through the new 1.5.2 Arduino IDE. Getting that to work would make all the difference in the world, since it means you don’t need to install anything other than the IDE (on any platform) and then simply drop in the ide-hardware folder.

Very odd: a Flash Board which works on one Mac with avrdude, but fails with the IDE, which also uses avrdude. Time to set up the scope in Logic Analyser mode! Lots of wires, lots of configuration to identity all the signals, and then the fun starts: locating the spot in the data stream which would allow me to see what was going on.

Here is a peek at the SPI bus, decoded where it is reporting the ATtiny84 signature bytes:


Looks ok. Next, instead of looking at the SPI bus, I looked at the 9600 baud serial data:


It’s a slower time scale, since the bit stream is not as fast, thus allowing to capture more information (2 M samples are not that much when sampling at fixed time intervals, since you have to sample often enough to reliably decode the data stream).

I even suspected the power supply lines, and had a look at VCC as an analog trace:


And although it’s not very clean, this wasn’t the problem either. Drat…

Since the verification was failing, I decided to capture the data read-back in the second half of the programming process. The way to do this, is to set the trigger on the RESET line going low at the start, and then delaying the capture by about 4 seconds. As you can see in that last screenshot, this is the point where avrdude switches from writing data into flash memory to reading the data back.

And sure enough, the data read back was wrong. Not much, but just some bits flipped – consistently, and reproducably!

It wasn’t a hardware problem after all! A wild good chase: final episode tomorrow…

JeeNode Micro breakout

In AVR, Hardware on Mar 24, 2013 at 00:01

While messing around with a bunch of JeeNode Micro’s here, I decided to create a little convenience breakout board for it:


Sooo many ways to mess up – it definitely helps to avoid silly mistakes! This particular board doesn’t do that much, but it’s still a nice convenience when you’re drowning in cables and trying to debug some of the nastier bits, such as the Flash Board ISP programmer:


The above setup was the result of what turned out to be a wild goose chase – stay tuned…

JN Micro v3 reference

In AVR on Mar 23, 2013 at 00:01

As promised, here is the reference card for the JeeNode Micro v3:


(Click on the image to see the full-size version)

Note: the definitions in Ports.h were incorrect for the JeeNode Micro. Please make sure you update to the latest version of JeeLib if you intend to use the Port or PortI2C classes with the ATtiny84 on the JeeNode Micro!

The thing with diagrams such as these, is that they are hampered by file format wars, with all full-featured drawing apps going out of their way to lock people in. In this case, the diagrams where created by Marco Bakker in Visio on Windows – which I don’t have – so I imported it into OmniGraffle for Mac – which is in turn useless on other platforms.

Luckily, PNG and PDF have become de-facto standards which every browser knows how to display – so rendering the diagrams at least produces results everyone can use.

Maybe one day, we’ll all be using SVG editors. There’s a web-based one called SVG-edit, but it doesn’t seem to be quite there yet w.r.t. importing the SVG from OmniGraffle. There is also the multi-platform Inkscape, which does import the SVG very nicely (just one minor font size issue, as far as I can tell). But the UI is a bit quirky on the Mac, since it has to conform to the XQuartz conventions of the X.Org X Window System (aka X11).

I’ve updated the Pinouts page on the Hardware wiki to include these new Quick Reference Cards, and have attached the design and output files in different formats.

JeeNode v6 reference

In AVR on Mar 22, 2013 at 00:01

What started out as a great initiative on the forum has now become a complete “Quick Reference Card” for the JeeNode v6. With many many thanks to @mars for working out all the details and tweaks and for sharing his design files:


(Click on the image to see the full-size version)

Here is the legend and the Creative Commons attribution:

legend   attribution

Tomorrow, I’ll post the other reference card for the JeeNode Micro (after a bit more checking), along with links to the final version in several formats on the wiki.

Programming the JNµ – at last!

In AVR, Hardware, Software on Mar 21, 2013 at 00:01

Yesterday’s post shows that it the JNµ can be easily be programmed using the standard Arduino IDE, if you get all the pieces right (isn’t that always the case?):

Screen Shot 2013-03-19 at 14.34.08

The easiest way to get the connections right, is to assemble a custom cable (more):


What about the software setup? Well, that too is now very simple:

  • download and install a copy of the Arduino IDE 1.5.2
  • create a folder called “hardware” inside your IDE’s “sketchbook” folder
  • download or – preferably – clone the new ide-hardware package from GitHub
  • rename it to “jeelabs” (!) and move it inside the “hardware” folder

The new jcw/ide-hardware project on GitHub was adapted from the arduino-tiny project on Google Code. I didn’t want to wait for that project (it hasn’t been updated for over 6 months), and decided to create a fork on GitHub instead. I did find two other such forks – rambo/arduino-tiny and james147/arduino-tiny, but neither of them appears to offer a major advantage – and since I had to rearrange things to make it work with the IDE 1.5.x structure, there does not seem to be much point in basing things off those forks.

The result of all these steps is that you now have the following new options:

  • use the Tools -> Programmer menu to select “Arduino as ISP”
  • select “JeeNode Micro” from the Tools -> Boards menu to build for that setup
  • use the Tools -> Burn Bootloader menu to set up the fuses
  • use the standard “Upload” button to upload a sketch using this ISP-based setup:

    Screen Shot 2013-03-20 at 16.42.05

Those two warnings are harmless, and can be ignored (I don’t know how to avoid them).

That’s it!

Update – Fixed a problem with setting fuses, make sure you use latest code from github.

Programming the JNµ – again

In AVR, Hardware on Mar 20, 2013 at 00:01

Given that the JeeNode Micro is based on an ATtiny84 and not the standard ATmega used in Arduino’s and clones, it has always been quite tricky to make things work on the JNµ.

No more:

Screen Shot 2013-03-19 at 14.34.08

The new (and still somewhat experimental) Arduino IDE 1.5.2 adds much better support for other process cores, making it possible to set up things as shown above, and then you can just use the “Upload” button to get things loaded into a JeeNode Micro.

It’s all based on setting up a JeeNode (or other 3.3V-level Arduino clone) to run as ISP programmer, and then a few wires to hook it up to a “target” JeeNode Micro. The IDE now includes an ISP programmer setup, but I tend to use the Flash Board for this – in combination with the isp_flash.ino sketch, which was recently updated to run at 9600 baud i.s.o. 19200, to make it compatible with what the IDE expects by default:


For more information about ISP programming, see this post and tag on this daily weblog.

Tomorrow, I’ll describe the steps needed to set things up. Stay tuned…

Direct relay switching – final

In Hardware on Mar 19, 2013 at 00:01

To conclude this little excursion into relay switching, here is the final circuit:


I’ve re-wired everything to use two DIO pins, and removed the extra 10 Ω resistor:


Bottom side:


The sketch can now switch both I/O pins using a single instruction, and avoid the brief shorting out of the I/O pins on the ATmega:

void setup() {
  digitalWrite(5, 0);
  digitalWrite(6, 0);
  pinMode(5, OUTPUT);
  pinMode(6, OUTPUT);

static void setLatch (bool on) {
  // change both I/O pins in the same clock cycle
  if (on)
    PORTD |= bit(5) | bit(6);
    PORTD &= ~ (bit(5) | bit(6));

void loop() {

This also gets rid of the inductive kickback, or rather, it “diffuses” that effect into the slow capacitor charge and discharge ramps. All in all, I think this setup is a viable circuit for low-power switching of this tiny relay.

I’ve ordered a few more of these units, and intend to try them out with live 230 VAC mains switching. Note that this is pushing it, and that it will most definitely not meet all rules and regulations for mains separation and isolation – but for just controlling small lights, and other small devices such as power adapters and chargers, I think it’s still usable, and am willing to try it – at least for personal use around the house here at JeeLabs.


Direct relay switching with a cap

In Hardware on Mar 18, 2013 at 00:01

Let’s explore this direct relay switching a bit further …

As suggested by @tmk in one of the comments, it is also possible to drive this thing with a single I/O level change by inserting a capacitor in the circuit, i.e. using just 2 I/O pins:


The idea is that the (opposite) charge and discharge currents pulses will be sufficient to make the latching relay switch state. The electrolytic cap takes the place of the pulse software, which can now be simplified to just switch high and low:

static void setLatch (bool on) {
  if (on) {
    bitSet(PORTD, 6); // D.6
    bitSet(PORTC, 2); // A.2
  } else {
    bitClear(PORTD, 6); // D.6
    bitClear(PORTC, 2); // A.2

The extra 10 Ω resistor in series with the cap is used to measure the current in the circuit and show it on the scope:


I’ve added the integral of the current in red, which allows calculating the total charge consumed by this circuit when switching the relay. A quick visual estimate tells me that the surface under the current pulse is about 50 ms x 50 mV, and since I’m measuring over 10 Ω, that corresponds to 50 ms x 5 mA = 250 µC. The actual integral is more like 350 µC. This corresponds to drawing 350 µA for one second, every time the relay is switched – which would presumably be done only a few times a day, so this is still within the capabilities of a battery-powered JeeNode.

There’s a strange hump at the start – here it is again, zoomed in:


I suspect that it is caused by relay contact bounce, and the change in inductance as the latching relay core toggles to its alternate state. The relay’s inductance is about 95 mH, according to my low-cost LCR meter – i.e. a 40 Hz resonant frequency combined with that 100 µF cap, so this can’t be a resonance issue (that would have to be in the 25 ms range).

Tomorrow, the last post about this topic, with the final circuit and code…

Direct relay switching – part 2

In Hardware on Mar 17, 2013 at 00:01

Yesterday’s post turns out to uncover a lot of trouble spots and mistakes on my end w.r.t. switching small relays. Get ready for some scope shots ahead, to see what’s going on…

But first, the code I’ve been using:

void setup() {}

static void setLatch (bool on) {
  pinMode(5, OUTPUT);  // DIO2
  pinMode(15, OUTPUT); // AIO2
  pinMode(6, OUTPUT);  // DIO3
  pinMode(16, OUTPUT); // AIO3

  digitalWrite(5, on ? 0 : 1);
  digitalWrite(15, on ? 0 : 1);
  digitalWrite(6, on ? 1 : 0);
  digitalWrite(16, on ? 1 : 0);

  pinMode(5, INPUT);
  pinMode(15, INPUT);
  pinMode(6, INPUT);
  pinMode(16, INPUT);

void loop() {

Basic idea: set the pins as outputs, set the levels right, wait 3 ms, then set the pins as input again, i.e. in high-impedance mode, to stop driving the relays.

There are two very serious problems with this code. The first one is that the I/O pins are set to inputs at the end. This means there is no good path for the inductive kick energy to go, leading to this very nasty waveform across the relay – same as shown yesterday:


Note the extreme voltage levels across the coil – way beyond the ± 3.3V you’d want to see in a good 3.3V totem-pole configuration.

It turns out that there is a surprisingly simple solution for this – don’t make the I/O pins inputs, but put them all the same output level, so that no current flows through the relay (as before), but with the I/O pins still being outputs, i.e. able to conduct current.

Here’s is the entire pulse again, at maximum scale this time:


(the lines are so thick because the input is set to peak-detect mode in the scope)

No more funny business! A fairly clean on and off transition, just a bit of “sagging” as the two I/O pins in parallel struggle (successfully) to supply the required current. It looks like the inductive kickback is now absorbed by the output pins which are all set to “0”, i.e. conducting to GND.

I’m surprised by how well this seems to work. I’m guessing that the CMOS switches in the ATmega’s pin driver are able to conduct in both directions when enabled, and that the ESD protection diodes are absorbing any voltage excursions outside the 0 .. 3.3V range.

But a much bigger problem probably, is that the original sketch was shorting out the DIO and AIO pins against each other – producing a very bad intermediate voltage level:


The time scale is completely different now, and as you can see, the voltage level goes through a strange middle state. This is caused by the fact that the digitalWrite() code is very slow, compared to the actual speed of the ATmega. It’s taking 4 µs to process each call, and as the first one is set to one value, the other pin connected to it can still be in the other state. So this is – briefly – shorting out two I/O pins, set to different values!

Very very bad – this may well reduce the lifetime of the ATmega µC. As it turns out, my choice of I/O pins makes it impossible to set both I/O pins simultaneously. But the least I can do without re-soldering, is to use the much faster bitSet() and bitClear() calls – even though this makes the code slightly harder to understand:

static void setLatch (bool on) {
  if (on) {
    bitSet(PORTD, 5); // D.5 = DIO2
    bitSet(PORTC, 1); // A.1 = AIO2
  } else {
    bitSet(PORTD, 6); // D.6 = DIO3
    bitSet(PORTC, 2); // A.2 = AIO3


  bitClear(PORTD, 5); // D.5 = DIO2
  bitClear(PORTC, 1); // A.1 = AIO2
  bitClear(PORTD, 6); // D.6 = DIO3
  bitClear(PORTC, 2); // A.2 = AIO3

The switching times are now dramatically closer together, more like 0.12 µS in fact:


Note that you’re looking at the superposition of both the rising and the falling switching waveforms in this case, as an easy way to see both in one screen shot.

For more information about inductive kickback and protection diodes: there’s an article by Douglas Jones – it’s full of useful information, even though it’s geared towards driving DC motors and stepper motors. The same principles apply when driving the coil in a relay.

So the conclusion seems to be that this specific relay could be driven from 4 I/O pins, without requiring any further circuitry. And since it’s a latching relay, this can easily be done on battery power: it merely takes a 3 ms pulse to make it switch, i.e. about the same amount of energy as sending out one RF12 packet!

Direct relay switching

In Hardware on Mar 16, 2013 at 00:01

There are small relays which are just weak enough to be switched directly by the ATmega’s I/O pins, even at 3.3V. I’ve been chasing and trying out these types of relays for some time, especially the latching types which only require power to switch, not to stay on.

Today, I received a very nice surprise package from @eljonco, who sent me a nice little latching relay (from eBay) which I’ve immediately mounted on a Proto Board:


Bottom side connects DIO2 + AIO2 to one side of the coil, and DIO3 + AIO3 to the other:


Why both? Because one pin drive is not strong enough to make these units switch at 3.3V:


…whereas driving them both in parallel works, resulting in a clear clicking sound and a very nasty “inductive kick-back” switching pulse across the coil on power-off:


There are horrific things going on here, electrically speaking, which I’ll look into later.

For now, let me just explain what’s going on and how this single-coil latching relay is being switched, using four I/O pins (port 2 and 3 on a JeeNode in this case):

JC's Grid, page 68

The relay core can be switched with a brief pulse, reversing the polarity to reverse the action. So we connect it to two different I/O pins and drive it as an H-bridge. When both pin levels are the same (or when the I/O pins are floating), nothing happens. By putting one pair in “0” state and the other in “1” state, current flows through the coil.

So to turn the relay on, the actions are:

  • set DIO2 = AIO2 = 0
  • set DIO3 = AIO3 = 1
  • wait 3 ms
  • set DIO3 = AIO3 = 0 (same as DIO2 and AIO3), or make them float, i.e. inputs

To turn it off, we simply exchange the 0’s and 1’s in the above.

More on this tomorrow. Stay tuned…

Unwanted RF

In Hardware on Mar 15, 2013 at 00:01

Houten is not a good place to be if you want to analyse signals in the 100 MHz range …

After looking into some switching regulator artefacts for a while, I’ve concluded that there is nothing wrong with my circuit after all. It’s all due to a nearby FM broadcasting tower!

(367 m high and about 10 km away: the Gerbrandytoren in Lopik – for the Dutch readers)

Getting rid of RF electromagnetic waves is hard. You need a Faraday cage, but you also have to prevent all unwanted RF from entering via measurement and power cables – and it has to work at all the frequencies you’re examining!

I doubt I’ll find a way to get rid of this, but the least I can do is to measure it:


Whoa – nice FM antenna!


Now let’s do the right thing and use a coax cable with internal 50 Ω termination in the scope turned on, and the input sensitivity increased to 1 mV per division:


Again, nice antenna. The “noise floor” is now 20 dB (100x) lower – a much cleaner signal:


And finally, just to make sure it’s really the loop that is picking up the signal, let’s replace that loop by a 50 Ω terminator on the input side as well:


Ah, now we’re getting to the bottom of this – a 30 .. 35 dB reduction:


Note that we’re still picking up very weak FM signals (no cable shielding is perfect). But it’s clear that the incoming signals are coming from antenna-type pick-up of electric fields, not some fault in the measuring setup, or that switcher I was investigating.

Let’s just hope I never have to really chase and analyse signals in this frequency range!

Heat imaging at JeeLabs

In Musings on Mar 14, 2013 at 00:01

Yesterday’s images were just a diversion from the task of “measuring the house”, of course. Here are some outside thermal images, taken mid-February:



But the really interesting bit comes from sucking air out of the house using the blower door on a cold day, to see where cold air comes leaking back in:



You have to take the measurement scale into account, but some of the (over 200!) images made were absolutely shocking. There are some major air leaks in and around the house!

As final test, smoke was used: you turn the blowers around to create a slight over-pressure, turn on a big smoke generator (of the kind used in shows on-stage), get out of the house, and watch the smoke appear – it was literally everywhere, even in the neighbour’s houses!

This sort of testing needs to be pre-announced to the local fire department…

Hot spots on a JeeNode USB

In Hardware on Mar 13, 2013 at 00:01

Recently, a blower door test was done here at JeeLabs, as a way to measure – not guess – how air currents are affecting the house, and what sort of insulation measures would be most effective. This was a fascinating and very extensive test, combined with a thorough scan of the entire house – inside and outside – using a high-end thermal imaging camera.

While the guys from Bergenbos were doing their tests, I asked them to take some close-ups of a JeeNode which had been running RF12demo for a while (i.e. not in sleep mode):



Fantastic resolution, as you can see. No big surprises, but nice pictures nevertheless…

What’s a measly 2 cm?

In Hardware on Mar 12, 2013 at 00:01

As promised, one last post about decoupling capacitors. Unlike yesterday, this one is about that 0.1 µF ceramic capacitor again – the kind which gets used all over the place in through-hole designs such as the JeeNode:

Screen Shot 2013-03-10 at 19.49.17

Let’s compare the 200 kHz .. 20 MHz sweep signal absorption when connected far away from the cap (white), i.e. with long leads, vs. connected right next to the cap (yellow):


Whoa! – a substantial effect at higher frequencies if we include the long leads!

If you try to think about this in a simple “DC electricity” way, then 2 cm of extra wire is nothing: let’s for the argument’s sake assume that electricity travels at the speed of light. Then 30 cm = 1 ns, i.e. 2 cm is less than 70 picoseconds. A 16 MHz frequency has a 62.5 ns period, i.e. 19 meter wavelength, which completely dwarfs those measly 2 cm wires.

But propagation is not what’s messing things up here, not with those “low-end” 16 MHz ATmega’s I’m experimenting with anyway. The effect shown above comes from any wire adding some parasitic impedance (and capacitance) to the circuit. Plus some transmission line effects, probably. As you can see, those little extra 2 cm cause a noticeable degradation of the 0.1 µF cap’s decoupling capability.

This also applies to traces on a PCB, by the way. The guideline to add de-coupling caps as close as possible to the source of the switching disturbances is a serious matter!

With thanks to Martyn, for helping me understand the issues presented in these posts.

PS. Part of the impedance increase in all the screen shots made in the last few posts has been identified as a bad cable from my signal generator to the test setup. Doh! – with a much better cable (90% coax with clips at the end), the extra rise towards 20 MHz is gone. Still, this does not significantly affect the general shape or outcome of these experiments.

Resonator and crystal

In Hardware on Mar 11, 2013 at 00:01

Today, some more experiments with a sweep frequency to see how different components respond to it, beyond the cap-which-is-also-an-inductor of yesterday.

The first one I’d like to show is the 16 MHz resonator used in JeeNodes:


As you can expect it has a very sharp impedance change at its 16 MHz frequency:


(The bottom half is a greatly zoomed-in section from the far right of the entire sweep)

The blue area sticking out the back is the current through the resonator. As you can see, there is a very noticeable yet narrow range of frequencies at which things change. Plus a little parasitic inductance, indicated by a modest rise in signal amplitude near the 20 MHz end of the range.

An even more pronounced frequency-dependent component is the crystal, as used in just about every digital device around us these days. Crystals can have several different “modes” of oscillation, and can in fact resonate on different frequencies if the oscillator circuit around it is not designed properly.

Here, you can in fact see two modes of oscillation: a series-resonant “dip”, followed by a parallel-resonant “peak”. Note that this is a 10 MHz crystal, taken from an RFM12B module, with the total sweep scale from left to right being only 9.99 .. 10.01 MHz:


The resonant frequency seems to be about 150 ppm low, which is not surprising, since the crystal is not being driven by a real oscillator at all, nor with the proper capacitive loading – it’s simply “resonating along” with the frequencies applied to it.

These pictures as not just gimmicks. If you think about it, the behaviours shown above in a way almost define what these resonators and crystals do. It is precisely this “little” effect on impedance which allows us to create oscillators that resonate at very specific frequencies. Without them, we’d still be living in Tesla’s and Marconi’s age of sparks: maybe enough to get some morse code across, but a far cry from fitting dozens of HD television channels into separate adjacent frequency bands – or a fiberoptic cable, for that matter.

Tomorrow, I’ll have one final surprising result to show you w.r.t. “parasitics” …

It’s no longer a capacitor!

In Hardware on Mar 10, 2013 at 00:01

Yesterday’s post was about plotting the response of a decoupling capacitor versus frequency. It might not seem like much, but this is in effect exactly what a Spectrum Analyser with “Tracking Generator” does! You put a signal with a known frequency and amplitude into a circuit, and you look at the amplitude of the signal that comes out. As you saw, these plots give instant insight in what an analog circuit is doing to signals. And if we were to somehow also measure the phase shift of the signal, we’d in fact have a Vector Network Analyser – an instrument which usually costs more than an average car!

But let’s go back to decoupling…

First, let me show you the exact circuit setup I’ve been using:


The green dotted line is the AWG signal generator, and it has an internal resistance of 50Ω. You can ignore the 1Ω resistor, it was intended to measure current through the cap, but it turns out that the 50Ω helps us get the same sort of information.

Imagine this circuit hooked up next to a digital chip, with high-frequency “noise” reaching the top of the cap. As you can see in yesterday’s plot, the 0.1 µF cap becomes more and more a conductor as the frequency increases – which is exactly what we want, because that means the remaining voltage will consist of just the remaining low frequency changes, which are more easily dealt with by the power supply source.

Another way to look at this circuit is as a low-pass RC filter. It lets the lower frequencies through, and shorts the higher frequencies to ground.

The plots so far have all been from 1 kHz to 1 MHz. Let’s now raise the frequency sweep range a bit – from 200 kHz to 20 MHz (just ignore the blue trace):


Now that is odd – the amplitude starts to rise again with frequencies nearing 20 MHz! In fact, there seems to be a “saddle point” roughly in the middle. This is about 2 Mhz (the scale is still logarithmic, so every 5 divisions is now a factor 10 with these sweeps).

What’s going on here?

The answer is that all electrical circuits and components have parasitic effects. In this case, the capacitor also has some inductance. An inductance (i.e. a coil which generates a magnetic field) is just the opposite of a capacitor: it’s impedance rises with frequency.

So this 0.1 µF cap is in fact not able to short out high frequencies at all – it leaves them unaffected. Note that with an ATmega running at 16 MHz, we’re very solidly in that range of frequencies where the decoupling cap is becoming less effective!

To give you an idea how odd these caps behave: let’s add a 0.01 µF capacitor in parallel. You’d expect the result to be equivalent to a 0.11 µF cap – with the saddle point simply moving to a different place on the plot, right? Not quite:


They each do their thing and have their effects super-imposed, generating a double saddle. That, by the way, is why the more demanding circuits use exactly this very same approach to decouple various frequencies at the same time – just put some different caps in parallel.

Tomorrow, I’ll take a few other familiar components through this sweep setup…

Frequency sweep

In Hardware on Mar 9, 2013 at 00:01

This is the continuation of yesterday’s post about decoupling capacitors. As promised, some more information now about a test setup which can visualise some interesting – and quite surprising – effects.

The first part of the idea is to generate a “sweep” across a frequency range, i.e. a signal which increases in frequency from 1 kHz to 1 MHz within 10 seconds, and then repeats:


Such a signal can be generated with a “frequency generator”, in my case this AWG.

The second part is to inject this signal into the “Device Under Test” – i.e. a 0.1 µF ceramic capacitor in our case – and let an oscilloscope “sweep” across in lock-step with that signal:


Since the frequency increases with time, we end up with a display of the frequency on the X axis versus the signal amplitude on the Y axis:


The scope was set to display in peak-detection mode, to make the display “fill” with a band of the oscillating signal. The second signal is used to trigger each scan, which takes 10 seconds to complete (plus one second on each side for the previous and next sweeps).

As you can see, the amplitude of what gets through the cap decreases. Decreases? – yes, well, I simplified things a bit. There’s a resistor in series and the setup is actually measuring the voltage over the cap. As its impedance decreases, it shorts out the signal more and more. I’m deliberately skimming over some details to keep things short for this weblog post.

Looks like all is peachy, right? Not so fast…

First, let’s change the way things get measured a bit. Instead of linearly increasing the frequency from 1 kHz to 1 MHz, I’m going to switch to a logarithmic scale. This means that every 3.33 seconds, the frequency gets increased by a factor of ten by the AWG. So while the previous image had the frequency increase by some 100 kHz per division, now we have 33.3 KHz in the middle of the screen instead of 500 KHz:


It’s exactly the same result, but it doesn’t emphasise the higher frequencies as much. Note that we’re not really interested in the exact values, more in the shape of these curves – so I’ll omit the on-screen info from now on.

So why is that frequency response flattening out in such a clear way? Stay tuned…

Update – All arrangements have been made: JeeDay will be held in Houten, with excellent train access, on Friday evening April 19th and Saturday all-day April 20th.

Decoupling caps

In Hardware on Mar 8, 2013 at 00:01

As I wait for some fresh inspiration (oh, and some more JeeNode Micro v3’s) to arrive, let’s go into something that has intrigued me for a long time:

> What’s that “decoupling” stuff all about when designing electronic circuits?

The following information comes from a fantastic discussion with martynj, exploring ways to make this visible on the scope. I really learned a great deal – thank you, sir!

As you probably know, all chip datasheets have these little capacitors connected near the main chips, “for decoupling”. You’ll often see schematics with several of them:



What is decoupling? And decoupling what exactly?

Well, the problem is that power supply lines can get very noisy – lots of fluctuations you don’t really want, and if they get really bad, the chips will start to malfuction, mistaking a “0” for a “1” level or vice-versa. Not to mention the fact that high frequency signals will radiate into free air, turning a circuit on a PCB into an illegal transmitter – whoops!

Let’s not go into the “deeper” stuff: parasitic capacitance & inductance, dielectrics, etc. Instead, let’s just run some experiments with these humble little 0.1 µF “caps”:


As you’ll soon see, they are not capacitors at all – obnoxious little buggers, they are!

But first, the basic idea: due to digital switching, chips tend to generate a lot of very sharp current transitions. On an ATmega running at 16 MHz, these pulses are likely to be most pronounced at around 16 or 32 MHz, i.e. both the up and the down flanks of the main clock causing all semiconductors inside to switch.

Drawing a lot of current very abruptly is like suddenly pulling on a pendulum: it doesn’t immediately follow your request, but lags and has some trouble meeting the requested change. And when released, it tends to overswing, making the problem even worse.

Same with current changes – the power supply can’t quite match it, and so the voltage near the chip drops (and later overshoots) while trying to keep up. It’s the same effect as the lights dimming briefly when turning on a very power-hungry appliance in the house. It’s all about inductance if you really want to know, but let’s just ignore that for now.

Capacitors near the chip are like tiny (but very responsive!) reservoirs of energy, and they can be very effective in compensating for this dip.

So the general advice is: put decoupling caps between VCC and GND, as near the chip as possible, and the cap will “dampen” the demand to give the much longer connections to the power supply time to replenish the energy. Capacitors are great for this – and the closer they are to the consumer, the better they can handle even extremely brisk “spikes”.

Another way of saying this is: capacitors have an infinite resistance at 0 Hz (DC), i.e. they do not leak DC power, but as the frequency increases, they pass more and more current. This “frequency-dependent resistance” is actually called impedance – so as frequency increases, the cap’s impedance decreases. Spikes are – by definition – high-frequency. And the lower the impedance, the more these spikes get… s(h)orted out!

Tomorrow I’ll describe a little test setup to clearly demonstrate the effect of higher and higher frequencies being more and more absorbed by that 0.1 µF capacitor shown above.

Programming the JNµ v3 – part 2

In AVR, Software on Mar 7, 2013 at 00:01

It’s a long and winding road, that path to finding an easy way to program the JeeNode Micro v3, as mentioned two days ago. With emphasis on the “easy” adjective…

Let me at least describe the process details of what I’ve been using so far:

  1. A custom cable to connect the JNµ to an ISP programmer
  2. Installed copy of the “arduino-tiny” project from Google Code
  3. A trick to get the Arduino IDE compile the sketch into a HEX file
  4. Setting up the JeeNode + Flash Board as an ISP programmer
  5. Using avrdude directly, to set the fuses and upload the sketch into the JNµ

It works, in fact it works fine once you’ve set it all up – but easy? Hm…

Let’s go through each of these in turn.

1. Custom cable

This was described a few days ago. Let me just add that it’s very easy to take apart the connectors on the Extension Cable, once you know the trick:


It’s all a matter of gently lifting the plastic tabs, then the wires come sliding out. Note that this is the wrong end of the cable for our case – but the other end works the same.

2. Arduino-tiny

These files on Google Code contain all the necessary code to trick the Arduino IDE into compiling for ATtiny processors, including the ATtiny84 used in the JNµ.

It works with IDE 1.0.x as is, if you follow the instructions, but I’ve been reorganising it to meet the requirements of the new multi-architecture IDE 1.5.x series, which would support this as a straight drop-in. I’m still working on it, but it means you can then just press the “Upload” button – a lot simpler than the process being described in this post!

3. Hex files

Normally, the Arduino IDE takes care of everything: compilation as well as uploading. However, for this setup, we need to get hold of the raw HEX file that gets generated before uploading. There’s a preference in the IDE for that:

Screen Shot 2013-03-06 at 10.22.12

Now the nasty bit: you have to compile (“run” in Arduino IDE terms, go figure!) the sketch to see the output that gets generated, and then copy the full path from the output window:

Screen Shot 2013-03-06 at 10.23.36

In this case, it’s that “/var/folders/…/radioBlip2.cpp.hex” thing.

4. ISP programmer

To turn the JeeNode + Flash Board combo into an ISP programmer, upload the isp_flash.ino to it. Make sure you get the very latest version, which uses 9600 Baud for communication, as that makes it compatible with the “Arduino as ISP” programmer:

Screen Shot 2013-03-06 at 15.36.07

As mentioned before, you don’t really need the Flash Board at this stage – a couple of wires will also work fine (see these posts about all the ways this can be done).

5. Avrdude

The last step is to call the “avrdude” uploader from the command line, specifying all the arguments needed, and pasting in that full file name. In my case, it was something like:

avrdude -v -v -pattiny84 -cstk500v1 -P/dev/tty.usbserial-A600K04C \
  -b9600 -D -Uflash:w:/var/folders/.../radioBlip2.cpp.hex:i

Oh, and if this is the first time you’re programming this ATtiny84 chip, then you also have to first set the fuses properly, as follows:

avrdude -v -v -pattiny84 -cstk500v1 -P/dev/tty.usbserial-A600K04C \
  -b9600 -e -U lfuse:w:0xC2:m -U hfuse:w:0xDF:m -U efuse:w:0xFF:m

That’s the internal RC oscillator with fastest startup, and the brown-out level set to 1.8V.

As I said, this is the messy way to do things. I’ll try to have something simpler soon!

Retro kitsch

In Musings on Mar 6, 2013 at 00:01

Here’s a crazy book for ya’ – one which you can’t quite judge by its cover, in fact:


Well… I fell for it, and decided to get one:


Just think of what this represents: 500 GB of storage in the space of a single book.

Not bad for compression: half a million books in the size of… a book? :)

Programming the JNµ v3 – part 1.5

In AVR, Hardware on Mar 5, 2013 at 00:01

After having created the cable, I was going to set up the AVRISP MkII programmer…


Things didn’t work out as planned, alas. When having some problems (unrelated, and due to a wrong part choice, as it turned out), I wanted to make sure that the AVRISP was really set up properly, and decided to install AVR Studio 6.0 …

Hm. Since I don’t have any computers running Windows, this had to be done in a virtual machine (running under Parallels in my case). Easy, right? Yeah, well, I do keep an image around for cases like these, so I downloaded the installer from Atmel, and a little later it was indeed running. Lost some 4 GB of disk space (nuts), and it installed all sorts of stuff as part of the deal (nuts), but hey, that’s how software should be… apparently (nuts!).

Plugging in the AVRISP, I got a message from AVR Studio that I had to upgrade its firmware (it worked fine with avrdude, but hey, who am I to argue). This was not an option, but a requirement (nuts).

So I proceded with the upgrade… piece of cake, right?

Whoops! As part of the upgrade, the device drops off the USB bus, at which point Parallels ceased to recognise it. Unrecognised devices can’t be attached to a Windows VM.

So the end result is: a “bricked” AVRISP programmer, which can probably only be restored on a non-VM install of Windows. Nuts!

I’m going to switch to the Flash Board, which works and has a normal upgrade process:


Note that I’m not using the on-board EPROM or button. A few wires could also be used.

It’s easy to use as a programmer, when you’re used to the Arduino IDE and avr-gcc way of doing things: install the flash.ino sketch on a JeeNode, insert this thing on top, and you end up with an ISP programmer which is recognised by avrdude by using the parameters “-p stk500v1” and “-B 19200”, and the serial port as “-P …”.

So much for closed source & lock-in – stuff I can’t control. ISP mode resumes tomorrow!

JNµ battery tests

In AVR, Hardware on Mar 4, 2013 at 00:01

With the LiPo battery “blipping” along happily for years now, I’ve started two new tests, based on the JeeNode Micro v3. Here’s the boost unit, running an updated radioBlip2.ino:


This one converts the incoming 1.2 .. 1.4 V from a single-cell Eneloop AA battery to roughly 3.0 V, as you can see in this overview in HouseMon (using an updated “radioBlip” driver):

Screen Shot 2013-03-04 at 00.59.28

The other new node runs off a CR2032 coin cell, which is struggling a bit to maintain a stable voltage each time a packet gets sent out, as indicated by the 3.20 -> 3.08 V drop. In case you’re wondering how a node can report its battery voltage after already having sent the data, simple: measure the voltage and send it out in the next packet.

The technique used to measure one’s own battery voltage was described in a post last year. Measuring the battery voltage in this way does not consume very much power in itself, since the ATtiny is placed in a special low-power “ADC-only” sleep mode while doing so:


The energy used for the measurement is the area under those last four little “blips” after the main packet transmission: there’s almost no penalty in performing this measurement before and after each packet transmission. Over time, hopefully the trend of the before + after voltages can be used as predictor for the amount of life left in them little coin cells…

PS – The main change in the radioBlip2 sketch, apart from adapting it to work on the ATtiny, is that the packet now includes a “sender ID” byte. This way all the battery test nodes can re-use the same node ID here at JeeLabs – since the number of ID’s used is starting to become a concern (there are 30 unique ID’s in each net group). For nodes like these, which only send, it’s no problem at all to “re-use” node ID’s this way.

Update – I’ve added a “BOOST” option to report the battery input voltage, which is much more useful. It’s measured with a jumper from PWR to the AIO pin (PA0, analog 0).

Programming the JNµ v3 – part 1

In AVR, Hardware on Mar 3, 2013 at 00:01

The new JeeNode Micro v3 announced yesterday does some very subtle things w.r.t. how it deals with the SPI bus during ISP programming (don’t you love those silly acronyms?). I’ll have a lot more to tell about this in a future weblog post, but for now let’s just focus on getting a new sketch into the ATtiny84 chip!

As mentioned yesterday, there is no longer a standard 2×3 ISP header on this board. It’s just a 4-pin header, with GND and VCC obtained from the 8-pin header on the side of the board. For reference, here is the ISP header pinout as most ISP programmers expect it:

ISP 6 way pinout

In my case, I wanted to use the modified AVRISP mkII programmer, so I’ve set up a little cable + breadboard hack, as follows:


The 2×3 to 6-pin (actually 5-pin) header was constructed from an Extension Cable. Here is the hookup, using 2×3 long header pins to connect the 2 female ends together:


And here’s the breadboard side of things, in close-up:


Note the sixth wire (providing or sensing VCC power), which goes to either +3V or PWR on the main header. This flexibility will in fact turn out to be quite convenient when dealing with direct-power vs boost-power versions of the JeeNode Micro.

So much for the hardware side of things. The software side turned out to be a lot trickier, and I’d like to take a little more time to work out the best way to do things. After a short intermezzo tomorrow, will come part 2 – stay tuned!

Meet the JeeNode Micro v3

In AVR, Hardware on Mar 2, 2013 at 00:01

There’s a new kid in town, the JeeNode Micro v3 with ATtiny84 + optional boost converter:


(Can you spot the differences? That’s without and with buck/boost converter…)

It’s even shorter than the v2, and has all the components on one side again:


The main reason for the reduced size is the reduced number of header pins. One of the biggest changes in that regard, is that v3 no longer has a full 2×3 ISP header – just the 4 data pins. The idea is that GND and +3V can be connected to the 8-pin “port+” header.

As it so happens, there’s a fifth pad on the bottom side of the PCB which connects to GND. This can be put to good use for experimentation, by connecting some headers as follows:


Note that the 6th pin was removed from the female header – this is very easy to do: just pull on the pin from the bottom with some pliers and it’ll slide out.

That leaves just power to be connected up for SPI programming. More details tomorrow…

Taking the JNµ to 3.0 V

In AVR, Hardware on Mar 1, 2013 at 00:01

As mentioned in yesterday’s post, lower supply voltages lead to lower power consumption.

There’s a new JeeNode Micro v3 revision coming which applies this logic to try and get the power consumption as low as possible. It also has changed slightly in its pinout:

Screen Shot 2013-02-26 at 09.30.47

The main physical changes are: it’s now 16 x 43 mm, and the ISP pins are no longer 2×3.

Let’s just focus on the power side of things for now, though. The JNµ v3 can be operated from several power sources:

  • Direct power: this runs without regulation, and requires a 2.2V .. 3.8V power source
  • Boost power: using the same circuit as the AA Power Board, runs from 0.9V .. 5.0V

The boost power circuit is very flexible as you can see, but it adds a certain amount of quiescent power consumption of its own, making the whole setup draw more idle current than when it is directly powered. What you do get with the boost power hookup, is that it’ll try to stay running as long as possible, down to a 0.5 V input under ideal conditions, so this thing will squeeze the last drop of energy out of its power source. Great for “empty” AA’s!

Boost power supplies are very clever and useful circuits, but they can’t break the laws of physics: a 100% efficient boost supply will draw 15 mA when asked to generate a 3.3 V @ 5 mA output from a 1.1 V input source. The current it draws will be triple that which it produces (more like four times, in fact, since the real circuit is far from 100% efficient).

Note that – in the light of yesterday’s story – it would really make no sense to pump up the voltage to say 5V, only to end up with a more wasteful circuit. That’s not just a quadratic increase, those losses would in fact be proportional to the third power of the voltage!

With a boost circuit, it really pays to work with as low an output voltage as possible. Which is why the JeeNode Micro v3 now comes with a 3.0V version of the boost chip, not 3.3V.

Let’s do the math, assuming a 100% efficient boost supply:

  • a 3.0 V booster draws 3.0 / 3.3 ≈ 91% of the current in comparison to a 3.3 V one
  • running at 3.0 V instead of 3.3 V consumes ≈ 83% of the power (with a resistive load)
  • combined, that means we’ll be using 3.0 ^ 3 / 3.3 ^ 3 ≈ 75% of the power at 3.0 V
  • so the total power consumed is 25% less at 3.0 V compared to what it’d take at 3.3V

That translates to 3 more months on a battery lifetime of 1 year – quite a bit!

This doesn’t just apply to the boost regulator: coin cells often have a 3.0V rating, and so do 2 alkaline AA batteries in series. Note also that 2 rechargable AA batteries will still work fine, as these will supply 2.4..2.6 V, well within the direct power range of the JNµ v3.

Note that for all the standard JeeNodes, nothing changes: these will continue to be populated with 3.3V regulators. One key reason for this is that they are already operating slightly beyond their specs (“overclocked”) with the clock running at 16 MHz. We shouldn’t push our luck with these settings, which are standard in the Arduino world. So the JN, JN SMD, and JN USB will all continue to be operated at 3.3V – no changes!

As for the many different JeePlugs: these all run at 3.3V, and the chips used on these plugs are normally qualified to run at either 3.0 .. 3.6V or 2.7 .. 3.3V, so we’re still fully in range.

Is it all win, then? Well, yes, but note that the noise margins are slightly reduced at 3.0V, which can slightly affect the accuracy of analog pin measurements and the cable lengths at which I2C chips still operate reliably. These effects are going to be minimal, I expect, but it’s something to watch out for.

Update – As pointed out in the comments, the above calculations are not correct. There is a small effect on booster circuit efficiency in all this, but the power savings are closer to 20% than 25% – not quite 3 months per year, as I claimed.

Dropping below 3.3 V

In AVR, Hardware on Feb 28, 2013 at 00:01

One of the departures from the Arduino world has always been that JeeNodes operate at 3.3 V, whereas the standard Arduino’s all ran at 5 V. Things are changing, with the ARM-based “Arduino Due”, for example, which also runs at 3.3 V. There’s not really a choice: ARM chips all run at 3.3V or less (3.6V max actually, which leaves a nice safety margin).

So what’s this trend towards lower voltages all about, eh?

Well… you may have seen an older post in the Easy Electrons series (whoa, over two years ago.. time flies!). If you think about it for a moment, all the electric energy pumped into a circuits will end up being converted into some other form of energy: a teeny tiny bit of radiated RF energy in the case of a JeeNode, but mostly heat, really.

Odd as it may seem, all that electronic stuff is just a way to turn current into heat. There is a purpose for this, since we’re usually interested in the side effects: some physical sensing and some computation, leading to the information we’re interested in. But the electricity consumption is just an (unavoidable) side-effect.

Whenever we can lower the voltage and/or current consumption, we’ll end up consuming less power as a result (W = I x V). Lowering the voltage on a circuit often leads to a dramatic reduction in power, since part of a circuit is always resistive. Ohm’s law says that voltage = resistance x current, or to put it differently: current = voltage / resistance. So halving the voltage (V) over a resistor also halves the current (I) it draws. The effect: half the voltage leads to a quarter of the power consumption, in purely resistive circuits:

JC's Grid, page 65

Unfortunately, you can’t just use 0.1 V to power a circuit: bipolar semiconductors such as diodes and transistors have a 0.6V threshold voltage, anything below that renders them useless. In the real world, running digital circuits at under 1.0 V is rarely done. Even that 1.8 V level down to which an ATmega and ATtiny can be operated is pushing some limits.

Another reason is that lower voltages make capacitive effects more dominant. The lower you go, the longer it takes to charge or discharge a capacitor. Which explains why the ATmega’s maximum clock speed must drop to 4 MHz with a supply voltage of 1.8 V.

But lowering the clock speed is somewhat self-defeating, as this means that the ATmega and ATtiny have to remain powered-up longer before going back into deep-sleep mode! Lots of trade-offs. Still: a low supply voltage is often a good idea – generally speaking.

Tomorrow, I’ll describe the choices made for the next revision of the JeeNode Micro…

HouseMon 0.5.1

In Software on Feb 27, 2013 at 00:01

It feels a bit odd to give this piece a software a version number, since it’s all so early still, but I’m releasing it as HouseMon 0.5.1 anyway, to set a baseline for future development.

Note that the instructions so far were all about development. To get an idea of the true performance on a Raspberry Pi, you should start up as follows (make sure HouseMon is not running already): cd ~/housemon && SS_ENV=production node app.js – this’ll take a few minutes while SocketStream combines and “minifies” everything the first time around. After that, all web accesses will be a lot snappier than in development mode.

But let’s not get ahead of ourselves… tons of work to do before HM becomes useful!

I suspect that this project will lead to more questions than answers at this stage, but I guess that’s what you get when following the “release early, release often” approach of open source software. There seems to be no other way to go through this than just bite bullet, get a basic release out in the wild, and see how it holds up when installed in more places.

Here’s an attempt to map out what’s in the 0.5.x release:


The dotted lines indicate that logged and archived data is being saved but not used yet (logged data is the raw incoming text from serial interfaces, etc – archived data is per-sensor information, aggregated hourly).

This diagram is still too complicated for my tastes. Part of the process is getting to grips with complexity: multiple sensor values per reading, “de-multiplexing” different types of packets from the ookRelay, mapping node ID’s to locations, ignoring some repeated data which contains no extra information, and more real-world messiness…

There is currently very little error checking and error messages can (will!) be cryptic.

Which is all a long-winded way of saying that HouseMon is still in its very early stages. Let’s see how it evolves, and please do comment and hack on it if you are interested.

Note that HouseMon is a moving target. The “develop” branch is already moving ahead:

Screen Shot 2013-02-24 at 10.11.57

As you can see, it’s all made up of “briqs” which can be added and enabled as needed.

PS. I’ve added some documentation on how to track changes in the development branch.

Issues on GitHub

In News, Software on Feb 26, 2013 at 00:01

Since doing more and more with GitHub, I’ve been getting increasingly impressed with the features and workflow of that website. One of the things which really turned me around, was its issue tracker. This is already in use on projects such as EtherCard, as you can see:

Screen Shot 2013-02-19 at 22.39.00

The way things are implemented, including mail integration, keyboard shortcuts, and the fact that issue pages update in real time (making them little real-time chat environments) has convinced me that it really is the best place to deal with these things for all JeeLabs projects. This means I’m also not going to enable issue tracking in Redmine after all.

I’m not too concerned about lock-in, because all source code, its history, and even the entire issue tracking dataset is available for download from their site via a REST API.

I’ve also been using GitHub issues quite a bit for HouseMon lately, and have just started using its “milestones” feature, to help organise and plan the pile of work ahead:

Screen Shot 2013-02-19 at 22.20.22

Sooo… if you have a problem with JeeLib, Ethercard, etc. and want to report a bug, feel free to do so on the GitHub site. This doesn’t replace or interfere with the discussion forums on the JeeLabs “Café” in any way, as these continue to be the place for discussion and general questions and support. But for obvious bugs and clear-cut enhancement requests, GitHub is the place to be from now on…

As some people have started to discover, GitHub is also the fastest way to get patches and new features implemented – the mechanism for this is called “forking” and “submitting a pull request” (me pulling your changes into the official libraries, that is).

Here is a nice write-up about the process, written by the developers of “ThinkUp” – but it really applies to any project on GitHub.

As for release versioning, I’ll be using this approach for HouseMon from now on. This means that if you want to follow along and see the latest changes, you need to enter git checkout develop (once). The “master” branch will only update on the next release.

PS. Still struggling with all the terms and “features” of git. Thanks to @tht on the forum, I’m looking into SourceTree (for Mac), which acts as GUI wrapper around git commands, but I can’t say the coin has dropped quite yet. Things like merge vs rebase sound logical, but I’m sure I’ll get it wrong several more times before getting the hang of it…

Combined usage patterns

In Software on Feb 25, 2013 at 00:01

It’s going to be an interesting puzzle to try and deduce what all the different consumers are, just by looking at detailed electricity consumption graphs, such as this one:

Screen Shot 2013-02-18 at 16.03.20

I’ve already determined that the “double dip” patterns, such as the one at 15:41 to 15:52, is the thermostat + boiler “hunting” around to maintain its setpoint temperature.

The trouble of course, is that this whole house measurement approach doesn’t measure invidual appliances, but – as the term says – it can only report the total of all appliance consumption patterns. In this particular graph, you can see a second consumer kicking in around 15:15 and switching off again at 15:56. Fortunately though, this one is very characteristic as well: there’s a motor turning on, causing a brief very sharp blip at the start. It’s the kitchen fridge, in fact – and as the graph shows, it draws about 80 W.

In the general sense, this Knapsack problem is impossible to solve with certainty. But that doesn’t mean we can’t get a good insight of what’s going on around the house anyway.

There are several more known sources and consumption patterns around here, and once HouseMon is a bit further along, I hope to be able to identify them all in the collected data to be able to obtain fairly good estimates of the monthly consumption for each one of them. Also, once these patters are sufficiently accurately known, it should be possible to subtract them from the graph so to speak, and get a better view of the rest of the measurements.

JeeDay => April 20

In Musings on Feb 24, 2013 at 00:01

It’s been four and a half years of fun since I had this crazy idea to start JeeLabs, and it’s been four years also since the JeeNode was born. An excellent reason to celebrate, eh?

Coming April 19th and 20th (Friday evening and Saturday), I’m going to kick off JeeDay:

Meet face-to-face with fellow PhysComp / WSN / JeeStuff enthusiasts and JC + Martyn. Get the latest news, share your ideas and show off your project (or pictures of it). Discussions, presentations, hands-on sessions – it’s all possible, if we organise ourselves and our time appropriately!

The topics we could cover include things like:

  • Wireless Sensor Networks
  • Ultra-low power nodes in the Arduino world
  • Home monitoring and home automation
  • JeeLabs products Q & A
  • Solutions for dealing with AC mains
  • Funky sensors and clever displays
  • How to lower your energy bill
  • Soldering and measurement techniques
  • Hands-on with an oscilloscope
  • Designing and manufacturing PCBs
  • Enclosures, laser-cutting, 3D printing
  • Hack sessions? Debug sessions?
  • Bring and show your projects, especially if in-progress
  • Ideas for future projects and products
  • Presentations, presentations, presentations

Whoa, that list could go on forever… a huge set of topics!

The location will be in Utrecht or in Houten (5 min by train from Utrecht), which is located in the middle of the Netherlands. There are lots of accommodation nearby for those who want to stay overnight. Come and visit the Netherlands, you’ll enjoy it!

We can extend this to Sunday, if I can find a suitable venue and if there is enough interest, although perhaps that’s a bit too ambitious for such a first event.

Fees would be just to cover costs, drinks, etc. Also some sandwiches or pizza to get us through the day. Should all be doable for €15 .. €25.

I have no idea yet how many people would be interested and might be able to come, so I’ve set up a meeting scheduler – if you’re considering participating, please, please, please do add your name and indicate the time range – 10? 20? 50? 100? people – Let’s find out!

Further details will be added to the JeeDay 13.04 wiki page, as preparations progress. The sooner you respond, the more chances that I can figure out a proper venue and how to make it all happen. And… if you have any tips or suggestions, please get in touch now!

It’ll be great to meet face-to-face, it can be informative for all, and it’ll definitely be fun! :)

Small oops on the LED Node v2

In Hardware on Feb 23, 2013 at 00:01

The new LED Node v2 has been re-done slightly, by fixing the labels of the FTDI connector on the back side, but unfortunately Mr. Murphy pulled another one on me by making me ignore another issue at the time these new boards were ordered.

The current release of the LED Node v2 has a “disconnected ground plane”, meaning that some large copper areas on the back are not attached to ground!

The circuit is correct, in the sense that the board will function as is, but for better noise suppression and higher range of the on-board RFM12B radio, it’s best to make a small modification. Sooo… time to get out the hobby knife and start scratching!

Here is the back of the LED Node v2, unmodified (the silk screen may differ slightly):


What you have to do is scratch off the solder mask, without cutting the trace:


The result will be a bit of bare copper on both the trace and the ground plane next to it. Then, apply a bit of solder to connect everything together. Note that doing so may require quite a bit of heat, as the ground plane is hard to heat up – just keep your soldering iron on the spot until the solder flows freely:


That’s it. The ground of the circuit and the ground plane are now electrically connected, at least under the RFM12B module.

While I’m at it, here’s another modification – unlike the previous “mod”, this one should not be done in a standard setup, with 12V feeding the LED strips as well as the LED Node itself. This is a change you need only consider if you are using voltages higher than 12V, i.e. for 24V LED strips, or when running off batteries which go up to 14V or more.

The reason for this change, is that the MCP1702 regulator of the LED Node is rated only up to about 13V. Anything higher will damage it, even if applied briefly – I’ve seen several cases of this in the past.

The solution is to cut the trace between the power connector and the LED Node, but keep in mind that you then have to power the LED Node logic from the FTDI connector or one of the PWR pins on the headers at all times:


Just use a sharp hobby knife to cut through the trace this time:


So much for hacking the LED Node!

DIJN.12 – Final checks and unattended use

In Uncategorized on Feb 22, 2013 at 00:01

Welcome to the last instalment of Dive Into JeeNodes. Let’a make an unattended setup!

Everything is working now. The only step that remains is to automate things a bit further, so that the RPi will automatically start up HouseMon when powered up. This turns it into a fire-and-forget system, so that it becomes a permanent service on your LAN.

Auto-startup is convenient, but it means we also have to think a bit about how to upgrade HouseMon. This is where the nodemon utility comes in: it can be used to start up a Node.js application, and restart it whenever certain source files change. This is mostly intended as development tool, but at this stage where HouseMon is still so young and evolving rapidly, it’s actually going to be quite practical – even in an unattended mode.

Install the nodemon package by entering the command: npm install nodemon -g

The “-g” flag causes nodemon to be installed in a central location instead of as part of HouseMon, so that we can type in “nodemon” from the command line.

Now we’re ready to configure an unattended setup. Copy and paste these commands:

    cd ~/housemon
    echo 'cd ~/housemon' >
    echo 'PATH=/usr/local/bin:$PATH' >>
    echo 'nodemon >nohup.out 2>nohup.err &' >>
    chmod +x
    echo '@reboot ~/housemon/' | crontab

This creates a “” script which will be used to start up HouseMon while you’re not around and sets up an entry in the “cron” table which will run that script right after reboot, even when you are not logged in.

Warning: this loses any previous crontab entries, if this is not a fresh Raspbian install.

Reboot now, using sudo reboot to start the ball rolling. After a few minutes, the RPi will be up and running again, and you will be able to visit the HouseMon server via your web browser – no need to log in for that!

Screen Shot 2013-02-10 at 13.57.11

Congratulations: you have created your own personal Wireless Sensor Network, with a JeeNode sending out light readings once a second over wireless, to a JeeLink connected to a stand-alone Raspberry Pi, and via HouseMon running on Node.js, you’re able to watch the current light level in real time, from any web browser with access to your LAN.

Is this a home monitoring / home automation system? Heh.. not quite, but it is definitely an important first step towards such a system. All the foundations are in place, yearning to be filled-in and extended in numerous directions. The load on a Raspberry Pi looks fine:

Screen Shot 2013-02-21 at 15.36.54

This also concludes this initial series of “Dive Into JeeNodes”. The goal was to set up a basic – but fully functional – system, as a baseline for lots and lots of further explorations. As far as I’m concerned, there will be many more posts building upon everything that has been accomplished so far. For now, I’d like to leave this to settle down a bit, and to reconcile loose ends – such as going through these 12 instalments using Windows. Since I don’t use Windows myself, I’m hoping that someone else will chime in with details, so that the exact steps to get going can be documented in a follow-up post or how-to page in the project.

Cheers for now, I hope you’ve enjoyed this “PhysComp+WSN fun pack” DIJN series!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS – I’ve set up an SD card image pre-configured with Raspbian and HouseMon 0.5.1, so if you want to bypass all the setup work, download the hm051.img.gz file (550 MB!), and follow the instructions in DIJN.05 to set up that SD card. Then insert the JeeLink and SD card, and power up the RPi. It’ll start with HouseMon running – including the demo page.

PPS – Another milestone, this is weblog post #1250. Onwards! :)

DIJN.11 – Connect the light sensor

In Uncategorized on Feb 21, 2013 at 00:01

Welcome to the eleventh instalment of Dive Into JeeNodes. Let there be light!

Everything until now was nice, but wasn’t really about sensing the environment. For that, we need to hook up some sensors, obviously. One of the simplest sensors around is the Light Dependent Resistor, or LDR, as it’s usually called. It does exactly what its name says: vary its own internal resistance, depending on the amount of light falling on it.

To read it out with an ATmega, all we need to do is connect the LDR between ground and an analog I/O pin, and enable the ATmega’s internal pull-up resistor for that I/O pin to get a voltage drop over the LDR. Let’s first hook up the LDR:


As you can see, I’ve connected the LDR to the analog pin of Port 1 on the JeeNode (a JeeNode USB in this case – but any type of JeeNode will do).

The next step is to upload the proper code to the JeeNode, so that it actually does the measurement and transmits the result over wireless. This is a task for the Arduino IDE. We’ve already used it to load the initial test1 sketch, and now we can replace it with test2:

#include <JeeLib.h>

#define LDR 0 // the LDR will be connected to AIO1, i.e. analog 0

void setup () {
  // this is node 1 in net group 100 on the 868 MHz band
  rf12_initialize(1, RF12_868MHZ, 100);

  // need to enable the pull-up to get a voltage drop over the LDR
  pinMode(14+LDR, INPUT_PULLUP);

void loop () {
  // measure analog value and convert the 0..1023 result to 255..0
  byte value = 255 - analogRead(LDR) / 4;

  // actual packet send: broadcast to all, current counter, 1 byte long
  rf12_sendNow(0, &value, 1);

  // let one second pass before sending out another packet

(rf12_sendNow was recently added to JeeLib, make sure you’re using a recent version)

If you have followed along to the letter of the series, then the JeeNode will still be hooked up, and the IDE will still have the proper serial port setting for it. If not, please go back and make sure you can upload a new sketch. It’s a matter of choosing the proper serial port, selecting the examples/DIJN/test2 sketch in the IDE, and clicking on the Upload button. If all went well, you’ll see this:

Screen Shot 2013-02-16 at 11.40.21

That’s it. If you now look at the Demo page in HouseMon again via your browser, you’ll see actual light readings. Put your hand over the LDR and see the value decrease in real-time (within a second, in this case, since that’s how often the JeeNode sends out a new packet).

As you’ll see, the LDR is very sensitive. It has to be pretty dark for the value to drop under 100 (it ranges from 0..255). The range is also not linear or calibrated in any way.

Congratulations… now you have a real WSN! Tomorrow we’ll take care of some loose ends.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.10 – Set up a demo web page

In Uncategorized on Feb 20, 2013 at 00:01

Welcome to the tenth instalment of Dive Into JeeNodes. Let’s show real-time data!

With HouseMon running on the Raspberry Pi, we can start to think how to show incoming data on a web page. The following approach is the most basic one possible: get the received packets into Node.js, i.e. the RPi server, “publish” each through a WebSocket connection to any attached client, and figure out how to get the received value onto the web page.

Let’s start from the back. This is the web page we’ll be setting up:

Screen Shot 2013-02-09 at 12.22.13

Note that I’m running the server on my Mac in these screen shots, hence the “localhost” in the URL, but this will work in exactly the same way with HouseMon running on the RPi. Note also that the following three files are already included in HouseMon – they are just shown to explain what is going on.

That page is produced by a file called client/templates/demo.jade, with this contents:

        h1 Demo
        h3 {{value}}

Lots of stuff going on behind the scenes, clearly, but this is what Jade looks like with Foundation CSS, and how AngularJS ties a dynamic “value” variable into a web page. It’s sort of like a template (but not quite, due to the two-way binding nature of all this).

The second piece of the puzzle is a file called client/code/modules/

        module.exports = (ng) ->

          ng.controller 'DemoCtrl', [
            ($scope) ->
              $scope.$on 'ss-demo', (event, value) ->
                $scope.value = value

This one is a bit dense: it defines an AngularJS “controller” which will listen to incoming “ss-demo” events, and set the “value” variable we referred to in that Jade file above.

The last code we need, is in a file called briqs/ It defines this “Briq”: =
  name: 'demo'
  description: 'This demo briq is used by the "Dive Into JeeNodes" series'
  menus: [
    title: 'Demo'
    controller: 'DemoCtrl'

state = require '../server/state'
ss = require 'socketstream'

exports.factory = class
  constructor: ->
    state.on 'rf12.packet', packetListener
  destroy: -> 'rf12.packet', packetListener

packetListener = (packet, ainfo) ->
  if is 1 and is 100
    value = packet.buffer[1]
    ss.api.publish.all 'ss-demo', value

A “Briq” is something which can be installed at run time in HouseMon. Once installed, that file above will run on the server side. Briqs are a new mechanism I came up with (it’s no rocket science, but at least the name is new), and it’s still very early days. Once I figure out how, briqs will be turned into self-contained directories with the above three source files all in one place. For now, each of those files needs to be placed in a very specific location, for SocketStream to be able to find and use these files.

So how do we “install” the above demo in HouseMon, you may ask…

Easy. Go to the “Admin” page in HouseMon, via the Admin tab in the upper right corner:

Screen Shot 2013-02-09 at 13.18.11

No installed briqs yet, but you can see some briqs which are currently included in HouseMon in the second list. We need to install two briqs: one to set up that demo page, and one to connect to the JeeLink.

Click on the “demo” entry in the list to get this (whoops, I already installed everything):

Screen Shot 2013-02-09 at 12.24.52

Once you’ve clicked on install, you’ll see a new “Demo” tab appear in the upper right. It won’t do much, though, because we’re not receiving any data yet!

Click on the “rf12demo” entry, and fill in the serial port of your JeeLink. On the RPi, it’ll most probably be “/dev/ttyUSB0”, so just enter “ttyUSB0”. Then click install.

Go to the “Demo” page, and if your JeeNode is powered up, you’ll see an updating value.

Bingo – you’re looking at data coming from your Wireless Sensor Network in real-time.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

a s d f – j k l ;

In Musings on Feb 19, 2013 at 00:01

What can I say? Some things are worth the pain and the dip, I guess…

Exactly two weeks ago, I mentioned that one of the things I wanted to try was to learn and touch-type. You know, using this curious contraption without looking at it all the time…


Actually, the one I use has an extra key, the “~” / “`” key is in a different place, and the Shift and Return keys have a different size. And to make matters, ehm, more “interesting”, I even followed the suggestion to change the Caps-lock key into an alternate Control key (consider for a moment what that does to typing “TODO” in capitals as a touch-typist!).

Right now, after two weeks, I’m still typing quite a bit slower than I used to, am making tons and tons of mistakes (especially for the hard / far-away keys, and all the tricky Ctrl / Shift combinations), and I’m probably hitting the backspace key almost as often as all the other keys combined to correct my never-ending mistakes.

The worst bit is that this way of typing is still much more distracting than it used to be.

But… you know what? I really, really think it’s going to work and pay off!

Weird as it sounds, and hard as it feels, the letter keys are slowly starting to fall into place. Even after a mere two weeks (I did have some prior experience, to be fair). The thing that is taking most time to adjust to, is not the change in overall speed and effort, but the fact that typing speed is much more variable right now, depending on whether I’m typing letters or hitting other keys, and especially whether the Ctrl and/or Shift keys are involved (not to mention the Mac’s Option and Command keys!). So what seems to be happening is that words already come out faster than I used to type, but everything else really really takes a lot of time and mental effort. In short: punctuation and symbols are a pain, and with programming languages that means lots and lots of keystrokes are… t e d i o u s !

If you write a lot of prose, it’s pretty obvious that touch typing will be a wise investment. But now I’m starting to think that even with something as different as typing code, i.e. a notation full of weird characters, and even typing commands in Vim (where all sorts of key combinations have to be used all the time, and where mistakes are awkward!), the task of learning to type without looking at your fingers or the keyboard is indeed worth the effort.

(Speaking of Vim: it’s easier to type Ctrl-[ than it is to hit the far-away Escape key!)

It’s not even really about looking less up and down, as I thought. It’s about muscle memory, about off-loading an activity to a different part of your brain, and about focus. While writing this weblog post, which is obviously mostly text, I already feel more “relaxed” while typing in this new way. It’s like having an extra assistant – albeit still clumsy ;)

Pretty crazy stuff, if you think about it. I’m not making it up. This IS working!

Up next: the final posts of the Dive Into JeeNodes series – stay tuned…

Who needs a database?

In Software on Feb 18, 2013 at 00:01

Yesterday’s post was about the three (3!) ways HouseMon manages data. Today I’ll describe the initial archive design in HouseMon. Just to reiterate its main properties:

  • archives are redundant – they can be reconstructed from scratch, using the log files
  • data in archives is aggregated, with one data point per hour, and optimised for access
  • each hourly aggregation contains: a count, a sum, a minimum, and a maximum value

The first property implies that the design of this stuff is absolutely non-critical: if a better design comes up later, we can easily migrate HouseMon to it: just implement the new code, and replay all the log files to decode and store all readings in the new format.

The second decision was a bit harder to make, but I’ve chosen to save only hourly values for all data older than 48 hours. That means today and yesterday remain accessible from Redis with every single detail still intact, and for things like weekly, monthly, and yearly graphs it always resolves to hourly values. The simplification over progressive RRD-like aggregation, is that there really are only two types of data, and that the archived data access is instant, based on time range: simple file seeking.

Which brings me to the data format: I’m going to use Plain Old Binary Data Files for all archive data. No database, nothing – although a filesystem is just as much a database as anything, really (as “git” also illustrates).

Note that this doesn’t preclude also storing the data in a “real” database, but as far as HouseMon is concerned, that’s an optional export choice (once the code for it exists).

Here’s a sample “data structure” of the archive on disk:


The “index.json” file is a map from parameter names to a unique ID (id’s 1 and 2 are used in this example). The “p123” directory has the data for time slot 123. This is in mult1ples of of 1024 hours, as each data files holds 1024 hourly slots. So “p123” would be unix time 123 x 1024 x 3600 = 453427200 (or “Tue May 15 00:00:00 UTC 1984”, if you want to know).

Each file has 1024 hourly data points and each data point uses 16 bytes, so each data file is 16,384 bytes long. The reason they have been split up is to make varying dataset collections easy to manage, and to allow optional compression. Conceptually, the archive is simply one array per parameter, indexed by the absolute hour since Jan 1st, 1970 at midnight UTC.

The 16 bytes of each data point contain the following information in little-endian order:

  • a 4-byte header
  • the 32-bit sum of all measurement values in this hour
  • the 32-bit minimum value of all measurements in this hour
  • the 32-bit maximum value of all measurements in this hour

The header contains a 12-bit count of all measurement values (i.e. 0..4095). The other 20 bits are reserved for future use (such a few extra bits for the sum, if we ever need ’em).

The average value for an hour is of course easy to calculate from this: the sum divided by the count. Slots with a zero count represent missing values.

Not only are these files trivial to create, read, modify, and write – it’d also be very easy to implement code which does this in any programming language. Simplicity is good!

Note that there are several trade-offs in this design. There can be no more than 4095 measurements per hour, i.e. one per second, for example. And measurements have to be 32-bit signed ints. But I expect that these choices will work out just fine, and as mentioned before: if the format isn’t good enough, we can quickly switch to a better one. I’d rather start out really simple (and fast), than to come up with a sophisticated setup which adds too little value and too much complexity.

As far as access goes, you can see that no searching at all is involved to access any time range for any parameter. Just open a few files, send the extracted data points to the browser (or maybe even entire files), and process the data there. If this becomes I/O bound, then per-file compression could be introduced – readings which are only a small integer or which hardly ever change will compress really well.

Time will tell. Premature optimisation is one of those pitfalls I’d rather avoid for now!

Data, data, data

In Software on Feb 17, 2013 at 00:01

If all good things come in threes, then maybe everything that is a triple is good?

This post is about some choices I’ve just made in HouseMon for the way data is managed, stored, and archived (see? threes!). The data in this case is essentially everything that gets monitored in and around the house. This can then be used for status information, historical charts, statistics, and ultimately also some control based on automated rules.

The measurement “readings” exist in the following three (3!) forms in HouseMon:

  • raw – serial streams and byte packets, as received via interfaces and wireless networks
  • decoded – actual values (e.g “temperature”), as integers with a fixed decimal point
  • formatted – final values as shown on-screen, formatted and with units, i.e. “17.8 °C”

For storage, I’ll be using three (3!) distinct mechanisms: log files, Redis, and archive files.

The first decision was to store everything coming in as raw text in daily log files. With rollover at midnight (UTC), and tagged with the interface and a time stamp in milliseconds. This format has been in use here at JeeLabs since 2008, and has served me really well. Here is an example, taken from the file called “20130211.txt”:

    L 01:29:55.605 usb-AH01A0GD OK 19 115 113 98 25 39 173 123 1
    L 01:29:58.435 usb-AH01A0GD OK 9 4 50 68 235 251 166 232 234 195 72 251 24
    L 01:29:58.435 usb-AH01A0GD DF S 6373 497 68
    L 01:29:59.714 usb-AH01A0GD OK 19 96 13 2 11 2 30 0

Easy to read and search through, but clearly useless for seeing the actual values, since these are the RF12 packets before being decoded. The benefit of this format is precisely that it is as raw as it gets: by storing this on file, I can improve the decoders and fix the inevitable bugs which will crop up from time to time, then simply re-parse the files and run them through the decoders again. Given that the data comes from sketches which change over time, and which can also contain bugs, the mapping of which decoders to apply to which packets is an important one, and is in fact going to depend on the timeline: the same node may have been re-used for a different sketch, with a different packet format over time (has rarely happened here, once a node has been put to permanent use).

In HouseMon, the “logger” briq now ties into serial I/O and creates such daily log files.

The second format is more meaningful. This holds “readings” such as:

    { id: 1, group:5, band: 868, type: 'roomNode', value: 123, time: ... }

… which might represent a 12.3°C reading in the living room, for example.

This is now stored in Redis, using the new “history” briq (a “briq” is simply an installable module in HouseMon). There is one sorted set per parameter, to which new readings are added (i.e. integers) as they come in. To support staged archive storage, the sorted sets are segmented per 32 hours, i.e. there is one sorted set per parameter per 32-hour period. At most two periods are needed to store full details of every reading from at least the past 24 hours. With two periods saved in Redis for each parameter, even a setup with a few hundred parameters will require no more than a few dozen megabytes of RAM. This is essential, given that Redis keeps all its data in memory.

And lastly, there is a process which runs periodically, to move data older than two periods ago into “archival storage”. These are not round-robin databases, in the sense of a circular buffer which gets overwritten as new data comes in and wraps around, but they do use a somewhat similar format on disk. Archival storage can grow infinitely, here I expect to end up with about 50..100 MB per year once all the log files have been re-processed. Less, if compression is used (to be decided once speed trade-offs of the RPi have been measured).

The files produced by the “archive” briq have the following three (3!) properties:

  • archives are redundant – they can be reconstructed from scratch, using the log files
  • data in archives is aggregated, with one data point per hour, and optimised for access
  • each hourly aggregation contains: a count, a sum, a minimum, and a maximum value

I’ll describe the archive design choices and tentative file format in the next post.

DIJN.09 – Install the HouseMon server

In Uncategorized on Feb 16, 2013 at 00:01

Welcome to the ninth instalment of Dive Into JeeNodes. We’re going to install HouseMon!

If you’ve followed all the steps up to this point, then getting HouseMon running will be a piece of cake. We need to do exactly three things:

  1. Download the HouseMon source code
  2. Install the packages used by HouseMon
  3. Launch the HouseMon server

All of this can be accomplished by typing just a few commands, as described here:

    git clone
    cd housemon
    git checkout 0.7.x    # <== this is ESSENTIAL
    npm install
    npm start

Here’s what the above will look like in the terminal:

Screen Shot 2013-02-08 at 23.33.55

I haven’t typed the final return yet, as this will generate lots of gibberish while npm does its thing. There is some package compilation involved, which means that the “npm install” step will in fact take a couple of about 10 minutes to complete.

The real fun begins when we enter that final command to start the server: “npm start”. Here’s what you’ll see after a few seconds, while the system initialises itself:

Screen Shot 2013-02-08 at 23.53.47

Now we can use the web browser to connect to HouseMon. In my case, the RPi is running as IP address, so I can point my browser to, and something like this will appear:

Screen Shot 2013-02-08 at 23.59.45

You’ll need to replace the IP address by the one assigned to your own RPi setup, but apart from that it should all work in exactly the same way. HouseMon is now up and running.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS. Note that HouseMon is still evolving a lot (read: things change and things do break, sometimes). So be prepared to update it often, with: cd ~/housemon && git pull

DIJN.08 – Set up Node.js and Redis

In Uncategorized on Feb 15, 2013 at 00:01

Welcome to the eighth instalment of Dive Into JeeNodes. Let’s install Node.js!

(Note: these instructions have been updated to use the Feb 9th Raspbian image)

Now that the basic Wireless Sensor Network is up and running, it’s time to start thinking about the server-side software, which in this series will be HouseMon. For that, we need to get three pieces of software up and running on the RPi: Node.js, Redis, and git.

Those last two are in fact very easy to install on Linux, because they are available via the apt/aptitude package manager. And while we’re at it, let’s update all packages to the latest version. Getting the latest updates is always a good idea on Linux – just type:

    sudo apt-get update && sudo apt-get upgrade

Hit return when asked whether you want to install the updates. The first time around, this is bound to take some 15..30 minutes, as the system may not have been updated for quite some time, but later in it should normally takes no more than a few minutes.

Ok, back to the task at hand, i.e. installing redis and git:

    sudo apt-get install redis-server git

Easy, eh? Welcome to the world of open source Linux software. There are over 35,000 packages available. One way to explore what there is, is to use the menu-based “aptitude” command (type “sudo aptitude” if you’re curious, but be careful what you wish for!).

With Node.js, things are a bit more complicated, because the current version of “Node” available via apt-get is a bit too old for us. Fortunately, we don’t have to go through an actual build from the source code, as I described last month (things move fast in this world!). There is now a pre-compiled version of Node which we can install fairly easily:

  • First, a preparatory step, enter:

    sudo usermod -aG staff pi && sudo reboot

    Don’t be alarmed by the reboot, it’s merely a lazy way to log out and back in, which is essential for this one-time configuration change of the RPi. Wait a minute or two, and then you can log back in via ssh, as usual.

  • Now, enter these commands exactly as shown below:

    curl$v/node-$v-linux-arm-pi.tar.gz | tar xz
    cp -a node-$v-linux-arm-pi/{bin,lib} /usr/local/
    rm -r node-$v-linux-arm-pi
    npm -v

(Note: ignore the “preserving times for …: Operation not permitted” error messages)

That’s it. You can copy and paste these commands line by line, or all in one go – the latter is usually more convenient. The final “npm -v” command verifies that “node” and “npm” work properly by reporting npm’s version number when you hit return, i.e. “1.3.x”.

Lots of powerful tools… all in preparation of being able to install and use HouseMon.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.07 – Attach a JeeLink to the RPi

In Uncategorized on Feb 14, 2013 at 00:01

Welcome to the seventh instalment of Dive Into JeeNodes. Hello RPi, meet the JeeLink!

With all the RPi setup out of the way, it’s time to hook it up to the hardware. Fortunately, most of the essential ingredients are already included in the RPi. The JeeLink is based on an FTDI chip, for which the driver will be auto-loaded by Linux when it’s plugged in.

Proceed as follows:

  1. Plug the JeeLink in one of the two USB ports on the RPi.
  2. Type the following command in Linux: dmesg | tail -20

You’ll get something like this:

Screen Shot 2013-02-10 at 10.41.26

This is the kernel log, reporting that it has recognised the inserted USB device, has loaded a kernel driver for it, and has created a new serial “device” in Linux, called “/dev/ttyUSB0”.

We’re connected!

There are many ways to communicate with the JeeLink. Here’s a quick check that it works:

  • Enter the command: stty 57600 raw -echo </dev/ttyUSB0
  • Then enter the command: cat </dev/ttyUSB0

This sets up the serial ports and shows what is coming in. This is what I get:

Screen Shot 2013-02-10 at 10.50.33

Yippie! The JeeLink has restarted and shown its usual greeting, and is reporting incoming packets. You can hit CTRL+c to stop the process.

The above commands are sufficient to prove that things work, but not convenient. For that, we need to install a serial terminal emulator. There are two we can use, minicom or screen:

  • For minicom, install it with this command: sudo apt-get install minicom
  • To launch minicom, type: minicom -b 57600 -o -D /dev/ttyUSB0
  • For a help page in minicom, type: CTRL+a, then z
  • To exit and close the connection in minicom, type: CTRL+a, then q

I tend to prefer screen, mostly because I’m used to it:

  • For screen, install it with this command: sudo apt-get install screen
  • To launch screen, type: screen /dev/ttyUSB0 57600
  • For a help page in screen, type: CTRL+a, then ?
  • To exit and close the connection in screen, type: CTRL+a, then \

At this point, all the hardware, all the connections, and all the wireless communication are now known to work. All that remains, is to set up the HouseMon server on the RPi and to connect the light sensor.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS. There may be a potential issue with the FTDI driver on RPi. Will need to test this later.

Historical data and graphing

In Uncategorized on Feb 13, 2013 at 00:01

There’s now a first proof-of-concept graph in HouseMon, based on D3 and NVD3:

Screen Shot 2013-02-12 at 10.02.39

Historical data is now stored in Redis. The plan is to keep the last 48 hours in Redis and then move it to some sort of aggregated data files, to complement the daily full log files.

Lots of features in NVD3 which I haven’t explored yet. Out of the box, it supports hovering over a datapoint with a call-out to display the exact value, as you can see.

D3 – and the NVD3 package built on top of it – use SVG, which is supported by at least Safari and Chrome. You can see that they really are drawing with vectors, by zooming in:

Screen Shot 2013-02-11 at 22.18.40

The PDF version that gets generated when printing the page in Safari has several quirks (the shape is drawn as a closed polygon), and in Chrome the graph shows up as a black rectangle, so there is a price to pay for using this sort of cutting-edge technology – but on screen it comes out nicely.

This is the client/templates/graphs.jade file which defines the above page layout:


        h3 Graphs

        h3: input(type='text',ng-model='query',placeholder='Search...')
        div(ng-repeat='row in status | filter:query | orderBy:"key"')
          |  {{row.location}} - {{row.parameter}}
          span(ng-show='row.unit')  ({{row.unit}})

And this is how historical data gets fetched from the server (part of

key = 'meterkast - Usage house'
info = $scope.status.find key

promise = rpc.exec 'host.api', 'rawRange', key, -86400000, 0
promise.then (values) ->
  [ offset, pairs ] = values
  if pairs
    series = []
    for i in [0...pairs.length] by 2
        x: offset + parseInt pairs[i+1]
        y: adjustValue parseInt(pairs[i]), info
    data = [
      values: series
      key: 'Usage House'
    nv.addGraph ->
      chart = nv.models.lineChart()
      formatter = d3.time.format '%X'
      chart.xAxis.tickFormat (d) -> formatter new Date (d)
      chart.xAxis.showMaxMin false
      chart.yAxis.showMaxMin false'#chart svg').datum(data).call(chart)

The “promise” you see in this code is a clever mechanism to deal with results which come back asynchronously, i.e. at a later time than when the RPC call returns. Most of the code deals with “massaging” the data, to get it into the proper format for NVD3.

It’s nice to make some visible progress. for a change!

Stay tuned for tomorrow, when the DIJN series resumes…

Update – I’ve switched to Flotr2, smaller and better defaults for the time axis.


In Musings on Feb 12, 2013 at 00:01

Wow. What a lovely time of year (especially from inside a comfy home):


The next two pictures were taken by Liesbeth, in a little walk around the neighbourhood:


Not much solar power right now (150W), despite the sun – panels are covered up in snow!


Regular transmissions to resume tomorrow…

Solar fingernails

In Hardware on Feb 11, 2013 at 00:01

A couple of weeks ago, I described a little indoor solar energy test setup, based on a couple of fingernail-sized CPC1824 solar cell chips, with 0.47 Farad supercaps for storage:


It has been an interesting setup to keep running for a while, and now that 20 days have passed, it’s sort of intriguing to see how well, or badly, these solar energy cells performed. To summarise, here is what the four different test were about:

  1. One CPC1824, very lightly loaded with 20 MΩ, i.e. a 0.15..0.20 µA current draw
  2. One CPC1824, lightly loaded with 2 MΩ, i.e. a 1.5..2.0 µA current draw
  3. One CPC1824, moderately loaded with 0.2 MΩ, i.e. a 15..20 µA current draw
  4. Two CPC1824’s in parallel, moderate load with 0.2 MΩ, again 15..20 µA current draw

In addition, there is a top-up voltage cycling once per ≈ 2 hours, to prevent any the supercaps from discharging much further than about 2.7V. This is not a realistic situation if this setup were to be used for real, but I wanted to prevent a total discharge of the supercaps, to try and find out whether the supercaps would get better (i.e. less leaky) over time, as the constant charge conditions those membranes full of useable electrons.

Here are the results of 20 days, superimposed one upon the other:


(click two times to see it in full scale – X axis is in minutes since midnight UTC)

  • #1 = blue, #2 = bright green, #3 = orange (dotted), #4 = red (longer dots)
  • the last day is marked in black for all four cases
  • around noon, all cells fall into the shadow, so energy generation almost stops
  • note how only the blue lines stay above 3.0V (except in the first 2..3 days)

This was a wintery mix of sunny and cloudy days, which has a huge effect on the charge / discharge curves. Looks like indoor solar with these tiny cells is not going to work…

One conclusion is that only the lightest load of 0.15..0.20 µA allows the supercap to retain a decent amount of charge through each night. On the other end of the range: with two solar cells in parallel, a 15..20 µA load might be able to stay alive for a few hours each day.

Ehm, well, just maybe – this will really depend on the startup power consumption curves!

DIJN.06 – Boot the Raspberry Pi

In Uncategorized on Feb 10, 2013 at 00:01

Welcome to the sixth instalment of Dive Into JeeNodes. We’re halfway there!

(Note: these instructions have been updated to use the Feb 9th Raspbian image)

This part is probably the most complex piece of the journey called DIJN. Then again, you’re about to set up a self-contained server which can be used for a truly astonishing number of tasks. Linux is everywhere nowadays, and this is probably one of the easiest, lowest-cost, and most risk-free ways to tap into that world which has driven – and evolved alongside – the Open Source Software trend for so long already.

One could probably spend a decade exploring all the software that runs on Linux, and a lot of it may well be affecting your daily life (ever heard of this thing called “internet”?). The good news of course is that you don’t have to spend years, but that whatever interests you is going to be totally in reach, if and when you choose to, eh, dive in…

Let’s get rollin’, shall we?

  1. Insert the SD card you prepared according to yesterday’s post into the RPi. Note that it has to be inserted with the label facing down.
  2. Connect the Ethernet cable on both sides to hook up the RPi to your LAN.
  3. Connect the 5V power adapter to the RPi’s micro-USB port. This will power it up.

LEDs will start flashing, the world will start spinning, and a loud roar will emerge as the positron drive starts up. Oh, wait… wrong movie. Here’s what will really happen:

  • The red LED marked “PWR” will turn on.
  • The other LEDs will blink in a various patterns – after some 30s they’ll look like this:


This indicates that everything has started up properly. Now it’s time to try and connect to it over Ethernet. How, depends on your platform. On Windows, you can use the putty program. On Mac and Linux, the built-in “ssh” command will be perfect.

Now is the time to shed your Fear Of The Command Line, if it’s all new to you. It’s very simple, really: the GUI we tend to use is like the body of a car, and the command-line its chassis. This DIJN series is about what goes on under the hood. That’s where the “real” stuff happens. There’s no other way – we need to look inside. We need to operate inside, in fact. Soooo… welcome to the machine, I’m sure you’ll like it. It’s fascinating in there :)

But first, we need to solve a small network puzzle: what IP address does the RPi have? There are two ways to find out: 1) have a monitor plugged into the RPi when it boots up, and the value should show up on the screen, or 2) consult the “DHCP client table” in your network router, which is usually the DHCP server for your local network. In my case, the IP address is, so that’s what I’ll use in the rest of this post.

The next step is to log into your RPi over the network, using ssh or Putty. Putty has its own GUI way of logging in, but in the Mac’s “Terminal” app or from the Linux command line, this is the way to do so (i.e. user name “pi”, IP address

ssh pi@

Here’s a transcript of what it will look like the first time around:

Screen Shot 2013-02-20 at 18.37.25

Bingo, you have logged into your RPi – welcome to the Linux command line!

You now need to go through some specific first-time RPi configuration adjustments, as hinted on that last line above. See also an earlier post for some more info (but don’t install Node.js just yet: that will be covered later on):

Screen Shot 2013-02-20 at 18.37.54

It’s a menu-based step, just type sudo raspi-config, and once finished, reboot the system (by typing the command sudo reboot). This last step is essential in this case.

Then wait a minute or two and log back in as before. You’ll again be connected to the Linux “shell”, patiently awaiting your command(s). If it’s all totally new to you, then trying out a few commands to familiarise yourself might be a good idea at this point. See this site for some tips and ideas. As long as you use the “sudo” command with caution, which gives you full “superuser” (admin) rights, there’s not much that can really go wrong. And if things ever get totally out of hand, you can always get back to this starting situation by re-formatting the SD card and starting over.

In the next step, we’ll make the RPi the central node of our Wireless Sensor Network.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.05 – RPi in a nutshell

In Uncategorized on Feb 9, 2013 at 00:01

Welcome to the fifth instalment of Dive Into JeeNodes. Linux, anyone?

(Note: these instructions have been updated to use the Feb 9th Raspbian image)

The Raspberry Pi is a single-board computer running Linux. What sets it apart from the others is its low cost and the fact that it was designed to be educational and hackable:


It’s really the Arduino story all over again, but for a different audience and with a completely different hardware and software underpinning. As a matter of fact, the “RPi” as it’s often called, can run the Arduino IDE on-board, which means these two platforms are really complementary in actual use.

The performance of this ARM board with on-board GPU is very impressive, but it’s no match for computers with a modern Intel processor (nor does it need to be, at that price!). As a ballpark figure, think of it as 1/10th to 1/20th the performance of a i5 or i7.

One way to use the RPi, is to hook up a USB keyboard and HDMI monitor, but in the context of DIJN, we’re really more interested in using this thing as a little server on the network. Note that a RPi can easily be used for both, and that all you need to handle different scenario’s is a couple of SD cards, each configured with the proper Linux system. You could also install a full GUI and still use it as a server, but the goal here is to start as simple as possible. There are already enough things to get to grips with as it is.

The Raspberry Pi Model B comes with built-in 10/100 MHz wired Ethernet, which is what will be used here. With a few extra steps, it’s also possible to plug in a WiFi dongle and connect to your in-house WLAN.

The main preparation needed, is to create an SD card with a Linux system on it. This is a bit of a chicken-and-egg problem, since the RPi cannot be used for it while it’s not yet up and running. Fortunately, the task is not too complicated, if you have a way to plug an SD card into your main Windows, Mac, or Linux workstation or laptop:

  1. Get a new SD card – 4 .. 8 Gb will be more than enough. There can be significant differences in performance between SD cards from different vendors, so if you’re keen on squeezing the max out of this setup, google for good choices. I’m getting excellent mileage out of Platinum’s 4 GB Class 10 SDHC cards.
  2. Download the latest SD image from here, which is based on Raspbian, i.e. Debian for the RPi. This is a 500 MB download. Unzip to get an SD “image” of just under 2 GB.
  3. Follow the instructions on this page, using the section which applies to your platform. You have to be careful, it’s possible to wipe-out your workstation / laptop hard drive if you make a mistake with disk names!

The end result is a bunch-of-bits, ready to make that little RPi board do fascinating things:


For the next step, you’re going to need a Raspberry Pi model B, a 5V power adapter with micro USB connector, and an Ethernet cable to hook it all up to your local network.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS – The DIJN.12 post has a link to a fully-configured Raspbian + HouseMon SD image.

DIJN.04 – Receive data with a JeeLink

In Uncategorized on Feb 8, 2013 at 00:01

Welcome to the fourth instalment of Dive Into JeeNodes. Let’s go wireless!

The JeeLink is a version of the JeeNode with some extra features, but the main reason I like to use it as central node is that it comes as a little pluggable unit – quite convenient, here it is in action on my MacBook, for example:


As delivered, JeeLinks and JeeNodes come with the RF12demo sketch pre-installed. This can be used for a variety of tasks, including basic configuration and basic packet reception.

But first, we need to be able to connect to the JeeLink over serial USB. This is similar to the uploading performed in the previous DIJN post:

  1. Plug in the JeeLink – it will briefly flash on its red and green LEDs.
  2. Start the Arduino IDE, and go to the Tools -> Serial Port menu entry. Then, make sure to select the JeeLink, which will show up as a second serial port (assuming you still have the other JeeNode plugged in).
  3. In the IDE, click on the right-most button, the one with the magnifying glass, to bring up the “Serial Monitor”.

That last step will open up a new window, with something like this:

Screen Shot 2013-02-07 at 14.05.31

If you only get gibberish, you need to adjust the baud rate popup in the lower right to 57600. If you don’t get anything: double-check that you selected the correct serial port.

What we need to do next, is to match up the wireless settings to be able to pick up the packets sent from that other JeeNode. There are a few settings:

  • The frequency band must be set to 868 MHz. Note that this may differ from the unit you have, and the frequency used in your part of the world, but all JeeLinks and JeeNodes can operate in any of the 433 / 868 / 915 MHz bands. The only difference is that the power and range will be dramatically reduced in bands for which that wireless module is not optimised. In this case, we don’t care (yet), since the first tests will be across less than 1 meter, which can easily be bridged in any frequency band.
  • The “net group” must match between all transmitters and receivers, to allow them to pick up each other’s packets. In this case, we need to use group “100”.
  • The “node ID” is normally different for each node, so they can tell each other apart. For a central node, it’s usually best to pick the special ID “31”.

The commands to set these parameters can be combined into a single line:

Screen Shot 2013-02-07 at 14.15.51

Once the commands have been entered, you will see a confirmation shown as follows:

 _ i31 g100 @ 868 MHz 

And then, if the sending JeeNode is still powered up, these – once a second:

OK 1 0
OK 1 1
OK 1 2
OK 1 3
OK 1 4
OK 1 5

The “OK” text indicates reception of a valid packet (you may also get some “?” packets, which are usually caused by picking up noise or other 868 MHz transmitters), the first byte indicates that this is a regular packet broadcast coming from node #1, and the second byte is the counter, incremented for each new packet, as you can see in the test1 source code.

Congratulations. You have created your own Wireless Sensor Network. Well, no actual sensors yet, but we’ll get to that soon enough. First, we need to look more deeply into this thing called a “Ra(h)spberry P(h)i” (here’s a little exercise to get you started).


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)


In Musings on Feb 7, 2013 at 00:01

About an hour’s drive south of Houten, in the Netherlands, lies the city of Eindhoven. This is where the Philips family started, and as a technological giant, the company has shaped the city over the decades. Now, it’s also the beating heart of a fascinating new “brainport”, with a University of Technology (TU/e) and a creative Design Academy (DAE).

Dutch Design is known around the world by now.

And then there’s Strijp – an area being developed where the Philips factories once where.

Wow – if only I could be 20 again, and live that life and learn from scratch today!

These trends fascinate me. The idea that future generations will grow up in such a fertile environment, changing everything in the world around them, down to the very fabric of living together and applying creativity and technology in ways never before imagined.

If you have 20 minutes, and turn down your speakers a bit… here’s a video describing the vision and work of the city planners and architects involved in the Strijp project:


Wow. Just wow. If this doesn’t lead to an explosion of creative makers and idealists applying technology to shape the world of the(ir) future, then I don’t know what will.

I do have to add that I have a special emotional bond to what is starting to happen there, as our daughter Myra and her boyfriend Pieter, both currently studying at the DAE, are about to settle into one of the first apartments being set up in those re-purposed Philips buildings at the “Torenallee”.

Yeah, ok, so I’m a proud papa, I guess… :)

But hey, whatever. It’s very exciting to see these trends, ehm, materialise in today’s world!

HouseMon resources

In AVR, Hardware, Software, Musings, Linux on Feb 6, 2013 at 00:01

As promised, a long list of resources I’ve found useful while starting off with HouseMon:

JavaScript – The core of what I’m building now is centered entirely around “JS”, the language behind many sites on the web nowadays. There’s no way around it: you have to get to grips with JS first. I spent several hours watching most of the videos on Douglas Crockford’s site. The big drawback is the time it takes…

Best book on the subject, IMO, if you know the basics of JavaScript, is “JavaScript: The Good Parts” by the same author, ISBN 0596517742. Understanding what the essence of a language is, is the fastest way to mastery, and his book does exactly that.

CoffeeScript – It’s just a dialect of JS, really – and the way HouseMon uses it, “CS” automatically gets compiled (more like “expanded”, if you ask me) to JS on the server, by SocketStream.

The most obvious resource,, is also one of the best ways to understand it. Make sure you are comfortable with JS, even if not in practice, before reading that home page top to bottom. For an intruiging glimpse of how CS code can be documented, see this example from the CS compiler itself (pretty advanced stuff!).

But the impact of CS goes considerably deeper. To understand how Scheme-like functional programming plays a role in CS, there is an entertaining (but fairly advanced) book called CoffeeScript Ristretto by Reginald Braithwaite. I’ve read it front-to-back, and intend to re-read it in its entirety in the coming days. IMO, this is the book that cuts to the core of how functions and objects work together, and how CS lets you write on a high conceptual level. It’s a delightful read, but be prepared to scratch your head at times…

For a much simpler introduction, see The Little Book on CoffeeScript by Alex MacCaw, ISBN 1449321046. Also available on GitHub.

Node.js – I found the Node.js in Action book by Mike Cantelon, TJ Holowaychuk and Nathan Rajlich to be immensely useful, because of how it puts everything in context and introduces all the main concepts and libraries one tends to use in combination with “Node”. It doesn’t hurt that one of the most prolific Node programmers also happens to be one of the authors…

Another useful resource is the API documentation of Node itself.

SocketStream – This is what takes care of client-server communication, deployment, and it comes with many development conveniences and conventions. It’s also the least mature of the bunch, although I’ve not really encountered any problems with it. I expect “SS” to evolve a bit more than the rest, over time.

There’s a “what it is and what it does” type of demo tour, and there is a collection on what I’d call tech notes, describing a wide range of design docs. As with the code, these pages are bound to change and get extended further over time.

Redis – This a little database package which handles a few tasks for HouseMon. I haven’t had to do much to get it going, so the README plus Command Summary were all I’ve needed, for now.

AngularJS – This is the most framework-like component used in HouseMon, by far. It does a lot, but the challenge is to understand how it wants you to do things, and altough “NG” is not really an opinionated piece of software, there is simply no other way to get to grips with it, than to take the dive and learn, learn, learn… Let me just add that I really think it’s worth it – NG can be magic on the client side, and once you get the hang of it, it’s in fact an extremely expressive way to create a responsive app in the browser, IMO.

There’s an elaborate tutorial on the NG site. It covers a lot of ground, and left me a bit overwhelmed – probably because I was trying to learn too much as quickly as possible…

There’s also a video, which gives a very clear idea of NG, what it is, how it is used, etc. Only downside is that it’s over an hour long. Oh, and BTW, the NG FAQ is excellent.

For a broader background on this sort of JS frameworks, see Rich JavaScript Applications by Steven Sanderson. An eye opener, if you’ve not looked into RIA’s before.

Arduino – Does this need any introduction on this weblog? Let me just link to the Reference and the Tutorial here.

JeeNode – Again, not really much point in listing much here, given that this entire weblog is more or less dedicated to that topic. Here’s a big picture and the link to the hardware page, just for completeness.

RF12 – This is the driver used for HopeRF’s wireless radio modules, I’ll just mention the internals weblog posts, and the reference documentation page.

Vim – My editor of choice, lately. After many years of using TextMate (which I still use as code browser), I’ve decided to go back to MacVim, because of the way it can be off-loaded to your spine, so to speak.

There’s a lot of personal preference involved in this type of choice, and there are dozens of blog posts and debates on the web about the pro’s and con’s. This one by Steve Losh sort of matches the process I am going through, in case you’re interested.

Best way to get into vim? Install it, and type “vimtutor“. Best way to learn more? Type “:h<CR>” in vim. Seriously. And don’t try to learn it all at once – the goal is to gradually migrate vim knowledge into your muscle memory. Just learn the base concepts, and if you’re serious about it: learn a few new commands each week. See what sticks.

To get an idea of what’s possible, watch some videos – such as the vim entries on the DAS site by Gary Bernhardt (paid subscription). And while you’re at it: take the opportunity to see what Behaviour Driven Design is like, he has many fascinating videos on the subject.

For a book, I very much recommend Practical Vim by Drew Neil. He covers a wide range of topics, and suggests reading up on them in whatever order is most useful to you.

While learning, this cheatsheet and wallpaper may come in handy.

Raspberry Pi – The little “RPi” Linux board is getting a lot of attention lately. It makes a nice setup for HouseMon. Here are some links for the hardware and the software.

Linux – Getting around on the command line in Linux is also something I get asked about from time to time. This is important when running Linux as a server – the RPi, for example.

I found the resource which appears to do a good job of explaining all the basic and intermediate concepts. It’s also available as a book, called “The Linux Command Line” by William E. Shotts, Jr. (PDF).

There… the above list ought to get you a long way with all the technologies I’m currently messing around with. Please feel free to add pointers and tips in the comments, if you think of other resource which can be of use to fellow readers in this context.

Orthogonal learning

In Musings on Feb 5, 2013 at 00:01

It’s been about 2 months now since I switched back to doing mostly software development – in an attempt to get a new home monitoring software going. That does not mean that electronics and hardware design are off the table – far from it, in fact – but the switch has turned out to be essential for me to break out of the box, and find the headroom needed to make things happen.

This is not just a post about HouseMon, however. What I’d like to describe, is the process I went through, and some first experiences and notes.

Picking a new programming language is no big deal, really, but in this case I intend to go in very deep. I don’t just want to program in CoffeeScript, I want to become productive in it. If this will take several months – as I expect – then so be it. As the saying goes:

“If it’s a job’s worth doing, it’s a job worth doing well.”

Note that I didn’t set out to use CoffeeScript, or Node.js, or AngularJS. Half a year ago, I wanted to use ZeroMQ and Lua, in fact. But there were tiny gaps and little doubts in my mind, and I thought it best to let the whole issue rest. Turns out that this strategy works really well (in cases where one can afford the extra delay, of course) – if a decision doesn’t solidify once taken… wait, don’t rush: it may be trying to tell you something!

Of course every switch will be painful, to some degree. The habitual is, by definition, more convenient. A switch to something new and unfamiliar is a step back, sometimes even a huge step back. And at the end of this post I’ll tell you about one more switch which is also excruciatingly painful for me this very moment…

Not all changes turn out well over time. There are several forum choices to prove it, as all long-time readers and participants at JeeLabs know, and which I still feel bad about.

So how does one pick new technology, apart from choosing based on requirements?

My answer to this is now: follow your passion, but don’t let it blind you…

Go for what interests you, go read and surf like crazy, and try the things which you like. Because unconsciously, your mind will make lots of choices for you, leaving you free to apply your reasoning for making trade-offs which need to be made anyway.

The second guideline IMO, is to never go down a rabbit’s hole – if all the decisions you have to make end up becoming inter-dependent, to the point that you have to accept a single package deal, then… stop. Choices are compromises. There is no silver bullet. If something looks too good to be true – then it probably is (hmmm… where have I heard that before?).

I’ve taken quite a few very far-reaching decisions, lately, w.r.t. HouseMon. But watch this:

  • CoffeeScript is not essential – it interoperates 100% with standard JavaScript
  • Node.js is not essential – it could be replaced by Ruby On Rails, or Django
  • SocketStream is not essential – it merely streamlines the use of WebSockets
  • Redis is not essential – it could have been ZeroMQ, or Mosquitto, or some DB
  • AngularJS is not essential – it could be replaced by Knockout, or Backbone.js
  • Safari is not essential – everything will work fine with Chrome or FireFox
  • the Arduino IDE is not essential – underneath it’s all done with avr-gcc
  • the JeeNodes are not essential – there are several µC alternatives out there
  • the RFM12B is not essential – again, there are alternatives one could use
  • the Mac I use is not essential – it’s merely my personal platform choice
  • Vim is not essential – it just happens to be the editor I’ve chosen to work with

The key phrase is “is not essential”, and the key concept is orthogonality – the choices made above are to a large degree (but not 100%) independent of each other!

If any of the above turns out to be a disappointment, I can still get rid of it, without the whole plan unraveling and blowing up in my face (I hope…).

Which brings me to the main point of this post: by having a certain amount of de-coupling in there, something else also becomes quite an important benefit… the task of learning new stuff has a reduced risk! – if any of the above hits a dead end, I will lose my investment in time and energy in part of what I had to go through. But not all.

It’s really the same as in the real world: don’t put all your eggs in the same basket, as the saying goes. If one of ’em crashes and breaks, it won’t be the end of the world that way.

With so many eggs, eh, I mean technology choices, forcing me to re-learn, there is one more which I’ve decided to go for. Long ago, as a kid I took a course in touch typing, but never found the courage to carry it through for programming – all those nasty characters and punctuation marks were (and are) scaring the heck out of me! Well, no more excuses and no more hunt-and-peck: this post was written in touch-typing mode. Let me just say that my hands are hurting like crazy right now, and that it took – a g e s – to write!

Tomorrow, I’ll post an annotated list of pointers, for many of the items listed above, about the information I found which is really helping me forward right now. You may not make all the same choices, but with a bit of luck there will be something in there for everybody.

DIJN.03 – Store code in a JeeNode

In Uncategorized on Feb 4, 2013 at 00:01

Welcome to the third instalment of Dive Into JeeNodes. Some real hardware now!

Both the remote sending side and the more nearby reeiving side of the wireless sensor link are handled by a JeeNode – which is essentially an Arduino-like microcontroller with some support circuitry, plus a low-power wireless radio module.

The JeeLink is a modified version of a JeeNode in a USB-stick form factor and enclosure.

Remote nodes

But let’s start with that remote JeeNode first. What we’re going to set up is a little self-contained unit with a light sensor, a battery pack to power the whole thing, and the proper software pre-loaded onto the ATmega328 microcontroller inside that JeeNode.

The whole point of this “node” is that it’s completely autonomous. You’ll set it up once, take it to the place where you want to perform the light level measurements, hook up the battery pack, and that’s it. Its only task is to periodically measure the current light intensity, and report it over wireless – so that we can pick up the signal and pass it to the Raspberry Pi in a central location in the house, collect the data, tie it into the web server, and make it available on the local Ethernet network for visualisation in the browser.

Let’s set up that node now – except that it won’t be the final measurement code. Let’s start with a simpler goal first: sending out a counter value once a second, to see if we can pick up that value. With a little blink “blip” to let us know that the node is doing its work. But no light sensor for now…

As it so happens, JeeLib now includes an example sketch which does exactly that – see test1.ino on GitHub for the code (make sure you have the latest version of JeeLib!).


To get this code into the JeeNode, we need to communicate with it. This is done by connecting the JeeNode to our workstation / laptop – the one where we installed the Arduino IDE. That connection is created via a USB cable and “BUB”, as follows:


The next step is to tell the IDE which serial port to use to reach our JeeNode:

Go to the “Tools -> Serial Port” menu in the IDE, and select the serial port under which the USB BUB has registered itself. The actual name will depend on the platform (Windows, Mac OSX, or Linux) and in the case of Mac OSX, also on the unique name of that particular USB BUB.

Now, we need to load the proper sketch into the Arduino IDE:

Go to the “File -> Sketchbook -> libraries -> JeeLib -> DIJN -> test1” menu. This will load the test sketch.

To check whether the code is in good shape, click on the leftmost round button with the checkmark in it (or go to the “Sketch => Verify / Compile” menu – same thing).

(rf12_sendNow was recently added to JeeLib, make sure you’re using a recent version)

If all is well, you’ll get one line from the compiler, reporting something like:

Binary sketch size: 3,958 bytes (of a 32,256 byte maximum)

The value may differ a bit, but this indicates that the sketch is ok and can be uploaded.

The last step is where the real upload takes place, and since it includes the compile step, you could in fact have skipped that last instruction above.

Click on the second button from the left (the one with the arrow pointing right).

If all is well, you’ll see some LEDs blink the USB BUB, and when all is done a promising message “Done uploading.” will appear in the IDE (in the middle green bar).

If you get an error, make sure you have selected the proper board type (menu “Tools -> Board -> Arduino Uno”). Still no luck? Check that the USB serial port connection, the cable, the BUB, and the JeeNode are all hooked up as shown above. When in doubt, disconnect and reconnect gently.

Getting through this first upload is an important milestone. There are a few trouble spots, depending on the platform you use. You may have to install the latest FTDI USB driver. You may have to check that the serial USB drivers are present. There is not much generic advice to give here – other than “google for the error message you get”, in the hope that others have run into the same issue and that you’ll find a page with solution or tip.

At this point, you should see a brief LED flash once a second. This indicates that the JeeNode is running the test sketch and sending out its counter packet once a second.

Congratulations, you have set up a live test node for your wireless sensor network!


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.02 – The Arduino IDE

In Uncategorized on Feb 3, 2013 at 00:01

Welcome to the second instalment of Dive Into JeeNodes. Let’s get going!

DIJN differs from a conventional laptop / workstation setup, in that it will be interfacing to the real world via microntrollers running our code. To get that code onto these devices, we need to cross compile the software on our (big) computer running Windows, Mac OSX, or Linux, and then upload it into the permanent memory of a microcontroller chip.

The Arduino has brought this approach to the masses, by providing an open-source IDE to develop, compile, upload, and run the software in the form of embedded software (called “sketches” in Arduino-speak). The success of the Arduino is partly due to the fact that these tools have become easy to install and use.

Before the Arduino era, cross compilers were complex, platform-specific, and too expensive for personal use in small projects. Now they are free, open source, extensively documented, and surrounded by a vibrant community.

Installing the IDE

As a first step, we need to perform that installation. This process is fully described on the Getting Started with Arduino page.

The recommended version is Arduino IDE 1.0.3, at the moment. Note that you don’t have to follow all the steps (1..7 on Windows and Mac OSX are enough), because we’re not going to connect any hardware or upload any software just yet.

Go ahead, perform the installation now. Should take no more than 15 minutes, normally.

If all went well, you should be able to launch the IDE and see something like this after you open the LED blink example sketch with File > Examples > 1.Basics > Blink. :


If you click on the leftmost round button, the one with the checkmark, then the bottom part of the window should also show that same text:

Binary sketch size: ...

Congratulations, you’ve installed a fairly large package and it’s all running as expected.

What is an IDE?

But, wait… what is this thing? And why do we need it?

The “Integrated Development Environment” is where we create the software that ends up running on our microcontroller(s) – be it an Arduino, a JeeNode (which can be used just like an Arduino Uno because it has the same ATmega328 microcontroller), an RBBB, or a range of other similar products.

The Arduino IDE is an editor, a compiler (translating the program source text into some obscure binary code which one particular type of microcontroller understands), an uploader, and a serial terminal interface – all rolled into one app. If your background is PHP or Python, you may want to read a bit more about compiled languages.

The IDE also defines various conventions for how and where to store different pieces of software if we have more than one microcontroller, how to deal with standard libraries obtained from other software developers, and which type of microcontrollers we can handle through this environment.

All of this is essential during development, but once the compiled code has been stored in the microcontroller, it’s not really needed anymore. In day-to-day use, the IDE plays no role – until we decide to make changes, add new features, extend the code… or fix bugs. Then the whole process kicks in again – we load the sketch we were working on back into the IDE, make the changes, compile, upload, try things out, and again: once it works, the IDE can be closed and left alone.

Note that there is no way back: you can’t get code out of a microcontroller and turn it into the source text you wrote. So an essential part of the IDE is to help you manage all those different tasks and versions used for different microcontrollers in your (usually growing) collection of remote sensor nodes. It’s a good idea to think about how you name stuff and where you put it, because you’ll want to find it again whenever you fire up the IDE.

So we write (or use someone else’s) code, as a “sketch” for a particular task, and we upload it into one or more microcontrollers in the different remote nodes. We give these sketches names, and we use the IDE to keep a copy of each of them, to be able to improve / refine / fix / alter the logic of certain nodes at any time in the future.

Some people spend all their time in the IDE (because that’s where lots of code is created). We usually call ’em software developers, although I think IDEalists sounds a lot nicer!


Software can grow to become quite complex. Yet some parts are going to be used over and over again in different sketches. In the case of wireless JeeNodes, the best example is the “RF12” wireless driver, i.e. a piece of software which takes care of sending and receiving wireless data packets.

This sort of “standard” extra functionality is usually made available as software libraries. In the Arduino IDE, such libraries can contain the re-usable code as well as example sketches to try everything out.

The library we want is called “JeeLib”, and is documented here.

We need to add this essential ingredient to our IDE. It has been developed and extended by JeeLabs and other people over the years, and the latest version of this code is available from GitHub – a popular place for a huge range of open source software projects you’re likely to want to use. The GitHub website is enormous, but easy to search and navigate.

Installation requires some care, because we must download the entire set of files and put it in a specific location for the Arduino IDE to be able to find it. There is a good introduction on how to install libraries.

To download the latest version of JeeLib, click on this link, which you can also find as a “ZIP” button in the top left of this page on GitHub. Unpack and rename the resulting folder to just “JeeLib”, then follow the instructions on that how to install page.

To check that things went right, you have to first work around a quirk of the IDE:

Quit the Arduino IDE and then restart it, so that it will pick up all library changes.

Lastly, see if you can locate the following entry in the Arduino IDE’s nested menus:

File -> Sketchbook -> libraries -> JeeLib -> DIJN -> test1

Found it? Terrific. You’ve completed the software installation for embedded Arduino and JeeNode use. As far as the software required for DIJN is concerned, your laptop / workstation is ready to rock.


(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.01 – Introduction

In Uncategorized on Feb 2, 2013 at 00:01

Welcome to the first instalment of Dive Into JeeNodes. This one is a bit lengthy, alas…

The purpose of these articles is to introduce you to the world of Physical Computing and Wireless Sensor Networks in an easy to follow way. We will create a low-cost setup to let you track the light level of some spot anywhere in your house and present this information on any computer, tablet, or mobile phone with access to your home network.


In more visual terms, this is what we’re aiming for, and what we need to to set up for it:

dijn01-essence.png   dijn01-diagram

Don’t worry too much about the detailed diagram on the right – it’s just to give you an idea of the pieces involved. Here’s a quick rundown of the hardware which will be used:

  • a remote wireless sensor node – which will be a JeeNode SMD
  • the sensor – an LDR, which changes resistance depending on current light levels
  • an USB BUB to load new code into the JeeNode
  • a central wireless node to collect the measurement data – this will be a JeeLink
  • a small Linux “bare board” computer – we’ll use a Raspberry Pi (with Raspbian)
  • your existing local wired (and optionally wireless) network
  • some cables, a USB power supply, batteries, and…
  • your time, your attention, and your enthusiasm, of course!

In case you’re wondering: the Wireless Starter Pack includes much of the above.

None of this is set in stone. It’s possible to replace the Raspberry Pi with another board, or even run that part on your exsiting workstation, laptop, or server. You could use two JeeNodes, or replace the JN SMD + USB BUB by a JeeNode USB, or even create your own variations – but to limit the scope of this DIJN series, the above will be used here.

Note very specifically that you will not need to solder anything for this setup, although it’s very likely that you’ll be itching to do so once the basic system is working – because that’s how you can get more sensors and remote control features in there.

The final result will be something you can leave on, consuming a fraction of a normal PC’s or even laptop’s power – fully unattended, running a dedicated built-in webserver where everything gets configured and where the up-to-date light level “readings” will be available.

It might not sound like much… but don’t jump to conclusions just yet!


This is a truly minimal setup. It ought to be possible to assemble this in say a weekend, even if you’ve never done any hardware or software development before. The total cost should remain well under €150, more or less evenly split between the two JeeNodes + sensor and the Raspberry Pi + power and cabling. It’s still a significant amount of money… for which you could also buy a game console, or go watch all the latest blockbuster movies in 3D.

So what’s the point?

Well, it might not look like much, but this little setup opens up a whole new world and offers access to a surprisingly broad range of state-of-the-art technologies:

  • our world is being filled with sensors at an astonishing pace – just think of all the new “senses” mobile phones have acquired in the last couple of years
  • wireless information exchange is becoming so ubiquitous, it’s not even funny anymore: we live in an always-connected age, and that trend is here to stay
  • software, in the form of built-in intelligence, is everywhere – from the smallest ultra-low power microcontrollers, to tiny functionally-complete computers running Linux
  • hardware is shrinking and spreading everywhere, and more and more based on self-contained extremely sophisticated low-cost electronic chips
  • web technology is advancing faster than ever, covering everything from “big” desktops, to laptops, tablets, and mobile phones

With the DIJN setup presented here, everything just mentioned becomes something you can investigate, explore, tinker with, alter, extend, improve upon, or simply… learn from!

Whether you want to do this out of geek curiosity, for general self-education, to refresh knowledge from the past, to increase your job opportunities, to enhance specific skill sets in… take a deep breath: embedded or web software, microcontroller or Linux hardware, basic electronics, advanced chip capabilities, miniaturisation, ultra-low power design, system integration, wireless networking, communication protocols, C / C++ programming, JavaScript, shell scripting, Linux command-line tools, … everything is in there, in that “little” setup, ready to go where your interests (and your patience) take you.

Not only is everything open source, and hence ready to be explored, it’s also virtually risk free: even if you were to damage something (which is surprisingly hard, with a few simple precautions), it would be in the context of a very limited and low-cost setup.

Just add an extra wireless node, and go solder things together for the first time in your life. Or look into that Linux stuff which drives the Raspberry Pi. Perhaps you’re curious about WebSockets and real-time software. It’s all there. It doesn’t bite. It’s probably easier to understand, because small systems have to be simple by design to maintain their low cost.

Yet at the same time, it’s all really state-of-the-art in many ways. The power levels and battery life achievable with JeeNodes is measured in years. The performance of the Raspberry Pi is such that it can actually drive a display with full-screen HD movies. And the Node.js-based web server technology we’re going to use is at the forefront of what the web has to offer today. This isn’t some mix of technologies cobbled together “just because it works”. Under the hood is what drives our technological world today, and a glimpse of what will be evolving into the technology of tomorrow.


The DIJN series of posts is aimed at being totally, completely, fully, truly practical. Every post (ehm, except this one) is about making things work. Concrete steps, describing everything needed to create that final setup. This is going to be as “hands on” as it gets…

Then again, not everything is going to be spelled out in baby steps, and where possible, pointers will be supplied to point to instructions elsewhere, such as how to set up the Arduino IDE, or how to prepare an SD card for use as bootable system in the Raspberry Pi.

The goal is really to reach that finish line, for everyone who’s interested, regardless of specific knowledge. I.e. a working system, spanning a surprinsigly wide range of topics and technologies, but by necessity a very simple system. The idea is that once you have a working setup, you also have the foundation for diving in deep, to explore whatever aspect interests you most, and to alter and extend as much as you like.

This series will not explain how everything works. Nor go into more advanced topics such as implementing ultra-low power modes in the sensor node. Or extending the web server with huge amounts of logic and web page presentation. That’s step 2 (and 3, and 4).

Winding down

This post ended up being much longer than planned. Let’s hope the next ones fare better. There will definitely not be a DIJN post every day – it’ll be spread out over this month, to allow adding some lighter material and other topics from/about JeeLabs. There are only so many hours in a day – and that applies to both reading and writing all this stuff :)

I’m quietly hoping that a few people will try and follow along right away though, and hopefully also comment on where information is incomplete or incorrect. But even if you don’t have the time or opportunity to tag along as this unfolds, please note that this series of posts will be available from the Dive Into JeeNodes page on the Café wiki.

There’s a lot to cover. And I hope there is something in here for everyone. Last but not least: please do comment and make suggestions. That’s how weblogs like this work best.


Dive Into JeeNodes

In AVR, Hardware, Software, Linux on Feb 1, 2013 at 00:01

Welcome to a new series of limited-edition posts from JeeLabs! Read ’em while they last!

Heh… just kidding. They’ll last forever of course, as does everything on this thing called internet. But what I’m going to describe in probably a dozen posts or so is the following:


Hm, that doesn’t quite explain it, I guess. Let me try again:

JC's Grid, page 63

So this is to announce a new “DIJN” series of weblog posts, describing how to set up your own Wireless Sensor Network with JeeNodes, as well as the infrastructure to report a measured light-level somewhere in your house, in real time. The end result will be fully automated and autonomous – you could take your mobile phone, point it to your web server via WiFi, and see the light level as it is that very moment, adjusting as it changes.

This is a far cry from a full-fledged home monitoring or home automation system, clearly – but on the other hand, it’ll have all the key pieces in place to explore whatever direction interests you: ready-made sensors, DIY sensors, your own firmware on the remote nodes, your own web pages and automation logic on the central server… it’s up to you!

Everything is open source, which in this context matters a lot, because that also means that you can really dive into any aspect of this to learn and explore the truly magical world of Physical Computing, Wireless Sensor Networks, environmental sensing and control, as well as state-of-the art web technologies.

The focus will be on describing every step needed to implement this from scratch. I’ll cover setting up all the necessary software and hardware, in such a way that if you know next to nothing about any of the domains involved, you can still follow along and try it out – whether your background is in software, electronics, wireless, or none of these.

If technology interests you, and if I can bring across even a small fraction of the fun there is in tinkering with this stuff and making new things up as you g(r)o(w) along, then that would be a very nice reward for everyone involved, as far as I’m concerned.

PS. “Dijn” is also old-Dutch for “your” (thy, to be precise). Quite a fitting name in my opinion, as this sort of knowledge is really yours for the taking – if you want it…

PPS. For reference: here is the first post in the series, and here is the overview.

Now what?

In Musings on Jan 31, 2013 at 00:01

(Warning, this post is a bit about playing the devil’s advocate…)

Ok, so now I have this table with incoming data, updated in real time – Ajax polling is so passé – with unit conversions, proper labeling, and locations associated with each device:

Screen Shot 2013-01-28 at 16.19.55 copy 2

As I’ve said before: neat as a gimmick, but… yawn, who wants to look at this sort of stuff?

Which sort of begs the question: what’s the point of home monitoring?

(Automation is a different story: rules and automation could certainly be convenient)

What’s the point? Flashy / fancy dashboards with clever fonts? Zoomable graphs? Statistics? Knowing which was the top day for the solar panels? Calculating average temperatures? Predicting the utility bill? Collecting bragging rights or brownie points?

Don’t get me wrong: I’ve got a ton of plans for all this stuff (some of them in other directions than what you’d probably expect, but it’s way too early to talk about that).

But for home monitoring… what’s the point? Just because we can?

The only meaningful use I’ve been able to come up with so far is to save on energy (oh wait, that’s now called “reducing the carbon footprint”). And that has already been achieved to a fairly substantial degree, here at JeeLabs. For that, all I need really, are a few indicators to see the main energy consumption patterns on a day-to-day basis. Heck, a couple of RGB LEDs might be enough – so who needs all these figures, once you’ve interpreted them, drawn some conclusions, and adjusted your behaviour?

Remote compilation

In Software on Jan 30, 2013 at 00:01

Now we’re cookin’ …

I’ve implemented the following with a new “compile” module in HouseMon – based on, which is also available for the Raspberry Pi via Debian apt:

JC's Grid, page 62

In other words, this allows uploading the source code of an Arduino sketch through a web browser, manage these uploaded files from the browser, and compile + upload the sketch to a server-side JeeNode / Arduino. All interaction is browser based…

Only proof of concept, but it opens up some fascinating options. Here’s an example of use:

Screen Shot 2013-01-29 at 21.54.13

(note that both avr-gcc + avrdude output and the exit status show up in the browser!)

The processing flow is fairly complex, given the number of different tasks involved, yet the way Node’s asynchronous system calls, SocketStream’s client-server connectivity, and Angular’s two-way webpage bindings work together is absolutely mind-blowing, IMO.

Here’s the main code, running on the server as briqs/ =
  name: 'jcw-compile'
  description: 'Compile for embedded systems'
  menus: [
    title: 'Compile'
    controller: 'CompileCtrl'

state = require '../server/state'
fs = require 'fs'
child_process = require 'child_process'
ss = require 'socketstream'

# TODO hardcoded paths for now
SKETCHDIR = switch process.platform
  when 'darwin' then '/Users/jcw/Tools/sketch'
  when 'linux' then '/home/pi/sketchbook/sketch'

# callable from client as: rpc.exec 'host.api', 'compile', path
ss.api.add 'compile', (path, cb) ->
  # TODO totally unsafe, will accept any path as input file
  wr = fs.createWriteStream "#{SKETCHDIR}/sketch.ino"
  fs.createReadStream(path).pipe wr
  wr.on 'close', ->
    make = child_process.spawn 'make', ['upload'], cwd: SKETCHDIR
    make.stdout.on 'data', (data) ->
      ss.api.publish.all 'ss-output', 'stdout', "#{data}"
    make.stderr.on 'data', (data) ->
      ss.api.publish.all 'ss-output', 'stderr', "#{data}"
    make.on 'exit', (code) ->
      cb? null, code

# triggered when bodyParser middleware finishes processing a file upload
state.on 'upload', (url, files) ->
  for file, info of files 'uploads',
      key: info.path
      file: file
      size: info.size
      date: Date.parse(info.lastModifiedDate)

# triggered when the uploads collection changes, used to clean up files
state.on 'store.uploads', (obj, oldObj) ->
  unless obj.key # only act on deletions
    fs.unlink oldObj.key

The client-side code needed to implement all this is even shorter: see the controller code and the page definition on GitHub.

There’s no error handling in here yet, so evidently this code is not ready for prime time.

PS – Verified to also work on the Raspberry Pi – yippie!

Blink Plug meets NG

In Hardware, Software on Jan 29, 2013 at 00:01

Last month, I described how to hook up a JeeNode with a Blink Plug to Node.js via SocketStream (“SS”) and a USB connection. Physical Computing with a web interface!

That was before AngularJS (“NG”), using traditional HTML and JavaCoffeeScript code.

With NG and SS, there’s a lot of machinery in place to interface a web browser with a central Node process. I’ve added a “blink-serial” module to HouseMon to explore this approach. Here’s the resulting web page (very basic, but fully dynamic in both directions):

Screen Shot 2013-01-28 at 19.17.40

The above web page is generated from this client/templates/blink.jade definition:

Screen Shot 2013-01-28 at 18.42.02

There are two more files involved – client/code/modules/ on the client:

Screen Shot 2013-01-28 at 18.43.41

… and briqs/, which drives the whole shebang on the server:

Screen Shot 2013-01-28 at 20.17.40

I’m not quite happy yet with the exact details of all this, but these 3 files are all there is to it, and frankly I can’t quite imagine how a bidirectional real-time interface could be made any simpler – given all the pieces involved: serial interface, server, and web browser.

The one thing that can hopefully be improved upon, is the terminology of it all and the actual API conventions. But that really is a matter of throwing lots of different stuff into HouseMon, to see what sticks and what gets messy.


PS – What’s your take on these screenshots? Ok? Or was the white background preferable?

PPS – Here’s another test, code inserted as HTML – suitable for copying and pasting:

# Define a single-page client called 'main'
ss.client.define 'main',
  view: 'index.jade'
  css: ['libs', 'app.styl']
  code: ['libs', 'app', 'modules']

Update – Yet another approach, using a a WordPress plugin (no Jade / Stylus support?):

# Define a single-page client called 'main'
ss.client.define 'main',
  view: 'index.jade'
  css: ['libs', 'app.styl']
  code: ['libs', 'app', 'modules']

Learning vim

In Software on Jan 28, 2013 at 00:01

This post won’t be for everyone. In fact, it may well sound like an utterly ridiculous idea…

After yesterday’s post, it occurred to me that maybe some readers will be interested in what these so-called “programmer’s editors” such as vim and emacs are all about.

When developing software, you need a tool to edit text. Not just enter it, nor even correct it, but really edit it. Now perhaps one day we’ll all laugh about this concept of having a bunch of lines with characters on a screen, and working with tools to arrange those characters in files in a very very specific way. But for the past decades up to this very day, that’s how software development happens. Don’t blame me, I’m just stating a fact.

I can think of a couple of tools to use for editing text:

  • a non-programmer’s text editor or word processor (NotePad? TextEdit? Word?)
  • GUI-based programmer’s editors (UltraEdit, BBEdit, gedit, TextMate, Sublime Text)
  • an Integrated Development Environment, such as Visual Studio, Eclipse, Komodo
  • command-based programmer’s editor – the main ones are really only Vim and Emacs

Everything but the last option is GUI based, and while all of those tools have lots and lots of “keyboard shortcuts”, these are usually added for the “power users”, with the main idea being that everything should be reachable via clicking on menu’s, buttons, and text. Mouse-based, in other words.

There’s nothing wrong with using the mouse. But when it comes to text, the fact is that you’re going to end up using the keyboard anyway. Having to switch between the two is awkward. The other drawback of using a mouse for these tasks, is that menu’s can only give a certain amount of choice – for the full breadth and depth of a command-based interface, you’re going to have to navigate through a fairly large set of clever visual representations on the screen. Navigation means “pointing” in this case. And pointing requires eye-hand coordination.

As I said yesterday, I’m going back to keyboard-only. Within reason, that is. Switching windows and apps can be done through the keyboard, and so can many tasks – even in a web browser. But not all. The trick is to find a balanced work flow that works well… in the long run. Which is – almost by definition – a matter of personal preference.

Anyway. Let’s say you want to find out more about vim…

In a nutshell:

  • you type commands at it: simple keys, control keys, short sequences of keys
  • each of them mean something, and you have to know them to be able to use them
  • luckily, a small subset of commands is all it takes for simple (albeit tedious) editing
  • it does a lot more than edit a file: jump around, see multiple files, start compiles, …
  • but also: integrate with syntax check, file search, symbol lookup, batch operations
  • it’s all about monospaced text files, and it scales to huge files without problems
  • vim (like emacs) is a world in itself, offering hundreds (thousands!) of commands
  • bonus: a stripped version of vim (or vi) is present on even the tiniest Linux systems

In vim, you’re either in insert mode (typing text), or in visual mode, or in shell-like command mode. Typing the same key means different things in each of these modes!

Make no mistake: vim is huge. Here’s the built-in help for just the CTRL+W command:

Screen Shot 2013-01-26 at 23.58.29

Yeah, vim also supports all sorts of color-scheme fiddling. But here’s the good news:

  • you only need to remember how to enter insert mode (“i” or “a”), how to exit it (ESC), how to enter command mode (“:”), how to save (“:w”+CR), and how to quit (“:q”+CR) – and most importantly: how to get to the help section (“:help”+CR).
  • a dozen or so letter combinations will get you a long way: just read up on what the letters “a” to “z” do in visual mode, and you’ll be editing for real in no time
  • there is logic in the way these commands are set up and named, there really is – even a few minutes spent on vim’s extensive built-in introductory help texts are worth it
  • some advanced features are awesome: such as being able to undo any change even after quits / relaunches, and saving complete session state per project (“:mks”)

If you try this for a few days (and fight that recurring strong urge to go back to the editor you’re so familiar with), you’ll notice that some things start to automate. That’s your spine taking over. This is what it’s really all about: delegating the control of the editor to a part of you which doesn’t need your attention.

When was the last time you had to think about how to get into your car? Or how to drive?

Probably the best advice I ever got from someone about learning vim: if you find yourself doing something over and over again and it feels tedious, then vim will have some useful commands to simplify those tasks. And that is the time to find them and try them out!

Vim (and emacs) is not about good looks. Or clever tricks. It’s about delegation.

Learning new tricks

In Software on Jan 27, 2013 at 00:01

There’s a reason why many people stick to a programming language and reuse it whenever possible: it’s hard work to learn another one! So once you’ve learned one well, and in due time really, really well, then you get a big payback in the form of a substantial productivity boost. Because – as I said – it’s really a lot of hard work to become familiar, let alone proficient, with any non-trivial technology – including a programming language or an OS.

Here’s me, a few weeks ago:


I was completely overwhelmed by trying to learn Node.js, AngularJS, SocketStream, CoffeeScript, and more. To the point of not being able to even understand how it all fits together, and knowing that whatever I’d try, I would stumble around in the dark for ages.

I’ve been through the learning curve of three programming languages before. I like to think that I got quite good at each of them, making code “sing” on occasion, and generally being able to spend all my time and energy on the problem, not the tools. But I also know that it took years of tinkering. As Malcolm Gladwell suggests in his book Outliers, it may take some 10,000 hours to become really good at something. Maybe even at anything?

Ten. Thousand. Hours. Of concentrating on the work involved. That sure is a long time!

But here’s the trick: just run, stumble, and learn from each re- / disrecovery!

That doesn’t mean it’s easy. It’s not even easier than the last time, unfortunately: you can’t get good at learning new stuff – you still have to sweat it through. And there are dead ends. Tons of them. This is the corner I painted myself into a couple of days ago:


Let me explain:

  • I thought I was getting up to speed with CoffeeScript and the basics of AngularJS
  • … even started writing some nice code for it, with lots of debug / commit cycles
  • big portions were still clear as mud to me, but hey… I got some stuff working!

The trouble is: instead of learning things the way all these new tools and technologies were meant to be used, by people far more knowledgeable than me, I started fitting my old mental models on top of what I saw.

The result: a small piece, working, all wrapped into code that made it look nice to me.

That’s a big mistake. Frameworks such as AngularJS and SocketStream always have their own “mindset” (well, that of their designers, obviously) – and trying to ignore it is a sure path to frustration and probably even disaster, in the long term.

Ya can’t learn new things by just mapping everything to your old knowledge!

Yesterday, I realised that what I had written as code was partly just a wrapper around the bits of NG and SS which I started to understand. The result: a road block, sitting in the way of real understanding. My code wasn’t doing much, just re-phrasing it all!

So a few days ago, I tore most of it all down again (I’m talking about HouseMon, of course). It didn’t get the startup sequence right, and it was getting worse all the time.

Guess what? Went through a total, frustratingly disruptive, re-factoring of the code. And it ended up so much simpler and better… it blew me away. This NG + SS stuff is incredible.

Here’s me, after checking in the latest code a few hours ago:


No… no deep understanding yet. There’s no way to grasp it all in such a short time (yes, weeks is a short time) – not for me, anyway – but the fog is lifting. I’m writing less code again. And more and more finding my way around the web. The AngularFun project on GitHub and this post by Brian Ford are making so much more sense now.

I’ve also decided to try and change one more habit: I’m going back to the vim editor, or rather the MacVim version of it. I’ve used TextMate for several years (now v2, and open source), but the fact is that pointing at things sucks. It requires eye-hand coordination, whereas all command-line and keyboard-driven editing activity ends up settling itself in your muscle memory. Editors such as vim and emacs are amazing tools because of that.

Now I don’t know about you, but when I learn new things, the last thing on earth I’d want is to spend my already-strained mental energy on the process instead of the task at hand.

So these days, I’m in parallel learning mode: with head and spine working in tandem :)

Bad LiPo charge chip

In Hardware on Jan 26, 2013 at 00:01

(Not planned this way, but still a nice follow-up on yesterday’s Junk USB post…)

While thinking about some minor tweaks for the JeeNode USB board, I wanted to try out a different LiPo charger chip – mostly to reduce costs, given than not everyone using the JN USB would be interested in the LiPo charge capability (it’s also fine as a JeeNode-with-built-in-USB-BUB after all).

So I had a look at the MCP73832T – in fact, Paul Badger and I went ahead and had a new board made with it:

DSC 3220

The good news: as a LiPo charger, it works absolutely fine.

The bad news: without an attached LiPo battery, it’s not usable.

It turns out that this chip uses some sort of charge/discharge cycle. This is what happens without LiPo attached:


IOW, it’s delivering 4.2V for a while, and then dropping the voltage to see whether the LiPo itself will fill in the gaps. A pretty clever way to figure out the state of the attached battery, if you ask me.

One way to use the chip without attached LiPo would be to bypass the Vin and Vout pins of the chip, i.e. just disable it altogether via a (solder-) jumper. Drawbacks: 1) you have to remember this, and act accordingly, 2) this means PWR would be 4.2V with LiPo attached, and 5V without, and 3) when bypassed, there would be no over-current protection for the USB port.

Especially that 3rd issue is bad – JeeNodes are about tinkering with stuff, and JN USB’s are about tinkering while attached to a computer USB port. Without over-current protection, tinkering can damage your computer – scary!

There is one more way to solve this, but it’s not very practical: add a big electrolytic cap which sort of takes the place of a battery. I used a 6800 µF (which pulls too much current on power-up, BTW). The result:


A voltage on the PWR pin which carries 3.8 .. 4.2V, with a 10 Hz ripple. Not great, but good enough to make the JeeNode’s internal regulator work. Except that a 6800 µF capacitor is huge and highly impractical, of course!

Sooo… back to the MAX1555 it is. That chip works differently: it senses the charge current and the output voltage.

Note to self: don’t replace chips without testing all essential modes they’ll be used in.

And good bye Mr. Murphy, how considerate of you to drop by once in a while…

Junk USB power

In Hardware on Jan 25, 2013 at 00:01

The P1 JeeNode which was recently hooked up to my smart meter stopped working – or maybe I switched USB adapter and never checked?

First thought was to check it out, and although that provided some useful insight, it didn’t solve the problem: node blinking furiously and – more importantly – no data packets!

Well, it turns out that this thing was the problem:


That’s not a Texas Instruments unit, or rather: it’s probably from a rejected batch which found its way to the “surplus” market again (got it from Pollin in Germany, long ago).

Hooking it up to the scope tells the whole story at a glance – first the unloaded output:


Not stellar, but not shocking either. Let’s zoom in on the start of the voltage bumps:


Mweh… nasty. But here’s the same output with a JeeLink plugged in:


Whoa, right… totally useless as a “5V supply”. Ready for the garbage bin… plunk!

JavaScript semantics

In Software on Jan 24, 2013 at 00:01

Some things are quite surprising in JavaScript / CoffeeScript:

    $ coffee
    > '1'+2
    > '1'-2
    > 1 < null
    > 1 > null
    > 1 < undefined
    > 1 > undefined
    > a = [1,2,3]
    [ 1, 2, 3 ]
    > a.b = 4
    > (x for x in a)
    [ 1, 2, 3 ]
    > (x for x of a)
    [ '0', '1', '2', 'b' ]
    > (x for x in 'abc')
    [ 'a', 'b', 'c' ]
    > (x for x of 'abc')
    [ '0', '1', '2' ]

It makes sense once you know it … but that’s the whole thing with being a newbie, eh?

That array-with-poperties behaviour is actually very useful, because it lets you create collections which can be looped over, while still offering members in an object-like manner. Very Lua-ish. The same can be done with Object.defineProperty, but that’s more involved.

For the full story on “array-like” objects, see this detailed page. It gets real messy inside – but then again, so does any other language. As long as its easy to find answers, I’m happy. And with JavaScript / CoffeeScript, finding answers and suggestions on the web is trivial.

On another note: Redis is working well for storing data, but there is a slight impedance mismatch with JavaScript. Storing nested objects is trivial by using JSON, but then you lose the ability to let Redis do a lot of nifty sorting and extraction. For now, I’m getting good mileage with two hashes per collection: one for key-to-id lookup, one for id-based storage of each object in JSON format. But I’m really using only a tiny part of Redis this way.

Solar… again – first results

In Software on Jan 23, 2013 at 00:01

Sorry for keeping you in suspense after presenting yesterday’s code. I had to let this run for a few days to collect enough data worth graphing…

Some raw output:

L 15:01:00.818 usb-A40117UK OK 20 90 27 224 25 128 26 224 26
L 15:02:01.232 usb-A40117UK OK 20 68 27 224 25 128 26 224 26
L 15:03:01.663 usb-A40117UK OK 20 64 27 224 25 128 26 224 26
L 15:04:02.075 usb-A40117UK OK 20 64 27 222 25 127 26 224 26
L 15:05:02.486 usb-A40117UK OK 20 64 27 195 25 128 26 224 26
L 15:06:02.901 usb-A40117UK OK 20 64 27 192 25 123 26 224 26
L 15:07:03.314 usb-A40117UK OK 20 63 27 192 25 124 26 224 26
L 15:08:03.728 usb-A40117UK OK 20 38 27 192 25 111 26 224 26
L 15:09:04.140 usb-A40117UK OK 20 32 27 192 25 104 26 224 26
L 15:10:04.557 usb-A40117UK OK 20 32 27 169 25 99 26 224 26

And here some readings with DIO1 turned on:

L 16:01:24.894 usb-A40117UK OK 20 30 51 0 51 128 51 32 51
L 16:02:25.293 usb-A40117UK OK 20 159 51 128 51 0 52 160 51
L 16:03:25.677 usb-A40117UK OK 20 0 52 224 51 107 52 0 52
L 16:04:26.061 usb-A40117UK OK 20 64 52 55 52 192 52 96 52
L 16:05:26.461 usb-A40117UK OK 20 128 52 113 52 0 53 160 52
L 16:06:26.849 usb-A40117UK OK 20 192 52 160 52 61 53 192 52
L 16:07:27.224 usb-A40117UK OK 20 224 52 192 52 96 53 0 53
L 16:08:27.622 usb-A40117UK OK 20 0 53 237 52 128 53 32 53
L 16:09:28.011 usb-A40117UK OK 20 32 53 12 53 160 53 64 53
L 16:10:28.406 usb-A40117UK OK 20 64 53 32 53 192 53 96 53

In decimal, this is what I see:

a0: 13952 a1: 13914 a2: 14144 a3: 14017

Here are the first eight hours after sunset – no energy coming in at all, just conditioning:


Top to bottom: orange = 2 MΩ, red = 20 MΩ, blue = 2-C 200 kΩ, green = 1-C 200 kΩ. Some variation probably due to resistor / supercap tolerances.

Since we’re measuring half the voltage of the supercaps, you can see that they have all reached around 2.8V at this point. The interesting bit will be to see whether the voltage ever rises in the time between DIO1 “on” cycles – indicating a charge surplus!

Here’s the next 24 hours, only the 2 and 20 MΩ loads benefit from this cloudy winter day:


Another day, some real sunlight, not enough yet to survive the next night, even w/ 20 MΩ:


Note that the only curves which hold any promise, are the ones which can permanently stay above that hourly on/off cycle, since that DIO-pin top-up won’t be there to bail us out when using these solar cells as a real power source.

I’ll leave it for now – and will collect data for some more days, or perhaps weeks if there’s any point with these tiny solar cells. Will report in due time … for now: patience!

PS. Note that these fingernail-sized CPC1824 chips are a lot more than just some PV material in an SOIC package. There appears to be a switching boost regulator in there!

Solar… again – the code

In Software on Jan 22, 2013 at 00:01

With the hardware ready to go, it’s time to take that last step – the software!

Here is a little sketch called slowLogger.ino, now in JeeLib on GitHub:

Screen Shot 2013-01-19 at 16.20.50

It’s pretty general-purpose, really – measure 4 analog signals once a minute, and report them as wireless packet. There are a couple of points to make about this sketch:

  • The DIO1 pin will toggle approximately once every 64 minutes, i.e. one hour low / one hour high. This is used in the solar test setup to charge the supercaps to about 2.7V half of the time. The charge and discharge curves can provide useful info.

  • The analog channel is selected by doing a fake reading, and then waiting 100 ms. This sets the ADC channel to the proper input pin, and then lets the charge on the ADC sample-and-hold capacitor settle a bit. It is needed when the output impedance of the signal is high, and helps set up a more accurate voltage in the ADC.

  • The analog readings are done by summing up each value 32 times. When dividing this value, you get an average, which will be more accurate than a single reading if there is noise in the signal. By not dividing the value, we get the maximum possible amount of information about the exact voltage level. Here, it helps get more stable readings.

As it turns out, the current values reported are almost exactly 10x the voltage in millivolts (the scale ends up as 32767 / 3.3), which is quite convenient for debugging.

Having said all this, you shouldn’t expect miracles. It’s just a way to get minute-by-minute readings, with hopefully not too many fluctuations due to noise or signal spikes.

Tomorrow, some early results!

Solar… again – setup

In Software on Jan 21, 2013 at 00:01

With the purpose out of the way, it’s time start building.

It’s all a big hack, and since I didn’t plan the board layout, it all ended up being hard to hookup to a JeeNode, so I had to tack on a Proto Board:


Those solar cells are tiny! – if they don’t work, I’ll switch to more conventional bigger ones.

On the back, some resistors and diodes:


I used a JeeNode USB (about 60 µA consumption in power-down, due to the extra FTDI chip), but am powering it all via a 3x AA battery pack to get full autonomy. The idea is to place this thing indoor, facing south through a window:


Tomorrow, the code…

Solar… again

In Hardware on Jan 20, 2013 at 00:01

These dark and cold winter days are not really effective for solar energy: the entire month of January will probably not generate more power energy than two peak days in July!

Still, it’s a good baseline to try things with. And one of the experiments I haven’t given up on is making nodes run off solar power, using a supercap to hold the charge. Maybe they’ll not last through the night, but that’s fine – there are still various uses (especially outdoor) where being able to run during daytime with nodes that never need to have their batteries changed would be really nice.

In previous attempts, I’ve always immediately tried to power the actual node, but now I’d like to try something simpler: a solar cell, a supercap, and a resistor as load. Like this:

JC's Grid, page 61

I’m using a tiny solar cell by Clare again, the CPC1824, with the following specs:

Screen Shot 2013-01-19 at 15.28.51

Not much, but then again, it’s a cell which is just the size of a fingernail. As SOIC-16 package, and with the specs of the available current next to it:

Screen Shot 2013-01-19 at 15.28.28   Screen Shot 2013-01-19 at 15.29.26

In fact, I’d like to try this out with 4 different resistive loads:

  • one cell, 2x R = 20 MΩ = 0.2 µA current draw at full output
  • one cell, 2x R = 2 MΩ = 2 µA current draw at full output
  • one cell, 2x R = 200 KΩ = 20 µA current draw at full output
  • two cells in paralel, 2x R = 200 KΩ = 20 µA current draw at full output

There may be a flaw in this approach, in that the leakage of the supercap could completely overshadow the current draw from the resistors. But my hope is that supercaps get better over time when kept charged. Hmmm… not sure it applies if they run down every night.

So the second part of the idea, is to alternate solar cell use and dumb charging – just to measure how that affects output voltage over time. One hour, DIO will be on, and put the supercap on about 2.7V, the other hour it’ll be off and the solar cell takes over. With a bit of luck, the output voltage changes might show a pattern, right?

I think it’s worth a try and have made a setup with 4x the above – more tomorrow…

Who needs numbers?

In Software on Jan 19, 2013 at 00:01

I’ve always been intrigued by those “control panels” with lots and lots of numbers on them. It seems so much like old science-fiction movies with unmarked buttons all over the place.

That and the terror of the linear scale felt like a good reason to try out something new:

Screen Shot 2013-01-17 at 18.02.20   Screen Shot 2013-01-17 at 18.04.34   Screen Shot 2013-01-17 at 18.09.28   Screen Shot 2013-01-17 at 18.08.48

Ok, so here’s the idea:

  • you’re looking at four copies of the display next to each other, with different readings
  • the vertical scale is power production (top half) and power consumption (bottom half)
  • green is production (+), red is consumption (-), and blue is the difference (+ or -)
  • it’s logarithmic, top and bottom are 5000 W produced and consumed, respectively
  • the green and red dots are also log-proportional, the blue circe has a fixed diameter
  • the center has a ± 50 W dead zone, i.e. everything in that range is collapsed to a line

Now the examples, from left to right:

  1. end of the afternoon, no solar power, about 260 W consumption
  2. induction cooking, consuming about 2500 W
  3. night-time, house consumption is 80 W
  4. daytime, 1500 W production, 200 W consumption

Those last two readouts were simulated since I didn’t want to wait for a sunny winter day :)

The data is shown LIVE, and I’m going to keep it around to see whether this is an intuitive way of presenting this sort of information. It’s all implemented using the HTML5 canvas.

PS – I’ve since switched to “sqrt” i.s.o. of “log” for scaling. Looks nicer with large values.

Still crude, but oh so cool…

In Software on Jan 18, 2013 at 00:01

Good progress on the HouseMon front. I’m still waggling from concept to concept like a drunken sailor, but every day it sinks in just a little bit more. Adding a page to the app is now a matter of adding a CoffeeScript and a Jade file, and then adding an entry to the “routes” list to make a new “Readings” page appear in the main menu:

Screen Shot 2013-01-17 at 01.01.00

The only “HTML” I wrote for this page is readings.jade, using Foundation classes:

Screen Shot 2013-01-17 at 01.03.14

And the only “JavaScript” I wrote, is in to embellish the output slightly:

Screen Shot 2013-01-17 at 00.18.51

The rest is a “readings” collection, which is updated automatically on the server, with all changes propagated to the browser(s). There’s already quite a bit of architecture in place, including a generic RF12demo driver which connects to a serial port, and a decoder for some of the many packets flying around the house here at JeeLabs.

But the exciting bit is that the server setup is managed from the browser. There’s an Admin page, just enough to install and remove components (“briqlets” in HouseMon-speak):

Screen Shot 2013-01-17 at 00.14.09

In this example, I installed a fake packet generator (which simply replays one of my logfiles in real time), as well as a small set of decoders for things like room node sketches.

So just to stress what there is: a basic way of managing server components via the browser, and a basic way of seeing the live data updates, again in the browser. Oh, and all sorts of RF12 capturing and decoding… even support for the Announcer idea, described recently.

Did I mention that the data shown is LIVE? Some of this stuff sure feels like magic… fun!

Arduino sketches on RPi

In AVR, Software, Linux on Jan 17, 2013 at 00:01

There’s an arduino-mk package which makes it simple to build and upload sketches on a Raspberry Pi. Here’s how to set it up and use it with a JeeLink, for example:

Install the package:

  sudo apt-get install arduino-mk

That gets all the required software and files, but it needs a tiny bit of setup.

First, create the library area with a demo sketch in it:

  mkdir ~/sketchbook
  cd ~/sketchbook
  ln -s /usr/share/arduino/
  mkdir blink
  cd blink

Then create two files – the blink.ino sketch for a JeeLink or JeeNode SMD / USB:

// Blink - adapted for a JeeNode / JeeLink

void setup() {
  pinMode(9, OUTPUT);

void loop() {
  digitalWrite(9, LOW);
  digitalWrite(9, HIGH);

… and a Makefile which ties it all together:

BOARD_TAG    = uno
ARDUINO_DIR  = /usr/share/arduino

include ../

(that file in the sketchbook/ dir is well-commented and worth a read)

That’s it. You now have the following commands to perform various tasks:

  make             - no upload
  make upload      - compile and upload
  make clean       - remove all our dependencies
  make depends     - update dependencies
  make reset       - reset the Arduino by tickling DTR on the serial port
  make raw_upload  - upload without first resetting
  make show_boards - list all the boards defined in boards.txt

The first build is going to take a few minutes, because it compiles the entire runtime code. After that, compilation should be fairly snappy. Be sure to do a make clean whenever you change the board type, to re-compile for a different board. The make depends command should be used when you change any #include lines in your code or add / remove source files to the project.

This setup makes things run smoothly, without requiring the Arduino IDE + Java runtime.

Remote node discovery – code

In Software on Jan 16, 2013 at 00:01

Ok, time to try out some real code, after yesterday’s proposal for a simple node / sketch / packet format announcement system.

I’m going to use two tricks:

  • First of all, there’s no need for the node itself to actually send out these packets. I’ve written a little “announcer” sketch which masquerades as the real nodes and sends out packets on their behalf. So… fake info to try things out, without even having to update any nodes! This is a transitional state, until all nodes have been updated.

  • And second, I don’t really care about sending these packets out until the receiving end actually knows how to handle this info, so in debug mode, I’ll just report it over serial.

Note that packets get sent to “node 0”, which cannot exist – so sending this out is in fact harmless: today’s RF12 listeners will ignore such packets. Except node 31, which sees these new packets come in with header byte 64 (i.e. 0x40). No RF12 driver change needed!

Here’s the essence of the test sketch:

Screen Shot 2013-01-15 at 15.12.08

With apologies for the size of this screenshot – full code is on GitHub.

Note that this is just for testing. For the actual code, I’d pay far more attention to RAM usage and place the actual descriptive data in flash memory, for example.

And here’s some test output from it on the serial port:

Screen Shot 2013-01-15 at 15.09.51

Yeah, that looks about right. Next step will be to pick this up and do something with it. Please give me a few days for that, as the “HouseMon” server-side code isn’t yet up to it.

To be continued soon…

PS. Another very useful item type would be the maximum expected time between packets. It allows a server to automatically generate a warning when that time is grossly exceeded. Also a nice example of how self-announcement can help improve the entire system.

Remote node discovery – part 2

In Software on Jan 15, 2013 at 00:01

Yesterday’s post was about the desire to automate node discovery a bit more, i.e. having nodes announce and describe themselves on the wireless network, so that central receivers know something about them without having to depend on manual setup.

The key is to make a distinction between in-band (IB) and out-of-band (OOB) data: we need a way to send information about ourselves, and we want to send it by the same means, i.e. wirelessly, but recognisable in some way as not being normal packet data.

One way would be to use a flag bit or byte in each packet, announcing whether the rest of the packet is normal or special descriptive data. But that invalidates all the nodes currently out there, and worse: you lose the 100% control over packet content that we have today.

Another approach would be to send the data on a different net group, but that requires a second receiver permanently listening to that alternate net group. Not very convenient.

Luckily, there are two more alternatives (there always are!): one extends the use of the header bits already present in each packet, the other plays tricks with the node ID header:

  1. There are three header combinations in use, as described yesterday: normal data, requesting an ACK, and sending an ACK. That leaves one bit pattern unused in the header. We could define this as a special “management packet”.

  2. The out-of-band data we want to send to describe the node and the packet format is probably always sent out as broadcast. After all, most nodes just want to report their sensor readings (and a few may listen for incoming commands in the returned ACK). Since node ID zero is special, there is one combination which is never used: sending out a broadcast and filling in a zero node ID as origin field. Again, this combination could be used to mark a “management packet”.

I’m leaning toward the second because it’s probably compatible with all existing code.

So what’s a “management packet”, eh?

Well, this needs to be defined, but the main point here is that the format of management packets can be fully specified and standardised across all nodes. We’re not telling them to send/receive data that way, we’re only offering them a new capability to broadcast their node/packet properties through these new special packets.

So here’s the idea, which future sketches can then choose to implement:

  • every node runs a specific sketch, and we set up a simple registry of sketch type ID’s
  • each sketch type ID would define at least the author and name (e.g. “roomNode”)
  • the purpose of the registry is to issue unique ID’s to anyone who wants a bunch of ’em
  • once a minute, for the first five minutes after power-up, the sketch sends out a management packet, containing a version number, a sequence number, the sketch type ID, and optionally, a description of its packet payload format (if it is simple and consistent enough)
  • after that, this management packet gets sent out again about once an hour
  • management packets are sent out in addition to regular packets, not instead of

This means that any receiver listening to the proper net group will be able to pick up these management packets and know a bit more about the sketches running on the different node ID’s. And within an hour, the central node(s) will have learned what each node is.

Nodes which never send out anything (or only send out ACKs to remote nodes, such as a central JeeLink), probably don’t need this mechanism, although there too it could be used.

So the only remaining issue is how to describe packets. Note that this is entirely optional. We could just as easily put that description in the sketch type ID registry, and even add actual decoders there to automate not only node type discovery, but even packet decoding.

Defining a concise notation to describe packet formats can be very simple or very complex, depending on the amount of complexity and variation used in these data packets. Here’s a very simple notation, which I used in JeeMon – in this case for roomNode packets:

  light 8 motion 1 rhum 7 temp -10 lobat 1

These are pairs of name + number-of-bits, with a negative value indicating that sign extension is to be applied (so the temp field ranges from -512..+511, i.e. -51.2..+51.1°C).

Note that we need to fit a complete packet description in a few dozen bytes, to be able to send it out as management data, so probably the field names will have to be omitted.

This leads to the following first draft for a little-endian management packet format:

  • 3 bits – management packet version, currently “1”
  • 5 bits – sender node ID (since it isn’t in the header)
  • 8 bits – management packet number, incremented with each transmission
  • 16 bits – unique sketch type ID, assigned by JeeLabs (see below)
  • optional data items, in repeating groups of the form:
    • 4 bits – type of item
    • 4 bits – N-1 = length of following item data (1..16 bytes)
    • N bytes – item data

Item types and formats will need to be defined. Some ideas for item types to support:

  • type 0 – sketch version number (a single byte should normally be enough)
  • type 1 – globally unique node ID (4..16 bytes, fetched from EEPROM, could be UUID)
  • type 2 – the above basic bit-field description, i.e. for a room node: 8, 1, 7, -10, 1
  • type 3 – tiny URL of an on-line sketch / packet definition (or GitHub user/project?)

Space is limited – the total size of all item descriptions can only be up to 62 bytes.

Sketch type ID’s could be assigned as follows:

  • 0..99 – free for local use, unregistered
  • 100..65535 – assigned through a central registry site provided by JeeLabs

That way, anyone can start implementing and using this stuff without waiting for such a central registry to be put in place and become operational.

Tomorrow, a first draft implementation…

Remote node discovery

In Software on Jan 14, 2013 at 00:01

The current use of the wireless RF12 driver is very basic. That’s by design: simple works.

All you get is the ability to send out 0..66 bytes of data, either as broadcast or directed to a specific node. The latter is just a little trick, since the nature of wireless communication is such that everything goes everywhere anyway – so point to point is simply a matter of receivers filtering out and ignoring packets not intended for them.

The other thing you get with the RF12 driver, is the ability to request an acknowledgement of good reception, which the receiver deals with by sending an “ACK” packet back. Note that ACKs can contain data – so this same mechanism can also be used to request data from another node, if they both agree to use the protocol in this way, that is.

So there are three types of packets: data sent out as is, data sent out with the request to send an ACK back, and packets with this ACK. Each of them with a 0..66 byte payload.

But even though it all works well, there’s an important aspect of wireless sensor networks which has never been addressed: the ability for nodes to tell other nodes what/who they are. As a consequence, I always have to manually update a little table on my receiving server with a mapping from node ID to what sketch that node is currently running.

Trivial stuff, but still a bit inconvenient in the longer run. Why can’t each node let me know what it is and then I don’t have to worry about mixing things up, or keeping that table in sync with reality every time something changes?

Well… there’s more to it than a mapping of node ID’s – I also want to track the location of each node, and assign it a meaningful name such as “room node in the living room”. This is not something each node can know, so the need to maintain a table on the server is not going to disappear just by having nodes send out their sketch + version info.

Another important piece of information is the packet format. Each sketch uses its own format, and some of them can be non-trivial, for example with the OOK relay sending out all sorts of packets, received from a range of different brands of OOK sensors.

It would be sheer madness to define an XML DTD or schema and send validated XML with namespace info and the whole shebang in each packet, even though that is more or less the level of detail we’re after.

Shall we use JSON then? Protocol buffers? Bencode? ASN.1? – None of all that, please!

When power is so scarce that microcoulombs matter, you want to keep packet sizes as absolutely minimal as possible. A switch from a 1-bye to a 2-byte payload increases the average transmit power consumption of the node by about 10%. That’s because each packet has a fixed header + trailer byte overhead, and because the energy consumption of the transmitter is proportional to the time it is kept on.

The best way to keep packet sizes minimal, is to let each node pick what works best!

That’s why I play lots and lots of tricks each time when coming up with a remote sketch. The room node gets all its info across in just 4 bytes of data. Short packets keep power consumption low and also reduce the chance of collisions. Short is good.

Tomorrow, I’ll describe an idea for node discovery and packet format description.

P1 revisited

In Hardware on Jan 13, 2013 at 00:01

The P1 port connection I set up recently for picking up data from the utility company’s smart meter isn’t working reliably yet, so I dove a bit deeper into it.

Here’s how I now think the P1 interface is implemented inside that smart meter:

JC's Grid, page 59

There’s no way to ascertain whether a PNP or NPN transistor is used, without opening the box (impossible without breaking a seal, so definitely not an option!). But given that NPN is more common, the above circuit is my best guess.

The resistance was measured between the DATA and GND pin. The resistance between DATA and REQUEST is usually over 1 MΩ, which indicates that the phototransistor is open. Makes sense: pull the data low regardless of the REQUEST line state, and pull it high when the internal IR LED in that optocoupler is lit. That also explains the awkward inverted TTL logic levels provided by this interface.

What you can see is that the REQUEST line is really nothing but the power supply to the isolated side of the optocoupler. But the surprise is the value of that pulldown resistor!

Let’s do the math: when REQUEST is 5V, and the phototransistor is conducting, it’ll have about a 0.2V collector-to-emitter voltage drop, leaving 4.8V to feed the 180 Ω resistor (I measured 4.75 V, so yeah, close enough).

Whoa… 4.8 V over 180 Ω is 26 mA, a pretty hefty current draw in µC terms!

My conclusions from all this are: there’s no external pull-up or pull-down needed after all, and my hunch is that it’ll work just as fine with REQUEST powered by 3.3V (reducing the current somewhat). All you have to make sure when working with this P1 interface, is that your REQUEST pin can supply those 25-odd milliamps.

If the above is accurate, then nothing forces us to actually use that resistor, by the way. We could simply connect the REQUEST and DATA pins and leave GND dangling. In fact, by re-using the phototransistor in a different way, one could even make it work in active-high mode again (only if no other P1 devices are connected, evidently).

Note that this is for the Landis & Gyr E350 (PDF) smart meter I have – others may differ!

Flukso with RFM12B

In Hardware, Linux on Jan 12, 2013 at 00:01

Some exciting new developments going on…


You’re looking at the final prototype of the latest Flukso meter, which can be connected to AC current sensors, pulse counters, and the Dutch smart metering “P1” port. Here’s the brief description from that website:

Flukso is a web-based community metering application. Install a Fluksometer near your fuse box and you will be able to monitor, share and reduce your electricity consumption through this website.

The interesting bit is that it’s all based on a Linux board with wired and wireless Ethernet, plus a small ATmega-based add-on board which does all the real-time processing.

But the most exciting news is that the new version, now entering production, will include an RFM12B module with the JeeNode-compatible protocol. A perfect home automation workstation. Yet another interesting aspect of this, is that Bart Van Der Meersche, the mastermind behind Flusko, is working on getting the Mosquitto MQTT broker running permanently on that same Flukso meter.

Here’s the basic layout (probably slightly different from the actual production units):

Screen Shot 2013-01-11 at 21.10.20

Flukso runs OpenWRT, and everything in it is based on the Lua programming language, which is really an excellent fit for such environments. But even if Lua is not something you want to dive into, the open-endedness of PubSub means this little box drawing just a few Watt can interface to a huge range of devices – from RF12 to WiFi to LAN, and everything flowing in and out of that little box becomes easily accessible via MQTT.

PS. I have no affiliation with Flukso whatsoever – I just like it, and Bart is a nice fellow :)

Audio DAC

In Hardware on Jan 11, 2013 at 00:01

Last year, I got one of these little low-cost audio Digital-to-Analog Converters on eBay:


Takes digital SPDIF coax in, and produces analog left and right channels out. I got them before the bigger picture of the audio chain in the living room got fleshed out, and in the end I don’t really need this type of converter anymore.

Besides, there was some hiss and hum with these, so it’s not really high-end anyway.

Here’s what’s inside:


That’s a Cirrus Logic CS8416 “192 kHz Digital Audio Interface Receiver”, which apparently has all the smarts to detect and decode a range of serial bit formats, plus a Cirrus Logic CS4344 “10-pin, 24-Bit, 192 kHz Stereo D/A Converter” to generate the L/R audio signals.

I’m a bit surprised by the hiss/hum issues, since the audio DAC is specified as -90 dB THD+N and 105 dB Dynamic Range, which to me sounds like it should be pretty good. Maybe not audiophile level, but hey… neither are my ears, anyway!

Test-driven design

In Software on Jan 10, 2013 at 00:01

Drat, I can’t stop – this Node.js and AngularJS stuff sure is addictive! …

I’m a big fan of test-driven development, an approach where test code is often treated as more important than the code you’re writing to actually do the work. It may sound completely nuts – the thought of writing tons of tests which can only get in the way of making changes to your newly-written beautiful code later, right?

In fact, TDD goes even further: the idea is to write the test code as soon as you decide to add a new feature or write a new bug, and only then start writing the real stuff. Weird!

But having used TDD in a couple of projects in the past, I can only confirm that it’s in fact a fantastic way to “grow” software. Because adding new tests, and then seeing them pass, constantly increases the confidence in the whole project. Ever had the feeling that you don’t want to mess with a certain part of your code, because you’re afraid it might break in some subtle way? Yeah, well… I know the feeling. It terrifies me too :)

With test scaffolding around your code, something very surprising happens: you can tear down the guts and bring them back up in a different shape, because the test code will help drive that process! And re-writing / re-factoring usually leads to a better architecture.

There’s a very definite hurdle in each new project to start using TDD (or BDD). It’s a painful struggle to spend time thinking about tests, when all you want to do is write the darn code and start using it! Especially at the start of a project.

I started coding HouseMon without TDD/BDD on my plate, because I first want to expose myself to coding in CoffeeScript. But I’m already checking out the field in the world of Node.js and JavaScript. Fell off my chair once again… so many good things going on!

Here’s an example, using the Mocha and should packages with CoffeeScript:

Screen Shot 2013-01-09 at 22.57.08

And here’s some sample output, in one of many possible output formats:

Screen Shot 2013-01-09 at 22.59.55

I’m using kind of a mix of TDD’s assertion-style testing, and BDD’s requirement-style approach. The T + F tricks suit me better, for their brevity, but the “should.throw” stuff is useful to handle exceptions when they are intentional (there’s also “should.not.throw”).

But that’s just the tip of the iceberg. Once you start thinking in terms of testing the code you write, it becomes a challenge to make sure that every line of that code has been verified with tests, i.e. code coverage testing. And there too, Mocha makes things simple. I haven’t tried it, but here is some sample output from its docs:

Screen Shot 2012-02-23 at 8.37.13 PM

On the right, a summary of the test coverage of each of the source files, and in the main window an example of a few lines which haven’t been traversed by any of the tests.

Tools like these sure make it tempting to start writing tests!

Ok, ok, I’ll stop.

Gotta start somewhere…

In Software on Jan 9, 2013 at 00:01

Ok, well, baby steps or not – there’s just no way to avoid it… I need to start coding!

I’ve set up a new project on GitHub, called HouseMon – to have a place to put everything. It does almost nothing, other than what you’ll recognise after yesterday’s weblog post:

Screen Shot 2013-01-08 at 23.34.09

This was started with the idea of creating the whole setup as installable “Briqs” modules, which can then be configured for use. And indeed, some of that will be unavoidable right from the start – after all, to connect this to a JeeNode or JeeLink, I’m going to have to specify the serial port (even if the serial ports are automatically enumerated one day, the proper one will need to be selected in case there is more than one).

However, this immediately adds an immense amount of complexity over just hard-coding all the choices in the app for just my specific installation. Now, in itself, I’m not too worried by that – but with so little experience with Node.js and the rest, I just can’t make this kind of decisions yet. There’s too big a gap between what I know and understand today, and what I need to know to come up with a good modular system. The whole thing could be built completely out of npm packages, or perhaps even the lighter-weight Components mechanism. Too early to tell, and definitely too early to make such key decisions!

Just wanted to get this repository going to be able to actually build stuff. And for others to see what’s going on as this evolves over time. It’s real code, so you can play with it if you like. And of course make suggestions and point out things on the issue tracker. I’ll no doubt keep posting about this stuff here whenever the excitement gets too high, but this sort of concludes nearly two weeks of continuous immersion and blogging about this topic.

It’s time for me to get back to some other work. Never a dull moment here at JeeLabs! :)

Web page layouts

In Software on Jan 8, 2013 at 00:01

Wow – maybe I’m the last guy in the room to find out about this, but responsive CSS page layouts really have come a long way. Here are a bunch of screen shots at different widths:

Screen Shot 2013-01-07 at 21.32.15

Screen Shot 2013-01-07 at 21.32.53

Screen Shot 2013-01-07 at 21.33.44

Screen Shot 2013-01-07 at 21.34.00

Note that the relative scale of the above snapshots is all over the map, since I had to fit them into the size of this weblog, but as you can see, there’s a lot of automatic shuffling going on. With no effort whatsoever on my part!

Here’s the Jade definition to generate the underlying HTML:

Screen Shot 2013-01-07 at 21.47.21

How cool is that?

This is based on the Foundation HTML framework (and served from Node.js on on a Raspberry Pi). It looks like this might be an even nicer, eh, foundation to start from than Twitter Bootstrap (here’s a factual comparison, with examples of F and B).

Sigh… so many choices!

Technology decisions

In Software on Jan 7, 2013 at 00:01

Phew! You don’t want to know how many examples, frameworks, and packages I’ve been downloading, trying out, and browsing lately… all to find a good starting point for further software development here at JeeLabs.

Driving this is my desire to pick modern tools, actively being worked on, with an ecosystem which allows me to get up to speed, leverage all the amazing stuff people keep coming up with, and yet stay firmly on the simple side of things. Because good stuff is what you end up with when you weed out the bad, and the result is purity and elegance … as I’ve seen confirmed over and over again.

A new insight I didn’t start out from, is that server-side web page generation is no longer needed. In fact, with clients being more performant than servers (i.e. laptops, tablets, and mobile phones served from a small Linux system), it makes more and more sense to serve only static files (HTML, CSS, JS) and generic data (JSON, usually). The server becomes a file system + a database + a relatively low-end rule engine for stuff that needs to run at all times… even when no client is connected.

Think about the change in mindset: no server-side templating… n o n e !

Anyway – here are all the pieces I intend to use:

Node.js – JavaScript on the server, based on the V8 engine which is available for Intel and ARM architectures. A high-performance standards-compliant JavaScript engine.

The npm package manager comes with Node.js and is a phenomenal workhorse when it comes to getting lots of different packages to work together. Used from the command line, I found npm help to be great starting point for figuring out its potential.

SocketStream is a framework for building real-time apps. It wraps things like (now being replaced by the simpler for bi-directional use of WebSockets between the clients and the server. Comes with lots of very convenient features for development.

AngularJS is a great tool to create very dynamic and responsive client-side applications. This graphic from a post on says it all. The site has good docs and tutorials, this matters because there’s quite a bit of – fantastic! – stuff to learn.

Connect is what makes the HTTP webserver built into Node.js incredibly flexible and powerful. The coin dropped once I read an excellent introduction on If you’ve ever tried to write your own HTTP webpage server, then you’ll appreciate the elegance of this timeout module example on

Redis is a memory-based key-value store, which also saves its data on file, so that restarts can resume where it left off. An excellent way to cache file contents and “live” data which represents the state of a running system. Keeping data in a separate process is great for development, especially with automatic server restarts when any source file changes. IOW, the basic idea is to keep Redis running at all times, so that everything else can be restarted and reloaded at will during development.

Foundation is a set of CSS/HTML choices to support uniform good-looking web pages.

CoffeeScript is a “JavaScript dialect” which gets transformed into pure JavaScript on the fly. I’m growing quite used to it already, and enjoy its clean syntax and conciseness.

Jade is a shorthand notation which greatly reduces the amount of text (code?) needed to define HTML page structures (i.e. elements, tags, attributes). Like CoffeeScript, it’s just a dialect – the output is standard HTML.

Stylus is again just a dialect, for CSS this time. Not such a big deal, but I like the way all these notations let me see the underlying structure and nesting at a glance.

That’s ten choices, for Ten Terrific Technologies. Here are the links again:

(S = server-side, C = client-side, S+C = both, T = translated on the server)

All as non-restrictive open source, and with all development taking place on GitHub.

It’s been an intense two weeks, trying to understand enough of all this to be able to make practical decisions. And now I can hardly wait to find out where this will take me!

UpdateBootstrap has been replaced by Foundation – mostly as a matter of taste.

Node.js on Raspberry Pi

In Software on Jan 6, 2013 at 00:01

After all this fiddling with Node.js on my Mac, it’s time to see how it works out on the Raspberry Pi. This is a running summary of how to get a fresh setup with Node.js going.


I’m using the Raspbian build from Dec 16, called “2012-12-16-wheezy-raspbian”. See the download page and directory for the ZIP file I used.

The next step is to get this image onto an SD memory card. I used a 4 GB card, of which over half will be available once everything has been installed. Plenty!

Good instructions for this can be found at – in my case the section titled Copying an image to the SD card in Mac OS X (command line). With a minor tweak on Mac OSX 10.8.2, as I had to use the following command in step 10:

sudo dd bs=1m if=~/Downloads/2012-12-16-wheezy-raspbian.img of=/dev/rdisk1

First boot

Next: insert the SD card and power up the RPi! Luckily, the setup comes with SSH enabled out of the box, so only an Ethernet cable and 5V power with USB-micro plug are needed.

When launched and logged in (user “pi”, password “raspberry” – change it!), it will say:

Please run 'sudo raspi-config'

So I went through all the settings one by one, overclocking set to “medium”, only 16 MB assigned to video, and booting without graphical display. Don’t forget to exit the utility via the “Finish” button, and then restart using:

$ sudo shutdown -r now

Now is a good time to perform all pending updates and clean up:

$ sudo -i
# apt-get update
# apt-get upgrade
# apt-get clean
# reboot

The reboot at the end helps make sure that everything really works as expected.

Up and running

That’s it, the system is now ready for use. Some info about my 256 MB RPi:

$ uname -a
Linux raspberrypi 3.2.27+ #250 PREEMPT \
                    Thu Oct 18 19:03:02 BST 2012 armv6l GNU/Linux
$ free
             total       used       free     shared    buffers     cached
Mem:        237868      51784     186084          0       8904      25972
-/+ buffers/cache:      16908     220960
Swap:       102396          0     102396
$ df -H
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.9G  1.6G  2.2G  42% /
/dev/root       3.9G  1.6G  2.2G  42% /
devtmpfs        122M     0  122M   0% /dev
tmpfs            25M  213k   25M   1% /run
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs            49M     0   49M   0% /run/shm
/dev/mmcblk0p1   59M   18M   42M  30% /boot
$ cat /proc/cpuinfo 
Processor       : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS        : 697.95
Features        : swp half thumb fastmult vfp edsp java tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xb76
CPU revision    : 7

Hardware        : BCM2708
Revision        : 0004
Serial          : 00000000596372ab

So far, it’s just a standard boilerplate setup. Yawn…


On to Node.js! Unfortunately, the build included in Debian/Raspbian is 0.6.19, which is a bit old. I’d rather get started with the 0.8.x series, so here’s how to build it from source.

But first, let’s use this simple trick to get write permission in /usr/local as non-root:

$ sudo usermod -aG staff pi

Note: you have to logout and back in, or reboot, to get the new permissions.

With that out of the way, code can be built and installed as user “pi” – no need for sudo:

$ curl | tar xz 
$ cd node-v0.8.16
$ ./configure
$ make
(... two hours of build output gibberish ...)
$ make install

That’s it. A quick check that everything is in working order:

$ node -v
$ npm -v
$ which node
$ ldd `which node`
  /usr/lib/arm-linux-gnueabihf/ (0x40236000) => /lib/arm-linux-gnueabihf/ (0x40074000) => /lib/arm-linux-gnueabihf/ (0x40036000) => /usr/lib/arm-linux-gnueabihf/ (0x40107000) => /lib/arm-linux-gnueabihf/ (0x4023f000) => /lib/arm-linux-gnueabihf/ (0x4007f000) => /lib/arm-linux-gnueabihf/ (0x400a7000) => /lib/arm-linux-gnueabihf/ (0x402b0000)
  /lib/ (0x400d2000)

Oh, one more thing – the RPi doesn’t come with “git” installed, so let’s fix that right now:

$ sudo apt-get install git

There. Now you’re ready to start cookin’ with node and npm on the RPi. Onwards!

PS. If you don’t want to wait two hours, you can download my build, unpack with “tar xfz node-v0.8.16-rpi.tgz”, and do a “cd node-v0.8.16 && make install” to copy the build results into /usr/local/ (probably only works if you have exactly the same Raspbian image).

The key-value straightjacket

In Software, Musings on Jan 5, 2013 at 00:01

It’s probably me, but I’m having a hard time dealing with data as arrays and hashes…

Here’s what you get in just about every programming language nowadays:

  • variables (which is simply keyed access by name)
  • simple scalars, such as ints, floats, and strings
  • indexed aggregation, i.e. arrays: blah[index]
  • tagged aggregation, i.e. structs:
  • arbitrarily nested combinations of the above

JavaScript structs are called objects and tag access can be blah.tag or blah['tag'].

It would seem that these are all one would need, but I find it quite limiting.

Simple example: I have a set of “drivers” (JavaScript modules), implementing all sorts of functionality. Some of these need only be set up once, so the basic module mechanism works just fine: require "blah" can give me access to the driver as an object.

But others really act like classes (prototypal or otherwise), i.e. there’s a driver to open a serial port and manage incoming and outgoing packets from say a JeeLink running the RF12demo sketch. There can be more than one of these at the same time, which is also indicated by the fact that the driver needs a “serial port” argument to be used.

Each of these serial port interfaces has a certain amount of configuration (the RF12 band/group settings, for example), so it really is a good idea to implement this as objects with some state in them. Each serial port ends up as a derived instance of EventEmitter, with raw packets flowing through it, in both directions: incoming and outgoing.

Then there are packet decoders, to make sense of the bytes coming from room nodes, the OOK relay, and so on. Again, these are modules, but it’s not so clear whether a single decoder object should decode all packets on any attached JeeLink or whether there should be one “decoder object” per serial interface object. Separate objects allow more smarts, because decoders can then keep per-sensor state.

The OOK relay in turn, receives (‘multiplexes”) data from different nodes (and different types of nodes), so this again leads to a bunch of decoders, each for a specific type of OOK transmitter (KAKU, FS20, weather, etc).

As you can see, there’s sort of a tree involved – taking incoming packet data and dissecting / passing it on to more specific decoders. In itself, this is no problem at all – it can all be represented as nested driver objects.

As final step, the different decoders can all publish their readings to a common EventEmitter, which will act as a simple bus. Same as an MQTT broker with channels, with the same “nested key” strings to identify each reading.

So far so good. But that’s such a tiny piece of the puzzle, really.

Complexity sets in once you start to think about setup and teardown of this whole setup at run time (i.e. configuration in the browser).

Each driver object may need some configuration settings (the serial port name for the RF12demo driver was one example). To create a user interface and expose it all in the browser, I need some way of treating drivers as a generic collection, independent of their nesting during the decoding process.

Let’s call the driver modules “interfaces” for now, i.e. in essence the classes from which driver instances can be created. Then the “drivers” become instantiations of these classes, i.e. the objects which actually do the work of connecting, reading, writing, decoding, etc.

One essential difference is that the list of interfaces is flat, whereas a configured system with lots of drivers running is often a tree, to cope with the gradual decoding task described a bit above.

How do I find all the active drivers of a specific interface? Walk the driver tree? Yuck.

Given an driver object, how do I find out where it sits in the tree? Store path lists? Yuck.

Again, it may well be me, but I’m used to dealing with data structures in a non-redundant way. The more you link and cross-link stuff (let alone make copies), the more hassles you run into when adding, removing, or altering things. I’m trying to avoid “administrative code” which only keeps some redundant invariants intact – as much as possible, anyway.

Aren’t data structures supposed to be about keeping each fact in exactly one place?

PS. My terminology is still a mess in flux: interfaces, drivers, devices, etc…

Update – I should probably add that my troubles all seem to come from trying to maintain accurate object identities between clients and server.

Plotting again, at last

In Software on Jan 4, 2013 at 00:01

The new code is progressing nicely. First step was to get a test table, updating in real time:

Screen Shot 2013-01-03 at 11.29.07


It was a big relief to figure out how to produce graphs again – e.g. power consumption:

Screen Shot 2013-01-03 at 10.01.55

The measurement resolution from the 2000 pulse/kWh counters is excellent. Here is an excerpt of power consumption vs solar production on a cloudy and wet winter morning:

Screen Shot 2013-01-03 at 12.00.02

There is a fascinating little pattern in there, which I suspect comes from the central heating – perhaps from the boiler and pump, switching on and off in a peculiar 9-minute cycle?

Here are a bunch of temperature sensors (plus the central heating set-point, in brown):

Screen Shot 2013-01-03 at 10.02.14

There is no data storage yet (I just left the browser running on the laptop collecting data overnight), nor proper scaling, nor any form of configurability – everything was hard-coded, just to check that the basics work. It’s a far cry from being able to define and configure such graphs in the browser, but hey – one baby step at a time…

Although the D3 and NVD3 SVG-based graphing packages look stunning, they are a bit overkill for simple graphing and consist of more code for the browser to load and run. Maybe some other time – for now I’m using the Flot package for these graphs, again.

Will share the code as soon as I can make up my mind about the basic app structure!

Processing P1 data

In Software on Jan 3, 2013 at 00:01

Last post in a series of three (previous posts here and here).

The decoder for this data, in CoffeeScript, is as follows:

Screen Shot 2012-12-31 at 15.28.22

Note that the API of these decoders is still changing. They are now completely independent little snippets of code which do only one thing – no assumptions on where the data comes from, or what is to be done with the decoded results. Each decoder takes the data, creates an object with decoded fields, and finishes by calling the supplied “cb” callback function.

Here is some sample output, including a bit of debugging:

Screen Shot 2012-12-31 at 13.43.10

As you can see, this example packet used 19 bytes to encode 10 values plus a format code.

Explanation of the values shown:

  • usew is 0: no power is being drawn from the grid
  • genw is 6: power feed into the grid is 10 x 6, i.e. ≈ 60W
  • p1 + p3 is current total consumption: 8 + 191, i.e. 199W
  • p2 is current solar power output: 258W

With a rate of about 50 Kbit/sec using the standard RF12 driver settings, and with some 10 bytes of packet header + footer overhead, this translates to (19 + 10) * 8 * 20 = 4,640 µS “on-air” time once every 10 seconds, i.e. still under 0.05 % bandwidth utilisation. Fine.

This is weblog post #1200 – onwards! :)

Encoding P1 data

In Software on Jan 2, 2013 at 00:01

After yesterday’s hardware hookup, the next step is to set up the proper software for this.

There are a number of design and implementation decisions involved:

  • how to format the data as a wireless packet
  • how to generate and send out that packet on a JeeNode
  • how to pick up and decode the data in Node.js

Let’s start with the packet: there will be one every 10 seconds, and it’s best to keep the packets as small as possible. I do not want to send out differences, like I did with the otRelay setup, since there’s really not that much to send. But I also don’t want to put too many decisions into the JeeNode sketch, in case things change at some point in the future.

The packet format I came up with is one which I’d like to re-use for some of the future nodes here at JeeLabs, since it’s a fairly convenient and general-purpose format:

       (format) (longvalue-1) (longvalue-2) ...

Yes, that’s right: longs!

The reason for this is that the electricity meter counters are in Watt-hour, and will immediately exceed what can be stored as 16-bit ints. And I really do want to send out the full values, also for gas consumption, which is in 1000th of a m3, i.e. in liters.

But for these P1 data packets that would be a format code + 8 longs, i.e. at least 25 bytes of data, which seems a bit wasteful. Especially since not all values need longs.

The solution is to encode each value as a variable-length integer, using only as many bytes as necessary to represent each value. The way this is done is to stored 7 bits of the value in each byte, reserving the top-most bit for a flag which is only set on the last byte.

With this encoding, 0 is sent as 0x80, 127 is sent as 0xFF, whereas 128 is sent as two bytes 0x01 + 0x80, 129 is sent as 0x01 + 0x81, 1024 as 0x08 + 0x80, etc.

Ints up to 7 bits take 1 byte, up to 14 take 2, … up to 28 take 4, and 32 will take 5 bytes.

It’s relatively simple to implement this on an ATmega, using a bit of recursion:

Screen Shot 2012-12-31 at 13.47.01

The full code of this p1scanner.ino sketch is now in JeeLib on GitHub.

Tomorrow, I’ll finish off with the code used on the receiving end and some results.

Picking up P1 data

In Hardware on Jan 1, 2013 at 00:01

Just in time for 2013, I hooked up the smart meter which was installed a month ago:


This connects DIO4 to the output of the P1 port, which has the following pinout:

Screen Shot 2012-12-31 at 11.25.43

The circuit is as follows:

JC's Grid, page 52

The request pin has to be pulled to 5V (it very likely just powers the isolated side of the built-in optocoupler). The 10 kΩ signal pull-up is needed to improve the rising flank of the signal, and the 10 kΩ resistor to the I/O pin prevents problems when the input signal rises above the 3.3V powering the ATmega.

Tomorrow, I’ll describe the software that goes along with this.