Computing stuff tied to the physical world

Archive for October 2012

Meet the wireless JeeBoot

In Software on Oct 31, 2012 at 00:01

This has been a long time coming, and the recent Elektro:camp meet-up has finally pushed me to figure out the remaining details and get it all working (on foam board!):

DSC 4220

Bottom middle is a JeeLink, which acts as a “boot server” for the other two nodes. The JeeNode USB on the left and the JeeNode SMD on the right (with AA Power Board) both now have a new boot loader installed, called JeeBoot, which supports over-the-air uploading via the RFM12B wireless module.

The check for new firmware happens when pressing reset on a remote node (not on power-up!). This mechanism is already quite secure, since you need physical access to the node to re-flash it. Real authentication could be added later.

The whole JeeBoot loader is currently a mere 1.5 KB, including a custom version of the RF12 driver code. Like every ATmega boot loader, it is stored in the upper part of flash memory and cannot be damaged by sketches running amok. The code does not yet include proper retry logic and better low-power modes while waiting for incoming data, but that should fit in the remaining 0.5 KB. The boot loader could be expanded to 4 KB if need be, but right now this thing is small enough to fit even in an ATmega168, with plenty of room left for a decent sketch.

The boot algorithm is a bit unconventional. The mechanism is driven entirely from the remote nodes, with the central server merely listening and responding to incoming requests in a state-less fashion. This approach should offer better support for low-power scenarios. If no new code is available, or if the server does not respond quickly, the remote node continues by launching the current sketch. If the current sketch does not match its stored size and CRC (perhaps because of an incomplete or failed previous upload attempt), then the node retries until it has a valid sketch. That last part hasn’t been fully implemented yet.

The boot server node can be a JeeLink, which has enough memory on board to store different sketches for different remote nodes (not all of them need to be running the same code). But it could also be another RFM12B-based setup, such as a small Linux box or PC.

This first test server has just two fixed tiny sketches built in: fast blink and slow blink. It alternately sends either one or the other, which is enough to verify that the process works. Each time any node is reset, it’ll be updated with one of these two sketches. A far more elaborate server sketch will be needed for a full-fledged over-the-air updatable WSN.

But hey, it’s a start, and the hardest part is now done!


Debugging with a scope

In Hardware, Software on Oct 30, 2012 at 00:01

Good, but not perfect… I was curious about the actual current consumption of this latest version of the homeGraph.ino sketch. So here’s another timing snapshot:

Annotated capture

This has trace persistence turned on, collecting multiple samples in one screen shot.

As you can see, the radio is turned on for ≈ 75 ms in this case (different JeeNode, slightly different watchdog timer period). Then after 30..35 ms the sketch goes to sleep again.

The second case is when nothing has been received after 150 ms: the “radioTimer” fires, and in this case the sketch turns off the radio as well, assuming there will be no packet coming in anymore.

But the interesting bit starts on the next packet coming in after one has been omitted. The logic in the code is such that the radio will be left on indefinitely in this case. As you can see, every late arrival then comes 100 ms later than expected: same radio-off + power down signature on the right-hand side of the screen capture, just much later.

Here is the code again:

Screen Shot 2012 10 21 at 22 41 49

And the main loop which calls this code:

Screen Shot 2012 10 21 at 22 06 18

That “if (!timingWasGood) …” code is probably wrong.

But it’s not so easy to fix, I’m afraid. Because the timing gets even more tricky if two or more packets are omitted, not just occasionally a single one. Sometimes, the radio doesn’t get switched off quickly – as you can see in this 30-minute capture at the end of the day, when power levels are starting to drop:

SCR56

Maybe I need to try out a software PLL implementation after all. The other reason is that there seems to be a fair amount of variation in watchdog clock timing between different ATmega’s. For one, I had to use 2800 as value for recvOffTime, with another 3000 worked a lot better. A self-locking algorithm would solve this, and would let us predict the next packet time with even more accuracy.

But this is still good enough for now. Normally the radio will only be on for about 100 ms every 3s, so that’s a 30-fold power reduction, or about 0.6 mA on average for the radio.

This still ought to run for a month or two on a 3x AA battery pack, at least for daytime use. Which brings up another idea: turn the whole thing off when it’s dark – the display is not readable without light anyway.

Oh well. Perfection is the enemy of done.


Final HomeGraph setup

In Hardware, Software on Oct 29, 2012 at 00:01

This concludes my little gadget to track home energy use and solar energy production:

DSC 4219

The graph shows production vs total consumption in 15-minute intervals for the last 5 hours. A summary of this infomation is shown at the bottom: “+” is total solar production in the last 5 hours, “-” is total energy consumption in that same period.

The actual consumption values are not yet correct because the home energy pulse counter is wired incorrectly, but they will be once that is fixed. The total home consumption is currently 1327 – 1221 + 7 = 113W, since the home counter is currently driven in reverse.

The graph is auto-scaling and I’m storing these values in EEPROM whenever it scrolls, so that a power-down or reset from say a battery change will only lose the information accumulated in the last 15 minutes.

Power consumption is “fairly low”, because the backlight has been switched off and the radio is turned off between predicted reception times. The mechanism works quite well when there is a packet every 3 or 6 seconds, but with longer intervals (i.e. at night), the sketch still keeps the receiver on for too long.

A further refinement could be to reduce the scan cycle when there are almost no new pulses coming in – and then picking up again when the rate increases. Trouble is that it’s impossible to predict accurately when packets will be skipped, so the risk is that the sketch quickly goes completely out of sync when packet rates do drop. The PLL approach would be a better option, no doubt.

But all in all, I’m quite happy with the result. The display is reasonably easy to read in daylight, even without the backlight. I’ll do a battery-lifetime test with a fresh new battery once the pulse counter wiring has been fixed.

The code has become a bit long to be included in this post – it’s available as homeGraph on GitHub, as part of the GLCDlib project. I’m still amazed by how much a little 200-line program can do in just 11 KB of flash memory, and how it all ends up as a neat custom gadget. Uniquely tailored for JeeLabs, but it’s all open source and easy to adapt by anyone.

Feel free to add a light sensor, a PIR motion detector, an RTC, whatever…

Onwards!


It’s all about timing – part 2

In Software on Oct 28, 2012 at 00:01

Yesterday’s optimisation was able to achieve an estimated 60-fold power consumption, by turning on the radio only when a packet is predicted to arrive, and going into a deep sleep mode for the rest of the time.

That does work, but the problem is that the sending node isn’t sending out a packet every 3 seconds. Sometimes no packet is sent because there is nothing to report, and sometimes a packet might simply be lost.

Yesterday’s code doesn’t deal with these cases. It merely waits after proper reception, because in that case the time to the next packet is known and predictable (to a certain degree of accuracy). If a packet is not coming in at the expected time, the sketch will simply continue to wait indefinitely with the radio turned on.

I tried to somewhat improve on this, by limiting the time to wait when a packet does not arrive, turning the radio off for a bit before trying again. But that turns out to be rather complex – how do you deal with uncertainty when the last known reception is longer and longer ago? We can still assume that the sender is staying in its 3s cycle, but: 1) the clocks are not very accurate on either end (especially the watchdog timer), and 2) we need a way to resync when all proper synchronisation is lost.

Here’s my modified loop() code:

Screen Shot 2012 10 21 at 22 06 18

And here’s the slightly extended snoozeJustEnough() logic:

Screen Shot 2012 10 21 at 22 06 04

Yes, it does work, some of the time – as you can see in this slow scope capture:

SCR51

Four of those blips are working properly, i.e. immediate packet reception, drop in current draw, and then display update. Two of them are missed packets (probably none were sent), which are again handled properly by turning the receiver off after 150 ms, instead of waiting indefinitely.

But once that first packet reception fails, the next try will be slightly too late, and then the receiver just stays on for the full 3 seconds.

And that’s actually just a “good” example I picked from several runs – a lot of the time, the receiver just stays on for over a dozen seconds. This algorithm clearly isn’t sophisticated enough to deal with periodic packets when some of ’em are omitted from the data stream.

I can think of a couple of solutions. One would be to always send out a packet, even just a brief empty one, to keep the receiver happy (except when there is packet loss). But that seems a bit wasteful of RF bandwidth.

The other approach would be to implement a Phase Locked Loop in software, so that the receiver tracks the exact packet frequency a lot more accurately over time, and then gradually widens the receive window when packets are not coming in. This would also deal with gradual clock variations on either end, and if dimensioned properly would end up with a large window to re-gain “phase lock” when needed.

But that’s a bit beyond the scope of this whole experiment. The point of this demonstration was to show that with extra smarts in the receiving node, it is indeed possible to achieve low-power periodic data reception without having to keep the receiver active all the time.

Update – With the following timing tweak, everything works out a lot better already:

Screen Shot 2012 10 21 at 22 41 49

The receiver is now staying in sync surprisingly well – it’s time to declare victory!


It’s all about timing

In Software on Oct 27, 2012 at 00:01

The previous post showed that most of the power consumption of the homeGraph.ino sketch was due to the RFM12B receiver being on all the time. This is a nasty issue which comes back all the time with Wireless Sensor Networks: for ultra-low power scenarios, it’s simply impossible to keep the radio on at all times.

So how can we pick up readings from the new homePower.ino sketch, used to report energy consumption and production of the new solar panels?

The trick is timing: the homePower.ino sketch was written in such a way that it only sends out a packet every 3 seconds. Not always, only when there is something to report, but always exactly on the 3 second mark.

That makes it possible to predict when the next packet can be expected, because once we do receive a packet, we know that we don’t have to expect one for another 3 seconds. Aha, so we could turn the receiver off for a while!

It’s very easy to try this out. First, the code which goes to sleep:

Screen Shot 2012 10 21 at 21 26 40

The usual stuff really – all the hard work is done by the JeeLib library code.

And here’s the new loop() code:

Screen Shot 2012 10 21 at 20 24 23

In other words: whenever a packet has been received, process it, then go to sleep for 2.8 seconds, then wake up and start the receiver again.

Here’s the resulting power consumption timing, as voltage drop over a series resistor:

SCR52

I can’t say I fully understand what’s going on, but I think it’s waiting for the next packet for about 35 ms with the receiver enabled (drawing ≈ 18 mA), and then another 35 ms is spent generating the graph and sending the new image to the Graphics Board over software SPI (drawing 7 mA, i.e. just the ATmega).

Then the µC goes to sleep, leaving just the display showing the data.

So we’re drawing about 18 mA for say 50 ms every 3000 ms – this translates to a 60-fold reduction in average current consumption, or about 0.3 mA (plus the baseline current consumption). Not bad!

Unfortunately, real-world use isn’t working out quite as planned… to be continued.


Reducing power consumption

In Software on Oct 26, 2012 at 00:01

The problem with yesterday’s setup is power consumption:

DSC 4213

There’s no way the sketch will run for a decent amount of time on just a single AA battery.

Let’s take some measurements:

  • total power consumption on a 5V supply (via the USB BUB) is 20 mA
  • total power consumption on 3.3V is almost the same: 19 mA
  • the battery drain on the Eneloop at 1.28V is a whopping 95 mA

(the regulator on the AA board was selected for its low idle current, not max efficiency)

That last value translates to a run time of 20 hours on a fully charged battery. Fine for a demo, but definitely not very practical if this means we must replace batteries all the time.

Let’s try to reduce the power consumption of this thing – my favourite pastime.

The first obvious step is to turn off the Graphics Board backlight – a trivial change, since it can be done under software control. With the dark-on-light version of the GLCD this is certainly feasible, since that display is still quite readable in ambient light. But the net effect with a 5V power is that we now draw 16.5 mA … not that much better!

The real power consumer is the RFM12B module, which is permanently on. Turning it off drops the power consumption to 6.0 mA (with the Graphics Board still displaying the last values). And putting the ATmega to sleep reduces this even further, down to 0.5 mA. Now we’re cookin’ – this would last some 6 months on 3 AA batteries.

Except that this variant of the “homeGraph” sketch is next to useless: it powers up, waits for a packet, updates the display, and then goes to sleep: forever… whoops!

But maybe there’s a way out – stay tuned…

Who needs a server?

In Hardware, Software on Oct 25, 2012 at 00:01

With all those energy pulses flying through the air, we don’t even need a central setup to display this information on a little self-contained Graphics Board display – like so:

DSC 4214

The “homeGraph.ino” sketch which drives it has just been added to GitHub:

Screen Shot 2012 10 21 at 15 20 54

There’s room for a little graph, but it’ll have to wait until the home usage counter has been rewired to measure the proper value (the value shown above is in fact the surplus).

Goodbye Mousetrap

In Hardware on Oct 24, 2012 at 00:01

With the new “homePower” setup now in place and working, it is time to say goodbye to a good companion – this ATmega168-based mousetrap which started it all, 4 years ago:

DSC 4206

Note the dead spider :) – this thing has been soaked a few times, with water dripping from the kitchen on the next floor, yet all it took was a clean wipe, some time to dry, and it just resumed its duties over and over again:

DSC 4210

Very few problems other than that corrosion (probably also inside the mini 3-pin jacks):

DSC 4207

Less than a dozen resets / power cycles over all these years.

This is the predecessor of what eventually became the JeeNode, based on an RBBB by Modern Device. It works on 5V and uses resistor-based level converters for the RFM12B.

The gas measurements have been erratic for some time now, no doubt due to those bad contacts, so I’m going to look for another way to read those values.

And with it ends also the life of the node on the other side, a modified Arduino Mini plugged into the Mac Mini server using an FTDI cable:

DSC 4208

Also based on perf board with Kynar wire-wrapping wire to connect it all together:

DSC 4209

The code used between these two nodes was based on an early version of the RF12 driver protocol, and since it all worked perfectly, I never saw a need to change it. But I’m still going to miss these periodic packets in the logfiles:

  L 09:05:02.167 usb-FTE50C3P HM3 45 43 17 151 73 20 6 0 242 200 0 0 0 0
  L 09:05:07.363 usb-FTE50C3P HM3 45 44 18 151 80 20 6 0 242 200 0 0 0 0
  L 09:05:12.290 usb-FTE50C3P HM3 45 45 19 151 47 19 6 0 242 200 0 0 0 0
  L 09:05:17.361 usb-FTE50C3P HM3 45 46 20 151 200 19 6 0 242 200 0 0 0 0
  L 09:05:22.720 usb-FTE50C3P HM3 45 47 21 151 240 20 6 0 242 200 0 0 0 0
  L 09:05:27.646 usb-FTE50C3P HM3 45 48 22 151 49 19 6 0 242 200 0 0 0 0

Goodbye, dear nodes, and thank you for those nearly 4 years of perfect service!

PS – Don’t forget Elektro:camp next Friday and Saturday!

Decoding the pulses

In Software on Oct 23, 2012 at 00:01

Receiving the packets sent out yesterday is easy – in fact, since they are being sent out on the same netgroup as everything else here at JeeLabs, I don’t have to do anything. Part of this simplicity comes from the fact that the node is broadcasting its data to whoever wants to hear it. There is no need to define a destination in the homePower.ino sketch. Very similar to UDP on Ethernet, or the CAN bus, for that matter.

But incoming data like this is not very meaningful, really:

  L 22:09:25.352 usb-A40117UK OK 9 2 0 69 235 0 0 0 0 103 0 97 18

What I have in mind is to completely redo the current system running here (currently still based on JeeMon) and switch to a design using ZeroMQ. But that’s still in the planning stages, so for now JeeMon is all I have.

To decode the above data, I wrote this little “homePower.tcl” driver:

Screen Shot 2012 10 21 at 01 52 07

It takes those incoming 12-byte packets, and converts them to three sets of results – each with the total pulse count so far (2000 pulses/KWh), and the last calculated instantaneous power consumption. Note also the “decompression” of the millisecond time differences, as applied on the sending side.

Calculation of the actual Watts being consumed (or produced) is trivial: there are 2000 pulses per KWh, so one pulse per half hour represents an average consumption (or production) of exactly one Watt.

To activate this driver I also had to add this line to “main.tcl”:

  Driver register RF12-868.5.9 homePower

And sure enough, out come results like this:

Screen Shot 2012 10 21 at 01 52 32

This is just after a reset, at night with no solar power being generated. That’s 7 Watt consumed by the cooker (which is off, but still drawing some residual power for its display and control circuits), and 105 Watt consumed by the rest of the house.

Actually, you’re looking at the baseline power consumption here at JeeLabs. I did these measurements late at night with all the lights and everything else turned off (this was done by staring at these figures from a laptop on wireless, running off batteries). A total of 112 Watt, including about 24 Watt for the Wireless router plus the Mac Mini running the various JeeLabs web servers, both always on. Some additional power (10W perhaps?) is also drawn by the internet modem downstairs, so that leaves only some 80 Watt of undetermined “vampire power” drawn around the house. Not bad!

One of my goals for the next few months will be to better understand where that remaining power is going, and then try to reduce it even further – if possible. That 80 W baseline is 700 KWh per year after all, i.e. over 20% of the total annual consumption here.

Here are some more readings, taken the following morning with heavy overcast clouds:

Screen Shot 2012 10 21 at 10 24 37

This also illustrates why the wiring error is causing problems: the “pow3” value is now a surplus (counting down), but there’s no way to see that in the measurement data.

I’ve dropped the packet sending rate to at most once every 3 seconds, and am very happy with these results which give me a lot more detail and far more frequent insight into our power usage around here. Just need to wait for the electrician to come back and reroute counter 3 so it doesn’t include solar power production.

Sending out pulses

In Software on Oct 22, 2012 at 00:01

With pulses being detected, the last step in this power consumption sketch is to properly count the pulses, measure the time between ’em, and send off the results over wireless.

There are a few details to take care off, such as not sending off too many packets, and sending out the information in such a way that occasional packet loss is harmless.

The way to do this is to track a few extra values in the sketch. Here are the variables used:

Screen Shot 2012 10 21 at 00 14 43

Some of these were already used yesterday. The new parts are the pulse counters, last-pulse millisecond values, and the payload buffer. For each of the three pulse counter sources, I’m going to send the current count and the time in milliseconds since the last pulse. This latter value is an excellent indication of instantaneous power consumption.

But I also want to keep the packet size down, so these values are sent as 16-bit unsigned integers. For count, this is not so important as it will be virtually impossible to miss over 65000 packets, so we can always resync – even with occasional packet loss.

For the pulse time differences, having millisecond resolution is great, but that limits the total to no more than about a minute between pulses. Not good enough in the case of solar power, for example, which might stop on very dark days.

The solution is to “compress” the data a bit: values up to one minute are sent as is, values up to about 80 minutes are sent in 1-second resolution, and everything above is sent as being “out of range”.

Here are the main parts of the sketch, the full “homePower.ino” sketch is now on GitHub:

Screen Shot 2012 10 21 at 00 36 00

Sample output, as logged by a process which always runs here at JeeLabs:

  L 22:09:25.352 usb-A40117UK OK 9 2 0 69 235 0 0 0 0 103 0 97 18

Where “2 0 69 235” is the cooker, “0 0 0 0” is solar, and “103 0 97 18” is the rest.

Note that results are sent off no more than once a second, and the careful distinction between having data-to-send pending and actually getting it sent out only after that 1000 ms send timer expires.

The scanning and blinking code hasn’t changed. The off-by-one bug was in calling setblinks() with a value of 0 to 2, instead of 1 to 3, respectively.

That’s it. The recently installed three new pulse counters are now part of the JeeLabs home monitoring system. Well… as far as remote sensing and reporting goes. Processing this data will require some more work on the receiving end.

Detecting pulses

In Software on Oct 21, 2012 at 00:01

With yesterday’s solar setup operational, it’s now time to start collecting the data.

The pulse counter provides a phototransistor output which is specified as requiring 5..27V and drawing a current up to 27 mA, so my hunch is that it’s a phototransistor in series with an internal 1 kΩ resistor. To interface to the 3.3V I/O pins of a JeeNode, I used this circuit:

Pulse power

That way, if the circuit has that internal 1 kΩ resistor, the pin will go from 0 to 2.5V and act as a logic “1”. In case there is no internal resistor, the swing will be from 0 to 5V, but with the 10 kΩ resistor in series, this will still not harm the ATmega’s I/O pin (excess current will leak away through the internal ESD protection diode).

No need to measure anything, the above will work either way!

I considered building this project into a nice exclosure, but in the end I don’t really care – as long as it works reliably. As only 3 input pins are used, there’s a spare to also drive an extra LED. So here’s the result, built on a JeePlug board – using a 6-pin RJ12 socket:

DSC 4204

The RJ12 socket pins are not on a 0.1″ grid, but they can be pushed in by slanting the socket slightly. The setup has been repeated three times, as you can see:

DSC 4205

To avoid having to chase up and down the stairs while debugging too many things at once, I started off with this little sketch to test the basic pulse counter connections:

Screen Shot 2012 10 20 at 22 08 26

All the code does, is detect when a pulse starts and blink a LED one, two, or three times, depending on which pulse was detected. Note how the two MilliTimer objects make it easier to perform several periodic tasks independently in a single loop.

Tomorrow: logic to track pulse rates and counts, and sending results off into the air.

PS. Off-by-one bug – will be fixed tomorrow.

Solar at last

In Hardware, News on Oct 20, 2012 at 00:01

As mentioned recently, last step was getting the SMA 5000TL inverter and hooking it up:

DSC 4200

DSC 4203

Here’s the before-and-after shot of the hookup:

DSC 4190

DSC 4192

Three pulse-counters to help me measure what comes in, what goes out, and the induction stove (measured separately). Still a minor wiring error, but that’ll be fixed soon.

So that’s the end of the story for now. Let the sun shine!

Solar Railway Station

In Hardware on Oct 19, 2012 at 00:01

On a recent trip to Germany, we visited Uelzen, about 100 km north of Braunschweig. Its railway station was “upgraded” (pimped?) some 12 years ago by an Austrian artist called Friedensreich Hundertwasser in what looks very Gaudi-like in style and appearance:

PA175185

He was interested in organic shapes, creating round and uneven fairy-tale like interiors:

PA175215

But the reason I’m mentioning all this, is that the station also includes this display panel:

PA175218

The roof is covered with solar panels, and has been generating electricity for the community since 1997. Almost half a megigawatt-hour produced so far, and still generating just under 8 Kilowatt on a half-cloudy October day.

Way to go! So gibt mann das gute Vorbild!

PS. Calculating back from the figures shown, this would appear to be just 42 m2 of panels, which seems off by an order of magnitude. Oh, wait… that’s not accounting for the panel’s conversion efficiency, I’m guessing that to be somewhere in the 10% range for those panels.


Push-scanning with coroutines

In Software on Oct 18, 2012 at 00:01

The “benstream.lua” decoder I mentioned yesterday is based on coroutines. Since probably not everyone is familiar with them (they do not exist in languages such as C and C++), it seems worthwhile to go over this topic briefly.

Let’s start with this completely self-contained example, written in Lua:

Screen Shot 2012 10 15 at 01 23 15

Here’s the output generated from the test code at the end:

  (string) abcde
  (string) 123
  (number) 12345
  (boolean) false
  (number) 987
  (boolean) false
  (number) 654
  (table) 
  (number) 321
  (table) 
  (number) 99999999
  (boolean) true
  (string) one
  (number) 11
  (string) two
  (number) 22
  (table) 
  (string) bye!

There is some weird stuff in here, notably how the list start, dict start, and list/dict end are returned as “false”, “true”, and the empty list ({} in Lua), respectively. The only reason for this is that these three values are distinguishable from all the other possible return values, i.e. strings and numbers.

But the main point of this demo is to show what coroutines are and what they do. If you’re not familiar with them, they do take getting used to… but as you’ll see, it’s no big deal.

First, think about the task: we’re feeding characters into a function, and expect it to track where it is and return complete “tokens” once enough data has been fed through it. If it’s not ready, the “cwrap” function will return nil.

The trouble is that such code cannot be in control: it can’t “pull” more characters from the input when it needs them. Instead, it has to wait until it gets called again, somehow figure out where it was the last time, and then deal with that next character. For an input sequence such as “5:abcde”, we need to track being in the leading count, then skip the colon, then track the actual data bytes as they come in. In the C code I added to the EmBencode library recently, I had to implement a Finite State Machine to keep track of things. It’s not that hard, but it feels backwards. It’s like constantly being called away during a meeting, and having to pick up the discussion again each time you come back :)

Now look at the above code. It performs the same task as the C code, but it’s written as if it was in control! – i.e. we’re calling nextChar() to get the next character, looping and waiting as needed, and yet the coroutine is not actually pulling any data. As you can see, the test code at the end is feeding characters one by one – the normal way to “push” data, not “pull” it from the parsing loop. How is this possible?

The magic happens in nextchar(), which calls coroutine.yield() with an argument.

Yield does two things:

  • it saves the complete state of execution (i.e. call stack)
  • it causes the coroutine to return the argument given to it

It’s like popping the call stack, with one essential difference: we’re not throwing the stack contents away, we’re just moving it aside.

Calling a coroutine does almost exactly the opposite:

  • it restores the saved state of execution
  • it causes coroutine.yield() to return with the argument of the call

These are not full-blown continuations, as in Lisp, but sort of halfway there. There is an asymmetry in that there is a point in the code which creates and starts a coroutine, but from then on, these almost act like they are calling each other!

And that’s exactly what turns a “pull” scanner into a “push” scanner: the code thinks it is calling nextChar() to immediately obtain the next input character, but the system sneakily puts the code to sleep, and resumes the original caller. When the original caller is ready to push another character, the system again sneakily changes the perspective, resuming the scanner with the new data, as if it had never stopped.

This is in fact a pretty efficient way to perform multiple tasks. It’s not exactly multi-tasking, because the caller and the coroutine need to alternate calls to each other in lock-step for all this trickery to work, but the effect is almost the same.

The only confusing bit perhaps, is that argument to nextChar() – what does it do?

Well, this is the way to communicate results back to the caller. Every time the scanner calls nextChar(), it supplies it with the last token it found (or nil). The conversation goes like this: “I’m done, I’ve got this result for you, now please give me the next character”. If you think of “coroutine.yield” as sort of a synonym for “return”, then it probably looks a lot more familiar already – the difference being that we’re not returning from the caller, but from the entire coroutine context.

The beauty of coroutines is that you can hide the magic so that most of the code never knows it is running as part of a coroutine. It just does its thing, and “asks” for more by calling a function such as nextChar(), expecting to get a result as soon as that returns.

Which it does, but only after having been put under narcosis for a while. Coroutines are a very neat trick, that can simplify the code in languages such a Lua, Python, and Tcl.

Collecting serial packets

In Software, Linux on Oct 17, 2012 at 00:01

As promised after yesterday’s “rf12cmd” sketch, here’s a Lua script for Linux which picks up characters from the serial port and turns them into Lua data structures:

Screen Shot 2012 10 14 at 17 31 37

Ah, if only things were that simple! On Mac OSX, the serial port hangs when opened normally, so we need to play non-blocking tricks and use the lower-level nixio package:

Screen Shot 2012 10 14 at 17 28 40

(note also the subtle “-f” vs “-F” flag for stty… welcome to the world of portability!)

Here’s some sample output:

  $ ./rf12show.lua 
  (table) 
    [1] = (string) hi
    [2] = (string) rf12cmd
    [3] = (number) 1
    [4] = (number) 100
  (table) 
    [1] = (string) rx
    [2] = (number) 868
    [3] = (number) 5
    [4] = (number) 19
    [5] = (12 bytes) BAC80801E702200000DA5121
  (table) 
    [1] = (string) rx
    [2] = (number) 868
    [3] = (number) 5
    [4] = (number) 3
    [5] = (4 bytes) 21DE0F00
  (table) 
    [1] = (string) rx
    [2] = (number) 868
    [3] = (number) 5
    [4] = (number) 19
    [5] = (8 bytes) 7331D61AE7C43901

That’s the startup greeting plus three incoming packets.

I’m using a couple of Lua utility scripts – haven’t published them yet, but at least you’ll get an idea of how the decoding process can be implemented:

  • dbg.lua – this is the vardump script, extended to show binary data in hex format
  • benstream.lua – a little script I wrote which does “push-parsing” of Bencoded data

Note that this code is far too simplistic for real-world use. The most glaring limitation is that it is blocking, i.e. we wait for each next character from the serial port, while being completely unresponsive to anything else.

Taking things further will require going into processes, threads, events, asynchronous I/O, polling, or some mix thereof – which will have to wait for now. To be honest, I’ve become a bit lazy because the Tcl language solves all that out of the box, but hey… ya’ can’t have everything!

Reporting serial packets

In Software on Oct 16, 2012 at 00:01

The RF12demo sketch was originally intended to be just that: a demo, pre-flashed on all JeeNodes to provide an easy way to try out wireless communication. That’s how it all started out over 3 years ago.

But that’s not where things ended. I’ve been using RF12demo as main sketch for all “central receive nodes” I’ve been working with here. It has a simple command-line parser to configure the RF12 driver, there’s a way to send out packets, and it reports all incoming packets – so basically it does everything needed:

  [RF12demo.8] A i1* g5 @ 868 MHz 
  DF I 577 10

  Available commands:
    <nn> i     - set node ID (standard node ids are 1..26)
                 (or enter an uppercase 'A'..'Z' to set id)
    <n> b      - set MHz band (4 = 433, 8 = 868, 9 = 915)
    <nnn> g    - set network group (RFM12 only allows 212, 0 = any)
    <n> c      - set collect mode (advanced, normally 0)
    t          - broadcast max-size test packet, with ack
    ...,<nn> a - send data packet to node <nn>, with ack
    ...,<nn> s - send data packet to node <nn>, no ack
    <n> l      - turn activity LED on PB1 on or off
    <n> q      - set quiet mode (1 = don't report bad packets)
  Remote control commands:
    <hchi>,<hclo>,<addr>,<cmd> f     - FS20 command (868 MHz)
    <addr>,<dev>,<on> k              - KAKU command (433 MHz)
  Flash storage (JeeLink only):
    d                                - dump all log markers
    <sh>,<sl>,<t3>,<t2>,<t1>,<t0> r  - replay from specified marker
    123,<bhi>,<blo> e                - erase 4K block
    12,34 w                          - wipe entire flash memory
  Current configuration:
   A i1* g5 @ 868 MHz

This works fine, but now I’d like to explore a real “over-the-wire” protocol, using the new EmBencode library. The idea is to send “messages” over the serial line in both directions, with named “commands” and “events” going to and from the attached JeeNode or JeeLink. It won’t be convenient for manual use, but should simplify things when the host side is a computer running some software “driver” for this setup.

Here’s the first version of a new rf12cmd sketch, which reports all incoming packets:

Screen Shot 2012 10 14 at 15 31 51

Couple of observations about this sketch:

  • we can no longer send a plain text “[rf12cmd]” greeting, that too is now sent as packet
  • the greeting includes the sketch name and version, but also the decoder’s packet buffer size, so that the other side knows the maximum packet size it may use
  • invalid packets are discarded, we’re using a fixed frequency band and group for now
  • command/event names are short – let’s not waste bandwidth or string memory here
  • I’ve bumped the serial line speed to 115200 baud to speed up data transfers a bit
  • there’s no provision (yet) for detecting serial buffer overruns or other serial link errors
  • incoming data is sent as a binary string, no more tedious hex/byte conversions
  • each packet includes the frequency band and netgroup, so the exchange is state-less

The real change is that all communication is now intended for computers instead of us biological life-forms and as a consequence some of the design choices have changed to better match this context.

Tomorrow, I’ll describe a little Lua script to pick up these “serial packets”.

Going solar – almost there!

In Hardware on Oct 15, 2012 at 00:01

It’s been over two years ago since I started looking for ways to collect solar energy on the roof, here at JeeLabs.

And then reality set in… the roofs of the houses in this street are covered with shingles from the late 70’s, and as we found out they contain asbestos – yuck! Then, earlier this year, we were told that nothing could be done on the roof unless those shingles were first officially removed by a specially-equipped third party:

Image 5

So now at last, those “bad” shingles have been replaced and panels have been mounted:

DSC 4177

That little bulge is a “Solar tube” which brings extra sunlight into the stairway.

That’s 2 x 5 panels on this side, and 3 x 4 on the other roof:

DSC 4183

(the 10 panels you saw in the first picture are mounted on the roof to the far left)

For a total of 22 x 240 = 5280 Wattpeak. Not that they will ever all generate full power at the same time, because these two roofs are facing west and east, respectively. But hey, together they should still give us more than our annual 3100 kWh consumption. JeeLabs is about to become a net electricity producer!

Just one more step: wait for the SMA 5000 inverter to arrive and get it hooked up…

Paper-based Google

In Musings on Oct 14, 2012 at 00:01

Look what got delivered (unannounced) here at JeeLabs the other day:

DSC 4171

That’s nearly 3000 pages of product and price information, in a 2 kg package :)

Crazy? Maybe. But there’s still some value in seeing so many products at one glance:

DSC 4172

What would be even nicer, IMO, is a paper version with QR codes so you can instantly tie what you see to a web page, with more information, full quantity pricing details, and engineering specs / datasheets. Or at least a direct link between each full page and the web – that’s just 4 digits to enter, after all.

It feels a bit old-fashioned to leaf through such a catalog, but hey… when you don’t know exactly what you need (or more likely: what range of solutions is available), then that can still beat every parametric search out there.

RS Components have an excellent selection of electronic and mechanical parts, BTW.

Mechanical relays

In Hardware on Oct 13, 2012 at 00:01

Hm… with such a nice switching 5V pulse, it’s sad to tear down yesterday’s circuit again.

One more experiment then. Let’s switch a little relay (the one used in the Relay Plug) at 10 Hz, and give it a serious beating. I’m going to put my dual power supply to work and put 5V @ 3A straight across the relay’s contacts. In other words, let’s see what happens when we switch the full 3A across those tiny relay contacts.

At first sight, it all looks reasonably ok:

SCR46

That’s 5V across the relay while it’s open, and about 0.3V when it closes. So these contacts seem to have a resistance of about 0.1 Ω. The overshoot when the relay opens up again is probably in the power supply, as it recovers from just having been shorted out at 3A. Note that any inductance in the wiring will have this same effect. Inductance is a bit like a flywheel – it wants to keep going after such massive current (i.e. magnetic field) changes.

It’s quite impressive to see how this little relay rattles away at ≈ 10 Hz, working just fine.

But those transitions look quite different when we zoom in:

SCR47

You’re seeing a mix of arcing and contact bounce, as all mechanical switches do.

Note that the switch hasn’t quite closed yet – there is still about 1.5V between the contact points, 30 µs after switching – due to contact bounce. This drops to 0.3V after some 400 µs, at which point the contacts are really firmly closed.

Here’s a much better example, this time using a 10 Ω resistor in series so the power supply just delivers 5V @ 0.5A without going into current-limiting mode:

SCR50

Opening is again not quite what it seems, once you zoom in:

SCR48

Two things happening here: the mechanical release of the switch (less resistance as the pressure on the contact decreases, followed by some arcing), and then a fairly linear ramp and some overshoot as the power supply recovers its 5V setting after having been shorted out. Think of a stretched rubber band, and how it “overreacts” when released.

So as you can see, a relay does some nasty things while switching on and off!

Digital opto switching

In Hardware on Oct 12, 2012 at 00:01

While on the subject of optocouplers, there’s another type besides “analog” ones and “digital” ones (which include a comparator), and that’s the opto-relay. Again with several kilovolts of isolation.

The Avago ASSR-1611 is an interesting one, for example, because it uses MOSFETs:

Screen Shot 2012 10 07 at 14 11 47

Basically, it lets you switch up to 60V at a few amps. To see how it performs, while I still have that linear ramp circuit up and running, I hooked it up – as a big mess-o-wires:

DSC 4173

It’s getting pretty crowded in there. The ASSR-1611 is on the left. Here’s its schematic:

Screen Shot 2012 10 07 at 14 07 07

The interesting bit is that there are diodes in there, so it can deal with alternating current when hooked up differently, i.e. by using pins 4 and 6 instead of 5 and 6.

First thing to notice, is that this thing behaves in a strange way when switched at 1000 Hz:

SCR43

It “sort of” triggers … slowly (keep in mind that turning on translates to shorting the output to ground, as before). And then it decides to turn off again very quickly. For some reason this repeats about 10 times per second.

To slow the triangle wave rate way down, I used a 10 µF capacitor instead of 0.1 µF:

SCR44

Aha! Much better. At about 0.7 mA (purple voltage over 1 kΩ = 0.7V), this solid state relay switches on, and once the current drops back to almost 0, it switches off. Note how these readings match the specs nicely: turning on, i.e. the blue line dropping to 0, takes a few ms, whereas turning off is virtually instant.

At a few Euro each, these chips are not really cheaper than mechanical relays, but when you only need to switch a few dozen Volt at a few Amps, then this solution still has the benefit that it switches far more cleanly – with no arcing or mechanical wear. And it’s totally quiet, of course…

Optocoupler current transfer

In Hardware on Oct 11, 2012 at 00:01

The past few days were about generating a linear ramp, in the form of a triangular wave, and as you saw, it was quite easy to generate – despite the lack of a function generator.

The result was a voltage alternating between about 0.6V and 3.0V in a linear fashion. Here’s why…

I want to see how the MCT62 optocoupler passes a signal through it. More specifically, how a linearly increasing voltage would come out. Let’s look at that chip schematic again:

Mct621

So the idea is to apply that linear ramp through a current-limiting resistor into the opto’s LED. Then we put the photo-transistor in a simple 5V circuit, with again a current limiting resistor between collector and 5V – like this:

JC s Grid page 35

From left to right:

  • apply a triangle wave to the LED, which varies from 0.6 to 3.0V
  • there’s a 1 kΩ rsistor in series, so the maximum current will stay well under 3 mA
  • the phototransistor is hooked up as a normal DC amplifier
  • there’s another 1 kΩ pullup, so this too cannot draw more than 5 mA current

Prediction:

  • when the LED is off, the output will stay at 5V, i.e. transistor stays off
  • until the input rises above the 1.2V threshold of the (IR) LED, not much happens
  • as the voltage rises linearly, so will the current through the LED
  • depending on the transfer function the transistor current will rise accordingly
  • and as a consequence, the output voltage will drop

So if that behavior is linear, then the output voltage should drop linearly. Let’s have a look:

SCR42

  • the YELLOW line is the triangle wave, as generated earlier
  • the PURPLE line is the voltage over the leftmost resistor
  • the BLUE line is the voltage on the transistor’s collector output
  • the RED line is the derivative of the BLUE line
  • the zero origin for all these lines in the image is at two divisions from the bottom

First of all, the purple line indeed rises slowly once we start rising above 1.0V, and it stays roughly 1.2V under the input signal (yellow line).

The blue line is the interesting one: it takes a bit of input current (i.e. LED light) for the transistor to start conducting, but once it does, the output voltage drops indeed. Once we’re above 2.0V, the blue line becomes quite linear. As indicated by the fact that the red line is fairly flat between horizontal divisions 5 and 7.

So in this range (and probably quite a bit above), we have a linear transfer from input current to output current. Or voltage … it’s all the same with resistors.

In terms of current, we can use the purple line: it’s flat with a diode current between 0.7 and 1.7 mA (and probably beyond).

The output voltage only drops to just over 2V, so the phototransistor is still far from reaching saturation (“conducting all out”).

So what’s the point of all this, eh?

Well, one thing this illustrates is that you can get a pretty clean signal across such an optocoupler, as long as you stay in the linear range of it all. There is no real speed limitation, so even audio signals could be sent across reasonably well – without making any electrical connection, just a little light beam!

It’s not hard to imagine how this could be done with discrete components even, sending the light to a glass fiber over a longer distance.

You can call it wireless signal transmission, albeit of a different type: optical!

Elektro:camp October 26 and 27

In News on Oct 10, 2012 at 00:01

A year has passed, and it’s time for a get-together again, in Leuven, Belgium this time:

Elektro camp 2012 10 final

What is it?

Elektro:camp is a place where geeks meet to talk about smart metering, smart homes, smart grids, and smart ideas. Everything in and around the house related to electricity and electronics, really. Oh, and software.

This is a “barcamp” so there’s no fixed agenda: we make it up as we go, and we all present our ideas and discuss the ideas of others. Trust me, if things go as they did last time, then we’ll be scrambling to find enough time to go through everything that pops up.

Wanna have lots of fun with a bunch of geeks? Wanna show what you’re working on? Wanna present some new ideas? Wanna meet up in person? Be there!

It’s at the Faculty of Engineering this time, and since it’s all done as volunteer effort, please register soon if you’re considering joining us. For questions and other info, please contact Bart Van Der Meerssche.

Looking forward to meet some of you in a couple of weeks!

Pretty good triangles

In Hardware on Oct 9, 2012 at 00:01

As of yesterday, we now have a “triangular wave generator”. Whee!

Except that I don’t want a voltage between 1.25V and 3.75V but slightly lower. One way to accomplish this is to lower the reference “1/2 Vcc” voltage used by the comparator and integrator circuits. So I added a 22 kΩ resistor in parallel on one half of the voltage divider:

+5V  <=>  10 kΩ  <=|=>  (10 kΩ and 22 kΩ in parallel)  <=>  GND
                   |
                  TAP

This online calculator says that the parallel value is 6.875 kΩ. Here’s the resulting signal:

SCR38

It’s now slightly asymmetric (we’re discharging faster than we’re charging), but more importantly, the signal now runs from about 0.6V to 3.0V, which is more in sync with what I need (more on that in an upcoming post).

Notice that on these screen shots, the waveforms look very nice and straight, although it’s hard to see just how linear those ramps really are.

This is where a scope with good math functions comes in. If you recall from mathematics, the derivative of a straight line is a constant. Or to put it differently: the straighter the line is, the closer its derivative should be to a constant value. Positive for upward slopes, and negative for downward slopes. Let’s zoom in a bit:

SCR37

(the red line’s origin is centered vertically, the yellow line is at 1 division from the bottom)

That red line is the scope’s calculated derivative of the yellow line (it’s really just a matter of calculating differences between successive points). As you can see, the upward slope is pretty straight from 1.3V to 2.9V. The downward slope less so, IOW the capacitor discharge is not quite as linear. The signal was averaged over 128 samples in this last screen.

Excellent. I now have the signal I need to perform my experiment. Stay tuned.

Triangular wave generator

In Hardware on Oct 8, 2012 at 00:01

Let’s build an oscillator, now that we can generate linear ramps. But to get there, we’ll need a second op-amp circuit (1/2 Vcc is the middle of two 10 kΩ R’s between Vcc and GND):

JC s Grid page 34

This is a comparator with hysteresis, or Schmitt trigger. Same rules as yesterday, of course.

In contrast to yesterday’s setup, there is no negative feedback here, but a resistor between “OUT” and “+”. So in this case, when the output rises, it’ll cause “+” to rise even more, instead of bringing it closer to the “-” input.

This circuit doesn’t work towards an equilibrium, it’s unstable. Let’s start with the output being VCC, i.e. +5V. What will the input need to be to make it change?

  • the desired change can only happen, when the “+” input drops below 2.5V
  • with OUT at +5, that’s 2.5V over 20 kΩ, i.e. 125 µA
  • to pull “+” under 2.5V, we need to draw at least 125 µA out through the 10 kΩ resistor
  • that’s 1.25V under 2.5V, i.e. with the IN level under 1.25V

So when IN drops under 1.25V, the op-amp has a change of heart so to speak, and starts bringing OUT down in an attempt to bring “+ and “-” back to the same level. The silly thing is that it can’t, because lowering OUT is never going to raise the “+” back up to 2.5V (still with me?). So the output just keeps keeps dropping all the way to its minimum, which will be more or less equal to 0V.

Let’s review what happened: we started dropping the input level, and once it reached 1.25V, the output violently flipped from +5V to 0V.

Now the situation is reversed: how far do we need to raise the input to make the output go up again? Well, that’ll be 3.75V – using the same reasoning as before, but now based on 2.5V (the “-” pin) + 1.25V (the voltage over the 10 kΩ resistor.

So what this circuit does is flip between 0 and 5V, at trigger points 1.25V and 3.75V.

Now the magic part: we tie the input of this circuit to the output of yesterday’s circuit, and vice versa, creating a control loop. If you look back at yesterday’s scope image, you’ll see that the triangular wave flips at… 1.23V and 3.72V. What a coincidence, eh? Nah… it’s all by design. The comparator drives the integrator by feeding it 0 and 5V levels, and it switches when the integrator output reaches 1.25V and 3.75V. Since the capacitor requires a little time to charge, this ends up being a nicely controlled oscillator. Perfect!

Here’s my test circuit (for a complete schematic, check this website, for example):

DSC 4170

Tomorrow, I’ll change the signal slightly and I’ll examine how linear those ramps are.

Triangular waves

In Hardware on Oct 7, 2012 at 00:01

For an upcoming experiment, I’m going to need a slowly rising voltage, and with my signal generator currently out for a check-up, it’s time to dive into some electronic circuitry again. Let’s try to generate a sawtooth or triangular wave signal with a few basic components. After my search for a simple sine wave generator, I’m happy to report that generating “ramp voltages” is actually a lot simpler.

The reason for this is that all you need to generate a linear ramp, is a constant current into a capacitor. This automatically produces a linear voltage ramp. The electrical notation for such a circuit is:

JC s Grid page 34 copy

The voltage over that capacitor will rise linearly over time. It looks so simple! (well, apart from figuring out how to build a constant current source, perhaps)

But that’s only half the story. What to do once the capacitor has been fully charged up? We need to discharge it again, clearly. One way to do this, is to put a transistor or MOSFET across the cap and periodically make it turn on (briefly!) to discharge the circuit again. Ok, so now we need a periodic pulse as well. Hmmm…

There is another solution: op-amps. The op-amp is a truly amazing little building block. What we need here, is to use two op-amps in different configurations: one as an integrator and one as a comparator.

Let’s start with the integrator, because that explains how we can get a linearly varying voltage out of the circuit:

JC s Grid page 34

An op-amp has two properties, both crucial for explaining this little 3-component circuit:

  • both inputs are very high resistance, so virtually no current flows in or out
  • the output is constantly adjusted to try and keep both inputs at the same voltage

That second property can also be described as: when “+” is higher than “-“, the output will go up (towards Vcc, the supply voltage), when it’s lower, the output will go towards 0 (the ground voltage in a single-supply setup).

Here we’re tying “+” to half Vcc, in this case 2.5V, so you can think of the op-amp as trying to do whatever it can to keep the “-” pin at 2.5V as well.

Let’s start with everything in perfect balance, then “+”, “-“, and “OUT” will all be at +2.5V, and the capacitor will have no charge. Now let’s take the input to +5V:

  • a constant current starts flowing into the resistor, since the other side is +2.5V
  • this current drives the “-” pin up, so the output will go down
  • how far down? well as much as needed to cancel out that incoming current
  • IOW, the same current is going to go into the capacitor, which starts to charge up
  • as the charge builds up, less current starts to flow into the capacitor
  • “no way” says the op-amp, and so it pulls its output lower
  • so a constant current flows into the cap, and the output drops lower and lower
  • at some point near 0V the op-amp reaches its limit, and the mechanism breaks down

To give a water analogy: think of pouring water into a glass at a constant rate while trying to keep the surface of the water the same. You have to gradually lower the whole glass to make this work. As you do, the glass (cap) will contain more and more water (voltage). You’ve integrated (collected) the water flow into the glass!

If we keep the input voltage high, nothing more will happen: the cap will end up being fully charged, and the op-amp can no longer do anything to keep the “-” input from rising all the way to the input voltage.

When we now drop the input voltage to 0V, the reverse happens. Current flows through the resistor in the other direction, and the cap starts discharging. Again, the op-amp will do whatever it can to maintain that “-” at 1/2 Vcc. It does this by raising its output pin causing the capacitor to discharge with that same constant current rate. And sure enough, the voltage over the capacitor drops linearly. Until we hit the limits, and the process stops.

Here’s the effect in action, as seen on an oscilloscope (R = 4.7 kΩ, C = 0.1 µF, and the op-amp is an OPA2340):

SCR35

The blue line is the input signal, the yellow line is the triangular wave output. Neat, eh?

Tomorrow, I’ll add an op-amp as comparator to make this circuit oscillate all by itself.

Murphy goes opto

In Hardware on Oct 6, 2012 at 00:01

It looks like Mr. Murphy has found some time to mess with things again…

The Optocoupler Plug is a little board to let you isolate two I/O pins. When you drive one end with a small voltage to light up a little LED inside, it shines that light onto a sensitive photo-transistor which then starts conducting. It’s effectively a tiny switch, driven by light.

The nice thing about light is that it lets you avoid an electrical connection. So this thing allows you to detect a (small) input voltage without actually making a connection to it. These units support over a thousand volt of isolation – perfect when messing with AC mains coupled circuits, for example.

But the Optocoupler plug is actually two plugs in one, because you can also use it as an output by using it in reverse: then, the two I/O pins on a port can be used as output to turn on the LEDs, and the phototransistors can then control some output circuit without having to actually connect to it.

Here’s the “dual-mode” configuration of the Opto-coupler Plug:

Jumpers

In one mode, it can be used as input (with current limiting resistors on the right), in the other it’s an output (current-limiting resistors on the left, and solder jumpers on the right).

The unit I picked for this board is an MCT62:

Mct62

Then we recently switched over to a HCPL-2631, without thinking much about it:

Opto 2631

Whoops! Different pinout, but also entirely different beast: the MCT62 contains a pair of independent simple “analog” type optocouplers. The HPCL-2631 on the other hand is a “digital” model with some built-in amplification. Which means that this thing needs a power supply, and to stay within the 8-DIP pinout, this can only be realised by tying a common “ground” together and re-using the other as supply voltage.

There’s a lot more to describe about this apparently simple device, but for now all I can say is “we messed up!”. Fortunately, only four people have been affected by this so far, and we’ve contacted each one of them to resolve the problem and send a replacement out. To each of you, my apologies for the confusion and for wasting your time if you’ve been trying to get those faulty Optocoupler Plug kits to work.

With thanks to Jupe Software for reporting the issue, and saving others more grief!

Pssst … this is post # 1111 !

What’s the deal with Bencode?

In Software on Oct 5, 2012 at 00:01

These past few days, I’ve explored some of the ways we could use Bencode as data format “over the wire” in both very limited embedded µC contexts and in scripting languages.

Calling it “over the wire” is a bit more down-to-earth than calling Bencode a data serialisation format. Same thing really: a way to transform a possibly complex nested (but acylic) data structure into a sequence of bytes which can be sent (or stored!) and then decoded at a later date. The world is full of these things, because where there is communication, there is a need to get stuff across, and well… getting stuff across in the world of computers tends to happen byte-by-byte (or octet-by-octet, if you prefer).

When you transfer things as bytes, you have to delimit the different pieces somehow. The receiver needs to know where one “serialised” piece of information ends and the next starts.

There are three ways to send multi-byte chunks and keep track of those boundaries:

  • send a count, send the data, rinse and repeat
  • send the bytes, then add a special byte marker at the end
  • send the bytes and use some “out-of-band” mechanism to signal the other side

Each of them has major implications and trade-offs for how a transmission works. With counts, if there is any sort of error, we’re hosed – because we lose sync and no guaranteed way to ever recover from it.

With the second approach, we need to reserve some character code as end marker. That means it can’t appear inside the data. So then the world came up with escape sequences to work around this limitation. That’s why to enter a quote inside a string in C, you have to use a backslash: "this is a quoted \" inside a string" – and then you lose the backslash. It’s all solvable, of course… but messy.

The third approach uses a different trick: we send whatever we like, and then we use a separate means of communication to signal the end or some other state change. We could use two separate communication lines for example, sending data over one and control information over the other. Or close the socket when done, as with TCP/IP.

If you don’t get this stuff right, you can get into a lot of trouble. Like when in the 60’s, telephone companies used “in-band” tones on a telephone line to pass along routing or even billing information. Some clever guys got pretty famous for that – simply inserting a couple of tones into the conversation!

Closer to home, as noted recently – even the Arduino bootloader got bitten by this.

So how about Bencode, eh?

Well, I think it hits the sweet spot in tradeoffs. It’s more or less based on the second mechanism, using a few delimiters and special characters to signal the start and end of various types of data, while switching to a byte-counted prefix for the things that matter: strings with arbitrary content (hence including any bit pattern). And it sure helps that we often tend to know the sizes of our strings up front.

With Bencode, you don’t have to first build up the entire message in memory (or generate it twice) to find out how many bytes will be sent – as required if we had to use a size prefix. Yet the receiver also can prepare for all the bigger memory requirements, because strings are still prefixed with the number of bytes to come.

Also, having an 8-bit clean data path really offers a lot of convenience. Because any set of bytes can be pushed through without any processing. Like 32-bit or 64-bit floats, binaries, ZIP’s, MP3’s, video files – anything.

Another pretty clever little design choice is that neither string lengths nor signed integers are limited in size or magnitude in this protocol. They both use the natural decimal notation we all use every day. A bigger number is simply a matter of sending more digits. And if you want to send data in multiple pieces: send them as a list.

Lastly, this format has the property that if all you send is numerical and plain ASCII data, then the encoded string will also only consist of plain text. No binary codes or delimiters in sight, not even for the string sizes. That can be a big help when trying to debug things.

Yep – an elegant set of compromises and design choices indeed, this “Bencode” thing!

Bencode in Lua, Python, and Tcl

In Software, Linux on Oct 4, 2012 at 00:01

Ok, let’s get on with this Bencode thing. Here is how it can be used on Linux, assuming you have Lua, Python, or Tcl installed on your system.

Let me show you around in the wonderful land of rocks, eggs, and packages (which, if I had my way, would have been called rigs … but, hey, whatever).

Lua

Lua’s Bencode implementation by Moritz Wilhelmy is on GitHub. To install, do:

    luarocks install bencode

(actually, I had to use “sudo”, this appears to be solved in newer versions of LuaRocks)

LuaRocks – a clever play on words if you think about it – is Lua’s way of installing well-known packages. It can be installed on Debian using “sudo apt-get install luarocks”.

The package ends up as as a single file: /usr/local/share/lua/5.1/bencode.lua

Encoding example, using Lua interactively:

    > require 'bencode'
    > s = {1,2,'abc',{234},{a=1,b=2},321}
    > print(bencode.encode(s))
    li1ei2e3:abcli234eed1:ai1e1:bi2eei321ee

To try it out, we need a little extra utility code to show us the decoded data structure. I simply copied table_print() and to_string() from this page into the interpreter, and did:

    > t = 'li1ei2e3:abcli234eed1:ai1e1:bi2eei321ee'
    > print(to_string(bencode.decode(t)))
    "1"
    "2"
    "abc"
    {
      "234"
    }
    {
      a = "1"
      b = "2"
    }
    "321"

(hmmm… ints are shown as strings, I’ll assume that’s a flaw in to_string)

Python

Python’s Bencode implementation by Thomas Rampelberg is on PyPi. Install it as follows:

    PKGS=http://pypi.python.org/packages/2.7
    easy_install $PKGS/b/bencode/bencode-1.0-py2.7.egg

It ends up as “/usr/local/lib/python2.7/dist-packages/bencode-1.0-py2.7.egg” – this is a ZIP archive, since eggs can be used without unpacking.

This depends on easy_install. I installed it first, using this magic incantation:

    wget http://peak.telecommunity.com/dist/ez_setup.py
    python ez_setup.py

And it ended up in /usr/local/bin/ – the standard spot for user-installed code on Linux.

Now let’s encode the same thing in Python, interactively:

    >>> import bencode
    >>> bencode.bencode([1,2,'abc',[234],{'a':1,'b':2},321])                        
    'li1ei2e3:abcli234eed1:ai1e1:bi2eei321ee'

And now let’s decode that string back into a data structure:

    >>> bencode.bdecode('li1ei2e3:abcli234eed1:ai1e1:bi2eei321ee')                  
    [1, 2, 'abc', [234], {'a': 1, 'b': 2}, 321]

Note: for another particularly easy to read Python decoder, see Mathieu Weber‘s version.

Tcl

Tcl’s Bencode implementation by Andreas Kupries is called Bee and it part of Tcllib.

Tcllib is a Debian package, so it can be installed using “sudo apt-get install tcllib”.

Ok, so installation is trivial, but here we run into an important difference: Tcl’s data structures are not “intrinsically typed”. The type (and performance) depends on how you use the data, following Tcl’s “everything is a string” mantra.

Let’s start with decoding instead, because that’s very similar to the previous examples:

    % package require bee    
    0.1
    % bee::decode li1ei2e3:abcli234eed1:ai1e1:bi2eei321ee
    1 2 abc 234 {a 1 b 2} 321

Decoding works fine, but as you can see, type information vanishes in the Tcl context. We’ll need to explicitly construct the data structure with the types needed for Bencoding.

I’m going to use some Tcl trickery by first defining shorthand commands S, N, L, and D to abbreviate the calls, and then construct the data step by step as a nested set of calls:

    % foreach {x y} {S String N Number L ListArgs D DictArgs} {
        interp alias {} $x {} bee::encode$y
    }
    % L [N 1] [N 2] [S abc] [L [N 234]] [D a [N 1] b [N 2]] [N 321]
    li1ei2e3:abcli234eed1:ai1e1:bi2eei321ee

So we got what we wanted, but as you can see: not all the roads to Rome are the same!

Decoding Bencode data

In AVR, Software on Oct 3, 2012 at 00:01

After the encoding and collection of Bencoded data, all the pieces are finally in place to actually process that incoming data again.

I’ve extended the EmBencode library and the serialReceive demo sketch to illustrate the process:

Screen Shot 2012 09 30 at 15 15 53

Let’s try it with the same test data, i.e. the following text (all on one line):

    5:abcde3:123i12345eli987eli654eei321ee
    i999999999ed3:onei11e3:twoi22ee4:bye!

Here is the output:

Screen Shot 2012 09 30 at 15 18 03

You can see the pieces being separated, and all the different parts of the message being decoded properly. One convenient feature is that the asString() and asNumber() calls can be mixed at will. So strings that come in but are actually numeric can be decoded as numbers, and numbers can be extracted as strings. For binary data, you can get at the exact string length, even when there are 0-bytes in the data.

The library takes care of all string length manipulations, zero-byte termination (which is not in the encoded data!), and buffer management. The incoming data is in fact not copied literally to the buffer, but stored in a convenient way for subsequenct use. This is done in such a way that the amount of buffer space needed never exceeds the size of the incoming data.

The general usage guideline for this decoder is:

  • while data comes in, pass it to the decoder
  • when the decoder says it’s ready, switch to getting the data out:
    • call nextToken() to find out the type of the next data item
    • if it’s a string or a number, pull out that value as needed
  • the last token in the buffer will always be T_END
  • reset the decoder to prepare for the next incoming data

Note that dict and list nesting is decoded, but there is no processing of these lists – you merely get told where a dict or list starts, and where it ends, and this can happen in a nested fashion.

Is it primitive? Yes, quite… but it works!

But the RAM overhead is just the buffer and a few extra bytes (8 right now). And the code is also no more than 2..3 KB. So this should still fit fairly comfortably in an 8-bit ATmega (or perhaps even ATtiny).

Note that the maximum buffer size (hence packet size) is about 250 bytes. But that’s just this implementation – Bencoded data has no limitations on string length, or even numeric accuracy. That’s right: none, you could use Bencode for terabytes if you wanted to.

PS – It’s worth pointing out that this decoder is driven as a push process, but that actual extraction of the different items is a pull mechanism: once a packet has been received, you can call nextToken() as needed without blocking. Hence the for loop.

Shop news

In News on Oct 2, 2012 at 00:01

How does this sequence continue?

    GG TW TK SG .. ..

Answer:

    MT CY

Get it? Think country codes…

And those last two (Malta & Cyprus) bring the total to 50 countries where JeeLabs Shop has delivered products. Woohoo! Thank you John & Andreas for reaching that milestone – welcome to the expanding community of JeeLabs experimenters!

(and thank you Martyn, for spotting this factoid)

Now that we’re on this topic anyway: the shop has just been upgraded to accommodate the new 21% VAT tax requirements in the Netherlands, effective as of October 1st. Death and taxes, as they say… the two inevitabilities in life!

After two years of total stability, we’ve had to raise the price on some items, to match costs and keep those ever-changing US$ exchange rates in check, but the good news is that all the major items such as JeeNodes, RBBB’s, and USB BUBs remain at the same level as before. This is where (modest) economies of scale start to kick in, so we’re not changing one bit of what has been working out so well.

Another point I’d like to single out here, is that Martyn and Rohan Judd have been doing a truly phenomenal job of steering a major shop transition over the summer into a pretty smoothly-running operation again – as it’s now all done from the UK. With a tip-of-the-hat to you fella’s, for making this happen in the best imaginable way – cheers!

There is more change cooking, but for the moment the only change you will see is that the shop’s email is now coming from order_assistance at jeelabs dot org. This address is from now on also the best place to get in touch, to make sure no message goes by unseen. Whether it is an inquiry about packages, questions about delivery times or problems with the supplied goods, we’ll take care of it. Yes, including me – I’m not going anywhere :-)

Oh, and then there’s that second disruption I mentioned a few days ago: the Café + Wiki at https://jeelabs.net/ have now been replaced by the new Redmine 2 system. The original wiki and doc pages can still be reached at “oldred.jeelabs.net”. For now.

Note that the forum is not affected. It’ll be switched over to Redmine later this year.

Collecting Bencode data

In AVR, Software on Oct 1, 2012 at 00:01

After having constructed some Bencode-formatted data, let’s try and read this stuff back.

This is considerably more involved. We’re not just collecting incoming bytes, we’re trying to make sense of it, figuring out where the different strings, ints, lists, and dicts begin and end, and – not obvious either perhaps – we’re trying to manage the memory involved in a maximally frugal way. RAM is a scarce resource on a little embedded µC, so pointers and malloc’s have to be avoided where possible.

Let’s start with the essential parsing task of figuring out when an incoming byte stream contains a complete Bencoded data structure. IOW, I want to keep on reading bytes until I have a complete “packet”.

The following serialReceive sketch does just that:

Screen Shot 2012 09 30 at 11 46 31

All of the heavy-lifting is done in the EmBencode library. What we’re doing here is giving incoming bytes to the decoder, until it tells us that we have a complete packet. Here’s the output, using the test data created yesterday:

Screen Shot 2012 09 30 at 11 46 07

Looks obvious, but this thing has fully “understood” the incoming bytes to the point that it knows when the end of each chunk has been reached. Note that in the case of a nested list, all the nesting is included as part of one chunk.

There’s more to this than meets the eye.

First of all, this is a “push” scanner (aka “lexer”). Think about how you’d decode such a data stream. By far the simplest would be something like this (in pseudo code):

  • look at the next character
  • if it’s a digit:
    • get the length, the “:”, and then grab the actual string data
  • if it’s an “i”:
    • convert the following chars to a signed long and drop the final “e”
  • if it’s a “d” or an “l”:
    • recursively grab entries, until we reach an “e” at the end of this dict / list

But that assumes you’re in control – you’re asking for more data, IOW you’re waiting until that data comes in.

What we want though, is to treat this processing as a background task. We don’t know when data comes in and maybe we’d like to do other stuff or go into low-power mode in the meantime. So instead of the scanner “pulling” in data, we need to turn its processing loop inside out, giving it data when there is some, and having it tell us when it’s done.

The code for this is in EmBencode.cpp:

Screen Shot 2012 09 30 at 12 01 52

It’s written in (dense) C++ and implemented as a finite state machine (FSM). This means that we switch between a bunch of “states” as scanning progresses. That state is saved between calls, so that we’ll know what to do with the next character when it comes in.

There’s a fair amount of logic in the above code, but it’s a useful technique to have in your programming toolset, so I’d recommend going through this if FSM’s are new to you. It’s mostly C really, if you keep in mind that all the variables not declared locally in this code are part of a C++ object and will be saved from call to call. The “EMB_*” names are just arbitrary (and unique) constants. See if you can follow how this code flows as each character comes in.

The above code needs 7 bytes of RAM, plus the buffer used to store the incoming bytes.

There are tools such as re2c which can generate similar code for you, given a “token grammar”, but in simple cases such as this one, being able to wrap your mind around such logic is useful. Especially for embedded software on limited 8-bit microcontrollers, where we often don’t have the luxury of an RTOS with multiple “tasks” operating in parallel.

Halfway there: we’ve identified the pieces, but we’re not yet doing anything with ’em!

PS – The new JeeLabs Café website is now alive and kicking.