Computing stuff tied to the physical world

Archive for September 2011

Relational data

In Software on Sep 30, 2011 at 00:01

This post strays solidly into the realms of software and database architecture…

I’m having a lot of fun with JeeMon, but it’s also starting to look like a bone which is slightly too large to chew on…

As in any application, there is the issue of how to manage, represent, and store data structures. This is an area I’ve spent quite a few years on in a previous life. The nice bit is that for JeeMon I get to pick anything I like – there are no external requirements. Time will tell whether that freedom won’t become a curse.

So now I’m exploring the world of pure relational data structures again:

Screen Shot 2011 09 29 at 14 28 25

What you’re looking at is the definition of 4 drivers (I’m leaving off a bunch more to keep this example limited). This was rendered as nested HTML tables, btw – very convenient.

Each driver has a description of the type of values it can produce, and each set of values is grouped (as in the case of ookRelay2, which can generate data for three different types of packets it is relaying). Drivers can support more than one interface (the roomNode driver supports both remote packets and direct serial connection, for example). And drivers can expose a number of functions (in this case they all only expose a “decode” function).

In the “flat” SQL world, you’d need 5 tables to represent this sort of information. But since I’m supporting relations-as-values, it can all be represented as a single relation in JeeMon (I call ‘em “views”, btw).

Note that this isn’t a hierarchical- or network-database. It’s still purely relational. There’s an “ungroup” operator to flatten this structure as many levels as needed, along with the usual “select”, “project”, “join”, etc. operators.

There’s also something else going on. Some of the information shown above is virtual, in the sense that the information is never actually stored into a table but gets extracted on-demand from the running application. The “values” relation, for example, is produced by scanning some simple lists stored (as dicts) in a Tcl variable:

Screen Shot 2011 09 29 at 15 12 46

And the “functions” relation is created on-the-fly by introspecting the code to find out which public functions have been exposed in each driver.

It’s a bit unusual to think of application data in relational terms. This lets you “join” a list with another one, and do strange things such as perform selection and projection on the list of all public functions defined in the application. But it’s also extremely powerful: one single conceptual mechanism to deal with information, whether present in application variables, i.e. run-time state, from a traditional database, or as the result of a remote request of some kind. Here’s another example: perform a join between a local (scalar) variable and a dict / map / hash (or whatever you want to call associative arrays), and you end up doing key lookup.

This approach turns (at least part of) an application on its head: you don’t need to register every fact that’s relevant into a database, you launch the app and it sets up “views” which will extract what they need when they need it, from the running app itself. Don’t call us, we’ll call you!

It might seem like a complicated way to do things (indeed, it is!), but there’s a reason for this madness. I’m looking for a design which can adapt to changes – even in the code – while running. If a driver is updated to add support for Ethernet, say, then it should reflect in these data structures without further explicit effort from the programmer. I’m not there yet by a long stretch, but this sort of experimentation is needed to get there. IMO, the magic lies in not copying facts but in seeking them out when needed, combined with smart caching.

My research project which pioneered this approach is called “Vlerq”, and I’m considering resuscitating it for use in JeeMon. Trouble is, it’s quite a bit too ambitious to implement fully (I’ve hit that wall before) – so my current experimentation is aimed at finding a workable path for the near future. Perfection is the enemy of done – and right now I’m seeing my “done” horizon racing further and further away. It’s like trying to exceed the speed of light – oh, wait, didn’t someone do that the other day?

Noise spectrum

In Software on Sep 29, 2011 at 00:01

Not only is there a glcdScope sketch which is useful for displaying voltage across the 0.1 Ω shunt in my 220V experiments, there’s also a little Spectrum Analyzer sketch to try out. Yummie!

Spectrum analysis tells you what frequencies are present in a given sample of a periodic signal. Lots of math there – just Google around for “Fast Fourier Transform” (FFT) and you’ll get it all on your plate if you’re curious.

But this stuff is a bit harder than a plain signal. Here’s what I’m seeing, with just a short wire connected, picking up some 50 Hz presumably (as with the scope test) – again with the “digital phosphor” persistence:

DSC_2655.jpg

In principle, this graph is ok: lots of signal at lower frequencies and progressively less at higher frequencies. After all, with a perfect 50 Hz sine wave and no noise, we’d expect a single peak at the start of the scale.

The repeated peaks every few pixels also look promising. With a bit of luck they are in fact the harmonics, i.e. the 100 Hz, 150 Hz, … multiples – which is what you get when a signal is repetitive but not exactly a sine wave. Harmonics are what makes music special – the way to distinguish a note played on the violin and on the piano.

But I was hoping for something else: a bit more peaks at the right hand side of the graph. This would indicate that there are high frequencies in the signal, the computer’s switching power supply close to this setup, for example.

And worse: I’m seeing a completely flat line when hooking this up to the 220V current shunt. Looks like this signal is too weak to play FFT games with (should the data be auto-scaled?).

Anyway, here’s the glcdSpectrum50 sketch:

Screen Shot 2011-09-27 at 19.04.17.png

It relies on the fix_fft.cpp file by Tom Roberts which does the FFT heavy-lifting (original is here).

Also, note that I’m using a slightly different algorithm this time to determine the average signal value: the average of the last 256 samples is used to compute the center value subtracted from the next 256 samples. The outcome should be similar, as long as the signal is indeed symmetric around this value.

All in all a nice try, but it didn’t really provide much new insight (yet?).

It just blew apart

In Hardware on Sep 28, 2011 at 00:01

Got quite a scare the other day:

DSC_2649.jpg

One moment I was cheerily plugging in and unplugging my 100 W light bulb for testing current measurements, and the next it came apart with a huge bang. No fuse tripped anywhere (as it would have in UK households).

It’s a good reminder of the amount of energy an AC power outlet can discharge at any time. In fact, the standard rating for individually-fused 220 V power groups is 16 Amps around here. That’s roughly 3600 W.

To get an idea of what that means – one Horsepower is the sustained power a horse can generate, which is about 740 Watt. So there are almost 5 horses in each power outlet, waiting to charge at you!

If you push that much energy into a liter of water, it’ll boil in 2 minutes flat. Doesn’t sound like much, eh? How about 1000 Amps at 3.3V? You’ll need copper wire with a diameter of 18 mm to handle that much current.

Compare that to a JeeNode in sleep mode drawing 10 µA, i.e. 33 microwatt. That’s 8 orders of magnitude less.

Anyway, I’ve gained some extra respect for 220V mains circuits.

Now, if only I could find a replacement … 100 W incandescent light bulbs are no longer sold in Europe.

GLCD scope on 220V

In Hardware, Software on Sep 27, 2011 at 00:01

I’m not willing to hook my Mac up to 220V through USB, whether through the DSO-2090 USB scope I have, or any other way – even if it’s tied to a galvanically isolated setup such as the recent current measuring setups. One mistake and it’d go up in smoke – I’d rather focus all my attention on keeping myself safe while fooling around with this AC mains stuff.

But there’s this little 100 KHz digital sampling scope sketch, based on the Graphics Board. Looks like it could be a great setup for this context, since it can run detached off a single AA battery.

It took some hacking on the original sketch to get a system which more or less syncs to the power-line frequency (well, just the internal clock, but that turns out to be good enough). Here’s what it shows with the input floating:

DSC_2639.jpg

Definitely usable. The three different super-imposed waves are most likely an artifact of the scanning, which takes place every 60 ms. One huge improvement for this type of repetitive readout is “digital phosphor”, like the professional DSO’s do: leaving multiple traces on the screen to intensify the readout. Single traces on a GLCD like this one end up with very little contrast, due to the lag of liquid crystals. What I do now, is leave pixels on for 10 scans before clearing the display. It’s not quite digital phosphor (where each scan fades independently), but it’s pretty effective as you can see. And this setup is a tad cheaper than that Agilent 3000X MSO I dream of…

Here’s a readout with this setup tied to the 0.1 Ω shunt in the AC mains line, with the power off:

DSC_2643.jpg

That’s three pixels of noise, roughly, i.e. some 10 mV.

With a 60 W light bulb turned on, we get this:

DSC_2641.jpg

Not bad at all! It looks like with a bit of smoothing and averaging, one could turn this into an ON / OFF signal.

Alas, the sensivity does leave to be desired. With a 25 W light bulb:

DSC_2644.jpg

That’s barely above the noise threshold. It might be difficult to obtain a reliable detection from this, let alone at lower power levels. The 1W power brick showed almost no signal, for example.

Note that with North-America’s 110V, the readout would be twice as sensitive, since it’s measuring current.

Still, these results look promising. Here is the <cough> DSO with digital phosphor </cough> sketch I used:

Screen Shot 2011-09-26 at 15.52.38.png

This code can be found as “glcdScope50″ example in GLCDlib on GitHub.

Fun stuff. Just look how simple it is to gain new insight: a few chips, a few lines of code – that’s all it takes!

Package management done right

In Software on Sep 26, 2011 at 00:01

Yeah, ok, this is probably a bit off topic…

I’m talking about installing Unix’y software on the Mac (which has a big fat GUI on top of a Posix-compliant OS).

There have been several projects in the past which have addressed the need to use a Mac as a Unix command-line system with all the packages coming out of the Linux, FreeBSD, etc. universes. The places where “tar”, “patch”, “make”, “autoconf”, “libtool”, and so on rule.

First there was Fink – implemented in Perl:

The Fink project wants to bring the full world of Unix Open Source software to Darwin and Mac OS X. We modify Unix software so that it compiles and runs on Mac OS X (“port” it) and make it available for download as a coherent distribution. Fink uses Debian tools like dpkg and apt-get to provide powerful binary package management. You can choose whether you want to download precompiled binary packages or build everything from source.

Then came MacPorts – implemented in Tcl:

The MacPorts Project is an open-source community initiative to design an easy-to-use system for compiling, installing, and upgrading either command-line, X11 or Aqua based open-source software on the Mac OS X operating system. To that end we provide the command-line driven MacPorts software package under a BSD License, and through it easy access to thousands of ports that greatly simplify the task of compiling and installing open-source software on your Mac.

And now there’s Homebrew – implemented in Ruby:

Homebrew is the easiest and most flexible way to install the UNIX tools Apple didn’t include with OS X. Packages are installed into their own isolated prefixes and then symlinked into /usr/local.

It’s interesting to note that all systems were written in a scripting language. Which makes perfect sense, given that they offer a huge jump in programmer productivity and no downsides to speak of in this context.

As for my pick: Homebrew gets it right.

Homebrew (i.e. the “brew” command) is installed with a one-liner:

    /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

This is a big deal, because the last thing you want is to use a package manager to overcome installation hassles, only to end up with… an extra step which introduces its own installation hassles!

The actual design of that one-liner is quite clever: it points to a script in a secure area, with full history of all changes made to that script, so you can make sure it won’t do weird (or bad) things with your system.

Homebrew doesn’t introduce new conventions (like /sw/ or /opt/, which need to be added to your exe search path). It’ll simply install the packages you ask for in /usr/local/bin etc (which means you don’t need to sudo all the time). And it does it intelligently, because it actually installs in /usr/local/Cellar and then puts symlinks in /usr/local/*. Which means there’s a way out of an installation, and there’s even a (drastic) way to get rid of all Homebrew-installed packages again: delete Cellar, and remove the dangling symlinks from /usr/local/*. Not that you’d ever need to – Homebrew supports uninstalls.

It’s succinct (and colorized):

Screen Shot 2011 09 24 at 0 22 56

No endless lists of compiler commands on my screen, telling me absolutely nothing, other than “CPU == busy!”.

It does the right thing, and works out all the dependencies:

    fairie:~ jcw$ brew search rrd
    rrdtool
    fairie:~ jcw$ brew install rrdtool
    ==> Installing rrdtool dependency: gettext
    ==> Downloading http://ftpmirror.gnu.org/gettext/gettext-0.18.1.1.tar.gz
    ######################################################################## 100.0%
    ==> Downloading patches
    ...
    ==> Summary
    /usr/local/Cellar/gettext/0.18.1.1: 368 files, 13M, built in 5.4 minutes
    ==> Installing rrdtool dependency: libiconv
    ==> Downloading http://ftpmirror.gnu.org/libiconv/libiconv-1.14.tar.gz
    ...
    ==> Summary
    /usr/local/Cellar/libiconv/1.14: 24 files, 1.4M, built in 70 seconds
    ==> Installing rrdtool dependency: glib
    ... etc, etc, etc ...
    ==> Installing rrdtool
    ==> Downloading http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.5.tar.gz
    ######################################################################## 100.0%
    ==> Patching
    patching file configure
    Hunk #1 succeeded at 31757 (offset 94 lines).
    Warning: Using system Ruby. RRD module will be installed to /Library/Ruby/...
    Warning: Using system Perl. RRD module will be installed to /Library/Perl/...
    ==> ./configure --prefix=/usr/local/Cellar/rrdtool/1.4.5 --mandir=/usr/local/Cel
    ==> make install
    /usr/local/Cellar/rrdtool/1.4.5: 148 files, 3.2M, built in 68 seconds
    fairie:~ jcw$ 

It does its work with “formulas”, i.e. Ruby scripts which describe how to fetch, build, and install a certain package. The formulas (formulae?) are managed on GitHub, which means that there is a massive level of collaboration going on: new packages, issue tracking, fixes, and discussion. Here’s what I got when self-updating it again after having done so one or two days ago:

    $ brew update
    remote: Counting objects: 294, done.
    remote: Compressing objects: 100% (103/103), done.
    remote: Total 250 (delta 174), reused 212 (delta 143)
    Receiving objects: 100% (250/250), 39.46 KiB, done.
    Resolving deltas: 100% (174/174), completed with 32 local objects.
    From http://github.com/mxcl/homebrew
       36f4400..99bc0b7  master     -> origin/master
    Updated Homebrew from 36f4400e to 99bc0b79.
    ==> New formulae
    apktool             hexedit             p11-kit             solid
    denyhosts           jbigkit             qi                  tinyfugue
    gearman-php         kbtin               shen
    graylog2-server     opencc              sisc-scheme
    ==> Updated formulae
    android-ndk         ffmpeg              libquicktime        python
    aqbanking           frink               libraw              python3
    audiofile           fuse4x              libvirt             sleepwatcher
    cassandra           fuse4x-kext         nasm                tomcat
    class-dump          gflags              nginx               transcode
    csshx               google-sparsehash   nss                 tsung
    dash                gpsbabel            pdfjam              xml-security-c
    dvtm                gwenhywfar          pixman*
    elasticsearch       libiconv*           pos
    $

But best of all IMO, the formulas use an almost declarative style. Here’s the one for re2c I just installed:

Screen Shot 2011 09 24 at 0 33 33

It’s not hard to guess what this script does. No wonder that so many people submit and tweak such formulas.

Note that there’s neither a description, nor explicit version handling in there (other than what can be gleaned from the download URL). Brew is an installer, not a catalog. Wanna know more? Visit the home pages. Brilliant.

It’s great to see the Ruby community adopt a concise, declarative, and Domain Specific Language style. Nothing new or revolutionary – has been done many times before – but nevertheless underappreciated, IMNSHO.

It’s all about dialogue

In Musings on Sep 25, 2011 at 00:01

Don’t know about you, but I’m having a great time with this weblog!

I’d like to go on a small excursion of what it’s all about, why it matters, and unpredictable stuff, such as the future.

This weblog started roughly three years ago. I love tinkering with technology, I love learning more about it, I love making new things (even if it’s only new for myself). Especially when it’s about mixing software, hardware, and mechanical stuff. I describe myself as an architect, a hacker, and a maker, and I’m proud of it. And I decided to write about it. One day I didn’t, the next day I did – it’s really that easy. You could start doing it too, any day.

A weblog is a publishing medium. Push. From me to you – whoever you are and wherever you are. As long as I enjoy writing it, and as long as you enjoy reading it, we both win.

One crucial aspect of this process is that we need to share the same interests. If I tried to write about culture, nature, politics, or music, chances are that we’d no longer be in sync (and I might have very little interesting to report!). We all differ, we all embody our own unique mix of interests, opinions, and experiences, and there’s no reason whatsoever to assume that a shared interest in technology means we share anything else. The great thing is: it doesn’t matter. We are linked by our humanity, and our diversity is our greatest asset. Vive la différence!

So how does this weblog thing work? Well, from what I’m writing about and have written in the past, you can tell where my passion lies. And you have the simple choice of reading and following the posts on this weblog – or not. From your comments and emails, I think I get an idea who (some of) you are. We’re in sync.

This process excites me. Because it transcends culture, age, background, and all those other aspects in which we differ (and don’t even know about each other). We can share our interests, learn from each other, exchange tips and ideas, and all it takes is an internet connection and the ability to read and write in English, even if that’s not everyone’s native language.

But weblogs publishing is an asymmetric process – there’s no real dialogue going on at all. I don’t really know who reads this. There might be thousands of readers coming back every day, or there might be just those who post comments – I wouldn’t know. I used to care about that, but I no longer do. I don’t collect stats and I don’t “track” visitors. It’s just another distraction and life’s too short. But more important to me, is motivation: my goal is not to have an “important” blog, a big readership, or lots of fans. Nor a big shop or many customers, for that matter. My goal is to have fun with technology, learn as much as I can, invent new stuff, and share to inspire others to do the same. It took me a long summer break to figure this out.

Of course I have my preferences, and of course there are areas where I know more and less about. The field is way too large to dive into every topic, let alone build up expertise in each – although I do consider myself reasonably open minded and knowledgeable about a decent range of technical domains. And those gaps? Well, that’s the challenge, of course: filling one little gap each day – day in, day out!

So what does this mean for the future?

I see no reason why any of this should stop. It’s proven to be sustainable for me, and there’s plenty of material to go into and talk about to last a lifetime. As you may have noticed, I’m moving away from a pure hardware focus in this weblog. The central theme will definitely remain “Physical Computing in and around the house”, but there’s more to it than the ATmega + RFM12B that form a JeeNode, and I’d like to explore a wider range of topics, including software and data processing, and probably also mechanical aspects (construction, CNC, 3DP, bots):

Topics

I do have a little request I’d like to make: whenever you read a post on this weblog and have a suggestion or insight which is relevant, please consider adding a comment. I tend to go with the flow (of ideas), and I tend to pick the easy low-hanging fruit first. Suggestions made in recent days on all this scary 220V power measurement stuff have helped me greatly to better understand what’s going on and to come up with more experiments to set up to try and figure it all out. I encourage you to point me in the right direction and to point out mistakes.

Who knows, it might lead to a post which is more useful to you. We’ll all benefit!

Hacking around in software

In Software on Sep 24, 2011 at 00:01

Here’s a web page I’ve been designing recently – one of many planned for the JeeMon system I’m working on:

Screen Shot 2011 09 20 at 13 00 26

A couple of notes:

  • there are three “master” tables on this page: Devices, Interfaces, and Drivers
  • devices are a 1:N mapping between drivers and interfaces
  • … except that some devices are remote, and not tied to a local interface
  • clicking on any row of these three tables displays details in the rightmost column
  • drivers are listed but not managed on this page, that’s a software admin task

I’m describing this here for a number of reasons (other than showing off what I’ve been spending my days on).

First of all, you can see that this more or less covers what needs to be set up to in a home monitoring and/or home automation system. I like to see everything at a glance, i.e. on a single page and with as little navigation as possible. With this setup, I hope to keep it all essentially on one page. Whereby the right-side column may vary widely, depending on the device / interface / driver.

But it doesn’t really need to be about home automation at all. A very similar setup could be used to hook up to devices which I’m experimenting with in the lab, or devices which act like measuring instruments, or control some aspect of an experiment. Anything related to Physical Computing, really.

The other thing that interests me is the “degree of variety” needed to cover all the cases. Many devices will simply collect or send out one or more values, but not all devices are like that.

The RF12demo sketch is essentially a gateway from and to an entire group of RFM12B-based nodes. Each node can have its own driver. The ookRelay2 sketch is similar: it collects data from quite different devices, each of them sending out their specific packets with their specific types of measurements. It differs from RF12demo, in that it contains decoders for all the devices it knows about. There are no separate KS300 drivers, etc (although perhaps there should be, since some code is currently duplicated in the CUL driver).

The autoSketch driver is yet another beast. It listens on a serial interface for a line of the form “[...]” and when it sees one, tries to load a driver with the name in brackets and hand over control to it. This is the reason why just about all my sketches start off with the following boilerplate code:

    void setup() {
        Serial.begin(57600);
        Serial.println("\n[mySketch]");
        ...
    }

When associated with the autoSketch driver, this will automatically load a “mySketch” driver, if present. Which in turn means that all you have to do is plug in the JeeNode (JeeLink, Arduino, etc) and JeeMon will automatically detect it and start communicating with it. IOW, plug-and-play – with a simple implementation.

This is why there’s a “Use autoSketch for new interfaces” checkbox. I could have called it “Enable plug & play”.

But although this web page is functional in my test setup, it’s far from finished. The hardest part I want to get right, is to make the page completely dynamic. Right now, a click on a row will update the details via Ajax, but nothing will happen when there is no user interaction. What I want is that the page automatically adjusts when a JeeLink is plugged in (on the web server side). What I really want, is to generalize that mechanism to everything shown on any web page. The challenge is to do this without tedious periodic polling or complete table refreshes, just “pushing” changes in an event-driven manner. Events on the server side, that is.

The visual layout, styling, and behavior of a page like this has become very simple with today’s tools:

  • jQuery is the core JavaScript library
  • jQuery UI is used as basic style engine (the tabbed group on the right, for example)
  • jQuery TableSorter adds support for (surprise!) table sorting via their headers
  • Knockout is used to manage a lot of the user-facing behavior in the browser
  • 960 grid regulates the page structure flow in CSS
  • there is a small amount of custom CSS and custom Javascript (about 20 lines each)
  • the entire page is implemented as a single Tcl source file (175 lines so far)

There are a few trivial utility functions in there, which are bound to migrate to a more general module once I better understand the trade-offs and benefits of re-use. And the rest is gobbledygook which looks like this:

Screen Shot 2011 09 23 at 22 19 41

This is the notation I came up with and described in my Playing with indentation post a short while back. This particular code describes the HTML text returned for drivers in the “details” column, via Ajax.

This notation gets expanded to HTML, but the real value of it is that describing the HTML on a page is fun, because it’s now trivial to insert grouping levels (i.e. mostly <div>’s) and rearrange stuff until it’s just right.

Shielding!

In Hardware on Sep 23, 2011 at 00:01

Is it time for a tin foil hat?

Well, a recent comment mentioned common-mode voltage swings, and I think there’s a point there. With some resistance and impedance between the point I’m measuring and the power source downstairs, it’s easy to see how the whole thing swings widely in its “common mode”. Even the slightest leakage path to a non-swinging voltage potential could introduce a very substantial voltage difference.

Here’s a new test (it’s not the time to make mistakes!):

DSC 2635

What this does is place a ground plane underneath the whole circuit, sitting on (my all-time favorite) foam board. The foil is only attached to the JeeNode’s ground, nothing else.

And lo and behold – it really makes a difference!

The signal is 4 mV with nothing connected, and 7 mV with the power cable plugged in (but the switch still off). Turning the switch on has various effects, depending on whether the 100 W light bulb is plugged in.

With the light bulb included, the voltages I see are 80 .. 120 mV. Without, the voltages are 150 .. 400 mV. Still large swings, but nowhere near the consistent 650 .. 700 mV I was seeing before.

But what was more surprising (well… not in hindsight), is that these measurements depend on which way the power plug is plugged in. These old plugs are not polarized, so it’s easy to turn them around. It lets me put the measurement circuit in series with either the “live” or the “neutral” wire this way.

What I really like is that I’m back to 4 .. 7 mV readings when nothing is hooked up. That’s just one or two least significant bits, since the ATmega’s 10-bit ADC measures with ≈ 3 mV step sizes. It’s impossible to expect any better. This is also essential to get a decent shot at detecting low power levels.

Hm… this outcome is very awkward. A fully conducting enclosure might be best, but it’s also the most tricky one to deal with. This is after all still a dangerous 220 V hookup. A metal box inside a plastic box, perhaps? Yuck.

A shielded cable might help. Electrical signal amplification would introduce more (active i.e. power-consuming) complexity. Still, an op-amp with a super diode would allow using a capacitor to do the averaging.

One piece of the puzzle

In Software on Sep 22, 2011 at 00:01

The measurement anomalies of the recent experiments are intriguing. I don’t want to just tweak things (well, up to a point) – I also want to explain what’s happening! Several insights came through the comments (keep ‘em coming!).

Let me summarize the measurement algorithm: I measure the < 1 VAC peak-to-peak voltage ≈ 5000 times per second, and keep track of a slow-moving average (by smoothing with a factor of 1/500-th). Then I calculate the arithmetic mean of the absolute difference between each measured value and that average. Results are reported exactly once every second.

Some notes:

  • noise should cancel out for both the average and the mean values, when the signal is large
  • noise will not cancel out if it is larger than the fluctuation, due to the absolute function

There’s also another problem:

JC s Doodles page 15

I’m not synchronizing the averaging period to the power-line frequency, nor taking zero crossings into account.

This matters, and would explain up to some 2% variation in the measurement. Here’s why:

  • each 1-second sampling period measures about 50 cycles
  • let’s assume it’s 49-cycles-and-a bit
  • the extra “bit” will be residuals at both ends
  • the 49 cycles will be fine, averaged from zero-crossing to zero-crossing
  • but the ends may “average” to anything between 0 and 100% of the true average for entire cycles
  • so 1 of the 50 cycles may be completely off, i.e. up to 2% error (always too low)

So it looks like there are two measurement issues to deal with: noise and the lack of zero-crossing syncing.

It doesn’t quite explain the extreme values I’ve been seeing so far, but for now I’ll assume that an excessive amount of noise is being picked up in my crude breadboarded circuit and all the little dangling wires.

Shielding, digital noise filtering, and syncing to zero-crossings… quite a few things to try out!

A simpler direct connection

In Hardware on Sep 21, 2011 at 00:01

Now that direct connection to AC mains is no longer out of the question (at least to establish a baseline), I decided to create a much simpler hookup:

DSC 2633

That’s a 0.1 Ω resistor in series with the load and two diodes, all in parallel. The diodes are not that useful for load protection (anything over 1000 W will be a problem), but they may help with transients and spikes:

DSC 2633  Version 2

Note that the sense wires are attached to the shunt resistor. If anything were to get yanked loose by accident, then it would be those wires, not the shunt itself. As usual, this is all hooked up via an isolation transformer.

A quick check tells me that this setup does not perform any better than the previous direct connection with the sketch I’ve been using all along.

Even without anything plugged in, I can affect the readings by moving my hands close to the circuit:

    reading:RF12-868.5.17:oneLong:value = 28250
    reading:RF12-868.5.17:oneLong:value = 28247
    reading:RF12-868.5.17:oneLong:value = 28184
    reading:RF12-868.5.17:oneLong:value = 28637
    reading:RF12-868.5.17:oneLong:value = 33743
    reading:RF12-868.5.17:oneLong:value = 77924
    reading:RF12-868.5.17:oneLong:value = 92240
    reading:RF12-868.5.17:oneLong:value = 50233
    reading:RF12-868.5.17:oneLong:value = 30663
    reading:RF12-868.5.17:oneLong:value = 28476
    reading:RF12-868.5.17:oneLong:value = 28176
    reading:RF12-868.5.17:oneLong:value = 28145
    reading:RF12-868.5.17:oneLong:value = 29082
    reading:RF12-868.5.17:oneLong:value = 34851
    reading:RF12-868.5.17:oneLong:value = 43977
    reading:RF12-868.5.17:oneLong:value = 54547
    reading:RF12-868.5.17:oneLong:value = 62388
    reading:RF12-868.5.17:oneLong:value = 61987
    reading:RF12-868.5.17:oneLong:value = 61366
    reading:RF12-868.5.17:oneLong:value = 62199

So the next step will be to add filtering. Either analog or digital, we’ll see!

Direct connection summary

In Hardware on Sep 20, 2011 at 00:01

The different circuits described in the past few days all had problems (the first ACS Hall-effect hookup worked reasonably well, but was not sensitive enough for my purposes).

Here’s the circuit as it is now:

DSC 2631

Bottom side:

DSC 2632

(a slightly different layout would have avoided the overhang, had I known the complete setup in advance)

There are 3 independent circuits on there connected in series (so the same current passes through each of ‘em):

JC s Doodles page 14

As you can see in the first image, there’s a little screw-less terminal block on the board. It lets me short out any combination of these setups – this was useful until everything had been built up, and also lets me rule out interference while focusing on a specific setup. Note also that this is an epoxy-based board, not the pressed-cardboard type which breaks easily and isolates less.

Let me just repeat that messing with AC mains voltages like this can be extremely dangerous. This circuit has all the wires completely exposed, and that’s intentional: I prefer to be reminded of the risks, instead of getting a false sense of security because it all looks ok! One way to deal with these risks is to completely stand back from the entire hookup and only transfer measurement data over wireless whenever applying power to this stuff.

Tomorrow I’ll describe a simpler setup to get to the bottom of the measurement anomalies I’ve been seeing so far. With many thanks for all your comments – there’s clearly a lot more to it than with simple logic level signals!

Direct 220V hookup

In Hardware on Sep 19, 2011 at 00:01

This next hookup is the most tricky one – measuring current draw via a direct connection to AC mains:

Direct connect

No more transformers, no more galvanic isolation. The 0.1 Ω resistor in series with the appliance generates 0.1 VAC of output for each 1A passed through it. The resistor handles up to 2W, which means it’ll be ok up to about 1000 W: the current will be 1000 / 220 ≈ 4.5 A, and the power generated as heat will be 0.45 V x 4.5 A ≈ 2 W.

Not that I intend to go that far. My main load test is still a 100 W light bulb.

The rest of the circuit is the same as before: one side connected to 1/2 VCC, the other end connected to an AIO pin. I’m using the same test sketch as before, sending out one reading per second.

So how does it perform? Not good, so far. Here’s the sketch running with no AC connected:

    reading:RF12-868.5.17:oneLong:value = 18070
    reading:RF12-868.5.17:oneLong:value = 14045
    reading:RF12-868.5.17:oneLong:value = 14117
    reading:RF12-868.5.17:oneLong:value = 13375
    reading:RF12-868.5.17:oneLong:value = 13359
    reading:RF12-868.5.17:oneLong:value = 38606
    reading:RF12-868.5.17:oneLong:value = 17196

That’s 13 .. 39 mV, and it’s all over the map. These readings are with no load but AC connected:

    reading:RF12-868.5.17:oneLong:value = 625570
    reading:RF12-868.5.17:oneLong:value = 631161
    reading:RF12-868.5.17:oneLong:value = 632028
    reading:RF12-868.5.17:oneLong:value = 632705
    reading:RF12-868.5.17:oneLong:value = 631796
    reading:RF12-868.5.17:oneLong:value = 632296
    reading:RF12-868.5.17:oneLong:value = 631882

Hm – 630 mV. Doesn’t make sense. Turning it off (still connected) gives this:

    reading:RF12-868.5.17:oneLong:value = 31128
    reading:RF12-868.5.17:oneLong:value = 30760
    reading:RF12-868.5.17:oneLong:value = 30961
    reading:RF12-868.5.17:oneLong:value = 30512
    reading:RF12-868.5.17:oneLong:value = 30292
    reading:RF12-868.5.17:oneLong:value = 34016
    reading:RF12-868.5.17:oneLong:value = 32847

And with the 100 W light bulb:

    reading:RF12-868.5.17:oneLong:value = 474201
    reading:RF12-868.5.17:oneLong:value = 475105
    reading:RF12-868.5.17:oneLong:value = 475399
    reading:RF12-868.5.17:oneLong:value = 474646
    reading:RF12-868.5.17:oneLong:value = 475105
    reading:RF12-868.5.17:oneLong:value = 474585
    reading:RF12-868.5.17:oneLong:value = 468425
    reading:RF12-868.5.17:oneLong:value = 463834

Now it’s 470 mV. I’m lost…

I understand how a very high-impedance circuit can pick up a signal even when not connected. But the above levels make no sense at all. On an open line, one could explain how noise pickup would completely throw off the ADC, especially when no load is plugged in.

But in this case, the input impedance towards the AIO pin is 0.1 Ω + 22 kΩ. As a test, I even removed the voltage divider, and connected the 0.1 Ω resistor directly between GND and AIO, well… with a 100 Ω resistor in series to avoid damaging the I/O pin. The only difference was that the off-state now reads as 4 mV, as expected. But load or no load, with AC connected, the reading still jumps to around 0.7 V.

Doesn’t look like the ATmega is broken, since it sends out packets and reacts to changes on its analog input pin.

I’ll look into filtering. I’ll also consider hooking up my scope, but this comes with a considerable risk – all those wires plugged into all sorts of things and a USB-powered scope which could easily fry my computer!

Perhaps there’s a bug in the sketch. A simple multi-meter hooked up across the 0.1 Ω resistor indicates levels between 4 and 12 mVAC – which seems about right.

RFM12B Command Calculator

In Software on Sep 18, 2011 at 00:01

I wanted to learn a bit more about how to implement non-trivial forms in HTML + JavaScript. Without writing too much HTML, preferably…

There are a few versions of a tool floating around the web, which help calculate the constants needed to configure the RFM12B wireless module – such as this recent one by Steve (tankslappa).

Putting one and one together – it seemed like an excellent exercise to try and turn this into a web page:

Screen Shot 2011-09-17 at 15.47.21.png

If only I had known what I had gotten myself into… it took more than a full day to get all the pieces working!

In fairness, I was also using this as a way to explore idioms for writing this sort of stuff. There is a lot of repetitiveness in there, mixed with a frustrating level of variation in the weirdest places.

Due to some powerful JavaScript libraries, the result is extremely dynamic. There is no “refresh”. There is not even a “submit”. This is the sort of behavior I’m after for all of JeeMon, BTW.

The calculator has been placed on a new website, called http://tools.jeelabs.org/. The page is dynamically generated, but you can simply save a copy of it on your own computer because all the work happens in the browser. The code has been (lightly) tested with WebKit-based browsers (i.e. Safari and Chrome) and with FireFox. I’d be interested to hear how it does on others.

If you want to look inside, just view the HTML code, of course.

As I said, it’s all generated on-the-fly. From a page which is a pretty crazy mix of HTML, CSS, JavaScript, and Tcl. With a grid system and a view model thrown in.

You can view the source code here. I’ve explored a couple of ways of doing this, but for now the single-source-file-with-everything-in-it seems to work best. This must be the craziest software setup I’ve ever used, but so far it’s been pretty effective for me.

Why use a single programming language? Let’s mash ‘em all together!

A 220 : 6.3 V transformer… backwards

In Hardware on Sep 17, 2011 at 00:01

A new day, a new setup:

DSC_2630.jpg

The schematic of this new setup was shown yesterday.

Wait – a regular transformer AND it’s connected the wrong way around?

My first reaction when I saw this article was that this couldn’t possibly work. After all, a 220 -> 6.3 VAC transformer has a 35:1 step down ratio, so when connected in reverse, that would simply generate 7.5 kiloVolt!

The first reason this doesn’t happen, is that the transformer is connected in series with the load. It’s not measuring voltage but current. This is in fact exactly what a current transformer does.

The second reason why the high voltage never gets generated, is the 0.1 Ω (2W) resistor placed in parallel with the 6.3 V winding. With a 1000 W load, there’s 4.5 A going through the resistor, which leads to a 0.45 V voltage across the transformer. Together with the 1:35 step-up, that’s “only” 15 VAC or so on the “primary” side.

There is a risk, though: if the 0.1 Ω parallel resistor were to ever get disconnected during use, then the secondary winding will instantly turn into a little fuse, heat up, and blow. And while it does, a very high voltage spike might come out the other end!

Still, 15 V is far too much for an ATmega. But this voltage is not directly connected between the I/O pin and ground. Instead, one side is connected to a 22 / 22 kΩ voltage divider, which nicely puts the AC signal in mid-range for the ADC. When the voltage exceeds the ± 1.65 V (VCC/2) swing supported by the I/O pins, it will start to flow through the ESR protection diodes of the ATmega. With 22 kΩ on the other end to limit the current, these currents will be less than 1 mA – this is not high enough to do any harm.

Anyway, let’s see what comes out:

    $ jeemon examples/driver-demo
    ...
    reading:RF12-868.5.17:oneLong:value = 8180
    reading:RF12-868.5.17:oneLong:value = 7952
    reading:RF12-868.5.17:oneLong:value = 7886
    reading:RF12-868.5.17:oneLong:value = 8004
    reading:RF12-868.5.17:oneLong:value = 7364
    reading:RF12-868.5.17:oneLong:value = 32577
    reading:RF12-868.5.17:oneLong:value = 171745  <- 100 W light bulb
    reading:RF12-868.5.17:oneLong:value = 174525
    reading:RF12-868.5.17:oneLong:value = 175552
    reading:RF12-868.5.17:oneLong:value = 175254
    reading:RF12-868.5.17:oneLong:value = 175263
    reading:RF12-868.5.17:oneLong:value = 175193
    reading:RF12-868.5.17:oneLong:value = 175308
    reading:RF12-868.5.17:oneLong:value = 175101
    reading:RF12-868.5.17:oneLong:value = 174102
    reading:RF12-868.5.17:oneLong:value = 171595
    reading:RF12-868.5.17:oneLong:value = 89046   <- off
    reading:RF12-868.5.17:oneLong:value = 12378
    reading:RF12-868.5.17:oneLong:value = 8650
    reading:RF12-868.5.17:oneLong:value = 9568
    reading:RF12-868.5.17:oneLong:value = 9532
    ...
    reading:RF12-868.5.17:oneLong:value = 13413
    reading:RF12-868.5.17:oneLong:value = 12650
    reading:RF12-868.5.17:oneLong:value = 26564
    reading:RF12-868.5.17:oneLong:value = 111262  <- 1 W power brick
    reading:RF12-868.5.17:oneLong:value = 205944
    reading:RF12-868.5.17:oneLong:value = 209868
    reading:RF12-868.5.17:oneLong:value = 210105
    reading:RF12-868.5.17:oneLong:value = 210249
    reading:RF12-868.5.17:oneLong:value = 210311
    reading:RF12-868.5.17:oneLong:value = 210668
    reading:RF12-868.5.17:oneLong:value = 209933
    reading:RF12-868.5.17:oneLong:value = 210106
    reading:RF12-868.5.17:oneLong:value = 210014
    reading:RF12-868.5.17:oneLong:value = 209756
    reading:RF12-868.5.17:oneLong:value = 235024  <- off
    reading:RF12-868.5.17:oneLong:value = 52646
    reading:RF12-868.5.17:oneLong:value = 21260
    reading:RF12-868.5.17:oneLong:value = 10945
    reading:RF12-868.5.17:oneLong:value = 9046
    reading:RF12-868.5.17:oneLong:value = 9381
    reading:RF12-868.5.17:oneLong:value = 9236
    reading:RF12-868.5.17:oneLong:value = 8695
    reading:RF12-868.5.17:oneLong:value = 9015
    ...

How can this be? A jump from 8 mV to 175 mV for the light bulb, and then 210 mV for a much lighter load?

It gets even weirder:

    reading:RF12-868.5.17:oneLong:value = 8642
    reading:RF12-868.5.17:oneLong:value = 8567
    reading:RF12-868.5.17:oneLong:value = 11567
    reading:RF12-868.5.17:oneLong:value = 32239
    reading:RF12-868.5.17:oneLong:value = 122999
    reading:RF12-868.5.17:oneLong:value = 180108
    reading:RF12-868.5.17:oneLong:value = 180393
    reading:RF12-868.5.17:oneLong:value = 180469
    reading:RF12-868.5.17:oneLong:value = 180344
    reading:RF12-868.5.17:oneLong:value = 180702
    reading:RF12-868.5.17:oneLong:value = 180036
    reading:RF12-868.5.17:oneLong:value = 180432
    reading:RF12-868.5.17:oneLong:value = 180197
    reading:RF12-868.5.17:oneLong:value = 179220
    reading:RF12-868.5.17:oneLong:value = 174530
    reading:RF12-868.5.17:oneLong:value = 172794
    reading:RF12-868.5.17:oneLong:value = 32629
    reading:RF12-868.5.17:oneLong:value = 13266
    reading:RF12-868.5.17:oneLong:value = 7944
    reading:RF12-868.5.17:oneLong:value = 7909

This time, there is no load! I just turned on power without anything attached, i.e. an open circuit!

There’s something completely wrong here. My hunch is that this is either due to some extreme level of noise or an improperly-dimensioned burden resistor.

Noise might explain it, since the code tracks maxima, including any high-frequency spikes there might be. Maybe a digital low-pass filter could help, but I haven’t found a good source for calculating the coefficients of a FIR filter.

Further investigation will be required – this setup is useless!

SMD current transformer

In Hardware on Sep 16, 2011 at 00:01

Here’s another attempt to detect whether an appliance is on and consuming power.

This time I’m going to use a tiny SMD transformer (DigiKey part# 811-1193-1-ND). It has a 1:100 turn ratio, with 0.007 Ω resistance in the primary “coil” (I think it’s just a single loop), and rated up to 10 A.

The problem with this part, which rules it out for serious use, is that it’s only meant for frequencies in the range 50 kHz to 500 kHz. Whoops, that’ll be off by three orders of magnitude when used with 50 Hz AC mains…

Oh well, since I had one lying around, least I can do is try it – right? Here’s the setup:

DSC_2629.jpg

The hookup is a bit odd, but will become clear in future posts. The SMD transformer is only a few millimeters square, between the green jumper block and the black mini-breadboard.

I’m using the current transformer (CT) as follows:

JC's Doodles, page 12.png

… except that this schematic is for another experiment, still pending. In this case, there is no 0.1 Ω resistor on the primary side (it wouldn’t make a difference, next to the primary’s 7 mΩ), but there’s a 100 Ω “burden resistor” on the secondary side. This matches the data sheet for getting a 1 V/A sensitivity. For info on CT’s and burden resistors, see this OpenEnergyMonitor page.

Here are some first results, using the same sketch as in yesterday’s post:

    OK 17 164 19 0 0
    OK 17 17 16 0 0
    OK 17 66 16 0 0
    OK 17 42 19 0 0
    OK 17 241 163 0 0   <- 100 W light bulb
    OK 17 107 95 0 0
    OK 17 49 93 0 0
    OK 17 77 94 0 0
    OK 17 134 94 0 0
    OK 17 244 93 0 0
    OK 17 7 94 0 0
    OK 17 151 93 0 0
    OK 17 27 93 0 0
    OK 17 124 90 0 0
    OK 17 119 24 0 0    <- off
    OK 17 125 24 0 0
    OK 17 36 16 0 0
    OK 17 246 15 0 0
    OK 17 109 16 0 0
    OK 17 33 18 0 0
    OK 17 50 16 0 0

These are raw bytes, i.e. a 32-bit long int, sent once a second and received via wireless. I’m powering the JeeNode with an AA Power Board and standing back during testing, the 220 VAC is now a bit too close for (my) comfort…

The average measurement value with no current is now about 16 x 256 = 4 mV.

The high value when the light has been turned on is about 94 x 256 = 24 mV, a 20 mV increase.

Not bad… but it’s less sensitive than the ACS714 Hall-effect sensor. I didn’t even try it with a 1 W load. This is a far cry from the claimed 1 V/A sensitivity, no doubt because the 50 Hz signal is so far off from the design specs of this SMD current transformer. Saturation effects, or something…

I’ll continue these experiments in the next few days with another current transformer which is designed to work at 50 Hz, as well as with a reversed 220 : 6.3 V PCB transformer. Stay tuned!

Power Measurement (ACS) – code

In Software on Sep 15, 2011 at 00:01

For yesterday’s setup, I had to write a bit of code. Here’s the main part:

Screen Shot 2011 09 14 at 12 38 44

(this “powerACS” sketch can be found here)

The idea is to keep reading the analog pin as often as possible (several thousand times per second), and then keep track of its moving average, using this little smoothing trick:

    avg = (499L * avg + value) / 500;

Each new measurement adds 1/500th weight to the average. I didn’t want to use floating point, so it takes some care to avoid long int overflow. Here’s how I scale the input (0 .. 1023) to microvolts (0 .. 3,300,000):

    uint32_t value = measure.anaRead() * (3300000L / 1023);

Note that 500 x 3300000 is under the max of a 32-bit int, so the above “avg” calculation is ok.

It’s easy to calculate the offset, once we know the average voltage. Keep in mind that we’re measuring 50 Hz AC, and that the result we’re after is the fluctuation – i.e. what you get when putting a multimeter in VAC measurement mode. The way to do that is to calculate the “average absolute difference from the average”:

JC s Doodles page 13

IOW, what I want is the average height of those red bars. I flipped the lower half up because we’re interested in the absolute difference (otherwise the bars would just cancel out).

Here’s the calculation:

    if (value > avg)
      total += value - avg;
    else if (value < avg)
      total += avg - value;
    ++count;

When the time comes to report a “range” reading, we calculate it as follows and reset the other variables:

    uint32_t range = total / count;
    total = count = 0;

Here’s some sample debug output on the serial port:

    [powerACS]
    ...
    2560650 2569286 5916 31973363 5404
    2576775 2569256 5911 32175178 5443
    2563875 2569266 5917 32888792 5558
    2567100 2570205 5911 33200506 5616
    2573550 2569935 5917 32798060 5543
    2580000 2565675 5910 183251691 31007  <- 100 W light bulb
    2573550 2565108 5916 356840654 60317
    2570325 2565479 5916 355916529 60161
    2512275 2565467 5910 355676919 60182
    2525175 2565688 5916 356664359 60288
    2486475 2566547 5909 354854663 60053
    2480025 2566631 5916 355567509 60102
    2480025 2568658 5910 355608745 60170
    2583225 2569370 5916 178542547 30179  <- off
    2576775 2569122 5915 33193018 5611
    2570325 2569274 5911 33236267 5622
    2560650 2568988 5917 32873935 5555
    2576775 2569290 5911 32733630 5537
    2567100 2568564 5917 33390738 5643
    ...
    2560650 2569179 5917 32917499 5563
    2576775 2568640 5911 32354114 5473
    2570325 2569121 5916 32820006 5547
    2576775 2568319 5917 33896284 5728
    2570325 2569996 5911 33120608 5603
    2567100 2568514 5917 37697489 6371    <- 1.1 W power brick
    2550975 2568582 5911 36764137 6219
    2567100 2568767 5916 37434853 6327
    2567100 2568813 5911 36272763 6136
    2563875 2568659 5917 37738504 6377
    2563875 2568355 5911 37911395 6413
    2567100 2568586 5917 37444158 6328
    2563875 2568672 5916 37607002 6356
    2580000 2568879 5911 37293083 6309    <- off
    2570325 2568330 5917 33082874 5591
    2563875 2567713 5911 33007567 5584
    2567100 2568380 5917 32770050 5538
    2580000 2568846 5910 33002639 5584
    2570325 2568844 5917 33201661 5611

You can see the 60 mV signal when the power consumption is 100 W. Which is a bit low: I would have expected 100 W / 230 V * 0.185 V/A = 80 mV. But it turns out that the lamp really is an 87 W lamp, so that makes it 70 mV. I’ll attribute the remaining difference to my crude setup, for lack of a better explanation.

What’s worse though, is that there is always a 5.5 mV “hum” when no current is flowing (even when the whole circuit isn’t anywhere near a power line). One possible explanation could be ADC measurement noise: the 10-bit ADC resolution is 3.3 mV. Then again, note how multiple measurements do lead to a much finer resolution: the 60 mV readout fluctuates by only a fraction of one millivolt. Noise actually improves resolution, when averaged over multiple readings.

Still, the 5.5 mV baseline value makes it hard to reliably measure really low power levels. The 1.1 W test was from an unloaded power brick, i.e. an inductive load. It causes the output to jump from 5.5 mV to only 6.3 mV, which is not a very distinct result. Another 0.3 W test load didn’t even register.

Conclusion: this detects current fairly well, was very easy to set up, and offers galvanic isolation protection. But it does require a hookup to AC mains power and it’s not super effective at really low power levels.

Power measurement (ACS)

In Hardware on Sep 14, 2011 at 00:01

The first try I’ll make at measuring AC mains power is with an Allegro ACS714 Hall-effect sensor (this one). I measures ± 5 Amps. It’s a bit expensive for wide-scale use, and it does require hook-up (i.e. AC mains pass-through). Hopefully a non-contact solution will present itself, eventually. For now, it’s a good baseline test.

The circuit provides 2.5 kV isolation, but of course that means nothing when everything is so close that you could easily touch the wrong pin by accident.

Here’s my setup:

DSC_2627.jpg

Everything is connected to my home-made isolation transformer. Which means grounding adds no safety and to remind me of the risk at all times, I’m using old 2-wire cabling and old ungrounded connectors. Not used anywhere else around here, so a mixup is not possible (those old power plugs won’t fit in today’s grounded sockets). If you look very closely, you can see that the splice is secured with two zip-lock ties. Cables are short so they don’t get tangled up, but long enough to avoid pulling on them.

As I was preparing the setup, I found out that the ACS714 needs 4.5 .. 5.5 V to operate, so the JeeNode + AA Power Board shown here won’t cut it. Oh well, I’ll switch to a 4x EneLoop battery pack (which is 5.2V).

Since I’m only interested in low-power for this test (under 100 W), the output voltage is within range of an ATmega running at 3.3V (the ACS714 output is 185 mV/A, centered around 2.5V).

The initial test worked flawlessly: the light went on! :)

Next, the software. It’s always the software which requires most work.

Sending RF12 packets over UDP

In Software on Sep 13, 2011 at 00:01

As I mentioned in a recent post, the collectd approach fits right in with how wireless RF12 broadcast packets work.

Sounds like a good use for an EtherCard + JeeNode pair (or any other ENC28J60 + RFM12B combo out there):

DSC_2625.jpg

The idea is to pass all incoming RF12 packets to Ethernet using UDP broadcasts. By using the collectd protocol, many different tools could be used to further process this information.

What I did was take the etherNode sketch in the new EtherCard library, and add a call to forwardToUDP() whenever a valid RF12 comes in. The main trick is to convert this into a UDP packet format which matches collectd’s binary protocol:

Screen Shot 2011-09-12 at 17.50.37.png

The sendUdp() code has been extended to recognize “multicast” destinations, which is the whole point of this: with multicast (a controlled form of broadcasting), you can send out packets without knowing the destination.

The remaining code can be found in the new JeeUdp.ino sketch (note that I’m now using Arduino IDE 1.0beta2 and GitHub for all this).

It took some work to get the protocol right. So before messing up my current collectd setup on the LAN here at JeeLabs, I used port 25827 instead of 25826 for testing. With JeeMon, it’s easy to create a small UDP listener:

And sure enough, eventually it all started to work, with RF12 packets getting re-routed to this listener:

Screen Shot 2011-09-12 at 20.04.08.png

Now the big test. Let’s switch the sketch to port 25826 and see what WireShark thinks of these packets:

Screen Shot 2011-09-12 at 17.15.16.png

Yeay – it’s workin’ !

The tricky part is the actual data. Since these are raw RF12 packets, with a varying number of bytes, there’s no interpretation of the data at this level. What I ended up doing, is sending the data bytes in as many collectd 64-bit unsigned int “counter values” as needed. In the above example, two such values were needed to represent the data bytes. It will be up to the receiver to get those values and convert them to meaningful readings. This decoding will depend on what the nodes are sending, and can be different for each sending JeeNode.

I’ve left the original web browser in as well. Here is the “JeeUdp” box, as seen through a web browser:

Screen Shot 2011-09-12 at 18.02.28.png

(please ignore the old name in the title bar, the name is now “JeeUdp”)

It’s not as automatic out of the box as I’d really like. For one, you have to figure out which IP address this unit gets from DHCP – one way to do so is to connect it to USB and open the serial console. The other bit is that you need to configure the unit to set its name, the UDP port, and the RF12 settings to use. There’s a “Configure” link of the web page to do this – a some point, I’d like to make JeeMon aware of this, so it can do the setup itself (via http). And the last missing piece of the puzzle is to hook this into the different drivers and decoders to interpret the data from these UDP packets in the same way as with a JeeLink on USB.

Ultimately, I’d like to make this work without any remote setup:

  • attach Ethernet and power to this box (any number of them)
  • each box starts reporting its status via UDP (including its IP address)
  • a central JeeMon automatically recognizes these units
  • you can now give a name to each box and fill in its RF12 configuration
  • packets start coming in, so now you can specify the type of each node
  • decoders kick in to pick up the raw data and generate meaningful “readings”
  • that’s it – easy deployment of JeeNode-based networks is on the horizon!

Not there yet, but all the essential parts are working!

A site for the home – part 2

In Software on Sep 12, 2011 at 00:01

I’ve been making good progress since the recent post.

Some things work, some things don’t when you use “off-the-shelf Open Source”. For example, I started using Twitters Bootstrap as CSS grid framework for page layout. It looks great. But there was too much interference with other components, such as jQuery DataTables. Well, either that or I simply don’t get it. So I ended up removing it all, in favor of the 960 Grid System (note how all JavaScript development seems to be happening on GitHub).

So now I’ve got a first rough cut at the “Niner” layout, i.e. 3×3 pages accessed via tabs on the bottom/right. Here’s a screen shot from an iPad:

Photo

The tabs don’t look so great (I’m so inept at GUI design that I don’t even try), but the general design is shaping up nicely. The area at the bottom left shows the last two lines of the log on the server, and it’s updating in real time. So is the table itself, using Server-Sent events.

With this sort of dynamics, all sorts of details start to jump out. My first approach was to have one page per URL, i.e. “http://127.0.0.1:8181/1″ for page 1, etc. But that causes a full page refresh, which is too disruptive, especially with that log area constantly updating.

Which is where JavaScript and Ajax come in, of course. So now navigation only updates the main area – not the bottom & right-side borders. I’m only testing with Safari/Chrome, and occasionally FireFox, BTW.

The indentation idea I described earlier is also working out nicely. Here’s the main page description which produces the HTML sent to the browser:

Screen Shot 2011 09 11 at 22 39 33

(as you can see, there are limits to syntax coloring when mixing several languages in a single source file!)

And here’s the description of the page with that table shown above:

Screen Shot 2011 09 11 at 22 40 15

Let me just say that I’m “over” HTML. Bye bye, <div> et al. Good riddance.

PS. Oh yes, and here’s the fun part – this defines the structure of the site:

Screen Shot 2011 09 12 at 0 02 40

Tabs and titles are automatically generated unless overridden in this layout.

Sensing power consumption

In Hardware on Sep 11, 2011 at 00:01

A small exploration of a circuit which went nowhere…

I’ve been looking for a low-cost way to detect whether an electrical appliance is drawing current. There would be several uses for that, from power-saving master/slave power strips, to helping figuring out what’s going on in the house by measuring the central meter, as I’m doing already.

Here’s my original (naïve) line of thought:

  • optocouplers are a great way to stay out of AC mains electrical trouble
  • many units will still work with currents down to well under 1 mA
  • all I want is detection – is a device ON or OFF?
  • the rest can be done with JeeNodes, wirelessly, as usual

Here’s the (pretty silly) first step, in series between power outlet and appliance:

JC's Doodles, page 11.png

A resistor to limit the current, and the opto-coupler to “light up” and produce a signal in a galvanically isolated manner. But don’t get your hopes up, this setup definitely won’t work!

  • First of all, the appliance will stop working, if the resistor is in the 100 kΩ range.
  • With resistance values much lower, the resistor will become a heater and self-destruct.
  • Besides, even a 100 W load will draw 0.5 A or so… way too much for the LED.
  • There’s also a small problem with the voltage being AC, not DC, but that’s easily solved with a diode.

It’s a pretty useful mental exercise to work through this and understand how voltage, current, and power interact. If you’re totally lost, you could check out this series of posts and this link.

Sooo… could this approach be made to work?

In this case, an indirect demonstration that it can’t possibly work is actually easier to argue: suppose there were a circuit with the opto-coupler’s LED in series between outlet and load. It only needs to cause a voltage drop of about 1.5V to get the IR led inside to light up. Now suppose the load draws 1000 W, i.e. nearly 5 A. Big problem: (due to P = E x I) the power consumed by the opto-coupler would be 7.5 W … an excessive amount of heat for such a tiny device (apart from the sheer waste!).

Even some form of magical “power reduction” would be tricky, since I do need to detect the power-OFF case, i.e. power levels down to perhaps 0.5 W of standby power consumption for modern appliances (some even less). For 220V, 0.5 W corresponds to an average current of only 2.2 mA.

Maybe there’s a clever way to do this, but I haven’t found it. Something with a MOSFET, mostly conducting, but opening up just a little perhaps 1000x per second, to very briefly allow the LED in an opto-coupler to detect the current (with an inductor to throttle it). Heck, an ATtiny could be programmed to drive that MOSFET, but now it’s all becoming an active circuit, which needs power, several components, etc.

The trouble with AC mains is those extremes – on the one hand you need decent sensitivity, on the other you need to deal with peak loads of perhaps 2000 W, with their hefty currents, and where heat can become a major issue. We’re talking about power levels ranging over more than three orders of magnitude!

Of course there is the current transformer. But those are a bit too expensive to use all over the place.

Another option would be to use a regular transformer in reverse, as in this article. With a little 220 => 6.3V transformer, the ratio would be 35:1. With 5A max, a 0.1 Ω resistor would have 0.5 W of heat loss and create a 0.5V difference, amplified up to 17.5 V. For a 2.2 mA “off” current (0.5 W standby), that voltage would drop to 7.7 mV, which is probably too low to measure reliably with the ATmega ADC. Still.. might be worth a try.

An interesting low-cost sensor is based on the Hall-effect to detect magnetic fields. Because wherever there is alternating current, there is a magnetic field.

With hall-effect sensors, the problem is again at the low end. Keep in mind that the goal is reliable detection of current, not measurement. Now, you could increase the sensitivity by increasing the intensity of the magnetic field – i.e. by creating a coil around the sensor, for example. Trouble is: since those wires need to also handle large power consumption levels, they are by necessity fairly thick. Thick wires make lousy coils, and are impossible to get close to the sensor.

There are other ways. Such as measuring light levels, in case of a lamp. Or heat in case of a TV (would be a bit sluggish). Or vibration in case of a fridge.

Maybe a hand-made 100-turn current transformer would work? That’s a matter of winding some thin enameled wire around a single mains-carrying insulated copper wire.

Hm… might give that a try.

The challenge is to find something reasonably universal, which does not require a direct galvanic connection to AC mains. It seems so silly: an ATmega can detect nanowatt levels of energy, yet somehow there is no practical path from AC power ON/OFF detection to that ATmega?

PS. A related (and more difficult) problem is “stealing” some ambient magnetic energy to power a JeeNode. You don’t need much while it’s asleep, but transmitting even just one tiny packet is another story.

A site for the home

In Software on Sep 10, 2011 at 00:01

I’m building a “home monitoring and automation” system. Not just the hardware “nodes”, but also the software side of things, i.e. a server running permanently. Web-based, so that it can be accessed from anywhere if needed.

I’ve got a lot of code lying around here, ready to be put to use for just that. It’s not that hard to create a website (although truly dynamic displays are perhaps a little harder).

One thing I’ve been struggling with for a long time, though, is the overall design. Not so much the look & feel, but the structure and behavior of it all.

I’m not too keen on systems with elaborate menu structures, tons of hoops dialogs to jump through, even scrolling messes with our innate sense of location. Been there, done that. Modes suck – they just create labyrinths.

What I’ve come up with is probably not for everyone (heh, nothing new here). An entire system, organized as nine pages. No more, no less. Instant navigation with a keyboard (digits, function keys, whatever), and trivial navigation with a mouse or touch screen:

Web layout

Even a laptop screen with 3+3 buttons (or a joystick) could navigate to any page with just two clicks.

Here is a first exploration of what these pages could be for (vaporware, yeay!):

Web pages

Each page is nearly the full screen, with only the bottom and right-side borders used. And each page can have any content, of course: tabs, sidebars, widget collections, it’s still all up in the air. There’s plenty of software to extend this system infinitely, using HTML, CSS, and JavaScript.

One of the things I’d like to do with this, is make the system automatically return to the main Status Overview page when idle for a minute or so. With the option to “pin” one of the other pages to prevent this from happening.

This might all seem very basic, inconsequential, and even arbitrary, but for me it’s a major step forward. I’ve been running around in circles for ages, because there was no structure in place to start assembling the different parts of the site. It’s all nice and well to develop stuff and tinker endlessly behind the scenes, but in the end this still needs to lead to a practical setup.

So now there’s a plan!

Playing with indentation

In Software on Sep 9, 2011 at 00:01

Bear with me – this isn’t directly related to Physical Computing, but I’ve been having fun with some code.

If you’ve ever written any web apps or designed a web site, chances are that you’ve also written some HTML, or at least HTML templates. Stuff like this, perhaps:

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset='utf-8' />
        <title>Flot graph demo</title>
        <script type='text/javascript' src='.../jquery.flot.js'></script>
        <script type='text/javascript'>
          $(function(){ ... });
        </script>
        <style type='text/css'>
          #placeholder { width: 600px; height: 300px; }
        </style>
      </head>
      <body>
        <div id='placeholder'></div>
      </body>
    </html>

Well, I’m getting a bit tired of all this nesting. The essence of it is buried far too deeply inside what I consider line noise. There are currently a number of developments related to HTML, CSS, and JavaScript which might change this sort of development considerably. Much of it seems to be driven by the Ruby and Rails community, BTW.

Some examples: for HTML, there is Haml. For CSS, there is Sass. And for JavaScript there is CoffeeScript.

All of these have in common (along with Python and YAML) that they treat indentation as a significant part of the notation. Before you raise an eyebrow: I know that the opinions on this are all over the map. So be it.

I wanted to try this with JeeMon, so after a bit of coding, I ended up with a “Significant Indentation Formatter” and then applied that to HTML. Here’s the above, re-written in a HAML-like notation:

    !html
      head
        meta/charset=utf-8
        title: Flot graph demo
        script/type=text/javascript/src=.../jquery.flot.js
        script/type=text/javascript
          $(function(){ ... });
        style/type=text/css
          #placeholder { width: 600px; height: 300px; }
      body>#placeholder

It generates the same HTML as above, but there’s less to write. Even less in fact, because I’ve also integrated it with JeeMon’s templating system:

    !html
      head
        meta/charset=utf-8
        title: Flot graph demo
        [JScript includes flot]
        [JScript wrap $js]
        [JScript style { #placeholder { width: 600px; height: 300px; } }]
      body>#placeholder

Here’s a slightly more elaborate KAKU send example which illustrates basic template looping:

    !html
      head
        meta/charset=utf-8
        title: KAKU send
        [JScript includes ui]
        [JScript wrap $js]
      body
        % foreach x {1 2 3 4}
          p.toggles
            button#on$x: On $x
            button#off$x: Off $x
        p#group
          label: Group:
          % foreach x {I II III IV}
            input#g$x/type=radio/name=g/value=$x
            label/for=g$x: $x
        p
          label: House Code:
          select
          % foreach x {A B C D E F G H I J K L M N O P}
            option/value=$x: $x

So far, this system doesn’t use standard conventions, is undocumented, hasn’t been used in anger or proven itself to handle more complex layouts, has almost no error handling, and has only been used by yours truly for a few hours. Still, I’ve adjusted all the examples in JeeRev to gain a bit more experience with this approach.

I like it. The above is pretty close to how I want to design the structures and sections of a web page.

Low-end Linux

In Hardware, Software on Sep 8, 2011 at 00:01

It’d be neat to use a very low-end embedded Linux system for running (part of) my home monitoring and automation system.

I’ve mentioned the BifferBoard before, twice in fact. With a BogoMips rating of 56, it’s as low end as it gets.

Unlike most low-end boards, which are ARM-based, the BifferBoard runs on a “S3282/CodeTek” which is 486SX-compatible, so many 32-bit Linux binaries will work as is. I’m running a fairly standard Debian Lenny on mine, very convenient. It needs 650 Mb, including a full gcc build chain.

IMO, there’s great value in such a low-power system. For one, with just 30 MB RAM and no swap support on the standard setup, you’re really forced to look at memory consumption. Running out of memory will cause a program to be terminated (it can easily be restarted with a watchdog shell script).

The other feature this board does not have is hardware floating point. It’s all emulated. Here’s what that means:

  $ ssh biffie jeemon tcl 'time {/ 1 2} 1000'
  36.735 microseconds per iteration
  $ ssh biffie jeemon tcl 'time {/ 1.0 2.0} 1000'
  336.717 microseconds per iteration
  $

Versus my lowly 1.4 GHz laptop:

  $ jeemon tcl 'time {/ 1 2} 1000'
  0.569055 microseconds per iteration
  $ jeemon tcl 'time {/ 1.0 2.0} 1000'
  0.546295 microseconds per iteration
  $

As you can see, it’s a whole different ballgame :)

Anyway, let’s try and get a bit more substantial code running on this setup. I’ll use JeeMon, since I’ve recently been doing some work to combine round-robin data storage with graphing in the browser:

Screen Shot 2011 09 07 at 17 10 29

The graphs on the right are 1-day / 1-week / and 2-month previews, but let’s ignore the actual graphs since there is not enough test data for them to be meaningful.

What I wanted to see, is whether this would run on a BifferBoard. Here’s my first attempt:

Screen Shot 2011 09 07 at 17 03 14

Whoops… out of memory. Not only that, it’s taking ages to generate the JSON data for each Ajax request coming from the browser. Here’s my laptop for comparison:

Screen Shot 2011 09 07 at 16 59 31

The laptop is generating data for 5 graphs (one is off-screen) in a fraction of a second, while the BifferBoard needs several minutes (before running out of memory).

Good. Because with such a slow process, it’s much easier to find the bottlenecks. It turned out to be in the round-robin storage module I recently coded (using a USB stick as mass storage device doesn’t help either). Two hours later, things started working a lot better:

Screen Shot 2011 09 07 at 19 02 40

Progress! Still too slow for practical use… at least it’s no longer running out of memory. But although the BifferBoard is no longer maxed out, it is struggling – here’s a “vmstat 3” display, with one line per 3s:

Screen Shot 2011 09 07 at 20 13 16

I now realize that with my current settings, all data streams are aggregated to a larger “bucket” every 5 minutes. This probably explains these periods of 100% CPU activity.

Memory use is reasonable (with all interactive SSH shells logged out):

  $ ssh biffie free
               total       used       free     shared    buffers     cached
  Mem:         30000      28248       1752          0        444       8040
  -/+ buffers/cache:      19764      10236
  Swap:            0          0          0
  $

There’s still a lot of floating point activity going on, because I store all data points as 64-bit doubles. This needs to be improved at some point. My hunch is that it will take care of the performance problem. But it’s nice to see that JeeMon can cope (sort of), even in such a very limited environment. A BifferBoard with JeeMon will definitely be sufficient as “slave” controller in a larger configuration.

Power consumption is pretty neat: 2.85 W under full load and 2.20 W when idle. Measured at the AC outlet, i.e. including 5V power supply losses.

There are numerous low-end Linux systems out there. If the above can be made to work reasonably well, then it should work with most of them… ARM- / PPC- / x86-based, whatever.

For now, that BifferBoard is a good way to keep me honest, and on edge :)

Versions à la GitHub

In Software on Sep 7, 2011 at 00:01

As mentioned a few days ago, I’m in the process of moving all my software development to GitHub.

In a nutshell: Git is a Version Control system which lets you keep track of changes to groups of files, and do it in such a way that many people can make changes and nothing will ever get lost. You might feel lost if you really make a mess of it, but every change is tracked and can be recovered and compared against any other point in time. Based on git, GitHub takes it all to the next level, by fostering collaboration and combining tons of useful features – for free, via their website. GitHub is to git as JavaScript is to assembler (the analogy is a bit rough).

If you’re writing code, there is no excuse for not using a VCS. Even as a single developer, even if no-one else ever gets involved, and even if you’re the most organized person on the planet. An organized developer? Hah! :)

Life without a VCS looks like this:

  • let’s write some code – great, it works!
  • let’s try out something more difficult now
  • drat, didn’t work, I hope my editor undo buffer lets me back out
  • let’s make a copy of what worked before
  • now let’s write some more code – great, it works!
  • … time passes …
  • time to write code again – hm, I messed up
  • where was that copy I made earlier, again?
  • drat, it doesn’t have my latest changes
  • do I remember which part worked, and what I added?
  • … time passes …
  • I want to continue on that first attempt again
  • hmm, guess I’ll start from scratch
  • no wait, let’s make a copy before I mess up
  • … time passes, the disk fills with copies, none of them work …

Now let’s revisit that with a VCS:

  • write some code, check it in – add a brief comment
  • try out something more difficult now
  • check it in, let’s add a note that it’s still experimental
  • hm, can’t seem to get it to work
  • oh, well, I’ll go back to the previous version – easy!
  • writing more code, having fun, we’re on a roll
  • let’s check it in right away
  • … disaster strikes, my laptop breaks down
  • get it fixed or get a new one, check out from GitHub
  • where was I? oh yes, got one good version and two bad ones
  • let’s switch to that failed attempt again
  • wait, let’s first merge the latest good changes in!
  • cool, now I’ve got an up-to-date set of files to work on
  • oh, darn, still can’t make it work, let’s save it with a note
  • … time passes …
  • ok, time to review the different attempts and take it from there

Note that there are no more copies of files. Only a repository (on your laptop and on GitHub – but you still need to back it up, like everything else). The repository contains the entire history of all the changes, including notes, backing out steps, merges, everything.

A lot of this is possible with any modern VCS, including the “subversion” setup I’ve been using until now. But GitHub makes it effortless, and throws in a lot of nice extras. You still need a “git” application on your laptop (there are several versions), and if you’re using a Mac, you can use GitHub’s own app to communicate with GitHub without having to go through a browser.

Here’s what a “diff” looks like, i.e. a comparison of the source code, showing only the differences:

Screen Shot 2011 09 02 at 21 53 17

One line was replaced, two more lines were added at the end. It’s totally obvious, right?

Here’s a history of the “commits” made, i.e. the times when a new version was sent to git and GitHub:

Screen Shot 2011 09 02 at 21 51 01

Almost everything is a link – you can compare versions, find out what else was changed at the same time, see what else any of the developers is working on, and of course download a full copy of any version you want.

The other mechanism I’d like to describe is branching and merging. That change shown above was not made by me but by “Vliegendehuiskat” (hello, Jasper :) – and the way it works is quite neat, if you’ve never seen collaboration via VCS in action.

What Jasper did, was create a “fork” of JeeLib, i.e. a personal copy which knows exactly where it came from, when, and from which version of which files. And then he made a small change (without even telling me!), and at some point decided that it would be a good idea to get this change back into the official JeeLib code. So he submitted a “pull request”, i.e. a request for me to pull his changed version back in:

Screen Shot 2011 09 06 at 22 25 56

This generated a new entry in GitHub’s issue tracker and an email to me. I liked his change and with one click, merged it back in. Only thing left to do was to “pull” those changes to my laptop copy as well.

GitHub has a graphical “network” view to display all commits, forks, branches, and merges:

Screen Shot 2011 09 06 at 22 29 10

Each dot is a commit. The black line is my code, the blue diversion shows Jasper’s fork, his commit, and my merge. Hovering over any of these dots will show exactly what happened.

This illustrates the simplest possible case of GitHub for collaboration. If you want to see how a entire team puts git and GitHub at the center of a large-scale collaborative software development effort, read this article by Scott Chacon (who works with and at GitHub, BTW).

I’ve been moving lots of stuff to GitHub these past few days, not all of it related to JeeLabs. Here are the three main projects, as far as physical computing goes:

Screen Shot 2011 09 02 at 21 48 41

Note another nice feature: you can instantly see how active a project is. JeeLib is the brand new combination of the Ports and RF12 libraries, while EtherCard and GLCDlib have been around a bit longer (and the full change history has been imported). It’s been over half a year since Steve and I worked on GLCDlib, as you can see – with a small blip before the summer, when I added font support.

There are a few features which I haven’t enabled at GitHub. One of them is their wiki – because having yet another spot on the web with pages related to JeeLabs would probably do more harm than good. The Café at http://jeelabs.net is as before the place to look for documentation. There are ways to merge this stuff, so maybe I’ll go there one day.

A warning: the “sketches” on GitHub have all been adjusted for the new upcoming Arduino 1.0 IDE, and do not work with Arduino 0022, which is still the main version today. So I’m slightly ahead of the game with this stuff.

I’m really looking forward to how the software will evolve – and you’re cordially invited to be part of it :)

PS. I’m updating to GitHub fairly often because I work on two different machines – and this is the easiest and safest way to keep stuff in sync.

System status

In Software on Sep 6, 2011 at 00:01

Yesterday, the WordPress server dropped out. Looks like some joker decided to grab the whole site, no throttling, nothing. Oh well, I guess that’s what fat pipes lead to.

In principle, WordPress should be able to handle such things, but I think it caused the underlying MySQL to run out of memory and abort.

A restart of the WordPress VM fixed it, once I found out. But this gives me an excuse to describe some things I’ve been starting to set up here.

One of of things I did this summer, was set up a collectd process on most of the machines running here – collectd is a bit Linux-centric, but one of the things it can do is grab various system statistics and broadcast it on the LAN over UDP. There are loads of plugins, and you can interface to it with scripts to feed it anything you like.

The nice thing about UDP broadcasts, is that senders don’t need to know anything about receivers. As with the RF12 driver, you just kick stuff into the network and then it’s up to others to decide what to do with it, if anything.

Collectd is also extremely low overhead, both in terms of CPU & memory load and in terms of network load. I’ve enabled it permanently on five systems, sending out 50..100 system parameters each, once every 30 seconds.

Here’s a partial view of the information being collected:

Screen Shot 2011 09 05 at 12 57 52

This was generated in the browser using JeeMon and jstree, as an exercise (I’m looking for a more convenient JavaScript widget to adjust a live tree using JSON).

Broadcasting is very convenient, as the collectd tool and the RF12 driver have shown, because there’s no setup. In theory, packets can get lost without acknowledgements and re-transmissions. In practice, some do indeed get lost, but very rarely. No problem at all for periodic / repetitive data collection.

I created a small script to display system status on my development Mac, using a tool called GeekTool. It can put information right on the desktop, i.e. as unobtrusively as you want, really. Here’s what I see when there are no windows obscuring the desktop background:

Screen Shot 2011 09 05 at 12 42 27

On the left is a periodic “ps“, the output on the right is from “ssh myserver jeemon overview.tcl“. This is a remote call to the server using a small script to pull info out of a text file and format it. The output shows that there are 5 systems online (running collectd, that is), and the script last ran at 12:41 in 39 ms.

Perhaps more interesting is that the server also collects this data from the LAN. To do this, I just launch JeeMon next to this “main.tcl” script:

    History group * 1w/5m 1y/6h
    collectd listen sysinfo

The first line is optional – it defines the parameters for historical round-robin data storage. So what I end up with, is the last 7 days of information, with details every 5 minutes, as well as up to one year of information aggregated per 6 hours.

The second line starts listening for collectd packets on the standard UDP port, tagging the results with the “sysinfo” prefix. As side effect, JeeMon will update a “stored/state.map” text file once a minute with the current values of all variables plus some other details. This is what was used to generate the above GeekTool on-screen status. There’s a built-in web server to query and access the saved historical data.

There’s no further setup. It’s up to the clients to decide what to collect and how often to send it. Data storage on the server is fixed, since this is stored in round-robin fashion (like rrdtool). It’s now collecting nearly 600 800 variables and using about 100 125 Mb of disk space. Data collection adds 2% load on a 2.66 GHz CPU.

This could also be used to collect home sensor data.

Re-using an LCD screen

In Hardware on Sep 5, 2011 at 00:01

This summer, someone gave me an old laptop which had stopped working. I couldn’t figure out what was wrong, so I decided to take it apart. Hehehe…

One of the parts coming out was a 15″ TFT LCD screen. Nothing stellar – 1280×800 – but still.

It turns out that there are several ways to get such a thing going again (thank you, Google). I ended up buying a DPMI/VGA interface board on eBay, complete with high-voltage supply and cables. It worked right out of the box:

DSC 2618

The display is not running at its native resolution here, because I couldn’t figure out how to force Win2000 to a specific screen resolution, but with Linux it did, so all is well.

I’d like to build a small & flat enclosure for this, hook it up to the Mac Mini, and hang it on the wall. Preferably with a distance sensor to control the whole thing so it only turns on when someone is standing directly in front of it. No need for a touch screen, a few buttons would be fine as home status display.

The whole combination runs off 12V, drawing 10 Watt when fully on and 0.3 W while asleep.

Yeay, reuse is fun!

The beat goes on

In Hardware on Sep 4, 2011 at 00:01

Today, a milestone was reached:

    $ jeemon examples/driver-demo/
    15:45:01.975 RF12demo config usb-A700fdxv RF12-868.5
    reading:RF12-868.5.3:radioBlip:ping = 493295
    reading:RF12-868.5.3:radioBlip:age = 365
    ^C

That’s nearly half a million packets, one every 64 seconds, all on a single LiPo battery charge:

Screen Shot 2011 09 03 at 15 51 53

The setup is very simple. A 1300 mAh LiPo battery (I just picked one up in a camera shop with a convenient size), attached to a JeeNode USB:

At 3.94 V the battery is still far from dead, so it’ll probably stay alive for quite a bit longer…

The sketch running on this node is:

Screen Shot 2011 09 03 at 16 14 28

(looks like I had lost the low-power stuff in the “radioBlip” code, now fixed on GitHub)

Lots of low-power nodes are holding out nicely here at JeeLabs, by the way. There are about half a dozen permanent room nodes, and with the latest roomNode sketch they just keep going. Some are running on 3x AA rechargeable EneLoop batteries, others use 3x AAA with even less juice, and all of them have been running with just a single charge since the beginning of this year.

The bits have moved

In News, Software on Sep 3, 2011 at 00:01

There are some changes planned in how things are going to be done around here. I want to streamline things a bit more, and make it easier for people to get involved.

One of the major changes is to move all JeeLabs open source software to GitHub:

Screen Shot 2011 09 02 at 22 01 27

The main reason for doing this, is that it makes it much easier for anyone to make changes to the code, regardless of whether these are changes for personal use or changes which you’d like to see applied to the JeeLabs codebase.

For the upcoming Arduino IDE 1.0 release (which appears to break lots of 0022 projects), I’ve moved and converted a couple of JeeLabs libraries so far:

  • Ports and RF12 have been merged into a single new library called JeeLib
  • the EtherCard and GLCDlib libraries have been moved without name change
  • all *.pde files have been renamed to *.ino, the new 1.0 IDE filename extension
  • references to WProgram.h have been changed to Arduino.h
  • the return type of the write() virtual function has been adjusted
  • some (char) casts were needed for byte to fix unintended hex conversion

If you run into any other issues while using this code with the new Arduino IDE 1.0beta2, let me know.

So what does all this mean for you, working with the Arduino IDE and these JeeLabs libraries?

Well, first of all: if you’re using IDE 0022, there’s no need to change anything. The new code on GitHub is only for the next IDE release. The subversion repositories and ZIP archives on the libraries page in the Café will remain as is until at least the end of this year.

New development by yours truly will take place on GitHub, however. This applies to all embedded software as well as the JeeMon/JeeRev stuff.

The new JeeLib should make it easier to use Ports and RF12 stuff – just use #include <JeeLib.h>.

Note that you don’t need to sign up with GitHub to view or download any of the JeeLabs software. The code stored there is public, and can be used by anyone. Just follow the zip or tar links in the README section at the bottom of the project pages, or use git to “clone” a repository.

To follow all my changes on GitHub, you can use this RSS feed. To follow just the changes to JeeLib in slightly more detail, this feed should do the trick.

One of the features provided by GitHub is “Issue Tracking”, i.e. the ability to file bugs, comment on them, and see what has been reported and which ones are still open. This too is open to anyone, but you have to be signed up and logged in to GitHub to submit issues or discuss them.

For questions and support, please continue to use the JeeLabs forum as before. But if you’re really pretty sure there’s a bug in there, you’re welcome to use the issue trackers (as you know, Mr Murphy and me tend to sprinkle bugs around from time to time, just to keep y’all sharp and busy ;)

And if you’d like to suggest a change, consider forking the code on GitHub and submitting a “pull request”. This is GitHub-speak for submitting patches. Small changes (even multiple ones) are more likely to go in than one big sweeping request to change everything. I’m open for suggestions (in fact, I’ve got a couple of patches from people still waiting to be applied), but please do keep in mind that code changes often imply doc changes, as well as making sure nothing breaks under various scenarios.

All in all, I hope that GitHub will help us all keep better track of all the latest changes to the software, work together more actively to fix bugs and add more functionality. I haven’t heard of GitHub ever going offline, but if it ever does, I’ll make sure that the latest code is also available from the JeeLabs servers as backup.

Update – Here’s an excellent article on how to collaborate via Git and GitHub.

The Murphy brothers

In Hardware on Sep 2, 2011 at 00:01

Let’s get this out of the way first…

About a week ago, it came to my attention that Mr. Murphy has been having a good time again, this time with the RBBB board by Modern Device.

Turns out that some units were accidentally shipped with a 3.3V regulator instead of a 5V unit:

RBBB regulator mixup

The effect is not dramatic, in the sense that the RBBB will work just fine with either regulator. But as soon as you start hooking up stuff, you may get weird behavior. The worst case would be to hook up stuff with its own 5V power supply, since that means the attached device could potentially put a higher voltage on the IO pins than the 3.3V supply voltage of the ATmega.

So how do we fix this? Well, if you’ve got an RBBB from JeeLabs in the past months (probably anywhere from just before the summer break until now), please check the regulator voltage. Let me know if it’s the wrong one and I’ll send replacement 5V regulators.

(Note that the RBBB’s L4931 and the JeeNode’s MCP1702 regulators have different pinouts!)

As for the current batch of RBBB’s I still have here, the store gets a little bit more complicated: these are all wrapped and sealed by Modern Device, and the bags are too thick to be able to read that lettering on the regulator. I’ve opened up almost a dozen bags and found a few 3.3V units in the older batch, and none in the newer one. There’s not much more I can do than replace those I know about, and repeat that offer to replace any regulators you come across yourself.

FWIW, I suspect that this problem is gone by now.

But there’s a second member in the Murphy family, still dropping by occasionally, it seems…

As you may remember, I switched all SMD-based ATmega328’s over to use OptiBoot a couple of months ago. As it turns out, the standard OptiBoot loader included with the Arduino IDE 0022 isn’t reliable under Windows – my hunch is that there’s some timing problem with the combination OptiBoot <=> FTDI USB interface. It appears to be solved by changing the watchdog timer from 500ms to 1s, see the discussions here and here.

There’s now a new Arduino IDE 1.0beta2 out, and I’m getting some signals that we’re not alone with these OptiBoot troubles, so for now I’ll just stick to that modified 1s OptiBoot code which I’ve been using for several months now, and I’ll hold off switching to OptiBoot on the DIP version of the ATmega328, as used in JeeNode kits. Failing bootstraps are no fun if you don’t have an ISP programmer and no time to try out various tricks to emulate one.

In a nutshell: all SMD chips from JeeLabs use the 1s OptiBoot (which runs at 115,200 baud), while all the DIP chips get pre-loaded with the Duemilanove bootstrap (which runs at 57,600 baud).

I’ll revisit this when the dust settles.

Onwards!

Back!

In Musings on Sep 1, 2011 at 00:01

Well, that was a pretty strange “break”, as far as summers go… mostly cloudy!

For the first half of the summer, I’ve learned to relax and not go online, even when at home. And for the second half, we’ve been going out, mostly short trips around the country – like a few days by train + bike in “Drenthe”:

And a quick trip to Paris:

Paris? Yes, that’s Paris. There’s a vineyard at the Montmartre, not far from the Sacré Coeur.

So is this:

Don’t believe me? Ok, here’s a more traditional image:

(all vacation pictures by Liesbeth, as usual)

Anyway, it’s good to be back. I’m looking forward to smell the solder fumes, burn my fingers, and see chips go up in smoke again – eh, something like that … ;)