Computing stuff tied to the physical world

Archive for April 2009

Receiving and decoding FS20

In Hardware on Apr 30, 2009 at 00:01

Here is a little kit by ELV which can receive FS20 signals and switch 4 independent open-collector outputs:

4-channel FS20 decoder

This board has an on-board regulator and requires 5..24V DC. Most of the components are SMD’s on the other side, the wire bridges show that this is a single-sided pcb.

Pretty obvious stuff… my reason to buy this was simply to get another 868 MHz receiver module!

RFM12 vs RFM12B

In AVR, Hardware on Apr 29, 2009 at 00:01

Here’s an RFM12 433 MHz module from Pollin, hooked up to an Arduino Duemilanove:

RFM12 (not RFM12B) @ 5V

One major difference between the RFM12 and the RFM12B is that the RFM12 can run at 5V, whereas the maximum operating voltage for the RFM12B is 3.8V (it can withstand up to 6V, which is good to know).

Alas, there seems to be some other difference which eludes me, because the RFM12 hooked up in the above picture only seems to be able to send. The RF12 demo code, which works fine for send and receive on RFM12B’s seems to do something wrong. Same behavior with another RFM12 module – so it looks like this is not due to a broken module. Send works, but nothing is coming in other than occasional garbage data.

The transmit part works fine when sending to an RFM12B, also in OOK mode: the RFM12 successfully controls both 433 and 868 MHz units (KAKU and FS20, respectively). But as packet receiver for other RFM12 or RFM12B modules … no joy so far.

Weird. Tips, anyone? Please let me know.

Update – Thanks to R. Max’s comment below, the problem has been solved: the RFM12 does not support arbitrary 2-byte sync patterns, it has to be 0x2D + 0xD4 – then it works fine!

Update 2 - See also the RFM12 vs RFM12B revisited page for a list of differences.

RF woes – solved!

In Software on Apr 28, 2009 at 00:01

Turns out my troubles with using the 868 MHz OOK radio next to the RFM12B module were caused by a silly software mistake (I mixed up the port assignments). No hardware or RF issues after all.

Latest sample output:

Picture 2.png

(the VOLT and BARO readings are bogus because the hardware is not connected)

As you can see, it’s receiving both packet types now. Still some trickiness with allocating the port signals properly, since some lines do require specific pins: the OOK receiver uses the analog comparator, but the ADC gets used as well, so I’ll need to adjust things a bit to use a pin change interrupt instead (used a crude workaround for now).

So now the basics are there to receive all types of signals with a single JeeHub: packets from other JeeNodes using the RFM12B, an 868 MHz OOK receiver for weather data and the FS20 remotes, and a 433 MHz OOK receiver for picking up KAKU remote commands (and possibly some other cheap weather sensors later).

Added bwired.nl data feed

In Software on Apr 27, 2009 at 00:01

The bwired.nl website has been promoting domotics “by example” for a long time now. Pieter Knuvers, the guy behind it all, added a new service to submit and display data from other fellow energy geeks like me…

The bwired upload service was announced (in Dutch) on Pieter’s weblog.

So I added the bwired.tcl Tcl code to JeeMon to generate the XML input he wants:

Picture 2.png

It’s trivial but way too messy stuff, mainly because I don’t have the proper abstractions in place yet to easily generate all sorts of aggregated data. In this case hourly averages and daily totals.

Results can be viewed here, name “jcw”.

Oh well, good exercise. Pretty much the same logic as for the Pachube feed.

JeeMon self-update mechanism

In Software on Apr 26, 2009 at 00:01

As I’ve mentioned before, JeeMon is a self-contained executable which requires no installation and runs on Windows, Mac OS X, and various Linux systems (including the ARM chip on the MMnet1001 modules used in the JeeHub). There are no special requirements or dependencies, no Java runtime, no .NET, no nothing (note: on Linux you have to check that the standard C++ runtime libraries are present).

The JeeMon builds are here – you only need to fetch one and launch it to use it.

This is based on a generic runtime called Tclkit which contains a full implementation of the Tcl programming language as well as the Metakit transaction-based embedded database. It’s hard to overestimate the amount of functionality included in these 1 .. 2 Mb executables.

The only specific code in JeeMon is the startup logic to turn it into a web-server with home monitoring functionality. It’s all inside one main.tcl file, the essence of which is shown here:

Picture 3.png

The code starts off with a few definitions and then loads the rest of the application. From file if present, else from a file loaded from the internet. The “fetchLatestLibrary” code by default also updates to the latest version, this can be disabled by deleting the “Jee-library.update” file.

By changing one URL shown above, JeeMon can be turned into any other type of Tcl application, because everything the application actually does comes from the downloaded file.

For those who don’t like this automatic mode, there is always the option to download all the pieces and run them as is. Both JeeMon and the Jee-library are based 100% on open source software. JeeMon is implemented in C/C++ and can be compiled from scratch using the standard gcc/automake tools, and Jee-library can be unwrapped to inspect every single bit of it – it is nothing but a packaged collection of plain text files and images. I’d be happy to document this if needed.

I suspect that most people won’t quite appreciate the implications of the way JeeMon is built and packaged until they try it out. And the reality is that most people won’t ever try out this stuff – fine with me. The technology underneath JeeMon is just a set of choices. In the end, the degree to which these choices become invisible in day-to-day use of this system is all that matters. And for a platform-independent modern programming environment, it doesn’t get much more general purpose and deployable than this…

Tracking JeeMon status

In Software on Apr 25, 2009 at 00:01

This is what I see on my screen these days:

Picture 1.png

This text is not inside a window but on the desktop, i.e. it gets covered up by all regular windows (there is always Exposé’s F11 shortcut to uncover it).

The text is barely readable, not only using a tiny font but also using 50% transparency. This is intentional: the text is not there to distract but to offer me a quick way to check on the status of my permanently-running setup on the JeeHub with embedded MMnet1001 Linux module.

It’s all done through a little utility called GeekTool, there are no doubt similar ones for Windows and Linux. Geektool periodically executes some shell commands and shows the output on the desktop.

The first listing displays the last 5 lines of the log output:

ssh propie tail -5 jee_out

The second listing is plain-text output generated by a new web page in JeeMon:

wget -q -O- http://propie/data/current

Adding that page to JeeMon was trivial (data collection details omitted):

Picture 2.png

Now, I can always see the latest readings and the status of the JeeMon server.

P.S. I will be away until May 3rd and won’t be able to respond to emails and comments until then. This weblog will be on auto-pilot for a week, enjoy!

RF12 protocol improvement

In AVR, Software on Apr 24, 2009 at 00:01

A small change was recently made to the RF12 wireless packet driver. This is the structure of a version 1 packet:

Picture 1.png

And this is the new structure in version 2:

Picture 2.png

Small change, big implications: the 16-bit CRC now includes the SYNC2 byte.

The SYNC2 byte is actually abused a bit by the RF12 driver, because it is set to the network group. Since RFM12B modules can look for a two-byte sync pattern with a configurable second byte, this is actually quite useful. By filtering on the network group, the receiver will completely skip packets which do not start with the proper group. This greatly reduces processor load when packets for different groups are being sent around in the same area.

There may still be some false syncs, since the RFM12B continues to look for the 2-byte pattern during the entire transmission, but it reduces overhead nevertheless.

The problem with version 1 packets is that a bit error in the group byte will generate a valid packet for another group. Extending the CRC tail to include the group fixes the problem, since the CRC will no longer be valid (almost never, that is).

This change is incompatible, i.e. packets sent with one version of the driver cannot be received by the other version. To make this less of an issue, the default group has been changed from 0x50 in v1 to 0x51 in v2.

With the new v2 driver, many network groups can coexist in the same area with no interference, other than having to share the total channel bandwidth.

New RF12 driver progress

In Software on Apr 23, 2009 at 00:01

The current RF12 is a bit simplistic, which means you may have to do a lot of work to get things across reliably. I’ve been working on a more sophisticated layer on top of the current RF12 driver code, which supports a reliable streaming mode.

Here’s a basic example which lets two nodes treat the connection as a serial link:

Picture 1.png

Everything typed on one node ends up on the other, in both directions. This is actually pretty hard to get 100% right, because it has to deal with packet loss as well as congestion (when you try to send data faster than the link can currently handle).

A far more interesting use of the streaming mode is to get all sorts of data across, grouped as a command with extra arguments. For example:

Picture 2.png

The “<<” operator pushes arguments into the stream. The send() call then starts the transmission, including an arbitrary 1-byte command code. Note how different data types can be specified for each argument.

On the receiving end, unwrapping is handled with the “>>” operator and a number of variables. The first value pulled out is actually the command code. Again, all conversions are automatically taken care of.

A basic implementation of the packetizing code is now working. But it needs more work to take care of lost and damaged packets and to make sure this gracefully slows down when the channel capacity is reached. Which could be quite low at times due to range or interference issues.

This example also illustrates how tedious it can be to format data as text using the Serial implementation: a 1-line “RF12 >> …” statement does the equivalent of a dozen “Serial.print(…)” calls.

P.S. The MuxShield and front-end are extremely convenient for debugging this sort of code.

RF woes

In Hardware on Apr 22, 2009 at 00:01

I’ve been trying to set up a JeeNode as forwarding gateway from the various 868 MHz OOK signals into the RFM12B packet format. The lower left block in the following diagram, that is:

RF woes.png

But I can’t seem to get it to work…

The 868 MHz OOk receiver and the RFM12B transceiver both work fine. In isolation. But when I run code which enables both receivers, the OOK module stops receiving proper data.

There was a mention somewhere about having to provide a really clean power level to the OOK radio, so I’ll probably try again with a separate regulator for it. Or maybe move the two radio modules further apart, since they are both to the same 868 MHz frequency band.

Can’t think of much else. Hm, again.

More OOK signal decoding

In AVR, Software on Apr 21, 2009 at 00:01

The ES-1 energy sensor described in yesterday’s post is now also recognized by the OOK reception code (the touch panel too, since it uses the FS20 protocol):

Picture 2.png

The main state machine code is still essentially the same:

Picture 1.png

Note that three completely different and independent pulse patterns are being recognized. These recognizers run in parallel, although evidently only at most one will end up with a successful reception at any point in time.

An updated sketch for the Arduino IDE can be found here.

Home control with FS20

In Hardware on Apr 20, 2009 at 00:01

There are many different types of FS20 units (by Conrad / ELV). Two new types of units came in recently. Here’s a power monitor which transmits power consumption via 868 MHz OOK in 5-minute intervals:

FS20 home control

It’s recognized by the CUL receiver, showing readings such as these:

E0205010000000000001B
E0205020000000000001A
E0205030000000000001A
E0205040100010001001A

Hmm, looks like a cumulative power consumption counter… I plugged in a lamp before that last reading. Maybe there’s both cumulative and real-time power use in there. I’m sure the protocol has been analyzed… and indeed, here it is.

One drawback is that the addressing scheme is limited to at most four ES-1 sensors.

And here’s a little touch-sensitive control panel which can send up to 6 different commands and is made for mounting into the wall:

FS20 home control

The panel can differentiate between short (on/off) taps and long (dimmer) taps, so one panel can actually emit up to 12 different signals. Again, this shows up in CUL:

F1000001204
F1000011217
F10000212F9
F1000031203

At € 36 and € 28, respectively, these units are not really affordable enough to sprinkle lots of them all over the house. Then again, maybe a few strategically placed power meters plus the total electricity meter pulses would be enough to figure out most power consumption patterns in the house…

The advantage of these of course is that they are plug-and-play. It ought to be easy to extend the current FS20 receiver code for the RFM12B so they recognize these extra bit patterns.

Production

In AVR, Hardware on Apr 19, 2009 at 00:01

It’s starting to look like a little assembly line over here…

Assembly line...

That’s five fresh JeeNodes … and they all worked right away, yippie!

Mounting the JeeNode Pulse

In AVR, Hardware on Apr 18, 2009 at 00:01

Here’s another experiment in creating a practical Pulse setup, using a fully populated JeeNode:

JeeNode Pulse mounting

The PWR/I2C and ISP/SPI connectors use wire-wrap pins mounted the other way:

JeeNode Pulse mounting

Here are four “peripheral boards”, ready to add all sorts of sensors:

JeeNode Pulse mounting

They look a bit like feet, ha ha:

JeeNode Pulse mounting

The nice bit is that since each of the boards uses one port and all ports are identical, it is very easy to use various sensor configurations.

Here’s the “carrier board”, mounted on part of the enclosure:

JeeNode Pulse mounting

All the pieces fit together nicely, of course:

JeeNode Pulse mounting

And lastly, the protective shell:

JeeNode Pulse mounting

Case closed… heh :)

Nine times faster

In Software on Apr 17, 2009 at 00:01

The graph pages in JeeMon are a lot faster now. Here are the “before” …

Picture 1.png

… and “after” timing statistics:

Picture 2.png

A nine-fold speed increase. Great.

The change is that hourly and daily values have been re-calculated and stored in the database, instead of constantly recalculating these on-the-fly. The last value is not stored, as it might be incomplete and could change when more readings come in.

Keep in mind that these measurements are made with the MMnet100 Linux module, which is a very low-cost / low-power ARM-based system. On a modern PC, the graph pages usually come out instantly, even without storing any condensed hourly or daily datasets.

The other thing to keep in mind is that all JeeMon processing takes place in scripted languages: Tcl on the server and JavaScript in the browser. And that this is handling non-trivial amounts of data, since JeeMon is basing all these graphs on 5-minute values collected for up to an entire year.

In short: 9x faster is good enough, the graph pages now finish loading within two seconds.

It’s nice to see hunches work out in reality…

Update – on the MMnet1001, the UBIFS flash filesystem has compression turned on as default for all files. It would seem to make more sense to keep Metakit database files uncompressed, so I did a “chattr -c Jee-database” and made sure to rewrite the file from scratch with decompressed data. However, it looks like this only shaves another 5% off the graph page times, bringing times down to from 2.0 to 1.9 seconds. Oh well, it was worth a try.

Switching AC

In Hardware on Apr 16, 2009 at 00:01

Got myself a couple of these relays from Pollin, for less than the coin next to it:

Bi-stable relay

They can switch 220V @ 16A, and work at 3V. The specialty is that this relay is bi-stable, IOW there are two 15 Ω coils and you send a short pulse through either of them to switch to the corresponding position. So it takes some power to switch, but after that the relay keeps its last position without power. Like a mechanical switch.

It does take some 200 mA – briefly – to switch, so the power supply has to be up to that.

My plan is to connect this to a JeeNode as follows:

Picture 3.png

Probably using a pair of BC547 transistors and 1N4148 diodes (or maybe a ULN2803, to control up to 4 relays). That way, each port would be able to drive one relay. A pretty cheap solution to control any 220 V appliance.

Haven’t decided yet how far to go into actual AC control, since the other option is to use off-the-shelf RF-controlled switches, such as the Conrad/ELV FS20 868 MHz series or the cheap KAKU 433 MHz switches. But relays can be more secure, if some encryption is added to the RF12 driver.

Fun note: I spent a lot of time as a teenager thinking about building a (feeble) computer with relays. After all, this is essentially a 1-bit memory. I’ve moved on since then … slightly :)

++Year

In News on Apr 15, 2009 at 00:01

Today is my birthday. Regular broadcasts resuming tomorrow.

jc8a.jpg

That’s me, a couple of decades back, around the time when the PDP-8 came out. No chips, no personal computers, no Internet, no Google, no mobile technology.

All prerequisites for the JeeNode :)

Inside JeeMon

In Software on Apr 14, 2009 at 00:01

I thought I’d share a bit here how JeeMon works and what it’s made of.

One of the main goals for JeeMon, is that it should support two modes of operation:

  • running on a personal computer as one of many processes, launched and stopped at will
  • running on a dedicated system which is always running, possibly with quite limited resources

There is not that much difference between these two modes, but it does mean JeeMon should present itself as a webserver and that its processing demands have to be low. Being portable to different platforms also helps.

JeeMon is essentially a webserver plus database, both embedded and combined into a single process. It also contains all the logic to collect readings from – and send commands to – a JeeHub via a USB or serial port.

I picked the technology I know well, some of which I have helped develop or developed myself over the past years:

  • the code is written in two languages: Tcl on the server and JavaScript in the browser
  • the Tcl runtime uses a highly portable and self-contained system called Tclkit
  • the web server is my own design, using a highly modular Mason-like system called Mavrig
  • the embedded database is Metakit, for very compact and efficient storage of data
  • the JavaScript code uses jQuery as library to simplify many tasks
  • interactive plots are generated in the browser using the Flot package
  • the JeeMon application is packaged as a Starpack into a single installation-free runtime
  • similarly, all Tcl code, Javascript code, and website content is wrapped up into a Starkit

[more details below...]

Read the rest of this entry »

JeeHub modularity

In Hardware on Apr 13, 2009 at 00:01

The current JeeHub setup is a bit of a mish-mash:

Picture 1.png

It includes the RFM12B radio of course, to receive data from all JeeNodes. But it also includes a 433 MHz OOK receiver, a BMP085 pressure / temperature sensor, and a DCF77 clock signal receiver.

Trouble is, these things interfere with each other …

The 433 MHz OOK receiver seems to severely degrade RFM12B reception – so I yanked it out. Didn’t have an immediate need to decode KAKU signals anyway.

But worse, the DCF77 receiver also fails to work inside the current JeeHub enclosure. I suspect that either radiated RF power or some messy interference on the power supply levels is preventing it from detecting the 77 kHz signal. I really want DCF77 as reference clock, but I haven’t been able to make any use of it so far.

It looks like there’s too much going on in the current JeeHub prototype, electrically or electro-magnetically.

This is silly – and there’s a very simple way out: move some data sources / sensors to a separate JeeNode. After all, the JeeHub has been doing one thing really well for some time now, i.e. collecting data from remote JeeNodes via the RFM12B:

Picture 2.png

All I need to do is add support for OOK receivers plus BMP085 to the pulse software, and then attach them to one of the remote JeeNodes. Would also make it possible to connect both 433 and 868 MHz OOK receivers.

Haven’t made up my mind about the DCF77 receiver though. Having it tightly integrated with the JeeHub still makes sense – it’s where all the data comes through and gets time-stamped. Maybe the DCF radio will start to work once I disconnect all the other cruft. If not, I’ll either move it physically away from the rest of the JeeHub or reassign it to a remote JeeNode as well.

Surprising, how physical / electrical aspects are starting to impose themselves…

Decoding multiple OOK signals

In AVR, Software on Apr 12, 2009 at 00:01

It is possible to decode multiple signals with a single 868 MHz OOK bit-by-bit receiver, like this JeeNode setup:

868 MHz reception

Here’s the output from a sample run, receiving data from an FS20 remote, a KS300 weather station, and two S300 weather sensors:

Picture 1.png

The trick is to maintain independent state for all the signal patterns. The pulses need not have the same lengths, as long as each recognizer only picks up what it wants and stops decoding when anything unexpected comes in.

Here is an interrupt-driven implementation, using the analog comparator to pick up bit transitions and timer 2 to measure pulse widths:

Picture 2.png

If no transition is seen within 1024 µsec, timer 2 overflows and resets all recognizers to their UNKNOWN state.

A sketch for the Arduino IDE can be found here.

JeeNode Pulse deployment

In Hardware on Apr 11, 2009 at 00:01

This is my plan for sensors in each room in the house:

pulse-deployment1.png

This setup uses a power supply, which is a bit inconvenient. But the ePIR motion sensor draws too much power to run long enough off a battery. And I haven’t yet figured out low-power use of the RFM12B anyway. With everything permanently on, total power consumption should be under 40 mA.

Am still looking for a nice power supply to use, Pollin has several, some as cheap as € 2. Given that the JeeNode has an on-board regulator, any 4..8 V DC supply will do.

The sensors listed above require 2 ports, so there is room for expansion – depending on room requirements:

  • more temp sensors, strung together via 1-wire
  • door and window contact sensors, water level, etc

The JeeNode is placed up against the ceiling to give the motion sensor an optimal view of the room. For really accurate temperature and humidity measurements, it might be preferable to move the SHT11 sensor down a bit – or alternately, to put the JeeNode near the power supply and run a 4-wire cable to the ePIR + light sensors.

Decisions, decisions …

Graph navigation

In Software on Apr 10, 2009 at 00:01

JeeMon lets you zoom in (“drill down”) on the electricity / gas / water consumption data. Here’s how:

Picture 1.png

That’s 4 graphs: 1-day details with 5-minute resolution, 2 weekly summaries, and an entire year.

Picture 2.png

The weekly graphs only differ in the level of detail: on the left is the daily change each hour, whereas the graph on the right shows daily totals.

Clicking on a day in either of the weekly graphs causes the daily graph to display that day. And a click on any week in the year overview switches both weekly graphs to that week.

With two mouse clicks on this page you can display the details of any day in the past year. To easily go back to the initial display, a “Go to Today” button appears once a different graph has been selected (same as a refresh).

Note also that selecting a time range in the daily graph calculates the total consumption in that period, and hovering over a bar in the bar graphs displays the exact value as a tooltip.

No other zooming or panning is available, just these fixed choices. Simple and effective, IMO.

JeeMon demo (alpha)

In News, Software on Apr 9, 2009 at 00:01

Today I’m releasing a first version of the open source JeeMon home monitoring application, along with a demo dataset. Warning: this is an early alpha version – all features and bugs are still likely to change, a lot!

There are builds for Windows (x86), Mac OS X (x86 and ppc), and Linux (x86 + x86_64 + some embedded systems, such as the JeeHub’s MMnet1001 module).

It takes 3 steps to run this demo – there is no installation:

  • Get a runtime for your computer from this area
  • On Unix-like systems, run this: chmod +x JeeMon-*
  • Launch JeeMon

Here is the debug output this generates on my laptop:

Picture 1.png

And here’s a Windows screenshot with similar output:

Picture 1.png

Now point your browser to http://localhost:8888/ to see what JeeMon has to offer.

When started for the first time, JeeMon downloads a few extra files from the internet: “Jee-demodata”, “Jee-library”, and “Jee-library.update” (under 2 Mb total). These files are obtained from public servers at Dropbox, I do not track downloads or statistics.

On every subsequent startup, JeeMon looks for updates and refreshes the “Jee-library” file if there is a new version, so by simply starting JeeMon again a few days later you can track its development progress. To disable automatic updates, delete the “Jee-library.update” file.

I’m releasing this code to give you an early glimpse into the Jee Labs kitchen – and to gauge the interest and figure out how to improve JeeMon. If the demo works for you, great. If it doesn’t, you can either try to figure out what the problem is, or delete all the Jee-* files and try again at a later date. The new mailing list announced yesterday is open to anyone wishing to comment and make suggestions.

Let me reiterate that this JeeMon alpha release is for infinitely curious and technically interested people, not for those looking for a finished home energy monitoring solution. Running this JeeMon demo is bound to raise more questions than I can deal with – but it’s all open source, so feel free to explore this as much as you like.

Whatever. Enjoy …

Update – the demo dataset now contains data since 2009-01-01, i.e. a total of over 3 months of readings.

New "jeelab" mailing list

In News on Apr 8, 2009 at 00:01

For those who want to share their interest / questions / ideas w.r.t. any of the Jee Labs projects, I’ve set up a mailing list at Google Groups:

Picture 2.png

See http://groups.google.com/group/jeelab/about – if you don’t want to sign in to Google to join this list, let me know and I’ll add your email address manually.

Update – the mailing list has been replaced by the Jee Labs discussion forums at http://talk.jeelabs.net/.

It gets a lot worse…

In Software on Apr 7, 2009 at 00:01

Yesterday’s post was about web server performance. Here’s the latest build of JeeMon with the same page:

Picture 1.png

That’s nearly 17 seconds to render the page now, versus 5 yesterday… whoops!

Here’s why I call this major progress anyway:

  • The “electricity.html” page no longer contains plot data values and is half its previous size.
  • All plot values are obtained via Ajax calls, running (mostly) in parallel.
  • This opens up the path to unobtrusive refreshes and automatic plot updates.
  • All plots now contain meaningful data, the previous version used some bogus values.

As you can see in the above graph, over 15 seconds is spent in waiting for the server to return all plot data.

And the reason this now takes so long is very simple: all values are calculated on-the-fly by the server from stored 5-minute values. That’s over 100,000 aggregated values to generate a simple 52-week bar graph!

All this should sort itself out once I start caching hourly and daily results in the database. For now, only the raw readings and the 5-minute aggregated counts get stored. The finecky bit is to make all aggregations / calculations consistent across arbitrary JeeMon restarts.

Here’s the latest version of the generated page:

Picture 2.png

Website debugging tool

In Software on Apr 6, 2009 at 00:01

Just to prove that JeeMon is far from ready for real-world use…

The Safari web browser has excellent tools to figure out where bottlenecks are. Here is an overview of where the loading time is going for the main electricity consumption page, as produced by JeeMon running on the MMnet1001 Linux module. That web page has several detailed graphs on it – here are the stats:

Picture 1.png

As you can see, all the time is consumed by the initial page generation on the server. That’s 4.64 seconds to generate the page – which is totally unacceptable. JeeMon serves that same page 50 times faster when running locally on my Mac laptop (!).

Here are the amounts of data transferred to serve that page:

Picture 2.png

The main page including the plot data is about 5 % of the total size – the other files are cached as static content, which is why they do not increase the transfer time.

There is a lot more information to be gleaned from these measurements, btw.

I have a hunch on where the bottlenecks in JeeMon are, but haven’t looked into it yet. Which is exactly how it should be: aim for solid / correct behavior first and use the proper instruments to measure performance and track the effects of optimizations later (I happen to use Safari, but similar tools exist for other browsers).

Stay tuned for updates, once I get to tweaking this…

JeeMon sneak preview

In Software on Apr 5, 2009 at 00:01

I’ve started work on a basic website design for JeeMon:

pastedGraphic-1.png

Here’s the gas consumption page:

pastedGraphic.png

The same JeeMon code is currently running on the “production” (ahem) JeeHub and on my development Mac.

These are real live webpages, with real (but not yet 100% correct) datasets. Still, this is just a tiny tip of a massive iceberg. Lots and lots of things left to do.

Well, at least it’s a pretty iceberg IMO ;)

Rethinking sensor nodes

In Software on Apr 4, 2009 at 00:01

The current “JeeNode Pulses” used around the house are getting a bit large in terms of code size – about 12 Kb of the available 14 Kb have been used up. A lot of this is due to debugging code, with each node generating a detailed report on its serial port. And then there’s the configuration code: displaying the current configuration on startup and interpreting commands entered on the serial port to change those settings.

Here’s serial output from a simple unit with 3 readings:

Picture 2.png

Most of this code becomes irrelevant once the nodes are installed and working properly, since the only thing that matters are the transmitted wireless packets. It seems a bit wasteful to have each node carry this logic around – and it’s inconvenient because settings can only be inspected and changed through a serial connection. Not so great for “remote” nodes…

A lot of this logic could be centralized, if only remote nodes were able to respond to wireless requests. But this requires some care: better not allow anyone anywhere to re-configure the JeeNodes in your house at will! IOW, we need a secure channel.

The simplest way is probably with a simple encryption protocol such as XXTEA, with a per-node “key” securely installed in EEPROM via the serial port. Once that is in place, we can securely send requests to any node to reconfigure and reboot it.

But why stop there? Why not generalize the entire transfer of state in both directions? What I’m thinking of is the following design:

  • Each node has a unique 128-bit key, known only the central server.
  • The central server can send encrypted configuration data, up to 60 bytes (1 packet).
  • The remote nodes can send readings data (perhaps up to 250 bytes).
  • Both sides keep track of the state of this data and only send differences.
  • The ack contains a CRC of the full state – if it is wrong, the full state will be re-sent.
  • Both the unique key and the configuration data are stored in EEPROM, and get copied to RAM on startup.
  • It might take a few packets to send all readings, but since only differences are sent, often a single packet will suffice.
  • The basic content of a packet would be: OFFSET, BITMAP, DIFFS, HASH.
  • The same mechanism could even be used to re-flash a remote node one day, given a suitable wireless bootstrap.

With such an approach, the code for JeeNode Pulses could become a lot simpler. No need to display text results on the serial port, since the central node has access to all relevant state (including all intermediate values which are not used as final readings). And no need to embed a command interpreter, since the server can produce the exact byte values needed for configuration – with a fancy GUI even, if needed.

The challenge now is to tie things together easily. How to define simple C code on the Pulse which is fully configurable on another system. This is sort of a poor-man’s SNMPhmm, I’ve got some ideas to try out.

Real-time tracking

In Software on Apr 3, 2009 at 00:01

Here’s a real-time graph of data collected by JeeMon:

And a snapshot, in case the above real-time graph isn’t working right now:

Picture 2.png

It’s generated via a new site called pachube – I’ve extended JeeMon to feed that site through simple REST-style HTTP requests. Since the graph only has 15-minute resolution, that’s how often it gets updated.

Here’s another real-time feed, showing our gas consumption in l/hr:

Pachube is still in beta. I’ve only just started looking into it. You can define any number of feeds, each having any number of data values. Not sure how it deals with missing values, nor what to do about data which is collected at different rates. The graphs are somewhat configurable, as you can see – this will no doubt still evolve further.

Pachube can poll a server you designate (“automatic” mode), or you can feed it new data at your own pace (“manual” mode), as is being done here. The advantage of the latter is that it gets through firewalls and avoids the security issues of running a public server.

It’s an interesting approach, apparently pachube aims to become some sort of free exchange for producers and consumers of location-specific real-time data.

I’ll leave this setup running for now, but don’t be surprised if this post stops showing an up-to-date graph, since I might stop this feed one day.

Bye bye readout

In Hardware on Apr 2, 2009 at 00:01

This is what remains when I took the ELV WS300 PC unit apart – the enclosure:

ELV WS300 PC

The electronics, with an ATmega168, a temp/humidity sensor (SHT10?), a barometric pressure sensor, a combined USB interface / battery holder, and a pretty nice custom LCD:

ELV WS300 PC

Oh, and an 868 MHz OOK receiver module:

ELV WS300 PC

This unit includes some dataflash memory to store all received measurements and can be connected to a PC for reading everything out. I took it apart because the Windows-only software really isn’t my cup of tea (a not-so-exciting front end with Postgresql as database backend) and because the 3 AA batteries where running out a bit too quickly (why doesn’t it use USB power when available?). Besides, I really wanted to get a better 868 MHz radio. The module in here seems just right: it can receive all the KS300 and S300 data and it runs at 3V.

Maybe I’ll try my hand at de-soldering one day – the sensors might be worth salvaging as well.

This is getting out of hand

In AVR, Software on Apr 1, 2009 at 00:01

I’ve installed a preliminary JeeNode pulse upstairs, with a few 1-wire temperature sensors to track hot-water and central heating a bit. Here’s the real-time read-out, coming from the JeeHub:

Picture 4.png

These readings are coming from the following sources, mostly via wireless:

  • Electricity / gas meter, from a prototype JeeNode.
  • Weather data from 1 KS300 + 2 S300 commercial sensors.
  • Barometer and battery check, inside the JeeHub.
  • Room data and central heating temperatures, from a JeeNode Pulse
  • Some test values, from a second JeeNode Pulse on my desk.

But this display clearly won’t scale visually. Besides, it’s just a big pile of numbers. I really need to figure out a way to present this information nicely!

It’s fun to watch, seeing values update in real time…