Computing stuff tied to the physical world

Archive for February 2013

Dropping below 3.3 V

In AVR, Hardware on Feb 28, 2013 at 00:01

One of the departures from the Arduino world has always been that JeeNodes operate at 3.3 V, whereas the standard Arduino’s all ran at 5 V. Things are changing, with the ARM-based “Arduino Due”, for example, which also runs at 3.3 V. There’s not really a choice: ARM chips all run at 3.3V or less (3.6V max actually, which leaves a nice safety margin).

So what’s this trend towards lower voltages all about, eh?

Well… you may have seen an older post in the Easy Electrons series (whoa, over two years ago.. time flies!). If you think about it for a moment, all the electric energy pumped into a circuits will end up being converted into some other form of energy: a teeny tiny bit of radiated RF energy in the case of a JeeNode, but mostly heat, really.

Odd as it may seem, all that electronic stuff is just a way to turn current into heat. There is a purpose for this, since we’re usually interested in the side effects: some physical sensing and some computation, leading to the information we’re interested in. But the electricity consumption is just an (unavoidable) side-effect.

Whenever we can lower the voltage and/or current consumption, we’ll end up consuming less power as a result (W = I x V). Lowering the voltage on a circuit often leads to a dramatic reduction in power, since part of a circuit is always resistive. Ohm’s law says that voltage = resistance x current, or to put it differently: current = voltage / resistance. So halving the voltage (V) over a resistor also halves the current (I) it draws. The effect: half the voltage leads to a quarter of the power consumption, in purely resistive circuits:

JC's Grid, page 65

Unfortunately, you can’t just use 0.1 V to power a circuit: bipolar semiconductors such as diodes and transistors have a 0.6V threshold voltage, anything below that renders them useless. In the real world, running digital circuits at under 1.0 V is rarely done. Even that 1.8 V level down to which an ATmega and ATtiny can be operated is pushing some limits.

Another reason is that lower voltages make capacitive effects more dominant. The lower you go, the longer it takes to charge or discharge a capacitor. Which explains why the ATmega’s maximum clock speed must drop to 4 MHz with a supply voltage of 1.8 V.

But lowering the clock speed is somewhat self-defeating, as this means that the ATmega and ATtiny have to remain powered-up longer before going back into deep-sleep mode! Lots of trade-offs. Still: a low supply voltage is often a good idea – generally speaking.

Tomorrow, I’ll describe the choices made for the next revision of the JeeNode Micro…

HouseMon 0.5.1

In Software on Feb 27, 2013 at 00:01

It feels a bit odd to give this piece a software a version number, since it’s all so early still, but I’m releasing it as HouseMon 0.5.1 anyway, to set a baseline for future development.

Note that the instructions so far were all about development. To get an idea of the true performance on a Raspberry Pi, you should start up as follows (make sure HouseMon is not running already): cd ~/housemon && SS_ENV=production node app.js – this’ll take a few minutes while SocketStream combines and “minifies” everything the first time around. After that, all web accesses will be a lot snappier than in development mode.

But let’s not get ahead of ourselves… tons of work to do before HM becomes useful!

I suspect that this project will lead to more questions than answers at this stage, but I guess that’s what you get when following the “release early, release often” approach of open source software. There seems to be no other way to go through this than just bite bullet, get a basic release out in the wild, and see how it holds up when installed in more places.

Here’s an attempt to map out what’s in the 0.5.x release:

hm050

The dotted lines indicate that logged and archived data is being saved but not used yet (logged data is the raw incoming text from serial interfaces, etc – archived data is per-sensor information, aggregated hourly).

This diagram is still too complicated for my tastes. Part of the process is getting to grips with complexity: multiple sensor values per reading, “de-multiplexing” different types of packets from the ookRelay, mapping node ID’s to locations, ignoring some repeated data which contains no extra information, and more real-world messiness…

There is currently very little error checking and error messages can (will!) be cryptic.

Which is all a long-winded way of saying that HouseMon is still in its very early stages. Let’s see how it evolves, and please do comment and hack on it if you are interested.

Note that HouseMon is a moving target. The “develop” branch is already moving ahead:

Screen Shot 2013-02-24 at 10.11.57

As you can see, it’s all made up of “briqs” which can be added and enabled as needed.

PS. I’ve added some documentation on how to track changes in the development branch.

Issues on GitHub

In News, Software on Feb 26, 2013 at 00:01

Since doing more and more with GitHub, I’ve been getting increasingly impressed with the features and workflow of that website. One of the things which really turned me around, was its issue tracker. This is already in use on projects such as EtherCard, as you can see:

Screen Shot 2013-02-19 at 22.39.00

The way things are implemented, including mail integration, keyboard shortcuts, and the fact that issue pages update in real time (making them little real-time chat environments) has convinced me that it really is the best place to deal with these things for all JeeLabs projects. This means I’m also not going to enable issue tracking in Redmine after all.

I’m not too concerned about lock-in, because all source code, its history, and even the entire issue tracking dataset is available for download from their site via a REST API.

I’ve also been using GitHub issues quite a bit for HouseMon lately, and have just started using its “milestones” feature, to help organise and plan the pile of work ahead:

Screen Shot 2013-02-19 at 22.20.22

Sooo… if you have a problem with JeeLib, Ethercard, etc. and want to report a bug, feel free to do so on the GitHub site. This doesn’t replace or interfere with the discussion forums on the JeeLabs “Café” in any way, as these continue to be the place for discussion and general questions and support. But for obvious bugs and clear-cut enhancement requests, GitHub is the place to be from now on…

As some people have started to discover, GitHub is also the fastest way to get patches and new features implemented – the mechanism for this is called “forking” and “submitting a pull request” (me pulling your changes into the official libraries, that is).

Here is a nice write-up about the process, written by the developers of “ThinkUp” – but it really applies to any project on GitHub.

As for release versioning, I’ll be using this approach for HouseMon from now on. This means that if you want to follow along and see the latest changes, you need to enter git checkout develop (once). The “master” branch will only update on the next release.

PS. Still struggling with all the terms and “features” of git. Thanks to @tht on the forum, I’m looking into SourceTree (for Mac), which acts as GUI wrapper around git commands, but I can’t say the coin has dropped quite yet. Things like merge vs rebase sound logical, but I’m sure I’ll get it wrong several more times before getting the hang of it…

Combined usage patterns

In Software on Feb 25, 2013 at 00:01

It’s going to be an interesting puzzle to try and deduce what all the different consumers are, just by looking at detailed electricity consumption graphs, such as this one:

Screen Shot 2013-02-18 at 16.03.20

I’ve already determined that the “double dip” patterns, such as the one at 15:41 to 15:52, is the thermostat + boiler “hunting” around to maintain its setpoint temperature.

The trouble of course, is that this whole house measurement approach doesn’t measure invidual appliances, but – as the term says – it can only report the total of all appliance consumption patterns. In this particular graph, you can see a second consumer kicking in around 15:15 and switching off again at 15:56. Fortunately though, this one is very characteristic as well: there’s a motor turning on, causing a brief very sharp blip at the start. It’s the kitchen fridge, in fact – and as the graph shows, it draws about 80 W.

In the general sense, this Knapsack problem is impossible to solve with certainty. But that doesn’t mean we can’t get a good insight of what’s going on around the house anyway.

There are several more known sources and consumption patterns around here, and once HouseMon is a bit further along, I hope to be able to identify them all in the collected data to be able to obtain fairly good estimates of the monthly consumption for each one of them. Also, once these patters are sufficiently accurately known, it should be possible to subtract them from the graph so to speak, and get a better view of the rest of the measurements.

JeeDay => April 20

In Musings on Feb 24, 2013 at 00:01

It’s been four and a half years of fun since I had this crazy idea to start JeeLabs, and it’s been four years also since the JeeNode was born. An excellent reason to celebrate, eh?

Coming April 19th and 20th (Friday evening and Saturday), I’m going to kick off JeeDay:

Meet face-to-face with fellow PhysComp / WSN / JeeStuff enthusiasts and JC + Martyn. Get the latest news, share your ideas and show off your project (or pictures of it). Discussions, presentations, hands-on sessions – it’s all possible, if we organise ourselves and our time appropriately!

The topics we could cover include things like:

  • Wireless Sensor Networks
  • Ultra-low power nodes in the Arduino world
  • Home monitoring and home automation
  • JeeLabs products Q & A
  • Solutions for dealing with AC mains
  • Funky sensors and clever displays
  • How to lower your energy bill
  • Soldering and measurement techniques
  • Hands-on with an oscilloscope
  • Designing and manufacturing PCBs
  • Enclosures, laser-cutting, 3D printing
  • Hack sessions? Debug sessions?
  • Bring and show your projects, especially if in-progress
  • Ideas for future projects and products
  • Presentations, presentations, presentations

Whoa, that list could go on forever… a huge set of topics!

The location will be in Utrecht or in Houten (5 min by train from Utrecht), which is located in the middle of the Netherlands. There are lots of accommodation nearby for those who want to stay overnight. Come and visit the Netherlands, you’ll enjoy it!

We can extend this to Sunday, if I can find a suitable venue and if there is enough interest, although perhaps that’s a bit too ambitious for such a first event.

Fees would be just to cover costs, drinks, etc. Also some sandwiches or pizza to get us through the day. Should all be doable for €15 .. €25.

I have no idea yet how many people would be interested and might be able to come, so I’ve set up a meeting scheduler – if you’re considering participating, please, please, please do add your name and indicate the time range – 10? 20? 50? 100? people – Let’s find out!

Further details will be added to the JeeDay 13.04 wiki page, as preparations progress. The sooner you respond, the more chances that I can figure out a proper venue and how to make it all happen. And… if you have any tips or suggestions, please get in touch now!

It’ll be great to meet face-to-face, it can be informative for all, and it’ll definitely be fun! :)

Small oops on the LED Node v2

In Hardware on Feb 23, 2013 at 00:01

The new LED Node v2 has been re-done slightly, by fixing the labels of the FTDI connector on the back side, but unfortunately Mr. Murphy pulled another one on me by making me ignore another issue at the time these new boards were ordered.

The current release of the LED Node v2 has a “disconnected ground plane”, meaning that some large copper areas on the back are not attached to ground!

The circuit is correct, in the sense that the board will function as is, but for better noise suppression and higher range of the on-board RFM12B radio, it’s best to make a small modification. Sooo… time to get out the hobby knife and start scratching!

Here is the back of the LED Node v2, unmodified (the silk screen may differ slightly):

DSC_4371

What you have to do is scratch off the solder mask, without cutting the trace:

DSC_4374

The result will be a bit of bare copper on both the trace and the ground plane next to it. Then, apply a bit of solder to connect everything together. Note that doing so may require quite a bit of heat, as the ground plane is hard to heat up – just keep your soldering iron on the spot until the solder flows freely:

DSC_4375

That’s it. The ground of the circuit and the ground plane are now electrically connected, at least under the RFM12B module.

While I’m at it, here’s another modification – unlike the previous “mod”, this one should not be done in a standard setup, with 12V feeding the LED strips as well as the LED Node itself. This is a change you need only consider if you are using voltages higher than 12V, i.e. for 24V LED strips, or when running off batteries which go up to 14V or more.

The reason for this change, is that the MCP1702 regulator of the LED Node is rated only up to about 13V. Anything higher will damage it, even if applied briefly – I’ve seen several cases of this in the past.

The solution is to cut the trace between the power connector and the LED Node, but keep in mind that you then have to power the LED Node logic from the FTDI connector or one of the PWR pins on the headers at all times:

DSC_4372

Just use a sharp hobby knife to cut through the trace this time:

DSC_4376

So much for hacking the LED Node!

DIJN.12 – Final checks and unattended use

In Uncategorized on Feb 22, 2013 at 00:01

Welcome to the last instalment of Dive Into JeeNodes. Let’a make an unattended setup!

Everything is working now. The only step that remains is to automate things a bit further, so that the RPi will automatically start up HouseMon when powered up. This turns it into a fire-and-forget system, so that it becomes a permanent service on your LAN.

Auto-startup is convenient, but it means we also have to think a bit about how to upgrade HouseMon. This is where the nodemon utility comes in: it can be used to start up a Node.js application, and restart it whenever certain source files change. This is mostly intended as development tool, but at this stage where HouseMon is still so young and evolving rapidly, it’s actually going to be quite practical – even in an unattended mode.

Install the nodemon package by entering the command: npm install nodemon -g

The “-g” flag causes nodemon to be installed in a central location instead of as part of HouseMon, so that we can type in “nodemon” from the command line.

Now we’re ready to configure an unattended setup. Copy and paste these commands:

    cd ~/housemon
    echo 'cd ~/housemon' >go.sh
    echo 'PATH=/usr/local/bin:$PATH' >>go.sh
    echo 'nodemon >nohup.out 2>nohup.err &' >>go.sh
    chmod +x go.sh
    echo '@reboot ~/housemon/go.sh' | crontab

This creates a “go.sh” script which will be used to start up HouseMon while you’re not around and sets up an entry in the “cron” table which will run that script right after reboot, even when you are not logged in.

Warning: this loses any previous crontab entries, if this is not a fresh Raspbian install.

Reboot now, using sudo reboot to start the ball rolling. After a few minutes, the RPi will be up and running again, and you will be able to visit the HouseMon server via your web browser – no need to log in for that!

Screen Shot 2013-02-10 at 13.57.11

Congratulations: you have created your own personal Wireless Sensor Network, with a JeeNode sending out light readings once a second over wireless, to a JeeLink connected to a stand-alone Raspberry Pi, and via HouseMon running on Node.js, you’re able to watch the current light level in real time, from any web browser with access to your LAN.

Is this a home monitoring / home automation system? Heh.. not quite, but it is definitely an important first step towards such a system. All the foundations are in place, yearning to be filled-in and extended in numerous directions. The load on a Raspberry Pi looks fine:

Screen Shot 2013-02-21 at 15.36.54

This also concludes this initial series of “Dive Into JeeNodes”. The goal was to set up a basic – but fully functional – system, as a baseline for lots and lots of further explorations. As far as I’m concerned, there will be many more posts building upon everything that has been accomplished so far. For now, I’d like to leave this to settle down a bit, and to reconcile loose ends – such as going through these 12 instalments using Windows. Since I don’t use Windows myself, I’m hoping that someone else will chime in with details, so that the exact steps to get going can be documented in a follow-up post or how-to page in the project.

Cheers for now, I hope you’ve enjoyed this “PhysComp+WSN fun pack” DIJN series!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS – I’ve set up an SD card image pre-configured with Raspbian and HouseMon 0.5.1, so if you want to bypass all the setup work, download the hm051.img.gz file (550 MB!), and follow the instructions in DIJN.05 to set up that SD card. Then insert the JeeLink and SD card, and power up the RPi. It’ll start with HouseMon running – including the demo page.

PPS – Another milestone, this is weblog post #1250. Onwards! :)

DIJN.11 – Connect the light sensor

In Uncategorized on Feb 21, 2013 at 00:01

Welcome to the eleventh instalment of Dive Into JeeNodes. Let there be light!

Everything until now was nice, but wasn’t really about sensing the environment. For that, we need to hook up some sensors, obviously. One of the simplest sensors around is the Light Dependent Resistor, or LDR, as it’s usually called. It does exactly what its name says: vary its own internal resistance, depending on the amount of light falling on it.

To read it out with an ATmega, all we need to do is connect the LDR between ground and an analog I/O pin, and enable the ATmega’s internal pull-up resistor for that I/O pin to get a voltage drop over the LDR. Let’s first hook up the LDR:

DSC_4370

As you can see, I’ve connected the LDR to the analog pin of Port 1 on the JeeNode (a JeeNode USB in this case – but any type of JeeNode will do).

The next step is to upload the proper code to the JeeNode, so that it actually does the measurement and transmits the result over wireless. This is a task for the Arduino IDE. We’ve already used it to load the initial test1 sketch, and now we can replace it with test2:

#include <JeeLib.h>

#define LDR 0 // the LDR will be connected to AIO1, i.e. analog 0

void setup () {
  // this is node 1 in net group 100 on the 868 MHz band
  rf12_initialize(1, RF12_868MHZ, 100);

  // need to enable the pull-up to get a voltage drop over the LDR
  pinMode(14+LDR, INPUT_PULLUP);
}

void loop () {
  // measure analog value and convert the 0..1023 result to 255..0
  byte value = 255 - analogRead(LDR) / 4;

  // actual packet send: broadcast to all, current counter, 1 byte long
  rf12_sendNow(0, &value, 1);

  // let one second pass before sending out another packet
  delay(1000);
}

(rf12_sendNow was recently added to JeeLib, make sure you’re using a recent version)

If you have followed along to the letter of the series, then the JeeNode will still be hooked up, and the IDE will still have the proper serial port setting for it. If not, please go back and make sure you can upload a new sketch. It’s a matter of choosing the proper serial port, selecting the examples/DIJN/test2 sketch in the IDE, and clicking on the Upload button. If all went well, you’ll see this:

Screen Shot 2013-02-16 at 11.40.21

That’s it. If you now look at the Demo page in HouseMon again via your browser, you’ll see actual light readings. Put your hand over the LDR and see the value decrease in real-time (within a second, in this case, since that’s how often the JeeNode sends out a new packet).

As you’ll see, the LDR is very sensitive. It has to be pretty dark for the value to drop under 100 (it ranges from 0..255). The range is also not linear or calibrated in any way.

Congratulations… now you have a real WSN! Tomorrow we’ll take care of some loose ends.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.10 – Set up a demo web page

In Uncategorized on Feb 20, 2013 at 00:01

Welcome to the tenth instalment of Dive Into JeeNodes. Let’s show real-time data!

With HouseMon running on the Raspberry Pi, we can start to think how to show incoming data on a web page. The following approach is the most basic one possible: get the received packets into Node.js, i.e. the RPi server, “publish” each through a WebSocket connection to any attached client, and figure out how to get the received value onto the web page.

Let’s start from the back. This is the web page we’ll be setting up:

Screen Shot 2013-02-09 at 12.22.13

Note that I’m running the server on my Mac in these screen shots, hence the “localhost” in the URL, but this will work in exactly the same way with HouseMon running on the RPi. Note also that the following three files are already included in HouseMon – they are just shown to explain what is going on.

That page is produced by a file called client/templates/demo.jade, with this contents:

    .row
      .twelve.columns
        h1 Demo
        h3 {{value}}

Lots of stuff going on behind the scenes, clearly, but this is what Jade looks like with Foundation CSS, and how AngularJS ties a dynamic “value” variable into a web page. It’s sort of like a template (but not quite, due to the two-way binding nature of all this).

The second piece of the puzzle is a file called client/code/modules/demo.coffee:

        module.exports = (ng) ->

          ng.controller 'DemoCtrl', [
            '$scope',
            ($scope) ->
              $scope.$on 'ss-demo', (event, value) ->
                $scope.value = value
          ]

This one is a bit dense: it defines an AngularJS “controller” which will listen to incoming “ss-demo” events, and set the “value” variable we referred to in that Jade file above.

The last code we need, is in a file called briqs/demo.coffee. It defines this “Briq”:

exports.info =
  name: 'demo'
  description: 'This demo briq is used by the "Dive Into JeeNodes" series'
  menus: [
    title: 'Demo'
    controller: 'DemoCtrl'
  ]

state = require '../server/state'
ss = require 'socketstream'

exports.factory = class
  
  constructor: ->
    state.on 'rf12.packet', packetListener
        
  destroy: ->
    state.off 'rf12.packet', packetListener

packetListener = (packet, ainfo) ->
  if packet.id is 1 and packet.group is 100
    value = packet.buffer[1]
    ss.api.publish.all 'ss-demo', value

A “Briq” is something which can be installed at run time in HouseMon. Once installed, that file above will run on the server side. Briqs are a new mechanism I came up with (it’s no rocket science, but at least the name is new), and it’s still very early days. Once I figure out how, briqs will be turned into self-contained directories with the above three source files all in one place. For now, each of those files needs to be placed in a very specific location, for SocketStream to be able to find and use these files.

So how do we “install” the above demo in HouseMon, you may ask…

Easy. Go to the “Admin” page in HouseMon, via the Admin tab in the upper right corner:

Screen Shot 2013-02-09 at 13.18.11

No installed briqs yet, but you can see some briqs which are currently included in HouseMon in the second list. We need to install two briqs: one to set up that demo page, and one to connect to the JeeLink.

Click on the “demo” entry in the list to get this (whoops, I already installed everything):

Screen Shot 2013-02-09 at 12.24.52

Once you’ve clicked on install, you’ll see a new “Demo” tab appear in the upper right. It won’t do much, though, because we’re not receiving any data yet!

Click on the “rf12demo” entry, and fill in the serial port of your JeeLink. On the RPi, it’ll most probably be “/dev/ttyUSB0”, so just enter “ttyUSB0”. Then click install.

Go to the “Demo” page, and if your JeeNode is powered up, you’ll see an updating value.

Bingo – you’re looking at data coming from your Wireless Sensor Network in real-time.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

a s d f – j k l ;

In Musings on Feb 19, 2013 at 00:01

What can I say? Some things are worth the pain and the dip, I guess…

Exactly two weeks ago, I mentioned that one of the things I wanted to try was to learn and touch-type. You know, using this curious contraption without looking at it all the time…

keyboard

Actually, the one I use has an extra key, the “~” / “`” key is in a different place, and the Shift and Return keys have a different size. And to make matters, ehm, more “interesting”, I even followed the suggestion to change the Caps-lock key into an alternate Control key (consider for a moment what that does to typing “TODO” in capitals as a touch-typist!).

Right now, after two weeks, I’m still typing quite a bit slower than I used to, am making tons and tons of mistakes (especially for the hard / far-away keys, and all the tricky Ctrl / Shift combinations), and I’m probably hitting the backspace key almost as often as all the other keys combined to correct my never-ending mistakes.

The worst bit is that this way of typing is still much more distracting than it used to be.

But… you know what? I really, really think it’s going to work and pay off!

Weird as it sounds, and hard as it feels, the letter keys are slowly starting to fall into place. Even after a mere two weeks (I did have some prior experience, to be fair). The thing that is taking most time to adjust to, is not the change in overall speed and effort, but the fact that typing speed is much more variable right now, depending on whether I’m typing letters or hitting other keys, and especially whether the Ctrl and/or Shift keys are involved (not to mention the Mac’s Option and Command keys!). So what seems to be happening is that words already come out faster than I used to type, but everything else really really takes a lot of time and mental effort. In short: punctuation and symbols are a pain, and with programming languages that means lots and lots of keystrokes are… t e d i o u s !

If you write a lot of prose, it’s pretty obvious that touch typing will be a wise investment. But now I’m starting to think that even with something as different as typing code, i.e. a notation full of weird characters, and even typing commands in Vim (where all sorts of key combinations have to be used all the time, and where mistakes are awkward!), the task of learning to type without looking at your fingers or the keyboard is indeed worth the effort.

(Speaking of Vim: it’s easier to type Ctrl-[ than it is to hit the far-away Escape key!)

It’s not even really about looking less up and down, as I thought. It’s about muscle memory, about off-loading an activity to a different part of your brain, and about focus. While writing this weblog post, which is obviously mostly text, I already feel more “relaxed” while typing in this new way. It’s like having an extra assistant – albeit still clumsy ;)

Pretty crazy stuff, if you think about it. I’m not making it up. This IS working!

Up next: the final posts of the Dive Into JeeNodes series – stay tuned…

Who needs a database?

In Software on Feb 18, 2013 at 00:01

Yesterday’s post was about the three (3!) ways HouseMon manages data. Today I’ll describe the initial archive design in HouseMon. Just to reiterate its main properties:

  • archives are redundant – they can be reconstructed from scratch, using the log files
  • data in archives is aggregated, with one data point per hour, and optimised for access
  • each hourly aggregation contains: a count, a sum, a minimum, and a maximum value

The first property implies that the design of this stuff is absolutely non-critical: if a better design comes up later, we can easily migrate HouseMon to it: just implement the new code, and replay all the log files to decode and store all readings in the new format.

The second decision was a bit harder to make, but I’ve chosen to save only hourly values for all data older than 48 hours. That means today and yesterday remain accessible from Redis with every single detail still intact, and for things like weekly, monthly, and yearly graphs it always resolves to hourly values. The simplification over progressive RRD-like aggregation, is that there really are only two types of data, and that the archived data access is instant, based on time range: simple file seeking.

Which brings me to the data format: I’m going to use Plain Old Binary Data Files for all archive data. No database, nothing – although a filesystem is just as much a database as anything, really (as “git” also illustrates).

Note that this doesn’t preclude also storing the data in a “real” database, but as far as HouseMon is concerned, that’s an optional export choice (once the code for it exists).

Here’s a sample “data structure” of the archive on disk:

    archive/
      index.json
      p123/
        p123-1.dat
        p123-2.dat
      p124/
        p124-1.dat

The “index.json” file is a map from parameter names to a unique ID (id’s 1 and 2 are used in this example). The “p123” directory has the data for time slot 123. This is in mult1ples of of 1024 hours, as each data files holds 1024 hourly slots. So “p123” would be unix time 123 x 1024 x 3600 = 453427200 (or “Tue May 15 00:00:00 UTC 1984”, if you want to know).

Each file has 1024 hourly data points and each data point uses 16 bytes, so each data file is 16,384 bytes long. The reason they have been split up is to make varying dataset collections easy to manage, and to allow optional compression. Conceptually, the archive is simply one array per parameter, indexed by the absolute hour since Jan 1st, 1970 at midnight UTC.

The 16 bytes of each data point contain the following information in little-endian order:

  • a 4-byte header
  • the 32-bit sum of all measurement values in this hour
  • the 32-bit minimum value of all measurements in this hour
  • the 32-bit maximum value of all measurements in this hour

The header contains a 12-bit count of all measurement values (i.e. 0..4095). The other 20 bits are reserved for future use (such a few extra bits for the sum, if we ever need ’em).

The average value for an hour is of course easy to calculate from this: the sum divided by the count. Slots with a zero count represent missing values.

Not only are these files trivial to create, read, modify, and write – it’d also be very easy to implement code which does this in any programming language. Simplicity is good!

Note that there are several trade-offs in this design. There can be no more than 4095 measurements per hour, i.e. one per second, for example. And measurements have to be 32-bit signed ints. But I expect that these choices will work out just fine, and as mentioned before: if the format isn’t good enough, we can quickly switch to a better one. I’d rather start out really simple (and fast), than to come up with a sophisticated setup which adds too little value and too much complexity.

As far as access goes, you can see that no searching at all is involved to access any time range for any parameter. Just open a few files, send the extracted data points to the browser (or maybe even entire files), and process the data there. If this becomes I/O bound, then per-file compression could be introduced – readings which are only a small integer or which hardly ever change will compress really well.

Time will tell. Premature optimisation is one of those pitfalls I’d rather avoid for now!

Data, data, data

In Software on Feb 17, 2013 at 00:01

If all good things come in threes, then maybe everything that is a triple is good?

This post is about some choices I’ve just made in HouseMon for the way data is managed, stored, and archived (see? threes!). The data in this case is essentially everything that gets monitored in and around the house. This can then be used for status information, historical charts, statistics, and ultimately also some control based on automated rules.

The measurement “readings” exist in the following three (3!) forms in HouseMon:

  • raw – serial streams and byte packets, as received via interfaces and wireless networks
  • decoded – actual values (e.g “temperature”), as integers with a fixed decimal point
  • formatted – final values as shown on-screen, formatted and with units, i.e. “17.8 °C”

For storage, I’ll be using three (3!) distinct mechanisms: log files, Redis, and archive files.

The first decision was to store everything coming in as raw text in daily log files. With rollover at midnight (UTC), and tagged with the interface and a time stamp in milliseconds. This format has been in use here at JeeLabs since 2008, and has served me really well. Here is an example, taken from the file called “20130211.txt”:

    L 01:29:55.605 usb-AH01A0GD OK 19 115 113 98 25 39 173 123 1
    L 01:29:58.435 usb-AH01A0GD OK 9 4 50 68 235 251 166 232 234 195 72 251 24
    L 01:29:58.435 usb-AH01A0GD DF S 6373 497 68
    L 01:29:59.714 usb-AH01A0GD OK 19 96 13 2 11 2 30 0

Easy to read and search through, but clearly useless for seeing the actual values, since these are the RF12 packets before being decoded. The benefit of this format is precisely that it is as raw as it gets: by storing this on file, I can improve the decoders and fix the inevitable bugs which will crop up from time to time, then simply re-parse the files and run them through the decoders again. Given that the data comes from sketches which change over time, and which can also contain bugs, the mapping of which decoders to apply to which packets is an important one, and is in fact going to depend on the timeline: the same node may have been re-used for a different sketch, with a different packet format over time (has rarely happened here, once a node has been put to permanent use).

In HouseMon, the “logger” briq now ties into serial I/O and creates such daily log files.

The second format is more meaningful. This holds “readings” such as:

    { id: 1, group:5, band: 868, type: 'roomNode', value: 123, time: ... }

… which might represent a 12.3°C reading in the living room, for example.

This is now stored in Redis, using the new “history” briq (a “briq” is simply an installable module in HouseMon). There is one sorted set per parameter, to which new readings are added (i.e. integers) as they come in. To support staged archive storage, the sorted sets are segmented per 32 hours, i.e. there is one sorted set per parameter per 32-hour period. At most two periods are needed to store full details of every reading from at least the past 24 hours. With two periods saved in Redis for each parameter, even a setup with a few hundred parameters will require no more than a few dozen megabytes of RAM. This is essential, given that Redis keeps all its data in memory.

And lastly, there is a process which runs periodically, to move data older than two periods ago into “archival storage”. These are not round-robin databases, in the sense of a circular buffer which gets overwritten as new data comes in and wraps around, but they do use a somewhat similar format on disk. Archival storage can grow infinitely, here I expect to end up with about 50..100 MB per year once all the log files have been re-processed. Less, if compression is used (to be decided once speed trade-offs of the RPi have been measured).

The files produced by the “archive” briq have the following three (3!) properties:

  • archives are redundant – they can be reconstructed from scratch, using the log files
  • data in archives is aggregated, with one data point per hour, and optimised for access
  • each hourly aggregation contains: a count, a sum, a minimum, and a maximum value

I’ll describe the archive design choices and tentative file format in the next post.

DIJN.09 – Install the HouseMon server

In Uncategorized on Feb 16, 2013 at 00:01

Welcome to the ninth instalment of Dive Into JeeNodes. We’re going to install HouseMon!

If you’ve followed all the steps up to this point, then getting HouseMon running will be a piece of cake. We need to do exactly three things:

  1. Download the HouseMon source code
  2. Install the packages used by HouseMon
  3. Launch the HouseMon server

All of this can be accomplished by typing just a few commands, as described here:

    git clone https://git.jeelabs.org/housemon.git
    cd housemon
    git checkout 0.7.x    # <== this is ESSENTIAL
    npm install
    npm start

Here’s what the above will look like in the terminal:

Screen Shot 2013-02-08 at 23.33.55

I haven’t typed the final return yet, as this will generate lots of gibberish while npm does its thing. There is some package compilation involved, which means that the “npm install” step will in fact take a couple of about 10 minutes to complete.

The real fun begins when we enter that final command to start the server: “npm start”. Here’s what you’ll see after a few seconds, while the system initialises itself:

Screen Shot 2013-02-08 at 23.53.47

Now we can use the web browser to connect to HouseMon. In my case, the RPi is running as IP address 192.168.1.127, so I can point my browser to http://192.168.1.127:3333/, and something like this will appear:

Screen Shot 2013-02-08 at 23.59.45

You’ll need to replace the IP address by the one assigned to your own RPi setup, but apart from that it should all work in exactly the same way. HouseMon is now up and running.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS. Note that HouseMon is still evolving a lot (read: things change and things do break, sometimes). So be prepared to update it often, with: cd ~/housemon && git pull

DIJN.08 – Set up Node.js and Redis

In Uncategorized on Feb 15, 2013 at 00:01

Welcome to the eighth instalment of Dive Into JeeNodes. Let’s install Node.js!

(Note: these instructions have been updated to use the Feb 9th Raspbian image)

Now that the basic Wireless Sensor Network is up and running, it’s time to start thinking about the server-side software, which in this series will be HouseMon. For that, we need to get three pieces of software up and running on the RPi: Node.js, Redis, and git.

Those last two are in fact very easy to install on Linux, because they are available via the apt/aptitude package manager. And while we’re at it, let’s update all packages to the latest version. Getting the latest updates is always a good idea on Linux – just type:

    sudo apt-get update && sudo apt-get upgrade

Hit return when asked whether you want to install the updates. The first time around, this is bound to take some 15..30 minutes, as the system may not have been updated for quite some time, but later in it should normally takes no more than a few minutes.

Ok, back to the task at hand, i.e. installing redis and git:

    sudo apt-get install redis-server git

Easy, eh? Welcome to the world of open source Linux software. There are over 35,000 packages available. One way to explore what there is, is to use the menu-based “aptitude” command (type “sudo aptitude” if you’re curious, but be careful what you wish for!).

With Node.js, things are a bit more complicated, because the current version of “Node” available via apt-get is a bit too old for us. Fortunately, we don’t have to go through an actual build from the source code, as I described last month (things move fast in this world!). There is now a pre-compiled version of Node which we can install fairly easily:

  • First, a preparatory step, enter:

    sudo usermod -aG staff pi && sudo reboot
    

    Don’t be alarmed by the reboot, it’s merely a lazy way to log out and back in, which is essential for this one-time configuration change of the RPi. Wait a minute or two, and then you can log back in via ssh, as usual.

  • Now, enter these commands exactly as shown below:

    v=v0.10.21
    cd
    curl http://nodejs.org/dist/$v/node-$v-linux-arm-pi.tar.gz | tar xz
    cp -a node-$v-linux-arm-pi/{bin,lib} /usr/local/
    rm -r node-$v-linux-arm-pi
    npm -v
    

(Note: ignore the “preserving times for …: Operation not permitted” error messages)

That’s it. You can copy and paste these commands line by line, or all in one go – the latter is usually more convenient. The final “npm -v” command verifies that “node” and “npm” work properly by reporting npm’s version number when you hit return, i.e. “1.3.x”.

Lots of powerful tools… all in preparation of being able to install and use HouseMon.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.07 – Attach a JeeLink to the RPi

In Uncategorized on Feb 14, 2013 at 00:01

Welcome to the seventh instalment of Dive Into JeeNodes. Hello RPi, meet the JeeLink!

With all the RPi setup out of the way, it’s time to hook it up to the hardware. Fortunately, most of the essential ingredients are already included in the RPi. The JeeLink is based on an FTDI chip, for which the driver will be auto-loaded by Linux when it’s plugged in.

Proceed as follows:

  1. Plug the JeeLink in one of the two USB ports on the RPi.
  2. Type the following command in Linux: dmesg | tail -20

You’ll get something like this:

Screen Shot 2013-02-10 at 10.41.26

This is the kernel log, reporting that it has recognised the inserted USB device, has loaded a kernel driver for it, and has created a new serial “device” in Linux, called “/dev/ttyUSB0”.

We’re connected!

There are many ways to communicate with the JeeLink. Here’s a quick check that it works:

  • Enter the command: stty 57600 raw -echo </dev/ttyUSB0
  • Then enter the command: cat </dev/ttyUSB0

This sets up the serial ports and shows what is coming in. This is what I get:

Screen Shot 2013-02-10 at 10.50.33

Yippie! The JeeLink has restarted and shown its usual greeting, and is reporting incoming packets. You can hit CTRL+c to stop the process.

The above commands are sufficient to prove that things work, but not convenient. For that, we need to install a serial terminal emulator. There are two we can use, minicom or screen:

  • For minicom, install it with this command: sudo apt-get install minicom
  • To launch minicom, type: minicom -b 57600 -o -D /dev/ttyUSB0
  • For a help page in minicom, type: CTRL+a, then z
  • To exit and close the connection in minicom, type: CTRL+a, then q

I tend to prefer screen, mostly because I’m used to it:

  • For screen, install it with this command: sudo apt-get install screen
  • To launch screen, type: screen /dev/ttyUSB0 57600
  • For a help page in screen, type: CTRL+a, then ?
  • To exit and close the connection in screen, type: CTRL+a, then \

At this point, all the hardware, all the connections, and all the wireless communication are now known to work. All that remains, is to set up the HouseMon server on the RPi and to connect the light sensor.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS. There may be a potential issue with the FTDI driver on RPi. Will need to test this later.

Historical data and graphing

In Uncategorized on Feb 13, 2013 at 00:01

There’s now a first proof-of-concept graph in HouseMon, based on D3 and NVD3:

Screen Shot 2013-02-12 at 10.02.39

Historical data is now stored in Redis. The plan is to keep the last 48 hours in Redis and then move it to some sort of aggregated data files, to complement the daily full log files.

Lots of features in NVD3 which I haven’t explored yet. Out of the box, it supports hovering over a datapoint with a call-out to display the exact value, as you can see.

D3 – and the NVD3 package built on top of it – use SVG, which is supported by at least Safari and Chrome. You can see that they really are drawing with vectors, by zooming in:

Screen Shot 2013-02-11 at 22.18.40

The PDF version that gets generated when printing the page in Safari has several quirks (the shape is drawn as a closed polygon), and in Chrome the graph shows up as a black rectangle, so there is a price to pay for using this sort of cutting-edge technology – but on screen it comes out nicely.

This is the client/templates/graphs.jade file which defines the above page layout:

.row
  .twelve.columns

    .row
      .eight.columns
        h3 Graphs
        #chart
          svg

      .four.columns
        h3: input(type='text',ng-model='query',placeholder='Search...')
        div(ng-repeat='row in status | filter:query | orderBy:"key"')
          input(type='checkbox')
          |  {{row.location}} - {{row.parameter}}
          span(ng-show='row.unit')  ({{row.unit}})

And this is how historical data gets fetched from the server (part of graphs.coffee):


key = 'meterkast - Usage house'
info = $scope.status.find key

promise = rpc.exec 'host.api', 'rawRange', key, -86400000, 0
promise.then (values) ->
  [ offset, pairs ] = values
  if pairs
    series = []
    for i in [0...pairs.length] by 2
      series.push
        x: offset + parseInt pairs[i+1]
        y: adjustValue parseInt(pairs[i]), info
    data = [
      values: series
      key: 'Usage House'
    ]
    nv.addGraph ->
      chart = nv.models.lineChart()
      formatter = d3.time.format '%X'
      chart.xAxis.tickFormat (d) -> formatter new Date (d)
      chart.xAxis.showMaxMin false
      chart.yAxis.showMaxMin false
      d3.select('#chart svg').datum(data).call(chart)

The “promise” you see in this code is a clever mechanism to deal with results which come back asynchronously, i.e. at a later time than when the RPC call returns. Most of the code deals with “massaging” the data, to get it into the proper format for NVD3.

It’s nice to make some visible progress. for a change!

Stay tuned for tomorrow, when the DIJN series resumes…

Update – I’ve switched to Flotr2, smaller and better defaults for the time axis.

Winter

In Musings on Feb 12, 2013 at 00:01

Wow. What a lovely time of year (especially from inside a comfy home):

DSC_4363

The next two pictures were taken by Liesbeth, in a little walk around the neighbourhood:

P2105571

Not much solar power right now (150W), despite the sun – panels are covered up in snow!

P2105554

Regular transmissions to resume tomorrow…

Solar fingernails

In Hardware on Feb 11, 2013 at 00:01

A couple of weeks ago, I described a little indoor solar energy test setup, based on a couple of fingernail-sized CPC1824 solar cell chips, with 0.47 Farad supercaps for storage:

DSC_4351

It has been an interesting setup to keep running for a while, and now that 20 days have passed, it’s sort of intriguing to see how well, or badly, these solar energy cells performed. To summarise, here is what the four different test were about:

  1. One CPC1824, very lightly loaded with 20 MΩ, i.e. a 0.15..0.20 µA current draw
  2. One CPC1824, lightly loaded with 2 MΩ, i.e. a 1.5..2.0 µA current draw
  3. One CPC1824, moderately loaded with 0.2 MΩ, i.e. a 15..20 µA current draw
  4. Two CPC1824’s in parallel, moderate load with 0.2 MΩ, again 15..20 µA current draw

In addition, there is a top-up voltage cycling once per ≈ 2 hours, to prevent any the supercaps from discharging much further than about 2.7V. This is not a realistic situation if this setup were to be used for real, but I wanted to prevent a total discharge of the supercaps, to try and find out whether the supercaps would get better (i.e. less leaky) over time, as the constant charge conditions those membranes full of useable electrons.

Here are the results of 20 days, superimposed one upon the other:

solar

(click two times to see it in full scale – X axis is in minutes since midnight UTC)

  • #1 = blue, #2 = bright green, #3 = orange (dotted), #4 = red (longer dots)
  • the last day is marked in black for all four cases
  • around noon, all cells fall into the shadow, so energy generation almost stops
  • note how only the blue lines stay above 3.0V (except in the first 2..3 days)

This was a wintery mix of sunny and cloudy days, which has a huge effect on the charge / discharge curves. Looks like indoor solar with these tiny cells is not going to work…

One conclusion is that only the lightest load of 0.15..0.20 µA allows the supercap to retain a decent amount of charge through each night. On the other end of the range: with two solar cells in parallel, a 15..20 µA load might be able to stay alive for a few hours each day.

Ehm, well, just maybe – this will really depend on the startup power consumption curves!

DIJN.06 – Boot the Raspberry Pi

In Uncategorized on Feb 10, 2013 at 00:01

Welcome to the sixth instalment of Dive Into JeeNodes. We’re halfway there!

(Note: these instructions have been updated to use the Feb 9th Raspbian image)

This part is probably the most complex piece of the journey called DIJN. Then again, you’re about to set up a self-contained server which can be used for a truly astonishing number of tasks. Linux is everywhere nowadays, and this is probably one of the easiest, lowest-cost, and most risk-free ways to tap into that world which has driven – and evolved alongside – the Open Source Software trend for so long already.

One could probably spend a decade exploring all the software that runs on Linux, and a lot of it may well be affecting your daily life (ever heard of this thing called “internet”?). The good news of course is that you don’t have to spend years, but that whatever interests you is going to be totally in reach, if and when you choose to, eh, dive in…

Let’s get rollin’, shall we?

  1. Insert the SD card you prepared according to yesterday’s post into the RPi. Note that it has to be inserted with the label facing down.
  2. Connect the Ethernet cable on both sides to hook up the RPi to your LAN.
  3. Connect the 5V power adapter to the RPi’s micro-USB port. This will power it up.

LEDs will start flashing, the world will start spinning, and a loud roar will emerge as the positron drive starts up. Oh, wait… wrong movie. Here’s what will really happen:

  • The red LED marked “PWR” will turn on.
  • The other LEDs will blink in a various patterns – after some 30s they’ll look like this:

DSC_4361

This indicates that everything has started up properly. Now it’s time to try and connect to it over Ethernet. How, depends on your platform. On Windows, you can use the putty program. On Mac and Linux, the built-in “ssh” command will be perfect.

Now is the time to shed your Fear Of The Command Line, if it’s all new to you. It’s very simple, really: the GUI we tend to use is like the body of a car, and the command-line its chassis. This DIJN series is about what goes on under the hood. That’s where the “real” stuff happens. There’s no other way – we need to look inside. We need to operate inside, in fact. Soooo… welcome to the machine, I’m sure you’ll like it. It’s fascinating in there :)

But first, we need to solve a small network puzzle: what IP address does the RPi have? There are two ways to find out: 1) have a monitor plugged into the RPi when it boots up, and the value should show up on the screen, or 2) consult the “DHCP client table” in your network router, which is usually the DHCP server for your local network. In my case, the IP address is 192.168.1.127, so that’s what I’ll use in the rest of this post.

The next step is to log into your RPi over the network, using ssh or Putty. Putty has its own GUI way of logging in, but in the Mac’s “Terminal” app or from the Linux command line, this is the way to do so (i.e. user name “pi”, IP address 192.168.1.127):

ssh pi@192.168.1.127

Here’s a transcript of what it will look like the first time around:

Screen Shot 2013-02-20 at 18.37.25

Bingo, you have logged into your RPi – welcome to the Linux command line!

You now need to go through some specific first-time RPi configuration adjustments, as hinted on that last line above. See also an earlier post for some more info (but don’t install Node.js just yet: that will be covered later on):

Screen Shot 2013-02-20 at 18.37.54

It’s a menu-based step, just type sudo raspi-config, and once finished, reboot the system (by typing the command sudo reboot). This last step is essential in this case.

Then wait a minute or two and log back in as before. You’ll again be connected to the Linux “shell”, patiently awaiting your command(s). If it’s all totally new to you, then trying out a few commands to familiarise yourself might be a good idea at this point. See this site for some tips and ideas. As long as you use the “sudo” command with caution, which gives you full “superuser” (admin) rights, there’s not much that can really go wrong. And if things ever get totally out of hand, you can always get back to this starting situation by re-formatting the SD card and starting over.

In the next step, we’ll make the RPi the central node of our Wireless Sensor Network.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.05 – RPi in a nutshell

In Uncategorized on Feb 9, 2013 at 00:01

Welcome to the fifth instalment of Dive Into JeeNodes. Linux, anyone?

(Note: these instructions have been updated to use the Feb 9th Raspbian image)

The Raspberry Pi is a single-board computer running Linux. What sets it apart from the others is its low cost and the fact that it was designed to be educational and hackable:

DSC_4359

It’s really the Arduino story all over again, but for a different audience and with a completely different hardware and software underpinning. As a matter of fact, the “RPi” as it’s often called, can run the Arduino IDE on-board, which means these two platforms are really complementary in actual use.

The performance of this ARM board with on-board GPU is very impressive, but it’s no match for computers with a modern Intel processor (nor does it need to be, at that price!). As a ballpark figure, think of it as 1/10th to 1/20th the performance of a i5 or i7.

One way to use the RPi, is to hook up a USB keyboard and HDMI monitor, but in the context of DIJN, we’re really more interested in using this thing as a little server on the network. Note that a RPi can easily be used for both, and that all you need to handle different scenario’s is a couple of SD cards, each configured with the proper Linux system. You could also install a full GUI and still use it as a server, but the goal here is to start as simple as possible. There are already enough things to get to grips with as it is.

The Raspberry Pi Model B comes with built-in 10/100 MHz wired Ethernet, which is what will be used here. With a few extra steps, it’s also possible to plug in a WiFi dongle and connect to your in-house WLAN.

The main preparation needed, is to create an SD card with a Linux system on it. This is a bit of a chicken-and-egg problem, since the RPi cannot be used for it while it’s not yet up and running. Fortunately, the task is not too complicated, if you have a way to plug an SD card into your main Windows, Mac, or Linux workstation or laptop:

  1. Get a new SD card – 4 .. 8 Gb will be more than enough. There can be significant differences in performance between SD cards from different vendors, so if you’re keen on squeezing the max out of this setup, google for good choices. I’m getting excellent mileage out of Platinum’s 4 GB Class 10 SDHC cards.
  2. Download the latest SD image from here, which is based on Raspbian, i.e. Debian for the RPi. This is a 500 MB download. Unzip to get an SD “image” of just under 2 GB.
  3. Follow the instructions on this page, using the section which applies to your platform. You have to be careful, it’s possible to wipe-out your workstation / laptop hard drive if you make a mistake with disk names!

The end result is a bunch-of-bits, ready to make that little RPi board do fascinating things:

DSC_4360

For the next step, you’re going to need a Raspberry Pi model B, a 5V power adapter with micro USB connector, and an Ethernet cable to hook it all up to your local network.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

PS – The DIJN.12 post has a link to a fully-configured Raspbian + HouseMon SD image.

DIJN.04 – Receive data with a JeeLink

In Uncategorized on Feb 8, 2013 at 00:01

Welcome to the fourth instalment of Dive Into JeeNodes. Let’s go wireless!

The JeeLink is a version of the JeeNode with some extra features, but the main reason I like to use it as central node is that it comes as a little pluggable unit – quite convenient, here it is in action on my MacBook, for example:

DSC_4357

As delivered, JeeLinks and JeeNodes come with the RF12demo sketch pre-installed. This can be used for a variety of tasks, including basic configuration and basic packet reception.

But first, we need to be able to connect to the JeeLink over serial USB. This is similar to the uploading performed in the previous DIJN post:

  1. Plug in the JeeLink – it will briefly flash on its red and green LEDs.
  2. Start the Arduino IDE, and go to the Tools -> Serial Port menu entry. Then, make sure to select the JeeLink, which will show up as a second serial port (assuming you still have the other JeeNode plugged in).
  3. In the IDE, click on the right-most button, the one with the magnifying glass, to bring up the “Serial Monitor”.

That last step will open up a new window, with something like this:

Screen Shot 2013-02-07 at 14.05.31

If you only get gibberish, you need to adjust the baud rate popup in the lower right to 57600. If you don’t get anything: double-check that you selected the correct serial port.

What we need to do next, is to match up the wireless settings to be able to pick up the packets sent from that other JeeNode. There are a few settings:

  • The frequency band must be set to 868 MHz. Note that this may differ from the unit you have, and the frequency used in your part of the world, but all JeeLinks and JeeNodes can operate in any of the 433 / 868 / 915 MHz bands. The only difference is that the power and range will be dramatically reduced in bands for which that wireless module is not optimised. In this case, we don’t care (yet), since the first tests will be across less than 1 meter, which can easily be bridged in any frequency band.
  • The “net group” must match between all transmitters and receivers, to allow them to pick up each other’s packets. In this case, we need to use group “100”.
  • The “node ID” is normally different for each node, so they can tell each other apart. For a central node, it’s usually best to pick the special ID “31”.

The commands to set these parameters can be combined into a single line:

Screen Shot 2013-02-07 at 14.15.51

Once the commands have been entered, you will see a confirmation shown as follows:

 _ i31 g100 @ 868 MHz 

And then, if the sending JeeNode is still powered up, these – once a second:

OK 1 0
OK 1 1
OK 1 2
OK 1 3
OK 1 4
OK 1 5
[etc...]

The “OK” text indicates reception of a valid packet (you may also get some “?” packets, which are usually caused by picking up noise or other 868 MHz transmitters), the first byte indicates that this is a regular packet broadcast coming from node #1, and the second byte is the counter, incremented for each new packet, as you can see in the test1 source code.

Congratulations. You have created your own Wireless Sensor Network. Well, no actual sensors yet, but we’ll get to that soon enough. First, we need to look more deeply into this thing called a “Ra(h)spberry P(h)i” (here’s a little exercise to get you started).

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

Eindhoven

In Musings on Feb 7, 2013 at 00:01

About an hour’s drive south of Houten, in the Netherlands, lies the city of Eindhoven. This is where the Philips family started, and as a technological giant, the company has shaped the city over the decades. Now, it’s also the beating heart of a fascinating new “brainport”, with a University of Technology (TU/e) and a creative Design Academy (DAE).

Dutch Design is known around the world by now.

And then there’s Strijp – an area being developed where the Philips factories once where.

Wow – if only I could be 20 again, and live that life and learn from scratch today!

These trends fascinate me. The idea that future generations will grow up in such a fertile environment, changing everything in the world around them, down to the very fabric of living together and applying creativity and technology in ways never before imagined.

If you have 20 minutes, and turn down your speakers a bit… here’s a video describing the vision and work of the city planners and architects involved in the Strijp project:

 

Wow. Just wow. If this doesn’t lead to an explosion of creative makers and idealists applying technology to shape the world of the(ir) future, then I don’t know what will.

I do have to add that I have a special emotional bond to what is starting to happen there, as our daughter Myra and her boyfriend Pieter, both currently studying at the DAE, are about to settle into one of the first apartments being set up in those re-purposed Philips buildings at the “Torenallee”.

Yeah, ok, so I’m a proud papa, I guess… :)

But hey, whatever. It’s very exciting to see these trends, ehm, materialise in today’s world!

HouseMon resources

In AVR, Hardware, Software, Musings, Linux on Feb 6, 2013 at 00:01

As promised, a long list of resources I’ve found useful while starting off with HouseMon:

JavaScript – The core of what I’m building now is centered entirely around “JS”, the language behind many sites on the web nowadays. There’s no way around it: you have to get to grips with JS first. I spent several hours watching most of the videos on Douglas Crockford’s site. The big drawback is the time it takes…

Best book on the subject, IMO, if you know the basics of JavaScript, is “JavaScript: The Good Parts” by the same author, ISBN 0596517742. Understanding what the essence of a language is, is the fastest way to mastery, and his book does exactly that.

CoffeeScript – It’s just a dialect of JS, really – and the way HouseMon uses it, “CS” automatically gets compiled (more like “expanded”, if you ask me) to JS on the server, by SocketStream.

The most obvious resource, http://coffeescript.org/, is also one of the best ways to understand it. Make sure you are comfortable with JS, even if not in practice, before reading that home page top to bottom. For an intruiging glimpse of how CS code can be documented, see this example from the CS compiler itself (pretty advanced stuff!).

But the impact of CS goes considerably deeper. To understand how Scheme-like functional programming plays a role in CS, there is an entertaining (but fairly advanced) book called CoffeeScript Ristretto by Reginald Braithwaite. I’ve read it front-to-back, and intend to re-read it in its entirety in the coming days. IMO, this is the book that cuts to the core of how functions and objects work together, and how CS lets you write on a high conceptual level. It’s a delightful read, but be prepared to scratch your head at times…

For a much simpler introduction, see The Little Book on CoffeeScript by Alex MacCaw, ISBN 1449321046. Also available on GitHub.

Node.js – I found the Node.js in Action book by Mike Cantelon, TJ Holowaychuk and Nathan Rajlich to be immensely useful, because of how it puts everything in context and introduces all the main concepts and libraries one tends to use in combination with “Node”. It doesn’t hurt that one of the most prolific Node programmers also happens to be one of the authors…

Another useful resource is the API documentation of Node itself.

SocketStream – This is what takes care of client-server communication, deployment, and it comes with many development conveniences and conventions. It’s also the least mature of the bunch, although I’ve not really encountered any problems with it. I expect “SS” to evolve a bit more than the rest, over time.

There’s a “what it is and what it does” type of demo tour, and there is a collection on what I’d call tech notes, describing a wide range of design docs. As with the code, these pages are bound to change and get extended further over time.

Redis – This a little database package which handles a few tasks for HouseMon. I haven’t had to do much to get it going, so the README plus Command Summary were all I’ve needed, for now.

AngularJS – This is the most framework-like component used in HouseMon, by far. It does a lot, but the challenge is to understand how it wants you to do things, and altough “NG” is not really an opinionated piece of software, there is simply no other way to get to grips with it, than to take the dive and learn, learn, learn… Let me just add that I really think it’s worth it – NG can be magic on the client side, and once you get the hang of it, it’s in fact an extremely expressive way to create a responsive app in the browser, IMO.

There’s an elaborate tutorial on the NG site. It covers a lot of ground, and left me a bit overwhelmed – probably because I was trying to learn too much as quickly as possible…

There’s also a video, which gives a very clear idea of NG, what it is, how it is used, etc. Only downside is that it’s over an hour long. Oh, and BTW, the NG FAQ is excellent.

For a broader background on this sort of JS frameworks, see Rich JavaScript Applications by Steven Sanderson. An eye opener, if you’ve not looked into RIA’s before.

Arduino – Does this need any introduction on this weblog? Let me just link to the Reference and the Tutorial here.

JeeNode – Again, not really much point in listing much here, given that this entire weblog is more or less dedicated to that topic. Here’s a big picture and the link to the hardware page, just for completeness.

RF12 – This is the driver used for HopeRF’s wireless radio modules, I’ll just mention the internals weblog posts, and the reference documentation page.

Vim – My editor of choice, lately. After many years of using TextMate (which I still use as code browser), I’ve decided to go back to MacVim, because of the way it can be off-loaded to your spine, so to speak.

There’s a lot of personal preference involved in this type of choice, and there are dozens of blog posts and debates on the web about the pro’s and con’s. This one by Steve Losh sort of matches the process I am going through, in case you’re interested.

Best way to get into vim? Install it, and type “vimtutor“. Best way to learn more? Type “:h<CR>” in vim. Seriously. And don’t try to learn it all at once – the goal is to gradually migrate vim knowledge into your muscle memory. Just learn the base concepts, and if you’re serious about it: learn a few new commands each week. See what sticks.

To get an idea of what’s possible, watch some videos – such as the vim entries on the DAS site by Gary Bernhardt (paid subscription). And while you’re at it: take the opportunity to see what Behaviour Driven Design is like, he has many fascinating videos on the subject.

For a book, I very much recommend Practical Vim by Drew Neil. He covers a wide range of topics, and suggests reading up on them in whatever order is most useful to you.

While learning, this cheatsheet and wallpaper may come in handy.

Raspberry Pi – The little “RPi” Linux board is getting a lot of attention lately. It makes a nice setup for HouseMon. Here are some links for the hardware and the software.

Linux – Getting around on the command line in Linux is also something I get asked about from time to time. This is important when running Linux as a server – the RPi, for example.

I found the linuxcommand.org resource which appears to do a good job of explaining all the basic and intermediate concepts. It’s also available as a book, called “The Linux Command Line” by William E. Shotts, Jr. (PDF).

There… the above list ought to get you a long way with all the technologies I’m currently messing around with. Please feel free to add pointers and tips in the comments, if you think of other resource which can be of use to fellow readers in this context.

Orthogonal learning

In Musings on Feb 5, 2013 at 00:01

It’s been about 2 months now since I switched back to doing mostly software development – in an attempt to get a new home monitoring software going. That does not mean that electronics and hardware design are off the table – far from it, in fact – but the switch has turned out to be essential for me to break out of the box, and find the headroom needed to make things happen.

This is not just a post about HouseMon, however. What I’d like to describe, is the process I went through, and some first experiences and notes.

Picking a new programming language is no big deal, really, but in this case I intend to go in very deep. I don’t just want to program in CoffeeScript, I want to become productive in it. If this will take several months – as I expect – then so be it. As the saying goes:

“If it’s a job’s worth doing, it’s a job worth doing well.”

Note that I didn’t set out to use CoffeeScript, or Node.js, or AngularJS. Half a year ago, I wanted to use ZeroMQ and Lua, in fact. But there were tiny gaps and little doubts in my mind, and I thought it best to let the whole issue rest. Turns out that this strategy works really well (in cases where one can afford the extra delay, of course) – if a decision doesn’t solidify once taken… wait, don’t rush: it may be trying to tell you something!

Of course every switch will be painful, to some degree. The habitual is, by definition, more convenient. A switch to something new and unfamiliar is a step back, sometimes even a huge step back. And at the end of this post I’ll tell you about one more switch which is also excruciatingly painful for me this very moment…

Not all changes turn out well over time. There are several forum choices to prove it, as all long-time readers and participants at JeeLabs know, and which I still feel bad about.

So how does one pick new technology, apart from choosing based on requirements?

My answer to this is now: follow your passion, but don’t let it blind you…

Go for what interests you, go read and surf like crazy, and try the things which you like. Because unconsciously, your mind will make lots of choices for you, leaving you free to apply your reasoning for making trade-offs which need to be made anyway.

The second guideline IMO, is to never go down a rabbit’s hole – if all the decisions you have to make end up becoming inter-dependent, to the point that you have to accept a single package deal, then… stop. Choices are compromises. There is no silver bullet. If something looks too good to be true – then it probably is (hmmm… where have I heard that before?).

I’ve taken quite a few very far-reaching decisions, lately, w.r.t. HouseMon. But watch this:

  • CoffeeScript is not essential – it interoperates 100% with standard JavaScript
  • Node.js is not essential – it could be replaced by Ruby On Rails, or Django
  • SocketStream is not essential – it merely streamlines the use of WebSockets
  • Redis is not essential – it could have been ZeroMQ, or Mosquitto, or some DB
  • AngularJS is not essential – it could be replaced by Knockout, or Backbone.js
  • Safari is not essential – everything will work fine with Chrome or FireFox
  • the Arduino IDE is not essential – underneath it’s all done with avr-gcc
  • the JeeNodes are not essential – there are several µC alternatives out there
  • the RFM12B is not essential – again, there are alternatives one could use
  • the Mac I use is not essential – it’s merely my personal platform choice
  • Vim is not essential – it just happens to be the editor I’ve chosen to work with

The key phrase is “is not essential”, and the key concept is orthogonality – the choices made above are to a large degree (but not 100%) independent of each other!

If any of the above turns out to be a disappointment, I can still get rid of it, without the whole plan unraveling and blowing up in my face (I hope…).

Which brings me to the main point of this post: by having a certain amount of de-coupling in there, something else also becomes quite an important benefit… the task of learning new stuff has a reduced risk! – if any of the above hits a dead end, I will lose my investment in time and energy in part of what I had to go through. But not all.

It’s really the same as in the real world: don’t put all your eggs in the same basket, as the saying goes. If one of ’em crashes and breaks, it won’t be the end of the world that way.

With so many eggs, eh, I mean technology choices, forcing me to re-learn, there is one more which I’ve decided to go for. Long ago, as a kid I took a course in touch typing, but never found the courage to carry it through for programming – all those nasty characters and punctuation marks were (and are) scaring the heck out of me! Well, no more excuses and no more hunt-and-peck: this post was written in touch-typing mode. Let me just say that my hands are hurting like crazy right now, and that it took – a g e s – to write!

Tomorrow, I’ll post an annotated list of pointers, for many of the items listed above, about the information I found which is really helping me forward right now. You may not make all the same choices, but with a bit of luck there will be something in there for everybody.

DIJN.03 – Store code in a JeeNode

In Uncategorized on Feb 4, 2013 at 00:01

Welcome to the third instalment of Dive Into JeeNodes. Some real hardware now!

Both the remote sending side and the more nearby reeiving side of the wireless sensor link are handled by a JeeNode – which is essentially an Arduino-like microcontroller with some support circuitry, plus a low-power wireless radio module.

The JeeLink is a modified version of a JeeNode in a USB-stick form factor and enclosure.

Remote nodes

But let’s start with that remote JeeNode first. What we’re going to set up is a little self-contained unit with a light sensor, a battery pack to power the whole thing, and the proper software pre-loaded onto the ATmega328 microcontroller inside that JeeNode.

The whole point of this “node” is that it’s completely autonomous. You’ll set it up once, take it to the place where you want to perform the light level measurements, hook up the battery pack, and that’s it. Its only task is to periodically measure the current light intensity, and report it over wireless – so that we can pick up the signal and pass it to the Raspberry Pi in a central location in the house, collect the data, tie it into the web server, and make it available on the local Ethernet network for visualisation in the browser.

Let’s set up that node now – except that it won’t be the final measurement code. Let’s start with a simpler goal first: sending out a counter value once a second, to see if we can pick up that value. With a little blink “blip” to let us know that the node is doing its work. But no light sensor for now…

As it so happens, JeeLib now includes an example sketch which does exactly that – see test1.ino on GitHub for the code (make sure you have the latest version of JeeLib!).

Uploading

To get this code into the JeeNode, we need to communicate with it. This is done by connecting the JeeNode to our workstation / laptop – the one where we installed the Arduino IDE. That connection is created via a USB cable and “BUB”, as follows:

dijn03-jeenode-connect

The next step is to tell the IDE which serial port to use to reach our JeeNode:

Go to the “Tools -> Serial Port” menu in the IDE, and select the serial port under which the USB BUB has registered itself. The actual name will depend on the platform (Windows, Mac OSX, or Linux) and in the case of Mac OSX, also on the unique name of that particular USB BUB.

Now, we need to load the proper sketch into the Arduino IDE:

Go to the “File -> Sketchbook -> libraries -> JeeLib -> DIJN -> test1” menu. This will load the test sketch.

To check whether the code is in good shape, click on the leftmost round button with the checkmark in it (or go to the “Sketch => Verify / Compile” menu – same thing).

(rf12_sendNow was recently added to JeeLib, make sure you’re using a recent version)

If all is well, you’ll get one line from the compiler, reporting something like:

Binary sketch size: 3,958 bytes (of a 32,256 byte maximum)

The value may differ a bit, but this indicates that the sketch is ok and can be uploaded.

The last step is where the real upload takes place, and since it includes the compile step, you could in fact have skipped that last instruction above.

Click on the second button from the left (the one with the arrow pointing right).

If all is well, you’ll see some LEDs blink the USB BUB, and when all is done a promising message “Done uploading.” will appear in the IDE (in the middle green bar).

If you get an error, make sure you have selected the proper board type (menu “Tools -> Board -> Arduino Uno”). Still no luck? Check that the USB serial port connection, the cable, the BUB, and the JeeNode are all hooked up as shown above. When in doubt, disconnect and reconnect gently.

Getting through this first upload is an important milestone. There are a few trouble spots, depending on the platform you use. You may have to install the latest FTDI USB driver. You may have to check that the serial USB drivers are present. There is not much generic advice to give here – other than “google for the error message you get”, in the hope that others have run into the same issue and that you’ll find a page with solution or tip.

At this point, you should see a brief LED flash once a second. This indicates that the JeeNode is running the test sketch and sending out its counter packet once a second.

Congratulations, you have set up a live test node for your wireless sensor network!

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.02 – The Arduino IDE

In Uncategorized on Feb 3, 2013 at 00:01

Welcome to the second instalment of Dive Into JeeNodes. Let’s get going!

DIJN differs from a conventional laptop / workstation setup, in that it will be interfacing to the real world via microntrollers running our code. To get that code onto these devices, we need to cross compile the software on our (big) computer running Windows, Mac OSX, or Linux, and then upload it into the permanent memory of a microcontroller chip.

The Arduino has brought this approach to the masses, by providing an open-source IDE to develop, compile, upload, and run the software in the form of embedded software (called “sketches” in Arduino-speak). The success of the Arduino is partly due to the fact that these tools have become easy to install and use.

Before the Arduino era, cross compilers were complex, platform-specific, and too expensive for personal use in small projects. Now they are free, open source, extensively documented, and surrounded by a vibrant community.

Installing the IDE

As a first step, we need to perform that installation. This process is fully described on the Getting Started with Arduino page.

The recommended version is Arduino IDE 1.0.3, at the moment. Note that you don’t have to follow all the steps (1..7 on Windows and Mac OSX are enough), because we’re not going to connect any hardware or upload any software just yet.

Go ahead, perform the installation now. Should take no more than 15 minutes, normally.

If all went well, you should be able to launch the IDE and see something like this after you open the LED blink example sketch with File > Examples > 1.Basics > Blink. :

dijn02-ide103-blink

If you click on the leftmost round button, the one with the checkmark, then the bottom part of the window should also show that same text:

Binary sketch size: ...

Congratulations, you’ve installed a fairly large package and it’s all running as expected.

What is an IDE?

But, wait… what is this thing? And why do we need it?

The “Integrated Development Environment” is where we create the software that ends up running on our microcontroller(s) – be it an Arduino, a JeeNode (which can be used just like an Arduino Uno because it has the same ATmega328 microcontroller), an RBBB, or a range of other similar products.

The Arduino IDE is an editor, a compiler (translating the program source text into some obscure binary code which one particular type of microcontroller understands), an uploader, and a serial terminal interface – all rolled into one app. If your background is PHP or Python, you may want to read a bit more about compiled languages.

The IDE also defines various conventions for how and where to store different pieces of software if we have more than one microcontroller, how to deal with standard libraries obtained from other software developers, and which type of microcontrollers we can handle through this environment.

All of this is essential during development, but once the compiled code has been stored in the microcontroller, it’s not really needed anymore. In day-to-day use, the IDE plays no role – until we decide to make changes, add new features, extend the code… or fix bugs. Then the whole process kicks in again – we load the sketch we were working on back into the IDE, make the changes, compile, upload, try things out, and again: once it works, the IDE can be closed and left alone.

Note that there is no way back: you can’t get code out of a microcontroller and turn it into the source text you wrote. So an essential part of the IDE is to help you manage all those different tasks and versions used for different microcontrollers in your (usually growing) collection of remote sensor nodes. It’s a good idea to think about how you name stuff and where you put it, because you’ll want to find it again whenever you fire up the IDE.

So we write (or use someone else’s) code, as a “sketch” for a particular task, and we upload it into one or more microcontrollers in the different remote nodes. We give these sketches names, and we use the IDE to keep a copy of each of them, to be able to improve / refine / fix / alter the logic of certain nodes at any time in the future.

Some people spend all their time in the IDE (because that’s where lots of code is created). We usually call ’em software developers, although I think IDEalists sounds a lot nicer!

JeeLib

Software can grow to become quite complex. Yet some parts are going to be used over and over again in different sketches. In the case of wireless JeeNodes, the best example is the “RF12” wireless driver, i.e. a piece of software which takes care of sending and receiving wireless data packets.

This sort of “standard” extra functionality is usually made available as software libraries. In the Arduino IDE, such libraries can contain the re-usable code as well as example sketches to try everything out.

The library we want is called “JeeLib”, and is documented here.

We need to add this essential ingredient to our IDE. It has been developed and extended by JeeLabs and other people over the years, and the latest version of this code is available from GitHub – a popular place for a huge range of open source software projects you’re likely to want to use. The GitHub website is enormous, but easy to search and navigate.

Installation requires some care, because we must download the entire set of files and put it in a specific location for the Arduino IDE to be able to find it. There is a good introduction on how to install libraries.

To download the latest version of JeeLib, click on this link, which you can also find as a “ZIP” button in the top left of this page on GitHub. Unpack and rename the resulting folder to just “JeeLib”, then follow the instructions on that how to install page.

To check that things went right, you have to first work around a quirk of the IDE:

Quit the Arduino IDE and then restart it, so that it will pick up all library changes.

Lastly, see if you can locate the following entry in the Arduino IDE’s nested menus:

File -> Sketchbook -> libraries -> JeeLib -> DIJN -> test1

Found it? Terrific. You’ve completed the software installation for embedded Arduino and JeeNode use. As far as the software required for DIJN is concerned, your laptop / workstation is ready to rock.

Onwards!

(This series of posts is also available from the Dive Into JeeNodes page on the Café wiki.)

DIJN.01 – Introduction

In Uncategorized on Feb 2, 2013 at 00:01

Welcome to the first instalment of Dive Into JeeNodes. This one is a bit lengthy, alas…

The purpose of these articles is to introduce you to the world of Physical Computing and Wireless Sensor Networks in an easy to follow way. We will create a low-cost setup to let you track the light level of some spot anywhere in your house and present this information on any computer, tablet, or mobile phone with access to your home network.

What

In more visual terms, this is what we’re aiming for, and what we need to to set up for it:

dijn01-essence.png   dijn01-diagram

Don’t worry too much about the detailed diagram on the right – it’s just to give you an idea of the pieces involved. Here’s a quick rundown of the hardware which will be used:

  • a remote wireless sensor node – which will be a JeeNode SMD
  • the sensor – an LDR, which changes resistance depending on current light levels
  • an USB BUB to load new code into the JeeNode
  • a central wireless node to collect the measurement data – this will be a JeeLink
  • a small Linux “bare board” computer – we’ll use a Raspberry Pi (with Raspbian)
  • your existing local wired (and optionally wireless) network
  • some cables, a USB power supply, batteries, and…
  • your time, your attention, and your enthusiasm, of course!

In case you’re wondering: the Wireless Starter Pack includes much of the above.

None of this is set in stone. It’s possible to replace the Raspberry Pi with another board, or even run that part on your exsiting workstation, laptop, or server. You could use two JeeNodes, or replace the JN SMD + USB BUB by a JeeNode USB, or even create your own variations – but to limit the scope of this DIJN series, the above will be used here.

Note very specifically that you will not need to solder anything for this setup, although it’s very likely that you’ll be itching to do so once the basic system is working – because that’s how you can get more sensors and remote control features in there.

The final result will be something you can leave on, consuming a fraction of a normal PC’s or even laptop’s power – fully unattended, running a dedicated built-in webserver where everything gets configured and where the up-to-date light level “readings” will be available.

It might not sound like much… but don’t jump to conclusions just yet!

Why

This is a truly minimal setup. It ought to be possible to assemble this in say a weekend, even if you’ve never done any hardware or software development before. The total cost should remain well under €150, more or less evenly split between the two JeeNodes + sensor and the Raspberry Pi + power and cabling. It’s still a significant amount of money… for which you could also buy a game console, or go watch all the latest blockbuster movies in 3D.

So what’s the point?

Well, it might not look like much, but this little setup opens up a whole new world and offers access to a surprisingly broad range of state-of-the-art technologies:

  • our world is being filled with sensors at an astonishing pace – just think of all the new “senses” mobile phones have acquired in the last couple of years
  • wireless information exchange is becoming so ubiquitous, it’s not even funny anymore: we live in an always-connected age, and that trend is here to stay
  • software, in the form of built-in intelligence, is everywhere – from the smallest ultra-low power microcontrollers, to tiny functionally-complete computers running Linux
  • hardware is shrinking and spreading everywhere, and more and more based on self-contained extremely sophisticated low-cost electronic chips
  • web technology is advancing faster than ever, covering everything from “big” desktops, to laptops, tablets, and mobile phones

With the DIJN setup presented here, everything just mentioned becomes something you can investigate, explore, tinker with, alter, extend, improve upon, or simply… learn from!

Whether you want to do this out of geek curiosity, for general self-education, to refresh knowledge from the past, to increase your job opportunities, to enhance specific skill sets in… take a deep breath: embedded or web software, microcontroller or Linux hardware, basic electronics, advanced chip capabilities, miniaturisation, ultra-low power design, system integration, wireless networking, communication protocols, C / C++ programming, JavaScript, shell scripting, Linux command-line tools, … everything is in there, in that “little” setup, ready to go where your interests (and your patience) take you.

Not only is everything open source, and hence ready to be explored, it’s also virtually risk free: even if you were to damage something (which is surprisingly hard, with a few simple precautions), it would be in the context of a very limited and low-cost setup.

Just add an extra wireless node, and go solder things together for the first time in your life. Or look into that Linux stuff which drives the Raspberry Pi. Perhaps you’re curious about WebSockets and real-time software. It’s all there. It doesn’t bite. It’s probably easier to understand, because small systems have to be simple by design to maintain their low cost.

Yet at the same time, it’s all really state-of-the-art in many ways. The power levels and battery life achievable with JeeNodes is measured in years. The performance of the Raspberry Pi is such that it can actually drive a display with full-screen HD movies. And the Node.js-based web server technology we’re going to use is at the forefront of what the web has to offer today. This isn’t some mix of technologies cobbled together “just because it works”. Under the hood is what drives our technological world today, and a glimpse of what will be evolving into the technology of tomorrow.

How

The DIJN series of posts is aimed at being totally, completely, fully, truly practical. Every post (ehm, except this one) is about making things work. Concrete steps, describing everything needed to create that final setup. This is going to be as “hands on” as it gets…

Then again, not everything is going to be spelled out in baby steps, and where possible, pointers will be supplied to point to instructions elsewhere, such as how to set up the Arduino IDE, or how to prepare an SD card for use as bootable system in the Raspberry Pi.

The goal is really to reach that finish line, for everyone who’s interested, regardless of specific knowledge. I.e. a working system, spanning a surprinsigly wide range of topics and technologies, but by necessity a very simple system. The idea is that once you have a working setup, you also have the foundation for diving in deep, to explore whatever aspect interests you most, and to alter and extend as much as you like.

This series will not explain how everything works. Nor go into more advanced topics such as implementing ultra-low power modes in the sensor node. Or extending the web server with huge amounts of logic and web page presentation. That’s step 2 (and 3, and 4).

Winding down

This post ended up being much longer than planned. Let’s hope the next ones fare better. There will definitely not be a DIJN post every day – it’ll be spread out over this month, to allow adding some lighter material and other topics from/about JeeLabs. There are only so many hours in a day – and that applies to both reading and writing all this stuff :)

I’m quietly hoping that a few people will try and follow along right away though, and hopefully also comment on where information is incomplete or incorrect. But even if you don’t have the time or opportunity to tag along as this unfolds, please note that this series of posts will be available from the Dive Into JeeNodes page on the Café wiki.

There’s a lot to cover. And I hope there is something in here for everyone. Last but not least: please do comment and make suggestions. That’s how weblogs like this work best.

Onwards!

Dive Into JeeNodes

In AVR, Hardware, Software, Linux on Feb 1, 2013 at 00:01

Welcome to a new series of limited-edition posts from JeeLabs! Read ’em while they last!

Heh… just kidding. They’ll last forever of course, as does everything on this thing called internet. But what I’m going to describe in probably a dozen posts or so is the following:

dijn-essence

Hm, that doesn’t quite explain it, I guess. Let me try again:

JC's Grid, page 63

So this is to announce a new “DIJN” series of weblog posts, describing how to set up your own Wireless Sensor Network with JeeNodes, as well as the infrastructure to report a measured light-level somewhere in your house, in real time. The end result will be fully automated and autonomous – you could take your mobile phone, point it to your web server via WiFi, and see the light level as it is that very moment, adjusting as it changes.

This is a far cry from a full-fledged home monitoring or home automation system, clearly – but on the other hand, it’ll have all the key pieces in place to explore whatever direction interests you: ready-made sensors, DIY sensors, your own firmware on the remote nodes, your own web pages and automation logic on the central server… it’s up to you!

Everything is open source, which in this context matters a lot, because that also means that you can really dive into any aspect of this to learn and explore the truly magical world of Physical Computing, Wireless Sensor Networks, environmental sensing and control, as well as state-of-the art web technologies.

The focus will be on describing every step needed to implement this from scratch. I’ll cover setting up all the necessary software and hardware, in such a way that if you know next to nothing about any of the domains involved, you can still follow along and try it out – whether your background is in software, electronics, wireless, or none of these.

If technology interests you, and if I can bring across even a small fraction of the fun there is in tinkering with this stuff and making new things up as you g(r)o(w) along, then that would be a very nice reward for everyone involved, as far as I’m concerned.

PS. “Dijn” is also old-Dutch for “your” (thy, to be precise). Quite a fitting name in my opinion, as this sort of knowledge is really yours for the taking – if you want it…

PPS. For reference: here is the first post in the series, and here is the overview.