Time to dive in. Let’s create a development setup for JeeMon on Mac OSX, in a new folder called “jee”:

    cd ~/Desktop
    git clone git://git.jeelabs.org/jeerev.git jee
    cd jee
    wget https://jeelabs.org/pub/jeemon/jeemon-macosx.zip
    unzip jeemon-macosx.zip

That gives me the following files:

There is more here than strictly needed for production use – just ignore most of this for now. The main bits are “jeemon” and the “kit/” sub-folder with all the JeeRev code in it.

On Linux, the commands will be almost the same, but you’ll need to get a different JeeMon zip file.

Since I don’t use Windows myself, I’ll have to rely on help / support from others (yes, you!) to get the details right. Thanks to @tankslappa, here’s a first report of an install on XP SP3:

cd %homepath%\desktop
“C:\Program Files\Git\bin\git” clone git://git.jeelabs.org/jeerev.git jee
Cloning into jee…
remote: Counting objects: 1810, done.
remote: Compressing objects: 100% (670/670), done.
remote: Total 1810 (delta 1187), reused 1742 (delta 1119)
Receiving objects: 100% (1810/1810), 1.45 MiB | 87 KiB/s, done.
Resolving deltas: 100% (1187/1187), done.
cd jee
wget https://jeelabs.org/pub/jeemon/jeemon-win.zip

Then unzip using your favorite zip manager, and you should be good to go (I’ll optimize further, one day).

Note: on Mac OSX and Linux, if “.” is not in your path, you’ll need to add it or type “./jeemon” i.s.o. “jeemon” everywhere it is mentioned below.

At this point, JeeMon is ready for use. There are a few built-in commands – here’s a quick sanity check:

    jeemon env general

The output provides some general details about the current runtime environment:

    GENERAL:
           JeeMon = v1.5
          Library = /Users/jcw/Desktop/jee/kit
         Encoding = utf-8
        Directory = /Users/jcw/Desktop/jee
       Executable = /Users/jcw/Desktop/jee/jeemon
      Tcl version = 8.6b1.1

If you got this far, then everything is working as intended. If not: you’ve hit a bug – please get in touch.

But the normal procedure is to simply launch it:

    jeemon

If this is the first time, you’ll get something like this:

    No application startup code found.
    21:22:29.157      app is now running in first-time configuration mode
    21:22:29.189      web server starting on http://127.0.0.1:8181/

Where “first-time configuration mode” means that JeeMon didn’t find a “main.tcl” rig, which is normally used to start up. To get past this step, you need to point your web browser to the indicated URL, which’ll show this page:

There’s not much point in selecting anything else but “YES”at this stage. This creates a 3-line “main.tcl” file:

    # default JeeMon startup file
    Log main.tcl {in [pwd]}
    Jm needs HomeApp

From now on, JeeMon will start up using the built-in “HomeApp” rig when there are no command-line args.

The next steps depend on what you’re after – you can either dive into Tcl programming and explore how JeeRev (i.e. the kit/ area) is structured, or you can try out some examples and get a more top-down impression of it all.

To explore Tcl, the thing to keep in mind is that JeeMon will act as a standard no-frills Tcl 8.6 programming environment when launched with a source file as argument (just like tclsh and tclkit). Here’s how to make a hello-world demo – create a file called “hello.tcl” with your favorite text editor and put the following code in it:

    puts "hello, world"

Then run that code using the command “jeemon hello.tcl“. You can probably guess what it does…

If you want to use a convenient interactive shell with scrollback, history, and debugging support, download the TkCon script and launch it using “jeemon tkcon.tcl” – this uses Tk to create a mini IDE.

For Tcl manuals, books, and demos, see http://www.tcl.tk/. To really dive in, check out this tutorial or this.

But chances are that you just want to get an idea of what JeeMon does in the context of Physical Computing, House Monitoring, or Home Automation. I’ll describe just two simple examples here, and will point to the JeeMon homepage for the rest. Note that everything described below is really part of JeeRev, i.e. the standard library used by JeeMon – or to put it differently, by the source files located inside the “kit/” folder.

First, let’s launch a trivial web server (you can see the code at examples/hello-web/main.tcl):

    jeemon examples/hello-web/

You’ll see a log output line, similar to this:

    18:23:46.744      web server starting on http://127.0.0.1:8181/

JeeMon is now running in async event mode, and keeps running until you force it to stop (with ^C, kill, whatever). Leave it running and go to the indicated URL in your browser. You’ll be greeted with a web page generated by JeeMon. Hit refresh to convince yourself that it really works.

Now quit JeeMon and check out the files in the “examples/hello-rf12/” folder. This example connects to a JeeNode/JeeLink running the RF12demo sketch, and displays incoming packets via its web server. But before launching JeeMon, you have to tell it what serial interface (COM or tty port) to use: copy “config-sample.txt” to “config.txt” and edit it to match your setup. Make sure the JeeNode/JeeLink is plugged in, then start JeeMon:

    jeemon examples/hello-rf12/

The output will look something like this:

    18:37:57.166      web server starting on http://127.0.0.1:8181/
    18:37:58.154    rf12? [RF12demo.8] A i1* g5 @ 868 MHz 
    [...]

And two dozen more lines which you can ignore. Now go to that same URL again with your web browser. If your config file defines the proper net group and you have some JeeNodes sending out packets in that net group, you’ll see them in your browser after a while. This is what I got within a few minutes, here at JeeLabs:

    #1 18:42:21 - OK 19 186 200 8 1 226 2 32 0 0 213 81 33
    #2 18:42:31 - OK 19 186 200 8 1 226 2 32 0 0 213 81 33
    #3 18:42:44 - OK 3 132 43 9 0
    #4 18:42:51 - OK 19 186 200 8 1 226 2 32 0 0 213 81 33
    #5 18:43:11 - OK 19 186 200 8 1 226 2 32 0 0 213 81 33
    #6 18:43:21 - OK 19 186 200 8 1 226 2 32 0 0 213 81 33
    #7 18:43:29 - OK 6 1 95 212 0
    [...]

Now might be a good time to look at the code in examples/hello-rf12/main.tcl – to see how it all works.

Wanna experiment? Easy: quit JeeMon, create a new directory “blah“, and copy the files from the example to it. Edit them as you like, then start JeeMon using “jeemon blah“. Go ahead, try it! – the worst that can happen is that you get error messages. Usually, error tracebacks will refer to JeeRev files in the “kit/” folder.

This concludes my intro for those who want to get their feet wet with JeeMon. We’ve only scratched the surface!

I’ll work out more examples on the JeeRev page as I start building up my own setup here at JeeLabs. If you want to try things and run into trouble (there will be rough edges!) – here are two places to post problems and bugs:

• the JeeRev issue tracker is for most issues, really – when in doubt, post your issues here
• the JeeMon issue tracker is only for platform-specific startup bugs in the JeeMon executable itself

Feel free to ask and post anything else on the forum. If it gets out of hand, I’ll set up a separate area for JeeMon.

A quick note about the Shop: I’m running into some shortages across the board (such as wire jumpers), which also affect the JX and WSP packs. All the missing bits are on order, but some of this might not get in before early December. My apologies for the delay and inconvenience! -jcw

Here’s another post about the JeeMonBusRev trilogy, i.e. the JeeMon project I started a few years ago.

First, let me go into why there are still two pieces and two names left in this project:

• JeeMon is two things at once: the name of the entire system, and the name of the executable file to start it up. This exe file contains a general-purpose set of tools – it has no clue about Physical Computing or Home Automation. It could as well be used to build an accounting system or a CAD system… if you wanted to.

• JeeRev is the name of the software which extends the JeeMon executable for domain specific use, i.e. its run time library. This is where the code lives which does know very specifically about Physical Computing and Home Automation. This is also where all the (software development) action is.

Do these names stand for anything? Yeah, probably – but I’ve stopped agonizing about cool acronyms.

JeeRev is used by JeeMon in one of two ways:

• Development mode – as a “kit/” folder full of source files, which JeeMon will automatically detect and use. This is what you get when you grab the code from GitHub. Easy to browse, tweak, and keep up-to-date.

• Production mode – as a single file called “jeemon-rev“. Normally, JeeMon will automatically download the latest official version of that file if it doesn’t exist yet (if there’s no “kit/” folder to put it in development mode). End users (i.e. normal people!) don’t care about JeeRev. They just download and launch JeeMon.

The precise startup process is fully configurable – it’s documented in JeeMon’s README file on GitHub.

I suggest forgetting about production mode for now. It’s not going to matter for quite some time, even though that aspect has had a major effect on the “structural design” of JeeMon. For now, everything will be happening in development mode, with JeeRev fully exposed – like a car kept in the garage with the hood open, so to speak.

To expose what’s in JeeRev, I have to describe one more mechanism – which is the way the different pieces of an application work together in JeeMon. Since I don’t know how familiar you are with recent implementations of Tcl, I’ll start from scratch and go through the basics – it’ll be easy to follow if you know some programming language:

Commands – everything in Tcl is based on commands. The line < puts "hello, world" > (without the <>’s) is treated as a call to the command puts with one argument. The difference with older versions of Tcl, is that commands can also be namespaces (it’s actually the other way around, and they’re called “ensembles”, but let’s not be picky for now). That means that a command such as the following can mean different things:

    Interfaces serial connect $device 57600

The simplest case would be that there is a command called “Interfaces” which gets called with 4 arguments. So you might expect to see a command definition somewhere in the code, which looks like this:

    proc Interfaces {type action device baudrate} {
      ...
    }

But there is another interpretation in Tcl 8.6 (and 8.5): it can also be used to call a “serial” command defined in the “Interfaces” namespace. And in this particular case, it’s in fact nested: a command called “connect” defined in the “serial” namespace, which in turn is defined in the “Interfaces” namespace. Perhaps like this:

    namespace eval Interfaces {
      ...
      namespace eval serial {
        ...
        proc connect {device baudrate} {
          ...
        }
        ...
      }
      ...
    }

Or, more succinctly:

    proc Interfaces::serial::connect {device baudrate} {
      ...
    }

This is like writing “Interfaces.serial.connect(…)” in Python, Ruby, or Lua. But not quite – it’s slightly more general, in that callers don’t need to know how code is structured to be able to perform certain operations. It also allows you to alter the internal organization of the code later, without having to change all the calls themselves. But it can also be a bit more tricky, because you can’t tell from the appearance of a call where the actual code resides – you have to follow (or know) the chain in the code.

This is a fairly important concept in Tcl. The command “a b c d e” can lead to very different code paths (even in the same app, if things are added or re-defined at run-time), depending on whether and how “a”, “b”, etc are defined. At some point, the command name lookup “stops” and the arguments “begin”.

Apologies for the lengthy exposé. I really had to do this to introduce the next concept, which is…

Rigs – this is a mechanism I’ve been refining for the past few years now, which unifies the above nesting mechanism with the way code is arranged in source files. To continue the Interfaces example: there’s a file in JeeRev called serial.tcl, and inside is – as you would expect – a “proc connect …” definition.

The point is: that “serial.tcl” source file is located in a folder called “Interfaces”. So the nesting of command detail is reflected in the way files and folders are organized inside the “kit/” area, i.e. JeeRev.

If you’re familiar with Python, you’ll recognize the module mechanism. Yep – “rigs” are similar (not identical).

Some rig conventions: 1) definitions starting with lowercase are public (i.e. “serial” and “connect”), whereas everything else is considered private to that rig, 2) “child” rigs can access the definitions in their “parent” rigs, and 3) if you don’t like the way a built-in rig works, you can supply your own and completely override it.

If you look at this in objected-oriented terms: rigs are essentially singletons, like Python / Ruby / Lua modules. Speaking of objects: the result of the “Interfaces serial connect …” call is a connection object – Tcl 8.6 finally includes a single OO framework (there used to be tons!), so things are becoming a lot more uniform in Tcl-land.

Thanks for reading this far, even though I haven’t said anything about what’s inside JeeRev yet.

But with this out of the way, that’s easy: JeeRev consists of a collection of rigs, implementing all sorts of features, such as serial communication (serial), a web server (Webserver), logging (Log), config files (Config), and the very important “state manager” (State) which provides an event-based publish / subscribe mechanism for representing the state of sensors and actuators around the house. There is a rig which manages automatic rig loading for JeeMon (Jm), and one which has some utility code I didn’t know where else to put (Ju).

There’s a rig called “app”, which orchestrates startup, general flow of events, and application-wide event “hooks”. This hook mechanism is similar to the Drupal CMS (written in PHP) – because good ideas deserve to prosper.

And lastly, if you supply a rig called “main”, then this will become the first point where your code can take control.

I can’t go into much more detail in this post, but I hope it gives an idea of the basic design of JeeMon + JeeRev.

Tomorrow, I’ll close this series with details about getting started with JeeMon – for those who care and dare!

So much for the big picture – but what’s this JeeMon?

JeeMon ties stuff together – it can talk to serial and network interfaces using event-driven asynchronous I/O. This means that it can talk to lots of interfaces at the same time (like what the Twisted framework does in Python, but more deeply integrated into the system).

JeeMon collects data – it includes a column-oriented embedded and transactional database. This means that it can store lots of measurement data in a very compact format, using variable int sizes from 1 to 64 bits (usually an order of magnitude smaller than SQL databases, which add extra overhead due to their generality and are less suitable for repetitive time-series data).

JeeMon talks to browsers – it has an efficient embedded co-routine based web-server. This means that it can be used with web browsers, delivering rich user interfaces based on JavaScript, HTML 5, CSS 3, Ajax, and SSE. By pushing the logic to the browser, even a low-end JeeMon server can create a snappy real-time UI.

JeeMon likes to network – with the built-in event-based background network support, it’s easy to connect to (or provide) a range of network services. This lets you do remote debugging and create distributed systems based on JeeMon, as well as extract or feed information to other network-based systems.

JeeMon has a pretty face – on desktop machines running Windows, Mac OSX, or Linux, the “Tk” toolkit is included to quickly create a local user interface which does not require a browser. This means that you can create distinctive front panels showing real-time data or perhaps extensive debugging information.

JeeMon is dynamic – it is based on the 100% open source Tcl scripted programming language, which is used by several Fortune 100 companies (if that matters to you). This means you can work in a mature and actively evolving programming environment, which gets away from the usual edit-compile-run cycle. Making changes in a running installation means you don’t have to always stop / restart the system during development.

JeeMon is general-purpose – the structure of JeeMon is such that it can be used for a wide range of tasks, from quick command-line one-offs to elaborate long-running systems running in the background. This is possible because all functionality is provided as modules (“rigs”) which are loaded and activated on-demand.

Technically, JeeMon consists of the following parts:

• the Tcl programming language implementation
• (on desktop versions) the Tk graphical user interface
• support for serial interfaces, sockets, and threads
• the Metakit embedded database (by yours truly)
• co-routine based wibble web server, by Andy Goth
• upgrade mechanism using Starkits (by yours truly)
• domain-specific code, collectively called JeeRev

That last item is the only part in this whole equation which is in flux. It creates a context aimed specifically at environmental monitoring and home automation. This is the part where I keep saying: Wait, it’s not ready yet!

Tomorrow, I’ll describe what’s inside this “JeeRev” …

For several years now, I’ve been working on a software project on the side, called JeeMon. Then something called JeeBus was created, and then JeeRev came along.

That’s a lotta names for a pile of unfinished code!

I’d like to explain what it’s all about, and where I want to take this (looking back is not so interesting or useful).

The Big Picture ™

With monitoring and control in the home and <cough> Home Automation </cough>, the goal is quite simple, really: to figure out what’s going on in the house and to make things happen based on events and rules.

In a nutshell: I want energy consumption graphs, and curtains which close automatically when it gets dark.

There are no doubt roughly one million different ways to do this, and ten million implementations out there already. JeeMon adds one more to the mix. Because I’ve looked at them all, and none of the existing ones suit me. Call it “Not Invented At JeeLabs” if you like, but I think I can help design / build a better system (please remind me to define “better” in a future weblog post, some day).

I’ve pondered for some time on how to present the big picture. Threw out all my initial attempts. This says it all:

Let’s call the yellow box… JeeMon ! Forget about JeeBus. Forget JeeRev (it’s still there, under the hood).

JeeMon is the always-on software “switchboard” which talks to all this hardware. It can run on a dedicated little server box or on the router. The devices can be connected via serial, USB, Ethernet, WiFi, FSK, OOK, whatever.

Functionality & Features

The primary tasks of JeeMon are to …

1. track the state of sensors around the house
2. display this real-time state via a built-in web server
3. interface to remotely operated devices around the house
4. present one or more web pages to control these devices
5. act as a framework which is easy to install, configure, use, and extend

Everything is highly modular, JeeMon is based on drivers and other extensions – collectively called “features”.

Some other tasks which are a natural fit for JeeMon:

• configuration of each sensor, device, and interface involved
• collecting and saving the full history of all sensor data and control actions
• presenting a drill-down interface to all that historical data
• managing simple rules to automatically make things happen
• access control, e.g. to prevent “junior” from wreaking havoc

That’s about it. It’s that simple, and it needs to be built. JeeMon has been “work in progress” for far too long.

Ok, I must admit that JeeMon has been a bit too ambitious in its original inception. It works quite nicely here at Jee Labs, but there are just too many hoops you have to jump through to make it happen on your own.

I’ll first explain why things are the way they are, and how that is supposed to work, and then I’ll present a different setup for the OOK Scope which jettisons all that machinery and lets you use the OOK Scope with nothing but a single source file and JeeMon.

The idea behind the “Serial” and “JeeSketch” rigs (code modules) in JeeMon, is that it should be possible to respond to changes in interfaces dynamically. So there’s a way to scan for USB interfaces periodically:

Serial periodicScan <cmd>

This will compare the list of USB devices it sees with the list it saw last time (once every 5 seconds by default). And then call the specified <cmd> whenever an interface is added or has gone away. Only FTDI interfaces are detected, by the way.

The next step is to decide what to do when a new USB device is attached. I’ve been using a convention for some time now, whereby every sketch starts by sending out its own name, with optional version and configuration details. For example, RF12demo will send out a string like this to the USB connection when it starts up:

[RF12demo.6] A i1 g5 @ 868 MHz

The trick is to wait for such a string when JeeMon detects the device and opens it. This is handled by the “JeeSketch” rig. Once the sketch type is known, JeeMon then tries to locate a “host.tcl” driver for it. It does this by looking in a directory, as configured at start up. I’ve been placing all my drivers in a directory called “sketches”, so my startup includes the following line:

JeeSketch register ./sketches

When RF12demo connects, JeeMon checks whether “./sketches/RF12demo/host.tcl” exists, and runs it.

Similarly, when plugging in a JeeNode running the OOK Scope sketch, it announces itself as:

[ookScope]

The script “./sketches/ookScope/host.tcl” then gets started, it creates a GUI window, takes over the communication link to process all incoming (binary) data bytes, and does its pulse histogram thing.

So far so good. It’s a nicely modular mechanism. I can add a new sketch, i.e. a “blah.pde” file for use in a JeeNode, and add a matching “host.tcl” script alongside to deal with the communication in any way it likes. Then, all I need to do is plug the device in and everything starts up. With a bit of care, everything is also shut down and cleaned up again when the device is removed.

Unfortunately, that’s not the end of the story. One of the important devices I want to support is a JeeNode or JeeLink running RF12demo. But how can JeeMon deal with remote nodes? After all, all they do is send packets to the central device. Each of these nodes will be running a sketch, and not all of them are necessarily the same. So we either need some sort of auto-discovery or some central configuration file. For a first implementation, I decided to use a configuration file to try and keep things, eh, simple. Which is why my startup code also contains these two commands:

set appDir [file dir [dict get [info frame 0] file]]
Config setup $appDir/config.txt

That’s a tricky way of making sure that the “config.txt” file will be picked up from the same directory as the source code, i.e. the “application.tcl” file.

I’ll refrain from describing the config.txt file in full detail. Let me just include an example which I’ve been using around here:

As with any such type of “registry”, you can see lots of little config details, for use in different modules and parts of the code. Even some obsolete stuff, in fact.

Does it work? Oh, sure, it works great and it’s very easy to extend for new modules and usage scenarios. Even node discovery works nicely, both coming on-line and dropping off-line, as seen in the voltmeter demo.

But there’s a problem with what I’ve described so far…

… there’s too much rope – to hang yourself. It’s brittle, it needs lots of documentation to use this stuff (unless you’re willing to dive into the JeeMon Tcl code), and it’s just too much trouble if you want to do something simple with JeeMon, like run the OOK Scope and nothing else. The entry curve is way too steep.

I can’t say I’ve figured out an alternative. There is a lot of ground to cover, and it is fairly hard to implement a system which dynamically adapts to interfaces getting plugged in and nodes coming (wirelessly) online.

But at the end of the day, all that extra bagage is unnecessary for simple cases.

Fortunately, JeeMon is flexible enough to adapt in any way I want. I don’t have to use any of the above machinery. So here’s the OOK Scope logic as a single “application.tcl” file. No scanning, no config, nothing:

The full code is available here. To run this version of the OOK Scope, download that file, make sure it is called “application.tcl”, and place it next to your JeeMon executable. Then launch JeeMon. Just make sure to have the JeeNode running ookScope.pde plugged in.

If more than one FTDI interface is found, you will be asked to pick one:

That’s it: ookScope.pde, application.tcl, plus JeeMon – should work on Windows, Mac OS X, and Linux.

Yesterday‘s post made it possible to send a message to a remote LCD screen. The remote node is set to constantly receive packets, and once a valid packet comes in, it displays the text.

There is no acknowledgement, so the sender does not know whether the remote node actually did receive the message, nor even whether it is turned on.

This can be solved by “turning the protocol around”: instead of sending packets to a remote receiver, the remote node periodically polls a central node to see if there is new data. The communication becomes bi-directional.

Let’s first change the code in the remote node to do this periodic polling (code here):

I’m using the “easy transmission” calls in the RF12 driver. They are used in a somewhat unusual way: by sending out empty packets, and then waiting for a valid acknowledgement packet to come in. This is done by checking the return value of rf12_easyPoll() – it will be 1 when an ack has been received.

But now there’s a problem. The JeeLink will immediately send an ACK back when it gets the data:

That’s not what we want, because the acknowledgement will contain no message text.

Instead, the RF12demo needs to be configured to use “collect” mode instead of “respond” mode. That will stop it from sending out the ACK:

1c

Now something else happens. Lots of packets will start coming in, in quick succession, because the remote node’s easy transmission setup is not getting any ACKs. So it keeps retrying:

We’ll need to change the JeeMon “application.tcl” script to use ACKs instead of direct packets (code here):

This change is fairly subtle. The SendToLCD code hasn’t changed. But it’s used in a completely different way now: instead of just sending out one packet, we set up a connection and keep it going – and therefore listening…

The trick is now to check what’s coming in, and when an empty packet from node 9 is coming in (“OK 41”, i.e. 32+9), then we send an ACK (128+9) with the message text. And since we’re done, we also quit the application.

There’s one small GOTCHA… it doesn’t work!

The problem is a bug in the RF12demo code, which prevents it from sending out packets that look like an ACK. I’ve fixed it now, so you’ll need to update your JeeLink or JeeNode – whichever you use as central node.

Once you’ve done that, it should work: power up both nodes and then launch JeeMon in the same way as before. If all is well, a message will again appear on the LCD screen:

Note that I’ve extended the remoteLCD sketch to support 2 lines of output: everything after a newline ends up on the second line of the display.

Sooo – similar result, but using a completely different mechanism.

This approach has as benefit that the remote node will stubbornly keep trying to obtain a message to display, even when packets get lost. But in its current form, that’s also a drawback: when JeeMon is not running, the LCD node will keep on retrying forever. With the easy transmission mechanism, it normally retries 8 times, once a second, but since we restart the packet every 5 seconds, it ends up constantly retrying, each second.

There are some other advantages with this remotely-initiated protocol. One of them is less important in this particular case: since the remote node is in control, it doesn’t have to keep the receiver on all the time, and can go into low-power sleep mode any time to save on battery consumption.

The second advantage is that the remote node doesn’t need to know where to get its data from. The easy transmission mechanism uses broadcasts, so whichever node decides to respond to it, gets its message displayed. As long as exactly one node always does, to prevent constant retries.

A third advantage is that the central node can detect when a remote node is up and when not, and act accordingly. This is a topic for another weblog post…

P.S. The Dimmer, Gravity, and Lux plugs have been added to the shop.

Yesterday‘s post got a message to an LCD display, using a JeeLink with RF12demo to send the packet over the air. It works, but the packet had to be manually encoded as decimal bytes. Not very convenient.

Today, I’ll use JeeMon to streamline this somewhat. There will be several iterations of this example, so that it can also be used as an introduction to JeeMon and the Tcl programming language it uses.

To that end: here is a list of Tcl links which may be useful if you want to learn more about Tcl. Having said that – the code below should be fairly clear, even if you don’t want to dive in further.

This first example is mostly to get started with JeeMon.

Read the Getting Started section to obtain a suitable version of JeeMon (Windows, Mac, several Linux’es).

Got that? Ok, make sure it launches and no errors appear. Most likely result of doing so should be a message with “Can’t start (yet), …“. Good. It works.

Let’s set up JeeMon to send a message to the remote LCD.

• Create a directory called “JeeMon-lcd1”.

• Inside, create a file “application.tcl” with a text editor, and place the following Tcl program code in that text file (you can download it here):

• Change the “usb-A600dVNz” in the above program to match the USB device you’re using to send the message with (can be either a JeeLink or a JeeNode). You can use the Arduino IDE to find out the proper name (probably COMn on Windows). Do not use the port of the remote LCD node, since that JeeNode is listening to incoming packets, not its serial port.

• Launch JeeMon from the command line, by typing this command: “JeeMon lcd1” – JeeMon will now load the code contained in the JeeMon-lcd1 directory.

If all went well, you should see two things happen:

• Some debug output will be printed:

  Sending: 72,101,108,108,111,44,32,87,111,114,108,100,33,9s
  

• the text “Hello, World!” appears on the LCD!

To be continued…

Update – only moments ago, a bug in the RF12 driver was fixed. Make sure to get the latest one and re-build / upload the remoteLCD.pde sketch again if needed.

One more post about setting up a GUI with JeeMon…

I wanted to have a basic real-time display of all the readings coming in, sorted by source. Like this:

(the size of this screen shot was reduced to fit this post)

Everything in this window is dynamic: rows get added when new sources start reporting their data, and columns get added when new parameter types are reported. I’m not storing any information persistently yet, which is why this status window looks as follows right after starting up JeeMon:

Not very exciting, but all you have to do is wait …

Let’s see what it took to create this GUI. First, I added a generic “Notifier” rig to JeeMon, and a “Reading” method for all sketches to use. The rooms sketch, for example, now reports its incoming results as follows:

Similar changes were made to the ookRelay and RF12demo “host.tcl” sources.

The rest of the GUI code is in “statusWindow.tcl”:

The tabular display is based on a very nice Tcl/Tk open source package called Tablelist by Csaba Nemethi, which has been added to the JeeMon core library.

The coupling with the rest of the system is accomplished by the “Notifier attach readings * …” call at the end of the setup code. This registers a callback which will be invoked when anything in the “readings” notification group changes. Notifiers are going to be used a lot in JeeMon because they provide such a manageable way to glue independent parts of an application together.

And lastly, adding the following line to the main application starts the ball rolling:

There’s quite a bit going on here which I won’t go into. I just wanted to illustrate how little code it takes to create a basic, fully dynamic, cross-platform, self-updating status display window. There – how’s that for adjectives!

The JeeMon story continues. Here’s a fun little panel for controlling some switches around the house:

These screen shots show Windows (7 and 2K), Mac OS X, and Linux Ubuntu, respectively – long live diversity!

This GUI was created with these configuration settings:

Those settings were loaded by adding this line to the main application code:

Which in turn loaded the following “ookRemote.tcl” source file I wrote, as module / rig:

That’s the entire app. Adding lipstick (i.e. a fancier visual design) is left as exercise for the reader.

Note how the design is split in a configuration section and a source code file – I’m probably going to use this approach often in JeeMon. The idea is that people who don’t care about the code behind all this stuff can just adjust the configuration file (using a visual interface, hopefully, one day). While those who like to tinker can view the full source code, and explore and extend it at will, and more importantly: ad infinitum.

It’s called open source for a reason!

PS. There’s another major reason for the split between configuration settings and source code files: turnkey deployment. You can put all the source code files in a ZIP archive called “JeeMon-rigs.zip”, so that all you need for a new installation with JeeMon, is: 1) your archive, 2) the proper JeeMon runtime, and 3) a config file to match the target setup. The config file is optional, btw – more on that another time…

JeeMon is a little web server / database / reporting tool I wrote a while back. It has been in use at Jee Labs for over a year now. Here is the electricity use for 2009:

Couple of glitches, such as incorrect readouts when power failed and the sending node got reset to zero counts.

Here’s the gas consumption for 2009:

Gas consumption (heating and hot water) is relatively high for this house, which is an open split-level design with lots of windows. Well isolated, but there’s simply no way to keep heat from rising up through the open stairways.

Here are the temperature and humidity readings for the past month:

These are commercial S300 and KS300 sensors, and they seem to have frequent glitches. It looks like those could easily be filtered out, though.

Still, we managed to get €600 back on the 2009 energy bill, and with about 3000 kWh and 2000 m3 we’re 30% below the average consumption in this residential neighborhood, for both electricity and gas.

Being aware of what’s going on makes a difference, IMO. It has become a habit to turn off all the lights when I leave a room, and closing curtains right away when it gets dark. When we go out, we turn down the thermostat. Who needs high tech, when common sense is all you need? It’s so obvious, yet so effective …

The JeeMon database for all of 2009 is 26 Mb, i.e. tiny when considering that this includes every reading and some aggregated series as well. In fact, it contains a lot more data than what’s shown in the above graphs.

I’ve got several ideas and plans for JeeMon in 2010. I want to make it far more modular, so that nearly all its current functionality becomes available as easy-to-extend plug-ins. And it needs to be fullly configurable – as it is, JeeMon is still little more than a one-off implementation. But it has served its purpose very nicely already.

The bwired.nl website has been promoting domotics “by example” for a long time now. Pieter Knuvers, the guy behind it all, added a new service to submit and display data from other fellow energy geeks like me…

The bwired upload service was announced (in Dutch) on Pieter’s weblog.

So I added the bwired.tcl Tcl code to JeeMon to generate the XML input he wants:

It’s trivial but way too messy stuff, mainly because I don’t have the proper abstractions in place yet to easily generate all sorts of aggregated data. In this case hourly averages and daily totals.

Results can be viewed here, name “jcw”.

Oh well, good exercise. Pretty much the same logic as for the Pachube feed.

As I’ve mentioned before, JeeMon is a self-contained executable which requires no installation and runs on Windows, Mac OS X, and various Linux systems (including the ARM chip on the MMnet1001 modules used in the JeeHub). There are no special requirements or dependencies, no Java runtime, no .NET, no nothing (note: on Linux you have to check that the standard C++ runtime libraries are present).

The JeeMon builds are here – you only need to fetch one and launch it to use it.

This is based on a generic runtime called Tclkit which contains a full implementation of the Tcl programming language as well as the Metakit transaction-based embedded database. It’s hard to overestimate the amount of functionality included in these 1 .. 2 Mb executables.

The only specific code in JeeMon is the startup logic to turn it into a web-server with home monitoring functionality. It’s all inside one main.tcl file, the essence of which is shown here:

The code starts off with a few definitions and then loads the rest of the application. From file if present, else from a file loaded from the internet. The “fetchLatestLibrary” code by default also updates to the latest version, this can be disabled by deleting the “Jee-library.update” file.

By changing one URL shown above, JeeMon can be turned into any other type of Tcl application, because everything the application actually does comes from the downloaded file.

For those who don’t like this automatic mode, there is always the option to download all the pieces and run them as is. Both JeeMon and the Jee-library are based 100% on open source software. JeeMon is implemented in C/C++ and can be compiled from scratch using the standard gcc/automake tools, and Jee-library can be unwrapped to inspect every single bit of it – it is nothing but a packaged collection of plain text files and images. I’d be happy to document this if needed.

I suspect that most people won’t quite appreciate the implications of the way JeeMon is built and packaged until they try it out. And the reality is that most people won’t ever try out this stuff – fine with me. The technology underneath JeeMon is just a set of choices. In the end, the degree to which these choices become invisible in day-to-day use of this system is all that matters. And for a platform-independent modern programming environment, it doesn’t get much more general purpose and deployable than this…

This is what I see on my screen these days:

This text is not inside a window but on the desktop, i.e. it gets covered up by all regular windows (there is always Exposé’s F11 shortcut to uncover it).

The text is barely readable, not only using a tiny font but also using 50% transparency. This is intentional: the text is not there to distract but to offer me a quick way to check on the status of my permanently-running setup on the JeeHub with embedded MMnet1001 Linux module.

It’s all done through a little utility called GeekTool, there are no doubt similar ones for Windows and Linux. Geektool periodically executes some shell commands and shows the output on the desktop.

The first listing displays the last 5 lines of the log output:

ssh propie tail -5 jee_out

The second listing is plain-text output generated by a new web page in JeeMon:

wget -q -O- http://propie/data/current

Adding that page to JeeMon was trivial (data collection details omitted):

Now, I can always see the latest readings and the status of the JeeMon server.

P.S. I will be away until May 3rd and won’t be able to respond to emails and comments until then. This weblog will be on auto-pilot for a week, enjoy!

The graph pages in JeeMon are a lot faster now. Here are the “before” …

… and “after” timing statistics:

A nine-fold speed increase. Great.

The change is that hourly and daily values have been re-calculated and stored in the database, instead of constantly recalculating these on-the-fly. The last value is not stored, as it might be incomplete and could change when more readings come in.

Keep in mind that these measurements are made with the MMnet100 Linux module, which is a very low-cost / low-power ARM-based system. On a modern PC, the graph pages usually come out instantly, even without storing any condensed hourly or daily datasets.

The other thing to keep in mind is that all JeeMon processing takes place in scripted languages: Tcl on the server and JavaScript in the browser. And that this is handling non-trivial amounts of data, since JeeMon is basing all these graphs on 5-minute values collected for up to an entire year.

In short: 9x faster is good enough, the graph pages now finish loading within two seconds.

It’s nice to see hunches work out in reality…

Update – on the MMnet1001, the UBIFS flash filesystem has compression turned on as default for all files. It would seem to make more sense to keep Metakit database files uncompressed, so I did a “chattr -c Jee-database” and made sure to rewrite the file from scratch with decompressed data. However, it looks like this only shaves another 5% off the graph page times, bringing times down to from 2.0 to 1.9 seconds. Oh well, it was worth a try.

I thought I’d share a bit here how JeeMon works and what it’s made of.

One of the main goals for JeeMon, is that it should support two modes of operation:

• running on a personal computer as one of many processes, launched and stopped at will
• running on a dedicated system which is always running, possibly with quite limited resources

There is not that much difference between these two modes, but it does mean JeeMon should present itself as a webserver and that its processing demands have to be low. Being portable to different platforms also helps.

JeeMon is essentially a webserver plus database, both embedded and combined into a single process. It also contains all the logic to collect readings from – and send commands to – a JeeHub via a USB or serial port.

I picked the technology I know well, some of which I have helped develop or developed myself over the past years:

• the code is written in two languages: Tcl on the server and JavaScript in the browser
• the Tcl runtime uses a highly portable and self-contained system called Tclkit
• the web server is my own design, using a highly modular Mason-like system called Mavrig
• the embedded database is Metakit, for very compact and efficient storage of data
• the JavaScript code uses jQuery as library to simplify many tasks
• interactive plots are generated in the browser using the Flot package
• the JeeMon application is packaged as a Starpack into a single installation-free runtime
• similarly, all Tcl code, Javascript code, and website content is wrapped up into a Starkit

[more details below…]

Read the rest of this entry »

JeeMon lets you zoom in (“drill down”) on the electricity / gas / water consumption data. Here’s how:

That’s 4 graphs: 1-day details with 5-minute resolution, 2 weekly summaries, and an entire year.

The weekly graphs only differ in the level of detail: on the left is the daily change each hour, whereas the graph on the right shows daily totals.

Clicking on a day in either of the weekly graphs causes the daily graph to display that day. And a click on any week in the year overview switches both weekly graphs to that week.

With two mouse clicks on this page you can display the details of any day in the past year. To easily go back to the initial display, a “Go to Today” button appears once a different graph has been selected (same as a refresh).

Note also that selecting a time range in the daily graph calculates the total consumption in that period, and hovering over a bar in the bar graphs displays the exact value as a tooltip.

No other zooming or panning is available, just these fixed choices. Simple and effective, IMO.

Yesterday’s post was about web server performance. Here’s the latest build of JeeMon with the same page:

That’s nearly 17 seconds to render the page now, versus 5 yesterday… whoops!

Here’s why I call this major progress anyway:

• The “electricity.html” page no longer contains plot data values and is half its previous size.
• All plot values are obtained via Ajax calls, running (mostly) in parallel.
• This opens up the path to unobtrusive refreshes and automatic plot updates.
• All plots now contain meaningful data, the previous version used some bogus values.

As you can see in the above graph, over 15 seconds is spent in waiting for the server to return all plot data.

And the reason this now takes so long is very simple: all values are calculated on-the-fly by the server from stored 5-minute values. That’s over 100,000 aggregated values to generate a simple 52-week bar graph!

All this should sort itself out once I start caching hourly and daily results in the database. For now, only the raw readings and the 5-minute aggregated counts get stored. The finecky bit is to make all aggregations / calculations consistent across arbitrary JeeMon restarts.

Here’s the latest version of the generated page:

Just to prove that JeeMon is far from ready for real-world use…

The Safari web browser has excellent tools to figure out where bottlenecks are. Here is an overview of where the loading time is going for the main electricity consumption page, as produced by JeeMon running on the MMnet1001 Linux module. That web page has several detailed graphs on it – here are the stats:

As you can see, all the time is consumed by the initial page generation on the server. That’s 4.64 seconds to generate the page – which is totally unacceptable. JeeMon serves that same page 50 times faster when running locally on my Mac laptop (!).

Here are the amounts of data transferred to serve that page:

The main page including the plot data is about 5 % of the total size – the other files are cached as static content, which is why they do not increase the transfer time.

There is a lot more information to be gleaned from these measurements, btw.

I have a hunch on where the bottlenecks in JeeMon are, but haven’t looked into it yet. Which is exactly how it should be: aim for solid / correct behavior first and use the proper instruments to measure performance and track the effects of optimizations later (I happen to use Safari, but similar tools exist for other browsers).

Stay tuned for updates, once I get to tweaking this…

I’ve started work on a basic website design for JeeMon:

Here’s the gas consumption page:

The same JeeMon code is currently running on the “production” (ahem) JeeHub and on my development Mac.

These are real live webpages, with real (but not yet 100% correct) datasets. Still, this is just a tiny tip of a massive iceberg. Lots and lots of things left to do.

Well, at least it’s a pretty iceberg IMO ;)

Here’s a real-time graph of data collected by JeeMon:

And a snapshot, in case the above real-time graph isn’t working right now:

It’s generated via a new site called pachube – I’ve extended JeeMon to feed that site through simple REST-style HTTP requests. Since the graph only has 15-minute resolution, that’s how often it gets updated.

Here’s another real-time feed, showing our gas consumption in l/hr:

Pachube is still in beta. I’ve only just started looking into it. You can define any number of feeds, each having any number of data values. Not sure how it deals with missing values, nor what to do about data which is collected at different rates. The graphs are somewhat configurable, as you can see – this will no doubt still evolve further.

Pachube can poll a server you designate (“automatic” mode), or you can feed it new data at your own pace (“manual” mode), as is being done here. The advantage of the latter is that it gets through firewalls and avoids the security issues of running a public server.

It’s an interesting approach, apparently pachube aims to become some sort of free exchange for producers and consumers of location-specific real-time data.

I’ll leave this setup running for now, but don’t be surprised if this post stops showing an up-to-date graph, since I might stop this feed one day.

The point of several projects here is to monitor energy consumption, and probably also some weather data because it’s so easy to add. Where “monitoring” means logging as well as keeping measurement data in a database for quick access. But how do you make sure that all “readings” get logged, yet still retain the ability to adjust and extend the underlying database and software? It would be awful if each non-trivial change required conversion.

Here are a few basic design decisions I took for JeeMon:

• Readings come in as text from the attached JeeNode, one line per reading.
• All readings are time-stamped and appended to a logfile, in text format.
• To keep things manageable, logfiles roll over to a new one on 0:00 GMT each day.
• The first entry in a new logfile is the name and size of the previous one.
• The database is treated as a cache: it can be reconstructed from the logs at any time.

And here’s a requirement I want to meet:

• It must be possible to restart JeeMon at any time without losing readings.

The idea then is to “catch up” with the logs on each restart where possible, and to clear the cache db and reload it from scratch when the software changes are more substantial.

So what does it take to not lose any readings? Keeping in mind that JeeMon will run either on a little embedded Linux module right next to the central JeeNode, or on a personal Mac, Windows, or Linux computer. Well, first of all, this is why the central JeeNode has a backup battery and dataflash memory: it stays on at all times, and continues to receive and collect packets from all the other JeeNodes even when there is no JeeMon running. The duration of this autonomous operation will be fairly modest: a few hours or perhaps one day. Enough to handle brief power loss, to mess around with configurations, and to re-connect things occasionally.

This means there are now three places where data gets added and must be kept in sync with the rest:

(Four places if you also count the final destination: dynamically updating browser windows)

• The dataflash memory gets a copy of each reading collected by the central JeeNode.
• All readings must end up in the log files when JeeMon is running.
• The cache database stores decoded values in an efficient internal format.
• And lastly, the browser(s) present more or less detailed results, summaries, and derived info.

JeeMon will start up in logfile “catch up” mode: the current content of the dataflash memory will be checked against the last logfile entry. The first task is then to request old data from the JeeNode to bring the log files up to date. During this time, the JeeNode continues to add new incoming data to the dataflash.

Once all missed data has been transferred, the JeeNode switches to “real-time” mode, saving new readings to dataflash and sending them out to JeeMon at the same time. In real-time mode, the log files will track all readings as soon as they come in.

That’s just part of the story, though. Now JeeMon enters “db sync” mode. With information from the cache db, new entries and new log files are scanned and processed, with new readings added to the database. Once that is done, JeeMon switches to normal real-time operation.

All log files are kept online. Rebuilding a database from scratch is as simple as deleting the current one and restarting JeeMon. Since a full rebuild might take quite some time, the internal JeeMon webserver always starts up in a “please-wait” mode and switches to the real server after real-time operation has been enabled.

Part of the above logic has now been implemented. JeeMon resumes its logs, syncs / rebuilds its database as needed, and processes new readings while running. The firmware in the JeeNode to save and replay readings to and from dataflash is still work-in-progress, however.

So while the JeeHub / JeeNode needs to stay powered up, I can now restart JeeMon at will. Which is great, because that streamlines development.

For some time, I’ve had a second set of calculations running which produces much more accurate and regular data than some of what’s on these past charts:

The difference with the graphs is that only readings which come in exactly at the time the counter changes are considered here. All retransmission (i.e. readings tagged with a late time stamp) are simply ignored. Only one retransmission happens to be in this example (the one marked “6+2”), but it still produces consistent results.

It looks like the gas measurements could be improved by tagging all readings in the sensor node instead of at the receiving end of the packets. That way retransmissions would still be tagged properly, even when readings arrive a bit late.

Hmm, this requires running a clock with milli-second accuracy on the sensor board for best results… not entirely trivial over extended periods of time. The sensor board will need to be synchronized to some other clock – either an on-board DCF77 radio or some data received from another node. And I’ll need to use a crystal instead of a resonator (0.005% vs 0.5% drift).

The gas measurements are still giving me trouble:

I’ve changed scales and plotted all values to better see what’s going on over a small time frame. The electricity consumption looks ok now (these are night-time values). But the gas measurements keep jumping up and down by a factor of around 5 all the time!

Maybe there’s a problem with the sensor readout – there are about 4 readings coming in per minute. Or perhaps the automatic idle transmission of packets every 10 seconds is interfering with this cycle?

I don’t know what’s going on… needs more work!

Here’s a more detailed view of the measurement quirks related to gas consumption:

The yellowish line is electricity usage, which hovers around 200 watt, while the blue line is the calculated gas consumption. What happens is that the heater cycles on and off approximately once every 15 minutes. When off, no gas is consumed so the meter just stops providing pulses. Then, after roughly 10 minutes it starts again at our heater’s standard rate of around 1.5 m³/hr. If you look very closely, you can see that the “zero” values are actually just above zero. That’s because whatever gas flowed until the rotation stops and whatever gas flows right after it starts all gets averaged-out over the entire period.

What needs to be done, I think, is to re-interpret the long averaged-out period as a period of continued flow at the original full rate, followed by zero flow:

I.e. the incoming data was treated as the single-hatched “B”, whereas it should be treated as the cross-hatched bar “A”. Both areas are the same.

Actually, this is not 100% accurate since we don’t know whether the meter stopped near the beginning or near the end of its revolution. But there is simply not enough information in the gas measurement data to disambiguate these cases.

However… as you can see, the electricity consumption also has some surprisingly regular small changes. Those measurements suffer from the same problem, but they have a much smaller impact due the the higher measurement rate. My hunch is that these small wattage changes are actually the electricity consumption of the heater, as it turns on and off!

But hey, who cares for now – these are just instant-by-instant consumption rates. The integral of these rates is the total consumption, and those are by definition “accurate” since I’m measuring the very same metering revolutions which the electricity and gas company use to charge me later on.

Update – Here are the results with the above adjustments:

And here’s the electricity and gas consumption for (part of) last Wednesday:

There are some quirks due to the way things are measured. The meter cannot accurately detect when gas consumption drops to zero, since that simply produces no readings. This can be resolved by adding fake readings when a very low consumption is reported, estimating the times when the gas flow stopped and then started again.

The electricity peaks at 19:45 are from the induction stove, the one around 20:15 is probably the close-in boiler.

This Tcl code generates the necessary HTML/JavaScript snippet to produce the plot:

All measurement data selection is handled by the “dataset loop” command, which takes a measurement parameter name, the selection range, and the names of the time and value variables to set during the loop.

Some weather & energy data has been coming in over here for over a month now, logged to text files, and patiently awaiting further processing. Here are the results of a few days of hacking around on the software side of things:

This is a test page served by a custom web server, with all data now stored in a database. I’ve used several bits of software which are dear and near to me:

• the database used is my own trusty Metakit
• the custom software is written in Tcl
• the web server is based on my Rig modules
• the plots are generated in the browser by Flot
• it’s all in two files using using Tclkit / Starkits

The code is portable and works on Mac OS X, Windows, and Linux. The system requirements are minimal, so it runs just fine on my NSLU2 and Bubba NAS boxes.

Things are starting to come together nicely…

Here is the latest display showing on my screen:

Updated in real time, with new readings coming in every few seconds for electricity and gas and every few minutes from the weather station (still sitting here in the office for now, so the values are bogus).