Computing stuff tied to the physical world

There IS a pattern

In Hardware, Software on May 11, 2010 at 00:01

If you’ve been following this daily weblog for a while, you may have noticed that it’s all over the place – as if I’m working on everything and nothing. At the same time!

But there’s a pattern, dear reader.

Not fully worked out, not fully planned, not static, but still… there is.

The pattern is that I’m currently trying to automate some of the stuff I need to do here to keep Jee Labs running. The shop has been growing steadily, which is great because it means I can keep doing this – which is exactly what I want to do. Indefinitely, preferably.

But the shop needs to run smoothly, so that I don’t end up becoming its slave. That means the daily production work needs to be automated as much as possible. Only then will it be possible for me to work on all sorts of fun projects, keep up this daily weblog, and fill it with – hopefully – interesting topics, day in day out.

This “self-automation” is why I created the Flash Board, for example. And why I’m redoing the reflow controller to work more reliably. I’ve also been automating like crazy recently to try and stay on top of this huge pile of parts called (haha!) inventory.

I have not lost track of the main focus of Jee Labs – the direction where it all started: energy use tracking and environmental monitoring around the house. It’s still my main focus. And now that the basic hardware works, with lots of configurations and sensors to play with, the next frontier is the software.

But software is a very finicky beast. With hardware, you hook up a few parts and it starts working – after some soldering and/or tinkering, evidently. Software is both primitive and complex in comparison: primitive, in the sense that you have to create these big house-of-cards constructions to get anywhere. Complex, because each of the ingredients is usually a massive chunk of code.

So I’m taking a lot of time to think through numerous aspects of the software. JeeMon is the house I’m building, but its core structure needs to support ideas which haven’t even been invented yet. In terms of software, that means it has to be very modular. I’ve currently got a few components in place, and the binding structure and modularity trade-offs are starting to become clear.

What I don’t have right now, is a clear enough view on the data storage side of things. So all the little JeeMon experiments so far have been side-stepping the issue of persistence (the IT word for “storage”). I’m showing things on screen. Great, but of limited utility.

What’s the big deal with persistence? Well, the moment software includes a storage mechanism, you get into the issue of how to make changes. Suppose you have a working system, and you want to change it in some fundamental way because of a new insight. How do you deal with the data it has already stored? It sounds trivial, but I think it’s everything but trivial – on a very fundamental level.

Storage is a big deal. It is crucial. And it comes up even with something as simple as displaying a moving average. How do you deal with a system restart when there are moving averages in graphs you want to show, for example?

Software development and persistence create opposing forces. Development means just that: to progress from one insight to the next as you go along and extend your understanding of the deeper issues in the problem space at hand. And then, ideally, to implement solutions in better – sometimes completely new – ways. As a developer, I constantly tear my software apart, to put it back together in improved ways (I probably do it 10x more often than most developers). This is a learning process, and the result – IMO – leads to simplicity, elegance, and almost as a side effect: robustness.

Persistence is the elephant in the room. It opposes change. Data saved on disk (or flash memory) has a structure, and changing that structure can be anything from awkward to nearly impossible. That’s why starting from scratch is so easy. That’s also why version 2 of anything in software can be so elusive. It’s not just data, btw – code is data too, in this context. Try folding a new idea into an existing bit of code …

Maybe I’m overstressing this a bit – m a y b e. But this is the main reason why I work on completely unrelated issues at times, such as streamlining the shop activities.

In the meantime, as background process, I keep exploring scenarios for the software and collecting insights from what others do or even just ask for.

So there you have it: ISP programmers, solder reflow controllers, even tangential activities such as 3D printing and CNC milling, they all get addressed here at Jee Labs. Meanwhile, I try to figure out the best way forward for many far-reaching design choices w.r.t. JeeMon.

The good news is that I think this detour is coming to an end. I think a simple, small, modular, and fun way of tying all sorts of hardware together via software is starting to shape up – in a vague hand-waving style right now, but that’s just a matter of pushing the ideas into code. And doing the grunt work.

Neither the ideas nor the code are the hard part. Ideas are cheap and plentiful. Code is easy and can be created gradually. No, the really hard part is to come up with a “pervasively modular” architecture. I.e. how to set up a system (and hence make choices) which can evolve, even in ways not yet imagined.

What I’m after is not a solution, but a tool. And that takes a bit more time…

As the well-known Chinese proverb goes:


I’m not a teacher, but the fish doesn’t interest me nearly as much as improving the whole process of fishing.

  1. Sounds like a very sensible policy to me. Many people rush into building systems and then find themselves painted into the corner a couple of months later.

    That is why good design in advance is essential. Try to think of everything that could possibly be added in the future, and make it flexible enough to handle all the other things you can’t think of right now. Several of which will inevitably become high priority before the month is out!

    Keep up the good work.

  2. What direction are you headed? Are you thinking about TCL modules, messaging (json + sockets) or light weight browsable storage (mongodb or couchdb) or something completely different ?

    Couchdb’s browsable enables playing with data as opposed to black box programmatic persistence…

    • On the low-end (tiny Linux boards), i don’t want to install anything other than the JeeMon runtime. So there, storage options are limited to Tcl and Metakit.

      On workstations, and probably also with the beefier NAS options, anything goes, really. Here too, it’s more a matter of what’s available and easy to install. JeeMon could feed readings into CouchDB or MongoDB, or use JSON or XML to feed any other setup, or hook into SQL databases, RRDtool, etc – it should all be very easy to do. It’d be great to create lots of different types of hookups, and I’d love to help make it happen. I can’t go into all directions evidently, but if someone wants to go in a specific direction, I’ll be happy to help set up the Tcl code needed to make JeeMon suitable. That’s an offer, btw…

      One basic approach could be to push (time,origin,parameter,value) tuples to any type of source, and to keep an HTTP server running on JeeMon at the same time, which can be queried with REST requests about details of its configuration. I.e. “origin” could be a numeric code, where you could access the info about that code by using it as key to query JeeMon for the details (name, location, since when, average feed rate, etc).

      I’m pretty serious about connecting JeeMon to anything you want to use.

  3. A tempting offer, unfortunately little time available. I like the embedded parts, with C/C++ of Jeenode/Buspirate/Cul etc. But on the workstation side things tend to get monolithic including parts I don’t like (FHMEM/Perl, NetHomeServer/Java or IRScope/IRWidgets, McHelper) where you cannot hookup similar but not identical hardware. So my dream would be JeeMon controlling USB/Serialembedded simple text commands And ‘rest of the world’ JeeMon via HTTP with Json. Basic user interface for controlling/debugging/browsing should be a webbrowser.

    So a JeeMon with a ‘json over HTTP’ control interface (and firebug etc support) would be a nice foundation IF i can also use it to set the paramters of the USB connected devices… But that may be just my preferences (loose hyperterm and go sockets;)

    • (FWIW, the offer wasn’t necessarily for right now or just for you… more a general “letter of intent” :)

      on the workstation side things tend to get monolithic including parts I don’t like

      Exactly. Let’s avoid the package deal lock-in which seems to be so prevalent, even with open source. JeeMon is a switchboard-type app. It’s not tied to JeeNodes, even though that’s what I will probably use it for, mostly. Likewise, JeeNodes are not tied to JeeMon. Plugs are not tied to JeeNodes. The RF12 driver is not tied to JeeStuff. You get the idea.

      If you want JSON as communication format, fine. Should fit it in quite easily. I can see how it would work for controlling devices, but it’s not yet clear to me how you would use it to report sensor network data. Polling is not always the most efficient use of resources, so external software polling JeeMon via an embedded HTTP server might not be the best way forward. That’s why I suggested setting JeeMon up to push its readings to other systems. But either way is possible – at the same time, even.

      We’ll see where this leads to. Real use will quickly clear up what works well and what doesn’t.

Comments are closed.