If you’ve been following this daily weblog for a while, you may have noticed that it’s all over the place – as if I’m working on everything and nothing. At the same time!

But there’s a pattern, dear reader.

Not fully worked out, not fully planned, not static, but still… there is.

The pattern is that I’m currently trying to automate some of the stuff I need to do here to keep Jee Labs running. The shop has been growing steadily, which is great because it means I can keep doing this – which is exactly what I want to do. Indefinitely, preferably.

But the shop needs to run smoothly, so that I don’t end up becoming its slave. That means the daily production work needs to be automated as much as possible. Only then will it be possible for me to work on all sorts of fun projects, keep up this daily weblog, and fill it with – hopefully – interesting topics, day in day out.

This “self-automation” is why I created the Flash Board, for example. And why I’m redoing the reflow controller to work more reliably. I’ve also been automating like crazy recently to try and stay on top of this huge pile of parts called (haha!) inventory.

I have not lost track of the main focus of Jee Labs – the direction where it all started: energy use tracking and environmental monitoring around the house. It’s still my main focus. And now that the basic hardware works, with lots of configurations and sensors to play with, the next frontier is the software.

But software is a very finicky beast. With hardware, you hook up a few parts and it starts working – after some soldering and/or tinkering, evidently. Software is both primitive and complex in comparison: primitive, in the sense that you have to create these big house-of-cards constructions to get anywhere. Complex, because each of the ingredients is usually a massive chunk of code.

So I’m taking a lot of time to think through numerous aspects of the software. JeeMon is the house I’m building, but its core structure needs to support ideas which haven’t even been invented yet. In terms of software, that means it has to be very modular. I’ve currently got a few components in place, and the binding structure and modularity trade-offs are starting to become clear.

What I don’t have right now, is a clear enough view on the data storage side of things. So all the little JeeMon experiments so far have been side-stepping the issue of persistence (the IT word for “storage”). I’m showing things on screen. Great, but of limited utility.

What’s the big deal with persistence? Well, the moment software includes a storage mechanism, you get into the issue of how to make changes. Suppose you have a working system, and you want to change it in some fundamental way because of a new insight. How do you deal with the data it has already stored? It sounds trivial, but I think it’s everything but trivial – on a very fundamental level.

Storage is a big deal. It is crucial. And it comes up even with something as simple as displaying a moving average. How do you deal with a system restart when there are moving averages in graphs you want to show, for example?

Software development and persistence create opposing forces. Development means just that: to progress from one insight to the next as you go along and extend your understanding of the deeper issues in the problem space at hand. And then, ideally, to implement solutions in better – sometimes completely new – ways. As a developer, I constantly tear my software apart, to put it back together in improved ways (I probably do it 10x more often than most developers). This is a learning process, and the result – IMO – leads to simplicity, elegance, and almost as a side effect: robustness.

Persistence is the elephant in the room. It opposes change. Data saved on disk (or flash memory) has a structure, and changing that structure can be anything from awkward to nearly impossible. That’s why starting from scratch is so easy. That’s also why version 2 of anything in software can be so elusive. It’s not just data, btw – code is data too, in this context. Try folding a new idea into an existing bit of code …

Maybe I’m overstressing this a bit – m a y b e. But this is the main reason why I work on completely unrelated issues at times, such as streamlining the shop activities.

In the meantime, as background process, I keep exploring scenarios for the software and collecting insights from what others do or even just ask for.

So there you have it: ISP programmers, solder reflow controllers, even tangential activities such as 3D printing and CNC milling, they all get addressed here at Jee Labs. Meanwhile, I try to figure out the best way forward for many far-reaching design choices w.r.t. JeeMon.

The good news is that I think this detour is coming to an end. I think a simple, small, modular, and fun way of tying all sorts of hardware together via software is starting to shape up – in a vague hand-waving style right now, but that’s just a matter of pushing the ideas into code. And doing the grunt work.

Neither the ideas nor the code are the hard part. Ideas are cheap and plentiful. Code is easy and can be created gradually. No, the really hard part is to come up with a “pervasively modular” architecture. I.e. how to set up a system (and hence make choices) which can evolve, even in ways not yet imagined.

What I’m after is not a solution, but a tool. And that takes a bit more time…

As the well-known Chinese proverb goes:

I’m not a teacher, but the fish doesn’t interest me nearly as much as improving the whole process of fishing.