With all the focus on sensors and embedded hardware, I’ve lost track a bit of the other side of the equation – monitoring all that incoming data, and (later on) using it to control devices. The receiving end – i.e. software running on a central PC (or embedded Linux box) has not kept up with the rest of the Jee world.
One reason was that my setup for collecting sensor data around the house has been running quite smoothly. Producing these graphs, updatable via a browser refresh:
See this, this, and this post for more details about how it was done. The JeeMon 2009 setup has been running almost non-stop on a low-power NAS, called the Bubba II (now replaced by newer models). There were only about half a dozen system restarts in these past two years, some due to power outages, some due to devices needing a reset, but that’s about it.
Trouble is… all development on this has stagnated. I did start on a JeeMon 2010 successor, but that has only been used for newer projects, such as the ookScope. In the end, I didn’t really want to disrupt my working data-collection setup and just kept the old one going. So now I’ve got two years worth of detailed logging and a local web site which is showing its age and not very useful for meeting my new requirements – let alone being able to perform more meaningful stats and controlling devices for home automation.
Last summer a student project was started at the nearby Utrecht University to try and come up with a general infrastructure which would be able to do a lot more than what I had. The result is a new system called “JeeBus” (more details coming soon on this weblog).
While JeeBus does provide a fair set of interesting features, one of the issues that kept bothering me is that normally in software development you have to either run the code or make changes to it. The “conventional” operation mode is to stop the server, edit the code, and then restart it to pick up the changes. With a bit of luck, there may be the option to re-install certain parts – but usually this is limited to “drivers” and “extensions”.
I find this stifling. Having to restart an app just to try out a one-line change is much more disruptive during active development than such a simple stop/re-start would suggest, because each time you also have to get the process back to the state you were working on. And with dynamic scripting languages, it’s a bit silly to have to jump through such hoops – which really stem from the edit-compile-run cycle of statically-bound environments, decades ago.
So I’ve started scratching my itch, and implemented a small core “hub” which starts up functionally empty with just enough capability to accept remote procedure calls (RPC‘s) and to inject plugins into a (local or remote) running process. The last version of each plugin is saved and is automatically loaded again after a restart. The result is a JeeMon process which starts off as a blank slate and evolves into a full-fledged app – web server, gui, hardware interface, background task, anything.
So far, development in a “live” process looks promising. There are less and less situations where I need to restart. I’ve set up a little tool to push all changed plugins to a remote hub, and that really completely changes the landscape of software development for me. No need to take down a real-time system anymore, which is what most of all this is about when it comes to physical computing and devices. Bugs generate stack traces, but the hub continues to function, so re-installing a fix usually solves the problem. And changing code in a working system has never been easier. This matters a lot, because I really really want to be able to “grow” a system over time.
Starting and stopping a process which is designed to run non-stop is odd. Let’s see if this new design will make it unnecessary in most cases – during active development as well as for tweaking a working setup at a later date.
