It all starts with baby steps. Let me just say that it feels very awkward and humbling to stumble around in a new programming language without knowing how things should be done. Here’s the sort of gibberish I’m currently writing:
This must be the ugliest code I’ve ever written. Not because the language is bad, but because I’m trying to convert existing code in a hurry, without knowing how to do things properly in JavaScript / CoffeeScript. Of course it’s unreadable, but all I care for right now, is to get a real-time data source up and running to develop the rest with.
I’m posting this so that one day I can look back and laugh at all this clumsiness :)
The output appears in the browser, even though all this is running on the server:
Ok, so now there’s a “feed” with readings coming in. But that’s just the tip of the iceberg:
- What should the pubsub naming structure be, i.e. what are the keys / topic names?
- Should readings be managed per value (temperature), or per device (room node)?
- What format should this data have, since inserting a decimal point is locale-specific?
- How to manage new values, given that previous ones can be useful to have around?
- Are there easy choices to make w.r.t. how to store the history of all this data?
- How to aggregate values, but more importantly perhaps: when to do this?
And that’s just incoming data. There will also need to be rules for automation and outgoing control data. Not to mention configuration settings, admin front-ends, live development, per-user settings, access rights, etc, etc, etc.
I’m not too interested yet in implementing things for real. Would rather first spend more time understanding the trade-offs – and learning JavaScript. By doodling as I’m doing now and by examining a lot of code written by others.
If you have any suggestions on what I should be looking into, let me know!
“Experience is what you get while looking for something else.” – Federico Fellini
Would it make sense to have (a pointer to) the location in your data structure that indicates where exactly data was gathered? A very precise one, down to where within the room it was gathered? That would allow for locale translation AND would make the sensor relocatable without invalidating the data. Localization and internationalization does so much more than language translation. As mostly the location will be the same, that could be a very simple enumerated entry in a table with high precision.
Have a look at this: http://book.mixu.net/
Hi, I would recommend a quick look at knockout js here: http://knockoutjs.com/ It simplifies the user interface in several interesting ways, especially through the concept of interactive variables that can automatically affect the ui.
Thx – I’ve used Knockout here. Am now looking into AngularJS, which has an even cleaner separation of content and logic, IMO. Both these systems are absolutely amazing.
Formatting data is IMO always a client concern. (Or if you are dealing with particularly low-processing-power clients, you can be nice and serve them a “value_formatted” (or “vf”, whatever :)) value along with the raw, “xx.yy” value.)
RRDtool is the “de-facto” choice for time-series data saving – however you need to know the tradeoffs you want to make re persistence in advance (ie. how many datapoints at whichever temporal resolution to save). Depending on how much data you expect to be saving, you could easily get away with binary files of the format “YYYYMMDD/measure_name” with, for instance, 64 bit records of (32-bit timestamp, 16.16fixed value) format. According to my back-of-the-napkin (well figuratively) math that’d mean 100 measurements at a 5 second resolution for approx. 14 megabytes a day.
RRDtool does this – but you do have to know the when in advance. If you store all values, then you can do it at your leisure, of course, but then it’s a space tradeoff.
Agreed on the formatting – the incoming data for a temperature is say 125 for 12.5°C, so for now I’m just saving things as raw ints, with the client side dealing with formatting as scaling.
I know about RRDtool. Probably a bit contrarian, but as Whisper illustrates, it’s not so hard to implement the essence of round-robin storage, and in fact have been running my own code here for over a year now. Nothing I’d use again, since it is written in Tcl. But I also store all incoming data as raw text logs, and intend to replay it through an RRD-like mechanism once I figure out what sort of aggregation and granularity I really want. This can then be repeated if better approaches come up along the way.
Where I find your complete code ?
It’s still a bit early for that, I’m just doodling and trying out a few things for now.