There’s now a first proof-of-concept graph in HouseMon, based on D3 and NVD3:
Historical data is now stored in Redis. The plan is to keep the last 48 hours in Redis and then move it to some sort of aggregated data files, to complement the daily full log files.
Lots of features in NVD3 which I haven’t explored yet. Out of the box, it supports hovering over a datapoint with a call-out to display the exact value, as you can see.
D3 – and the NVD3 package built on top of it – use SVG, which is supported by at least Safari and Chrome. You can see that they really are drawing with vectors, by zooming in:
The PDF version that gets generated when printing the page in Safari has several quirks (the shape is drawn as a closed polygon), and in Chrome the graph shows up as a black rectangle, so there is a price to pay for using this sort of cutting-edge technology – but on screen it comes out nicely.
This is the client/templates/graphs.jade file which defines the above page layout:
.row
.twelve.columns
.row
.eight.columns
h3 Graphs
#chart
svg
.four.columns
h3: input(type='text',ng-model='query',placeholder='Search...')
div(ng-repeat='row in status | filter:query | orderBy:"key"')
input(type='checkbox')
| {{row.location}} - {{row.parameter}}
span(ng-show='row.unit') ({{row.unit}})
And this is how historical data gets fetched from the server (part of graphs.coffee):
key = 'meterkast - Usage house'
info = $scope.status.find key
promise = rpc.exec 'host.api', 'rawRange', key, -86400000, 0
promise.then (values) ->
[ offset, pairs ] = values
if pairs
series = []
for i in [0...pairs.length] by 2
series.push
x: offset + parseInt pairs[i+1]
y: adjustValue parseInt(pairs[i]), info
data = [
values: series
key: 'Usage House'
]
nv.addGraph ->
chart = nv.models.lineChart()
formatter = d3.time.format '%X'
chart.xAxis.tickFormat (d) -> formatter new Date (d)
chart.xAxis.showMaxMin false
chart.yAxis.showMaxMin false
d3.select('#chart svg').datum(data).call(chart)
The “promise” you see in this code is a clever mechanism to deal with results which come back asynchronously, i.e. at a later time than when the RPC call returns. Most of the code deals with “massaging” the data, to get it into the proper format for NVD3.
It’s nice to make some visible progress. for a change!
Stay tuned for tomorrow, when the DIJN series resumes…
Update – I’ve switched to Flotr2, smaller and better defaults for the time axis.
I was hoping to see a cake with 4 candles on it today….. congratulations on 4 years of Jeenode’s.
https://jeelabs.org/2009/02/13/its-called-a-jeenode/
Hein
Woohoo! – hey, never occurred to me to keep track of that, thanks!
Great, and with Valentine’s Day coming up, it’ll be easy to remember from now on :)
Just traced back my first purchase of a JeeNode to June 2009, so when it was 4 months old, almost 4 years ago! I sometimes enjoy thinking I might have been one of the first to discover it!
Most definitely, Alex. Fond memories of all those pioneers who were bold enough to try this crazy stuff :)
I was re-reading your blog entries yesterday and then I noticed it;-) So next year there should be a JeeNode conference or some other gathering on the 13th of February 2014 to celebrate it’s first lustrum. ….
Hmmm… http://xkcd.com/833/
tut tut :-)
Ehm… is “work in progress” an acceptable excuse?
Note quite a repetition, but 20130213 is still a funny date…
And in 2009, it was a Friday – heh :)
(btw, we really ought to get something going w.r.t. meeting up well before then!)
Hi, you should also take a look at flotr2 which is an open source graphing engine, I have it running well on every browser I have tried and its super simple to configure and IMHO looks great.
Great tip – thanks. I’m not too impressed yet with nvd3’s time series handling and axis formatting. Good timing… things are still quite easy to change at the moment!
Do you have any experience with performance for say 10,000 point datasets?
Hi, I’m doing almost the same thing (graphing temperature datapoints over several days) and NVD3 is really slow at handling massive data. I’ve selected NVD3 for its Time series pane + zoom capability and its general styling. I think the bottleneck is on the NVD3 side and not D3 itself (see http://square.github.com/cubism/ and http://square.github.com/crossfilter/)
Flotr2 is is. Trivial change in my code and it appears to be smaller and faster. Looks great.
very cool can’t wait for the next DIJN installment — I am following along here with my Pi. Meanwhile my bike computer project is progressing slowly — located some cheap reed switches at the ‘dollar store’.
for all my “old” data I use rrdtool – http://oss.oetiker.ch/rrdtool/gallery/index.en.html , and a simple cron job every 5 mins to update pngs of the graphs… obviously not as whizzy as multi-layered web graphs but fine for relatively slow moving data over short and long time scales.
Being a round-robin database it never fills up, and can be set to do all the averaging, maximums, minimums etc automatically over whatever timescales you need….
Would Graphite (http://graphite.wikidot.com/) be suited for storage? It provides fixed sized storage for time-series data by reducing granularity over time.
Apparently I share my birthday with JeeNode, I’m honoured!
Well, first of all: happy birthday, then, Alex! :)
I’ve looked into rrdtool, but until now found it simpler to just re-implement it for this specific use (i.e. a tiny fraction of it), just like Graphite does. Got a simple setup almost going in Node.js now – the essence of count/sum/min/max aggregation is only a dozen lines of code. Under 1 MB per parameter per year (uncompressed), so it all looks quite feasible so far…
Well I plot about 5 different data sets each with a few hundred points and its pretty snappy. I don’t think you need to worry too much about performance problems.