Computing stuff tied to the physical world

Extracting data from P1 packets

In Software on Dec 1, 2012 at 00:01

Ok, now that I have serial data from the P1 port with electricity and gas consumption readings, I would like to do something with it – like sending it out over wireless. The plan is to extend the homePower code in the node which is already collecting pulse data. But let’s not move too fast here – I don’t want to disrupt a running setup before it’s necessary.

So the first task ahead is to scan / parse those incoming packets shown yesterday.

There are several sketches and examples floating around the web on how to do this, but I thought it might be interesting to add a “minimalistic sauce” to the mix. The point is that an ATmega (let alone an ATtiny) is very ill-suited to string parsing, due to its severely limited memory. These packets consist of several hundreds of bytes of text, and if you want to do anything else alongside this parsing, then it’s frighteningly easy to run out of RAM.

So let’s tackle this from a somewhat different angle: what is the minimal processing we could apply to the incoming characters to extract the interesting values from them? Do we really have to collect each line and then apply string processing to it, followed by some text-to-number conversion?

This is the sketch I came up with (“Look, ma! No string processing!”):

Screen Shot 2012 11 29 at 20 44 16

This is a complete sketch, with yesterday’s test data built right into it. You’re looking at a scanner implemented as a hand-made Finite State Machine. The first quirk is that the “state” is spread out over three global variables. The second twist is that the above logic ignores everything it doesn’t care about.

Here’s what comes out, see if you can unravel the logic (see yesterday’s post for the data):

Screen Shot 2012 11 29 at 20 44 49

Yep – that’s just about what I need. This scanner requires no intermediate buffer (just 7 bytes of variable storage) and also very little code. The numeric type codes correspond to different parameters, each with a certain numeric value (I don’t care at this point what they mean). Some values have 8 digits precision, so I’m using a 32-bit int for conversion.

This will easily fit, even in an ATtiny. The moral of this story is: when processing data – even textual data – you don’t always have to think in terms of strings and parsing. Although regular expressions are probably the easiest way to parse such data, most 8-bit microcontrollers simply don’t have the memory for such “elaborate” tools. So there’s room for getting a bit more creative. There’s always a time to ask: can it be done simpler?

PS. I had a lot of fun come up with this approach. Minimalism is an addictive game.

  1. Regular expressions aren’t that elaborate. You can transform a regular expression into a (deterministic) Finite State Machine that parses that regular expression. Off-the-shelf regular expression matchers do that in memory, but you can also transform a regular expression to code. You’ll then probably end up with something similar to what you’ve written.

    Some examples: http://youtu.be/RYNN-tb9WxI http://youtu.be/dlH2pIndNrU

    As a software developer, I like the minimalism! I wished my colleagues would appreciate minimalism (and elegance) a bit more…

  2. Nice work! Last friday Stedin installed exactly the same meters you have at my house. I’d already written some code to parse the P1 output but your approach is much more efficient! Because I also wanted to know the current tariff field (0-0:96.14.0) I modified your code so it also accepts values without a dot in it.

    Did you notice that the ‘actual electricity power’ (1-0:1.7.0) field isn’t compliant with DSMR2.2? The standard specifies this field should be in 1 Watt resolution, but these meters give a value with a 10 watt resolution :(

Comments are closed.