Yesterday’s post was about the desire to automate node discovery a bit more, i.e. having nodes announce and describe themselves on the wireless network, so that central receivers know something about them without having to depend on manual setup.
The key is to make a distinction between in-band (IB) and out-of-band (OOB) data: we need a way to send information about ourselves, and we want to send it by the same means, i.e. wirelessly, but recognisable in some way as not being normal packet data.
One way would be to use a flag bit or byte in each packet, announcing whether the rest of the packet is normal or special descriptive data. But that invalidates all the nodes currently out there, and worse: you lose the 100% control over packet content that we have today.
Another approach would be to send the data on a different net group, but that requires a second receiver permanently listening to that alternate net group. Not very convenient.
Luckily, there are two more alternatives (there always are!): one extends the use of the header bits already present in each packet, the other plays tricks with the node ID header:
There are three header combinations in use, as described yesterday: normal data, requesting an ACK, and sending an ACK. That leaves one bit pattern unused in the header. We could define this as a special “management packet”.
The out-of-band data we want to send to describe the node and the packet format is probably always sent out as broadcast. After all, most nodes just want to report their sensor readings (and a few may listen for incoming commands in the returned ACK). Since node ID zero is special, there is one combination which is never used: sending out a broadcast and filling in a zero node ID as origin field. Again, this combination could be used to mark a “management packet”.
I’m leaning toward the second because it’s probably compatible with all existing code.
So what’s a “management packet”, eh?
Well, this needs to be defined, but the main point here is that the format of management packets can be fully specified and standardised across all nodes. We’re not telling them to send/receive data that way, we’re only offering them a new capability to broadcast their node/packet properties through these new special packets.
So here’s the idea, which future sketches can then choose to implement:
- every node runs a specific sketch, and we set up a simple registry of sketch type ID’s
- each sketch type ID would define at least the author and name (e.g. “roomNode”)
- the purpose of the registry is to issue unique ID’s to anyone who wants a bunch of ’em
- once a minute, for the first five minutes after power-up, the sketch sends out a management packet, containing a version number, a sequence number, the sketch type ID, and optionally, a description of its packet payload format (if it is simple and consistent enough)
- after that, this management packet gets sent out again about once an hour
- management packets are sent out in addition to regular packets, not instead of
This means that any receiver listening to the proper net group will be able to pick up these management packets and know a bit more about the sketches running on the different node ID’s. And within an hour, the central node(s) will have learned what each node is.
Nodes which never send out anything (or only send out ACKs to remote nodes, such as a central JeeLink), probably don’t need this mechanism, although there too it could be used.
So the only remaining issue is how to describe packets. Note that this is entirely optional. We could just as easily put that description in the sketch type ID registry, and even add actual decoders there to automate not only node type discovery, but even packet decoding.
Defining a concise notation to describe packet formats can be very simple or very complex, depending on the amount of complexity and variation used in these data packets. Here’s a very simple notation, which I used in JeeMon – in this case for roomNode packets:
light 8 motion 1 rhum 7 temp -10 lobat 1
These are pairs of name + number-of-bits, with a negative value indicating that sign extension is to be applied (so the temp field ranges from -512..+511, i.e. -51.2..+51.1°C).
Note that we need to fit a complete packet description in a few dozen bytes, to be able to send it out as management data, so probably the field names will have to be omitted.
This leads to the following first draft for a little-endian management packet format:
- 3 bits – management packet version, currently “1”
- 5 bits – sender node ID (since it isn’t in the header)
- 8 bits – management packet number, incremented with each transmission
- 16 bits – unique sketch type ID, assigned by JeeLabs (see below)
- optional data items, in repeating groups of the form:
- 4 bits – type of item
- 4 bits – N-1 = length of following item data (1..16 bytes)
- N bytes – item data
Item types and formats will need to be defined. Some ideas for item types to support:
- type 0 – sketch version number (a single byte should normally be enough)
- type 1 – globally unique node ID (4..16 bytes, fetched from EEPROM, could be UUID)
- type 2 – the above basic bit-field description, i.e. for a room node: 8, 1, 7, -10, 1
- type 3 – tiny URL of an on-line sketch / packet definition (or GitHub user/project?)
- …
Space is limited – the total size of all item descriptions can only be up to 62 bytes.
Sketch type ID’s could be assigned as follows:
- 0..99 – free for local use, unregistered
- 100..65535 – assigned through a central registry site provided by JeeLabs
That way, anyone can start implementing and using this stuff without waiting for such a central registry to be put in place and become operational.
Tomorrow, a first draft implementation…
Just as an aside (especially considering you mention temp ;-) I regularly record greater than 50deg C in one of my enclosures, and am running code based on roomNode.
Following this with great interest.
I doubt it’s not as important, but do you have any thoughts on data going the other way – ie command to remote nodes? I suppose that is covered with sketch registration, and therefore knowing what commands can be sent to particular nodes.
We could have an item describing at least the fact that the node listens to commands or accepts ACK’s with a payload, and possibly a similar bitfield description or some sort of other format indicator.
But the more I think about it, the more it seems likely that we’ll never fit in everything anyway – not even field names. So perhaps the main task of all this is simply to organise the node-to-driver association through unique ID’s.
Tim, have mercy on your rechargeables !
Got it sorted with AC power, and big sealed lead acid batteries with solar power ;-)
Why not let sketch type IDs be UUIDs also? You can maintain a registry but let third parties register their own? (USB would be different if class codes weren’t 0..255)
For simple uses, I’d rather not alyways have to send around 16-byte uuid’s, but it’s certainly an option.
As for the registry – I definitely wouldn’t want to be in the loop of issuing ID’s – was thinking of a simple automated website page. It’s not about control but about collaboration, after all.
It would be good if you could also include what openenergymonitor have tried to define as standard also. See appendix 2 here http://openenergymonitor.org/emon/buildingblocks/rfm12b2
Thanks – I wasn’t aware of that. With only 30 node ID’s available in each group, it’s hard to come up with a scheme to handle every situation. I tend to use ID 1 for the central node, one other node ID fixed for testing, and everything else assigned ad-hoc.
Things like an on-line sketch URL could be stored in the Sketch type ID database. I see no need to have them in the eeprom
I think that message formats are very important. For my own nodes i automatically create C struct and json representations for all messages. The messages are send in binary form (via serial) to a raspberry pi. There nodejs translates them to ajax/json.
i use “structname bbh2b cmd id count temps” to describe data. I have a script that creates a c struct and a json object from this string.
The meaning of “bbh2f” is defined at http://docs.python.org/2/library/struct.html
Bitfields could easily be supported. I thought of adding a new format letter for bit.
In my environment every message begins with a 1 byte message identifier (cmd). This makes supporting new messages very easy
I like the goal to standardize communication & discovery, but I wonder if a sketch based solution is practial in the long run.
I might of course be the exception to the rule, but I have ZERO standard sketches. All are different because of there number of sensors attached or 433Mhz receiver/senders.
Using the sketch scenario I would have different sketch ID’s for a standard roomnode, one with 1 extra temp sensor, 1 with 2 extra temp sensors, etc.
Therefore I chose to use the data as the common denominator, ie a standard roomnode does not exist, it happens to be a node that sends out a temperature, humidity, movement and battery status. And each value has its own unique id (node number + sensor number).
A node with 6 DS temperature sensors and a humidity sensor just sends these 7 values independent or as a list to the central node. In this way I can simply add or remove a few sensors without any change on the central side.
On startup each node tells the central node how many sensors it has, and what their id’s are. This looks exactly like your management messages. If the central node receives an id that was previously unknown, it automatically defines this sensor.
None of my nodes are battery powered, so I don’t care about the extra power needed to send larger messages (each value has its sensor id send with it).
All things aside: I’m very curious about your solution as converting unique sketches to say unique sensors on the central side would also be made possible by this standardization!
Mars, you make me very curious about your local ecosystem! Any URLs by any chance?
Thanks for the detailed comments. Agree on sketch vs data identity. We don’t really need standard sketches – we need a way to decode their information. So if you choose to use a specific format which works across several of your nodes, then that is the ID you’d want to announce and re-send periodically.
In a way, all the item data is really determined by that ID, but there’s potential for code re-use and sharing in coming up with decent standard descriptions.
In my case, and with the new HouseMon (vapourware alert!), it’s really just a matter of getting hold of that ID for each node, so that the collecting server can pick the proper decoder (and eventually also encoder) for each packet.
Whatever comes out of all this, I see these management packets as just hints to help the association with less manual effort and less chance of error.
Is there already some kind of handy plug for a JeeNode that has DIP switches or something so I could just plug it into one of the ports (or I2C or SPI) to tell the JeeNode something?
One of the things I definitely would like to tell my JeeNodes is whether they are on batteries or wired power. I would probably also use a few bits for a node Id. I’m not sure what I would use the other DIP switch bits for right now, but it would be things like how often to identify themselves.
If I were using a full Arduino, it’d be simple to wire up something like that on a proto shield or whatever, but I’m wondering if anyone has already done something like that in a form factor convenient for JeeNode.
The Input Plug has 16bits available for programming by jumpers.
But wouldn’t you ‘tell’ a node by a variety of ways and then store this characterisation in local flash?
What about group information? I have some remote nodes that use the group relay sketch to get their data across to the central node. This works really well.
To identify a node I use both group id and node id. The group id for relayed packets is added by my own housemon equivalent based on the originating relay node.
Adding the group id would make sending commands to remote nodes possible as well. Relays can peek into packets to see if ‘their’ group is addressed and then resend it within their own local group.
Maybe we just need a more advanced relay packet format.
I’m finding the whole “node id” thing a bit superfluous. Are packets between arbitrary nodes really useful? Aren’t 90%+ of use-cases really star topologies between a central node and the rest of the nodes? It seems that a packet format that is designed around sensor and actuator IDs would be more useful. E.g., a packet could contain a number of frames that have length:sensor_actuator_id:data. This way it actually doesn’t matter which jeenode has which sensor. Registration can then be around the mapping of sensor/actuator type & location to sensor_actuator_id.
Node ID’s are not only for specifying a destination – they also identify the source, which is as important in a sensor network as anywhere else. And once you start controlling nodes, that ID becomes important to specify which node should act upon the command.
But I’m with you – this is why RF12 packets don’t contain both source and destination ID’s: in almost all cases, one or the other is sufficient. Most packets are broadcasts, in which case the header bits are used to indicate the source ID of the packet.
As for using sensor ID’s: you can do that today by using broadcast packets for everything, but I wouldn’t want to use longer ID’s (2 bytes probably) several times in a packet, when the header can have a node ID and the sensors a few bits (or position in the packet) to indicate their identity.
Lastly: with the current scheme, each node type can play different tricks with packet encoding, to squeeze the maximum out of the transmit power available. With a network-wide standard packet format, you’d lose the ability to send meaningful 0- or 1-byte packets, for example.