It’s probably me, but I’m having a hard time dealing with data as arrays and hashes…
Here’s what you get in just about every programming language nowadays:
- variables (which is simply keyed access by name)
- simple scalars, such as ints, floats, and strings
- indexed aggregation, i.e. arrays:
blah[index] - tagged aggregation, i.e. structs:
blah.property - arbitrarily nested combinations of the above
JavaScript structs are called objects and tag access can be blah.tag or blah['tag'].
It would seem that these are all one would need, but I find it quite limiting.
Simple example: I have a set of “drivers” (JavaScript modules), implementing all sorts of functionality. Some of these need only be set up once, so the basic module mechanism works just fine: require "blah" can give me access to the driver as an object.
But others really act like classes (prototypal or otherwise), i.e. there’s a driver to open a serial port and manage incoming and outgoing packets from say a JeeLink running the RF12demo sketch. There can be more than one of these at the same time, which is also indicated by the fact that the driver needs a “serial port” argument to be used.
Each of these serial port interfaces has a certain amount of configuration (the RF12 band/group settings, for example), so it really is a good idea to implement this as objects with some state in them. Each serial port ends up as a derived instance of EventEmitter, with raw packets flowing through it, in both directions: incoming and outgoing.
Then there are packet decoders, to make sense of the bytes coming from room nodes, the OOK relay, and so on. Again, these are modules, but it’s not so clear whether a single decoder object should decode all packets on any attached JeeLink or whether there should be one “decoder object” per serial interface object. Separate objects allow more smarts, because decoders can then keep per-sensor state.
The OOK relay in turn, receives (‘multiplexes”) data from different nodes (and different types of nodes), so this again leads to a bunch of decoders, each for a specific type of OOK transmitter (KAKU, FS20, weather, etc).
As you can see, there’s sort of a tree involved – taking incoming packet data and dissecting / passing it on to more specific decoders. In itself, this is no problem at all – it can all be represented as nested driver objects.
As final step, the different decoders can all publish their readings to a common EventEmitter, which will act as a simple bus. Same as an MQTT broker with channels, with the same “nested key” strings to identify each reading.
So far so good. But that’s such a tiny piece of the puzzle, really.
Complexity sets in once you start to think about setup and teardown of this whole setup at run time (i.e. configuration in the browser).
Each driver object may need some configuration settings (the serial port name for the RF12demo driver was one example). To create a user interface and expose it all in the browser, I need some way of treating drivers as a generic collection, independent of their nesting during the decoding process.
Let’s call the driver modules “interfaces” for now, i.e. in essence the classes from which driver instances can be created. Then the “drivers” become instantiations of these classes, i.e. the objects which actually do the work of connecting, reading, writing, decoding, etc.
One essential difference is that the list of interfaces is flat, whereas a configured system with lots of drivers running is often a tree, to cope with the gradual decoding task described a bit above.
How do I find all the active drivers of a specific interface? Walk the driver tree? Yuck.
Given an driver object, how do I find out where it sits in the tree? Store path lists? Yuck.
Again, it may well be me, but I’m used to dealing with data structures in a non-redundant way. The more you link and cross-link stuff (let alone make copies), the more hassles you run into when adding, removing, or altering things. I’m trying to avoid “administrative code” which only keeps some redundant invariants intact – as much as possible, anyway.
Aren’t data structures supposed to be about keeping each fact in exactly one place?
PS. My terminology is still a mess in flux: interfaces, drivers, devices, etc…
Update – I should probably add that my troubles all seem to come from trying to maintain accurate object identities between clients and server.
One thing I’m having difficulty with is the Redis key:value notation versus that of a RDBMS (which I grew up with from 1980 onwards). e.g the need to have both of these properties:
SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9
SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000
to be able to access the item either via uid or auth key.
I’m still sort of solving the “distributor” problem by describing the different interpretations for a particular device in a database record and having a central php process decoding all the different variants… the resulting data should then be logged in a database AND published to a red is database from the same php process.
The jnode stuff subscribes to the redis data for real time stuff. Anyway, that’s what I’m thinking. But you’re right, for an oldie like me it’s a lot to take on board…
Same relational mindset here, I’m used to primary, secondary, and foreign keys avoiding much of the key vs value asymmetry. And thinking in more than two columns…
Good to hear you don’t like to ‘walk the driver tree’. Treeview is imho a very bad user interface. USB and Ethernet also have this tree (nodes connected to hubs/switches/routers) but you use them within one ‘flat’ home network by using IP adresses. So you need some mapping to a list of devices. Maybe CSS/jQuery/HTML kan help you. If you don’t like the tree you can flatten it by adding #id and .class attributes as in http://ejohn.org/blog/xpath-css-selectors/
It sounds like you are needing classes with ‘multiple’ interfaces, and then typical driver-manager and class-factory style setups.
This is obviously simpler in a language like C#/Java as Javascript does not ‘natively’ support concept of multiple interfaces (say like COM does), but you can implement multiple interfaces for a javascript ‘class’ although it requires more imagination and is NOT typesafe so enforcement (compiler) cannot be provided.
Once you have a class that ‘implements’ its interfaces you can then both create and manage the nested hierarchy and provide a ‘flat’ way to talk to them (via browser).
Years ago (when i was a Java/DCOM programmer for the FS Sector) I used this design pattern in a large number of projects, and a few years ago did a ‘pet’ project for my then local charity (surprise! using javascript) using such a design pattern, so it is possible in Javascript. Those were however large systems and I know you like lightweight systems with no ‘fluff’.
btw: I’m not new to JN’s, having purchased some about a year ago. I implemented my ‘system’ then, using python (same pattern), but I recently also have been interested in javascript again via node.js on my little RPi’s. Infact i just purchased a second hand Joggler that I hope to act as a remote touch interface for my proposed new node.js/html5 automation controller interface – so I’m sure I will be hitting just the problems you are crawling over right now.
Needless to say, its possible but not as ‘clean/simple’ to do in javascript, as in a traditional OO language. I wished I had some simple examples but today is a problem for me (being away from home until Tues).
For interest, in my current python system I actually store my hierarchical setup/runtime ‘tree’ in redis and use that as a de-referencing point into my ‘running’ system where each driver heartbeats its runtime TTL key. As a result my app has no traditional config files and setup/config by browser is a breeze.
Finally (related to above), I have a love/hate relationship with RFM12Demo ;) and wondered if Jeelabs uses RFM12demo output structure untweaked as I have been recently reviewing the concept of a ‘schema’ within output data. My first though was just using the 2 bytes (‘OK’) to denote schema and match on more variations, this would make ‘MY’ runtime driver/decoder handling code much cleaner, (I currently use nodeid for this but its not self describing when another node joins the network) – perhaps I have kept my JN protocol too close to jeelabs and my requirements differ a little. (Perhaps you use a few hidden ‘bits’ to do something similar?).
Thanks for your elaborate comment. Will re-read a few times, as it touches on many important topics.
W.r.t. plain RF12demo: I’m just using that as test interface for now. Would still like to switch to a more packet-style protocol (Bencode perhaps), but for now it’s holding up fine.
I’m not adding in-band type information in RF12 packets to keep data overhead absolutely minimal, so there needs to be some definition on the receiver end as to which node has which packet format. Not a big deal, since there is more information that needs to be set up per node anyway, such as each node’s location (short codes as well as human-friendly localised names).
I’ve been monitoring JavaScript frenzy at Jeelabs with great interest, wondering to what it will lead. There’s deeper problem with all that stuff which is well known to anyone who was in the industry before JavaScript – it’s very poor language. Everyone else can learn that for example from Crockford’s lectures as posted on 2012-12-21 – just don’t listen to his marketing-speak, but the actual meaning of his words, and he clearly shows how many inconvenient or useless stuff was inherited from past, and proceeds to show that JS is apotheosis of poor design and project management, enforced upon the world.
All great things on the web were done in spite of JavaSript, not because of it. (AngularJS and other constraint-propagating stuff reminds of Garnet and Amulet, delayed couple of decades).
Essentially, I don’t want to argue with jcw, this comment is intended for readers who may start to think “I’m writing my code in X – maybe I should switch to JS and stuff?” No, you don’t have to – there likely will be more problems than it solves.
Although JS may be a poor foundation, you’ll be standing on the shoulders of giants when you use Node.js and npm and all the frameworks etc developed around that. I am now a couple of weeks into the journey. I used to develop all my stuff using the https://contao.org/en/ framework and CMS, with PHP and MySQL. The learning curve for developing apps in that environment was steep too and I’m now pretty proficient with that…
Yet it’s complete over-engineered, solid and very wordy overkill compared to the stuff I’m just learning about now… and I still like Contao because it was developed by Germans and therefore very stable.
I don’t consider JS a poor language – syntax is only a tiny part of the whole equation. In semantics, JS is surprisingly close to Scheme, and the async event model of Node.js is brilliant. I was already used to this in Tcl, but did not realise how essential lexical scoping and closures really are in this context.
In this young and vibrant Node community, people are pouring out new ideas and bringing them in from all the languages and frameworks they were familiar with. This cross-pollination and the meritocratic structure of it all look like excellent conditions for the best of breed to float to the top.