Computing stuff tied to the physical world

Archive for July 2010

Improved EtherNode

In Software on Jul 14, 2010 at 00:01

The EtherNode.pde sketch described in this post not so long ago has been extended a bit with some of the functionality of the RF12demo.pde sketch.

That way, an EtherNode can now also be used as sort of a replacement for RF12demo – through Ethernet i.s.o. USB: i.e. to collect data (by polling the server and scraping all new data from the home page) and to send out packets (via a GET request with a properly-formatted HTTP query string).

First of all, the configuration settings are now saved to EEPROM, so that the webserver will come back with the same settings after a restart:

Screen Shot 2010 07 13 at 23.13.11

As you can see, the “collect” mode has also been added. When enabled, the EtherNode will not reply with an ACK when receiving a packet. The issue here is that at most one node should respond with an ACK to broadcast packets. IOW, for each net group you use, you should normally have a single node running RF12demo or EtherNode with collect mode turned off. The others will be “lurking” (with collect mode enabled), i.e. they will be able to see all packets but they won’t reply.

Note: there is one other way to deal with collect mode and ACKs: if you have software such as JeeMon (or your own code) set up to explicitly generate an ACK, then again you should set collect mode on, so that the ATmega itself doesn’t generate the ACK as well. This is why this feature was called “collect mode” btw: the ATmega collects packets and passes it hrough, but it does not act as a full receiver which sends out ACKs.

Also new is that for debugging, you can now connect a serial console to see the full incoming requests:

Screen Shot 2010 07 13 at 23.12.22

The main screen hasn’t changed very much:

Screen Shot 2010 07 13 at 23.19.29

The only changes are a new link to a “send packet” page, and the fact that packets are now properly numbered from 0000 to 9999 (and then wrapping back to 0000).

The send packet page is new: it lets you send out a packet with up to 66 bytes of arbitrary data, to either a specific node, or as broadcast:

Screen Shot 2010 07 13 at 23.43.52

There is currently no way to request an ACK or send out ACKs via this mechanism. It’s just for sending out a data packet via a web request.

This is just the beginning of what’s possible with a JeeNode hooked up to Ethernet, of course. The sky is the limit, since everything else is a matter of uploading new software. Long live open source :)

What a year it’s been…

In Musings on Jul 13, 2010 at 00:01

One year ago, the first serious PCB designs were “taped out” (heh, if that isn’t an anachronism by now!) – this is when the first batch of JeeNode v3 boards was produced, with all the ports and pins that have by now become a standard around here.

One year later, there are 4 JeeNode variants and over 20 “plugs” / add-ons – all part of a happy JeeFamily :)

What’s next? Well, I don’t have a crystal ball. But I do know what’s coming next because of some recent projects behind the scenes … and I can tell you that there will be several new plugs starting mid-August.

Another announcement I’d like to make now, is that after the summer more of the production will be out-sourced (here in the Netherlands), to free my time for work on new hardware and software development.

As you probably know, Jee Labs is just me, moi, and myself – with a few great people helping out behind the scenes. The major difference with traditional companies is that I’m neither driven by a boss, nor (primarily) by revenue, but by interest. Which means that you can have a considerably larger influence on where Jee Labs is going than you might think… all you need to do is speak up, preferably in the discussion forum, and point out neat / useful / practical stuff. I won’t guarantee that I’ll follow everyone’s lead, but I’m as keen as anyone to go where the neat stuff is regarding physical computing.

Speaking of neat stuff…

Franz Achatz sent in a great email today, describing what he’s been doing, complete with pictures and screen dumps. Here’s the latest addition to his RFM12B-based WSN – a fridge sensor (posted with permission):

Dsc05480

All in a neat little box, with the GSM-type antenna sticking out:

Dsc05478

The sensor is a 1-wire Dallas sensor, to allow tracking the current temperature inside the fridge.

And here’s the software side of it, all created by Franz with the current JeeMon software:

Screen Shot Small

(Click here for the full-size image)

Given how young JeeMon currently is, I’m amazed to see just how much it can already be made to do…

The story I’d like you to take home from this is not how great JeeNodes or JeeMon are (they’re not, they are still far too young and simplistic), but how much freedom you have when everything is open source, hardware as well as software.

It’s time for me to start winding down (with 30..39°C of humid heat, it’s almost a necessity, even…). There will be one or two more queued-up posts on the weblog, and then it’ll be set to read-only mode. In fact, all of internet will become read-only a few days from now, as far as I’m concerned. I’ll be away only part of this summer, but even when I’m in I won’t respond to emails – sorry.

If you ever get bored, there are now 550 posts on this weblog – feel free to browse around, and enjoy :)

TTYL, as they say!

Serial communication vs packets

In Hardware, Software on Jul 12, 2010 at 00:01

When you hook two devices up via wires, you’ve got essentially two options: parallel, i.e. one wire for each bit you want to transmit & receive (example: memory cards inside a PC). Or serial, where information gets sent across bit by bit over only a few wires (examples: ethernet, USB, I2C). Parallel can achieve very high speeds with little circuitry, but serial is more convenient and cheaper for large distances.

Serial communication is very common. The model even carries through to the way we think about the “command line” – a stream of characters typed in, followed by a stream of output characters. Not surprising, since terminals used to be connected via RS232 serial links.

Wireless connections are also essentially serial: you rapidly turn a transmitter on and off (OOK), or you change its frequency of operation (FSK), to get the individual bits across.

But there’s a lot more to it than that.

With two devices connected together, you get a peer-to-peer setup with a link which is dedicated for them. This means they can send whenever they please and things will work. The same can be done with wireless: as long as only two devices are involved, one device can send whenever it likes and the other will receive the signal just fine (within a certain range, evidently).

With such a peer-to-peer setup, the serial nature of the communication channel is obvious: A sends some characters, and B will receive them, in the same order and (almost) at the same time.

But what if you’ve got more than two devices? Ah, now it gets interesting…

With wires, you could do this:

Screen Shot 2010 07 11 at 11.20.41

It’s easy to set up, but it’s pretty expensive: lots of wires all over the place (N x (N-1) / 2 for N devices) plus lots of interfaces on each device (N-1). With 10 devices, that would be 45 wires and 90 interfaces!

Worse still, this is very hard to use with wireless, where each “wire” would need to be a dedicated frequency band.

The solution is to share a single wire – called multi-drop:

Screen Shot 2010 07 11 at 11.24.58

Now there’s one wire, a couple of “taps”, and one interface per device. Much cheaper!

Trouble is, you’ve now created a “channel” which is no longer dedicated to each device (or “node” as it is usually called in such a context). They can’t just talk whenever they like anymore!

Whole new slew of issue now. How do you find out when the channel is available? What do you do when you can’t send something right away – save it up? How long? How much can you save up? What if someone else hijacked the channel and never stops transmitting? What if all nodes want to send more than the channel can handle? How do you get your information out to a specific node? Can all nodes listen to everything?

Welcome to the world of networking.

All of a sudden, simple one-on-one exchanges become quite complex. You’ll need more software to play nice on the channel. All nodes need the same software revision. And you’ve got to deal with being told “not now”.

Note that these issues apply to wired solution sharing the same channel (RS485, Canbus, USB, Ethernet) as well as all wireless networks.

Simple OOK transmitters used in weather station sensors just ignore the issue. They send whenever they want to, in an après moi le déluge fashion… (“what the heck, I don’t care whether my message arrives”). This usually works fairly well when transmissions are short, and when lost transmissions are no big deal – they’ll send out a new reading a few minutes later anyway.

Another aspect of this shotgun approach is that it’s a broadcast mechanism. The sending node transmits its messages into the air without interest as to who receives them, or whether there’s anyone listening even. All it needs to do is include a unique code, so that the receiver(s) will be able to tell who sent the message.

For weather sensors, the above is ok. For security / alarm purposes, it’s a bit unfortunate – missing an intrusion alert is not so great. So what the simplest systems do is to yell a bit louder: repeat the alert message many times, in the hope that at least one will arrive intact. No guarantees, yet some very common security systems seem to be happy with that level of reliability.

For more robust setups, you really need bi-directional communication, even if the payload only flows in one direction. Then each receiver can let the transmitter know when it got a packet intact.

There’s a lot more (software) complexity involved to use a channel effectively, to get data across reliably with “ACK” packets, to detect new and lost nodes, to deal with “congestion” and external causes of bad reception, etc.

With JeeNodes and wireless comms via the RFM12B module, the basic RF12 driver is somewhere in the middle between unchecked uni-directional transmission and fully checked self-adapting configurations.

So what does this all mean for the “end user” ?

Well, first of all: wireless communication can fail. A node can be out of range, or a badly-behaved machine can be sending out RF interference to such an extent that nothing gets across no matter what nodes do. Wireless communication can fail, it’s as simple as that! But with bi-directional communication, at least all nodes can find out whether things work or not.

The second key property of communication via a shared channel, is that you can’t just send whenever you like. You have to be able to either save things up until later, or discard messages to let future ones through.

This means that treating a wireless channel as a serial link is really a very bad idea. Keep in mind that the baudrate can drop to zero – this means that you must be prepared to save up infinitely much data for re-transmission. And the more you intend to re-transmit later, the longer you’re going to have to need that channel when it becomes available. That will frustrate all the other nodes trying to do the same thing.

One way around this, is to use a RF link with very high data rates. That way there will be a lot of slack when nodes want to catch up. But then you still need to be able to buffer all that data in the first place. Not a great idea for limited devices such as an ATmega…

The better way is to design the system to work well with occasional loss of packets. Take an energy meter, for example: don’t sent the pulse or rotation trigger, but keep a count and send the current count value. That way, lost packets will not affect the accuracy of the results, they will merely be updated less frequently when the RF link is down.

The RF12 driver used in JeeNodes was designed for the context of a little data, sent on a periodic basis. The difference with a serial link, is that you don’t get garbled text on the other side, but packets (i.e. chunks of data). All you need to keep in mind is that occasionally an entire packet won’t make it.

This design also deals with multiple nodes. Each incoming packet can have a “node ID” so receivers can tell everything apart. Packets never get mixed up or combined or split in any way. Each packet is a verified and consistent amount of data.

Couldn’t we implement a virtual serial link?

Well, yes – there are well-known techniques to implement a virtual circuit on top of a packet-based communication channel.

But doing so would be a bad idea, for reasons which have hopefully become clear from the above. A virtual circuit would either have to act as perfect channel (not feasible with finite data storage) or drop characters in unpredictable places. It is far more practical to impose a packet / chunk structure on the sender, and then be allowed to drop chunks with clearly-defined boundaries when the RF link is out of service or overloaded.

The moral of the story: think in packets when using JeeNode wireless comms – you’ll get a lot more done!

Update – see some good comments by John M below, about IP, UDP, TCP, and the OSI model which describes all the levels of abstraction involved with networking, and all the standard terminology.

RF12 communication

In Hardware on Jul 11, 2010 at 00:01

(This weblog post seems to accidentally have escaped into the wild a few days early…)

The RFM12(B) wireless radio modules from HopeRF, as used on the JeeNode, uses “Frequency Shift Keying” (FSK) as the way to get information across a wireless channel on the 433, 868, or 915 MHz band.

With wireless, there’s always a trade-off between speed and range. More speed, i.e. a higher data rate, lets you either get more data across in the same time or the same amount of data in less time, which might reduce battery consumption. But higher data rates require a larger frequency shift in the transmitter for the receiver to still be able to detect all the bit transitions reliably. A larger frequency shift wastes more power though (I think…). And a larger frequency shift means that the receiver has to accept more bandwidth to catch all the signal details.

Btw, another way to extend the range is to improve the antennas – see this discussion.

I’ll leave the narrow-band vs. wide-band details to the EE’s and amateur radio experts in this world, along with all the RF / HF calculations, because frankly I’m at the limit of my knowledge on these topics.

But what the above comes down to is that we’ve got essentially three parameters to fool around with when using RFM12B’s for wireless networking:

  • the data rate, which needs to be identical on both sides
  • the frequency shift on the transmitter side
  • the bandwidth on the receiver to filter out unwanted signals

Here are the relevant sections from the HopeRF RF12B datsheet:

Data rate

Screen Shot 2010 07 08 at 00.58.29

Frequency shift

Screen Shot 2010 07 08 at 01.00.43

Bandwidth

Screen Shot 2010 07 08 at 01.01.15

Screen Shot 2010 07 08 at 01.01.37

The challenge is to find “good” settings, which really depends on what you’re after. The settings used in the RF12 v3 driver are as follows:

  • Data rate = 49.142 KHz (see this discussion)
  • Frequency shift is set to 90 Khz
  • Bandwidth is set to 134 KHz

This was chosen partly on what I found around the web, and partly by pushing the data rate up while verifying that the range in the home would be sufficient for my own purposes (i.e. to reach the office from the electricty meter: a few concrete walls and floors).

It can probably be improved, but since such changes affect all the node in a net group, I never bothered after the initial tests were “good enough”.

The RF12 library now includes a new rf12_control() function, which allows making changes to these parameters. It’s a low-level option, but you could easily add some wrappers and an API to adjust parameters in a more intuitive way.

As mentioned in the forum, there’s a (Windows-only) tool to do the conversion to hex parameter settings. That ought to make it fairly easy to tweak these things, if you want to push the envelope.

Some people will no doubt be quite interested in such optimizations, so if you’ve found an interesting new combination of parameters, please consider sharing your findings in this forum discussion.

Having said all this, please keep in mind that these settings will still lead to fairly low data rates. The default data rate corresponds to ≈ 6 Kbytes/second of one-way data, assuming perfect conditions and 100% utilization (“hogging”) of the frequency band. With the official ISM rules imposed on the 868 MHz frequency band in Europe, each node is allowed to use only 1% of that rate – i.e. about 60 bytes per second of throughput! (there are no such restrictions @ 433 and 915 MHz – but 915 is not allowed in Europe).

There are alternate bands in the 860’ish MHz range, but I’ve never quite figured out what works where, so for now I’m sticking to this simple 1% rule. For day-to-day sensing and signaling purposes around the house, it’s actually plenty.

To put things into perspective: the IEEE 802.15.4 standard used by XBee’s has up to 16 channels of 250 Kbaud each at its disposal, when operated at 2.4 GHz. It’s a whole different ball game. And a different range: 2.4 GHz gets absorbed far more by walls than the sub-GHz bands (which is why mesh networking becomes a lot more important, which requires more resources, making it harder to keep overall battery consumption low, etc).

So you see: speed, range, complexity, cost – lots of tradeoffs, as with everything in this world!

Update – just got an email with a lot more info about the 868 MHz regulations (for the Netherlands, but I assume it’ll be the same across Europe). Looks like cordless headphones get 40 channels to pick from with 100% utilization (in the 864..868 MHz range), so you could pick one of those channels to avoid the 1% rule.

Support

In Musings on Jul 10, 2010 at 00:01

A couple of days ago an email titled “Au secours! LCD ne marche pas” came in, with this picture:

Foto am 29 06 2010 um 20.07 #2

Two JeeNode tinkerers at the Computational Art department of the UdK in Berlin, looking sad and disappointed. And they guessed – correctly! – that sending an email from Germany, written in English, and with a French title would draw my attention :)

The email contained enough technical details to be able to resolve everything with a quick email reply from me (it was the brightness pot, which can be finecky!).

Then this came back:

Foto am 29 06 2010 um 20.56

Yeay – they look happy again! :)

(above pictures shown with permission)

Isn’t that what support is all about? It’s easy to forget that at the end of the day, support is for (and by) people, not technology.

Thank you, Alberto and Petja, for making my day and reminding me.

Assembling the JeeSMD, part 2

In AVR, Hardware on Jul 9, 2010 at 00:01

Yesterday’s post was about assembling all the SMD components of the JeeSMD kit.

The last step is to program a sketch into the ATmega. This isn’t as straightforwards as with a JeeNode, because there’s no on-board FTDI or USB serial port hookup.

It’s fairly easy to create an FTDI connection, but even if you do, you’ll still need an ISP programmer to install a boot loader (see this recent post for some background).

So let’s hook up an ISP programmer first:

Dsc 1787

I’m using a somewhat weird setup: first of all, my cable connector was attached the wrong way around, so I always have to use this one in that weird folded-over position.

But a more important issue is that the ISP connection needs to use pins 1..6 of the 2×4-pin SPI/ISP connector on the JeeSMD. That doesn’t work with normal flat cable connectors, which assume 2×3 pins and are too wide to fit in a 2×4-pin header. My solution is to insert wire-wrap pins the wrong way around into the cable header. This effectively extends the connector, but now it won’t be as wide and it’ll fit just fine. Another solution would be to only solder 2×3 pins in the SPI/ISP position – you can always add two more pins later.

Once you’ve passed that hurdle, you can use any ISP programmer you like. There have been several posts about this on the weblog, as listed here.

Now, if you want to use FTDI, then presumably you just uploaded a bootloader into the ATmega, with all the proper fuse settings, etc. The next step then, is to somehow connect to a 6-pin FDTI header.

There are several ways to do this. The one I use nowadays, is through a Carrier Board, which includes the 6-pin FTDI connector:

Dsc 1786

The point about the FTDI connector, is that it’s almost trivial. All you need is 4 wires to GND, PWR, TX, and RX – plus a way to reset the board from the RTS signal. The clever way to generate a reset is to insert a 0.1 µF capacitor between the serial side RTS and the ATmega’s reset pin. Tiny trick, huge implications (does the name “Arduino” ring a bell?).

So how does the Carrier Board implement FTDI? Easy: it adds the capacitor. And you can easily do that yourself without a Carrier Board. Here’s how:

Screen Shot 2010 07 08 at 23.20.52

Note that what FTDI calls “RX” is connected to what the ATmega calls “TXD”, and vice versa. It’s all a matter of perspective… Once you have the FTDI connection set up, you can upload sketches using the Arduino IDE just as with any other board. All you need is a USB-BUB or some other equivalent USB-to-FTDI interface.

Congratulations: that’s all it takes to build and use the Arduino-compatible JeeSMD!

Assembling the JeeSMD

In AVR, Hardware on Jul 8, 2010 at 00:01

The JeeSMD is a kit with tiny “Surface Mounted Device” (SMD) components. SMD was designed for automated assembly with Pick & Place machines, but with a bit of patience it’s fairly easy to assemble a board by hand – see this post for an overview of what you will need.

You won’t be able to do this without at least a fine-tipped (0.4..0.6mm) soldering iron plus the following tools:

Dsc 1773

A magnifier lamp also helps, I know I couldn’t do this without one anymore!

This is a step-by-step guide on how to assemble the SMD kit, which consists of these parts:

Dsc 1772

The tiny ones (don’t sneeze!) are hardest to tell apart:

Jee smd Closeup

(thanks to Steve E for the macro shot – I just added some labels)

There are 2 10 kΩ resistors in there, although you only need one. That lets you get started without having to worry too much – if you mess up completely, just remove it and start over with the other one.

For a fantastic resource with detailed videos about hand-soldering SMD, see this Curious Inventor page.

So let’s get started. First thing to do is to apply flux wherever you’re going to solder things. The flux is essential because the flux in your solder wire will have evaporated longe before you can move your tip from the wire to the part.

I’m right-handed, so that’s how I hold my soldering iron. That leaves only my left hand for the tweezers – and no hand at all for the soldering wire:

Dsc 1774

Use your tweezers for all these parts, and don’t let go – once a 0603 part flies off or drops on the floor, your chances of finding it again are next to zero. Best is to work on a clean flat surface with everything around you removed.

The trick is to place the part and then push it down while you touch it with your iron with solder already on it. The moment a part is soldered down on at least one pin, it becomes a lot easier:

Dsc 1775

The matte “stain” you see around these pads is the flux, which has dried up but is still active.

(Remainder continued after the break…)

Read the rest of this entry »

Mail order

In Musings on Jul 7, 2010 at 00:01

One of the things I was totally ignorant about when starting out on the Jee Labs adventure, was the whole process of running an internet shop. The real physics and mechanics of it, not just some imagined “process”.

Of course it was clear from the outset that it would be about packaging and shipping stuff. But what does it come down to, on a day-to-day basis? Does it add much overhead? What do you need? Is it tedious / boring?

To start at the end: no, it’s actually fun! You get to give something to people. And a surprising number of names on the orders even come back once in a while, which tells me that someone, somewhere appreciated this and liked it enough to get even more JeeStuff. Which is very rewarding: I get to come up with stuff and make it, and then I get to “give” it to people all around the world (of course it’s a sale, but for me it still feels like giving).

So what happens after the obvious assembly of boards and packaging of kits, etc? Well, I pick all the required items, and put them in a padded envelope – with lots of sizes to pick from:

Dsc 1669

Little did I know about how much room all that “sealed air” takes up!

To keep shipping costs as low as possible, I try to always fit all the stuff into one envelope. In fact, I think I’ve only had to ship in a cardboard box once until now, although I do have some extra large envelopes (25x38x3 cm) for bigger workshop packages, etc. And there are limits to this type of frugal packaging, as someone pointed out:

Hpim0756

So much for that Carrier Box. What did that postman do? Stand on it?

This is now solved by sending Carrier Boxes in slightly larger padded envelopes, btw. Apparently that gives just enough added cushioning to prevent this sort of damage.

Oh, and here’s a fun detail – check out the Jee Labs postal scale:

Dsc 1759

Seriously. That’s how each package here is weighed, to determine how much postage to apply. I can’t think of a nicer way to honor the makers of roughly a century ago. A timeless piece of engineering, with cast iron foot and all. Here’s a puzzle for you, if you haven’t seen such a postal scale before: try to figure out how it can have two ranges: 0..50 g and 0..250 g – it’s a very clever yet simple trick, as with all great inventions.

The next step is to add postage, which is now done with these state-of-the-art (ahem) digital stamps:

Dsc 1347

The convenience being that I’ve got a whole pile of them printed in advance, for each rate – instead of having to transfer up to 8 stamps.

And then it’s off to the mailbox: a 3-minute walk if I get everything done before 17:00 (5 pm), or a 10-minute walk to a more central mailbox which gets emptied at 19:00 (7 pm).

So there you have it – a peek into the Jee Labs kitchen, eh, I mean shipping department :)

Software hell

In Software on Jul 6, 2010 at 00:01

I wish this were an exceptional tale. But it isn’t, it’s the norm…

I run a webserver for Jee Labs and a couple of other web sites and services.

Screen Shot 2010 07 02 at 23.55.19

I wanted to add support for logging the last hours of a new “#jeelabs” IRC channel. IRC might not be a good fit for me personally, but since several people have asked for it, I do want to support it. With say the last 3 days of that discussion logged as some public web-pages, I could drop by once in a while and try and add to the discussion going on at that time.

The “eggbot” system was suggested to me as a way to accomplish this:

Screen Shot 2010 07 02 at 23.55.50

Ok. Server is Debian, so “aptitude install eggdrop” should work, right? Debian is good at package management.

Wrong.

Turns out my server is still running Debian Etch, and “Etch is end-of-lifed”, as I just found out. The server ISP’s repository is gone. No updates, no installs, no security fixes, nothing.

Ok. Better upgrade Debian Etch to Debian Lenny first.

I’ve been there before, I know how to do it. Takes a few hours, but then it’s done. Server is running Lenny now, and all the main servers and services are still running fine (impressive!). This is a server in Germany, btw.

That was yesterday (well, ehm… night).

Today, quick check. Server ok. Weblog ok. Forum ok. Cool.

Oops. Ikiwiki isn’t working anymore. That’s the software I use to convert the Markdown pages into the Cafe web site. Fully scripted, based on Perl.

Ikiwiki wasn’t installed as Debian package, because the one in Debian was waaaay too old. Even on Lenny, it lags (by over a year – Debian Lenny still has 2.x, Ikiwiki has been at 3.x for some time now).

Ok, so let’s try to re-install it from source. By now, usually all my warning systems go into in high alert mode, because source installs are a different ball game (even with a “scripted” language such as Perl, how ironic!).

The Ikiwiki installation page is reassuringly short. Three steps. First two look pretty easy, right?

PERL5LIB=`pwd` PERL_MM_USE_DEFAULT=1 perl -MCPAN \
    -e 'CPAN::Shell->install("Bundle::IkiWiki")'
PERL5LIB=`pwd` PERL_MM_USE_DEFAULT=1 perl -MCPAN \
    -e 'CPAN::Shell->install("Bundle::IkiWiki::Extras")'

Wrong.

I’m now waiting for step 2 to complete.

More than 10,000 lines of text have scrolled over my terminal window so far. This sort of drivel:

Screen Shot 2010 07 03 at 00.05.40

More than two hours have passed.

More than 180 Mb of disk space has been consumed.

I’m forced to watch closely, because disk space is running low on the root partition. And this is running as superuser, so it’ll bring down the server if it fills up the disk.

Pinch me, am I really in the year 2010?

Screen Shot 2010 07 03 at 00.08.42

My complaint is not that some things are complex and require a lot of work and understanding.

My complaint is that some things are NOT complex and yet you get treated to an incredible amount of nonsense. Not to mention the fact that the problem isn’t solved until you invest in figuring out every detail of lots of different kinds of packages.

The IRC logging? Not solved yet. I can probably write a 200 line script with not a single dependency on other code which does what I need: track the last 100 hours of an IRC channel disscussion as a web page.

The Ikiwiki setup for the Jee Labs Cafe? Broken. One week of work to write a Markdown parser and generate the static pages myself, would be my estimate. Probably well under 1000 lines of code, all in a single script. Three orders of magnitude less disk space, and easier to adjust to my needs.

I’m still waiting for the second step of the Perl install to complete. It’s running a “BigInt” package without proper library support, apparently. Falling back to “Math::BigInt::FastCalc”, and running tests which take ages:

Screen Shot 2010 07 03 at 00.26.24

Could someone help me understand where a static wiki page generator needs to do cryptographically secure factorization of large primes or something?

I’m stumped. Three hours waiting now. I give up.

I’ve aborted the (non-) install. With 220 Mb disk space gone, and no clue how to get it back. Oh yes, wait, it’s all in “~/.cpan” of course – how obvious. Thank you, Ikiwiki & Perl, for breaking down. Thank you, Eggdrop, for making simple things hard. And thank you Debian Linux, for not giving me the option to go back to my previous setup.

And you know what ticks me off? All of the above can be avoided. It has been solved.

But we’re living in medieval times, clearly. Most people haven’t even heard of Starkits. What a cruel joke.

Ah, wait. Maybe I’m the only one who runs into issues like these. Maybe everyone else has servers which just work, and to which you can add functionality without getting stuck. Doing everything you want. And a breeze to upgrade, of course.

That must be it. It’s just me. I have totally unrealistic expectations.

Apologies for the rant. Normal transmissions will resume tomorrow…

Update – thanks to a backport tip in the comments, Ikiwiki is now working again.

A whole lotta boxes

In Hardware on Jul 5, 2010 at 00:01

There are tons ideas floating around here at Jee Labs (thanks in no small part to a few people joining in on the brainstorming side of things). Never a dull moment!

One direction I’d like to explore a lot further, is coming up with kits which include a really nice enclosure. Even geeks sometimes like to show off their creations in such a way that not all the guts are spilling out, with wires everywhere, and everyone else scared of messing up… or even just touching something.

Because I like to get a real physical feel for things at times, I ordered a large set of ABS enclosures in a wide range of shapes, varieties, sizes, and colors:

Dsc 1760

See? That’s what “research” looks like! ;)

Now there are many ways to create enclosures, also with acrylic panels, aluminum plates, and PCBs. Even 3D printing is starting to become an option. Not to forget my favorite foam board, and more environmentally friendly materials such as cardboard and wood.

In many cases, it’s possible to fit a project + board into an enclosure after the fact – especially if the enclosure is made specifically for that purpose. And sometimes you just get lucky and find a case which turns out to fit existing boards really well with only a little extra tweaking:

But sometimes it’s easier to pick an enclosure first, and then see whether a proper board can be designed to create a perfect fit.

This is what the JeeLink did, and it ended up becoming a really practical gadget, IMO:

Anyway, with this new “box full of boxes” at Jee Labs, I hope to be able to design more kits in the future which are not only fun to build, but also end up looking good and practical for daily use in and around the house. We’ll see.

“Making stuff fit” is often a neat puzzle. But it’s also easy to overlook details – always the crucial ones, of course.

Uploading? ISP? FTDI? Huh?

In AVR, Hardware, Software on Jul 4, 2010 at 00:01

There seems to be a fair bit of confusion in- and outside the Arduino world, and it’s spilling over to JeeNodes …

I’d like to go through some terms and buzzwords to try and clarify how to get your Arduino or JeeNode to do the thing you want it to do. I’m going to assume that you are familiar with the process of writing software (“code”), compiling it, and running it – at least on a Windows, Mac, or Linux computer (let’s call them all PC’s for now). With software for the ATmega, there are two approaches, depending on whether you use the Arduino IDE or not:

Screen Shot 2010 06 28 at 23.50.22

Both lead to the same result: a “hex” file with code that needs to be transferred from the PC to the ATmega.

The step which can be puzzling when starting out with physical computing and embedded hardware is how to get things across from your PC to that little Arduino or Arduino-like system you’re holding in your hand. And vice versa, since we often want to get results back or see some confirmation that things are working properly.

The confusion comes from the different conceptual levels involved, and things like:

  • do you mean how to connnect? – plug in a USB cable
  • do you mean how does the ATmega change its own code? – through a boot loader
  • do you mean via ISP (In System Programming)? – no, that’s normally not needed
  • do you mean via FTDI? – yes, that’s the name of the chip which is hooked up to USB
  • isn’t FTDI a connector? – yeah, that too, sort of… i.e. a 6-pin convention
  • isn’t a power + serial cable enough? – no, resets also play a key role
  • do I have to use the Arduino IDE? – no, you can also use a program called “avrdude”
  • what’s avrdude? – a program which can upload to a boot loader or an ISP programmer
  • do I need an ISP progranmmer? – nope, the boot loader does essentially the same thing
  • so why not just get rid of ISP? – because you need ISP to install the boot loader

Confused? Welcome to the club…

In case you’re wondering… the process is called “uploading” because the PC initiates this as outbound transfer:

Screen Shot 2010 06 29 at 01.51.50

To get your code (“sketch” in Arduino-speak) into an ATmega, you need three things, working together to make uploading happen under all possible conditions:

  1. An electrical connection – to power the ATmega, to communicate with it, and to restart the ATmega when it is stuck or busy doing something else.

  2. A common language / protocol – the communication must be understood by both sides, i.e. PC and ATmega.

  3. Software on both sides of the connection – Sending something when the other side isn’t listening, or listening while no-one is sending will not have the desired outcome.

Let’s go through each of these separately.

1. An electrical connection

There are several ways to make the connection. With an Arduino, or any similar board which has a USB connector, you can simply plug in the USB cable:

Screen Shot 2010 06 28 at 23.52.41

Some boards use a separate USB interface (“FTDI adapter”), allowing reuse of that interface for multiple boards:

Screen Shot 2010 06 28 at 23.54.08

The end effect is the same: a connection which powers the ATmega and allows communicating with it using a simple serial protocol. There’s also a trick in this hookup to let the PC reset the ATmega whenever it wants.

2. A common language / protocol

Ah, now it gets interesting. First thing to note is that there is no single common language / protocol!

That’s right. It all depends on what you want to do. Here we want to upload code from the PC to the ATmega. That requires exchanging “ISP commands” over the connection. But once uploading is done, we really want to reuse the connection as a regular two-way serial link.

The way it works is that the PC will reset the ATmega just before uploading new code. This activates a “boot loader” on the ATmega. Now both sides will be in sync (briefly) so they can exchange the necessary information to make an upload happen. IOW, right after reset the protocol is “ISP commands”. Once the upload is done, the connection can be reused for any protocol you like – as determined by the code that was uploaded.

3. Software on both sides of the connection

Back to the software now. We need to send ISP commands over the connection.

As you may have guessed, that’s exactly what the Arduino boot loader on the ATmega understands. When reset, the boot loader in the ATmega gets control. It waits and listens for incoming STK500 ISP commands. If none come in within a second or so, it relinquishes control to whatever was previously uploaded to the ATmega.

On the PC side, we need software which resets the ATmega and then immediately sends all the ISP commands needed to transfer and program the contents of a hex file.

This is what “avrdude” does. You can either use it implicitly from the Arduino IDE by starting an “Upload” from the menu, or you can launch it manually from the command line – avrdude needs a few options to tell it where the USB port is, what baudrate to use, the type of ATmega, the protocol to use (i.e. STK500), etc.

There’s more…

The above describes the different pieces and concepts involved in getting code into an ATmega. The beauty of it is that once it works, it really works well. Supplying power, uploading, communication, control, debugging … all with one simple USB cable. You only need to go back a few years to appreciate just how much this approach simplifies embedded development.

But there’s one important detail: the ATmega has to have a functioning boot loader. Placing a boot loader into an ATmega is a bit more complicated (and involves other things such as “fuses”). It’s a chicken-and-egg problem.

This is where the ISP programmer comes in. An ISP programmer is a fairly simple piece of hardware. In fact, you can make your own, as I’ve described in several posts on this weblog. See this and this post for some quick solutions which require nothing more than a working Arduino or JeeNode.

The good news, is that you usually don’t have to worry about installing the boot loader – it’s all done for you. Once. For the mechanism described above, you’ll never need an ISP programmer.

Some people actually prefer to use the ISP technique for uploading their sketches. In fact, sometimes you have no choice, i.e. when you need the serial port at all times, or when you can’t spare the 1..4 Kb required by the boot loader code, or when working with ATtiny chips which don’t support bootloaders.

In thoses cases, you’ll need a setup with an ISP programmer. But for most people doodling around with ATmega’s and the Arduino IDE, the above boot loader mechanism is usually very convenient and the easiest to get going.

Either way, it helps to understand the process. I hope the above was helpful in that direction.

Six lousy pins!

In Hardware on Jul 3, 2010 at 00:01

This has been a somewhat frustrating week. Just when June ended up being the biggest month ever for the shop (no doubt propelled by the special June discount), Mr. Murphy strikes again.

I’ve been waiting for several weeks now for a batch of pin headers. Had lots of them when I ordered, but since the 6-pin headers are included in just about everything, I still ended up running out sooner than expected.

This is the cuplrit – the 6-pin female port header connector:

Dsc 1764

It’s not easy to find alternatives, because I insist on having a clean 6-pin header, not some cut-off-from-a-long-strip header with very rough plastic edges on both sides.

All would have been well, if the shipment had gone its normal route. But it’s been in this country, at Schiphol Airport, for over a week now! And some lazy bum in customs seems to be sitting on it. The irony is that they do this to charge me 19% VAT – which I then get back (being a company) a few months later. More paperwork.

I’ve started shipping some packages without this item, but often even that is not an option. I can’t really send out new JeeNode kits without these headers, for example – it really wouldn’t solve anyone’s problem.

So there you go. If you’re among the several dozen people waiting for the goodies from Jee Labs to get sent to you: I’m just as eager as you to get it all resolved! – probably more, because here, all those frustrations tend to accumulate by proxy :(

Affected are: Plug Headers, JeePlug Packs, JeeNode kits, Wireless Starter Packs, and probably a few more.

(A separate issue is that the JeeNode USB won’t be back in stock for at least another week)

What can I do? Order lots and lots of them, evidently. Which is what I did (many thousands) – in the hope that this silly state of affairs won’t happen again. And offer my sincere apologies to everyone waitng.

Six lousy pins! And in two days I’ll probably be drowning in them…

Update – Just got a letter from the postal service: no invoice included. Need to supply info. By mail. So there is light at the end of the tunnel, but it’s going to take several more days. My apologies. All for some VAT, which gets refunded later. What a silly world.

Fixing a faulty ATmega (Arduino)

In AVR, Hardware, Software on Jul 2, 2010 at 00:01

After a recent post on how to repair an ATmega with a faulty (or even missing) bootloader and helping someone out with it, it occurred to me that this mechanism will work for any Arduino – not just JeeNodes.

Any 3.3V or 5V Arduino’ish system which lets you upload the isp_repair.pde sketch can be used to program any board with an ISP connector on it. The code is for ATmega328’s, because that’s all I use around here these days.

The trick is to hook up a few power and I/O lines in a specific way:

Screen Shot 2010 06 28 at 16.59.11

I’m using an Arduino Pro as example, but that’s just one of many you could use. Now connect these six wires:

  • ISP pin 1 = BLUE = Digital pin 4
  • ISP pin 2 = RED = VCC (+5V)
  • ISP pin 3 = GREEN = Analog pin 0
  • ISP pin 4 = PURPLE = Analog pin 3
  • ISP pin 5 = ORANGE = Digital pin 7
  • ISP pin 6 = BLACK = GND

Note: if your working board operates at 3.3V, then you should connect the RED wire to +3.3V, not +5V (that way signal levels and power supply will match). Also: the target board should not be connected to anything, since it will be powered through the ISP connector.

The only thing left to do is to upload the isp_repair.pde sketch to your working board, and open up a console window. You should see something like this:

That’s it – disconnect all the wires. The ATmega on the target board now has a boot loader and the standard pin 13 blink sketch. Ready again to accept whatever sketch you upload to it!

Uploading without avrdude

In AVR, Software on Jul 1, 2010 at 00:01

In the future, I’d like to upload new firmware to JeeNodes, Arduino’s, and other AVR boards through channels other than a serial port or USB. Uploading to a “slave plug” via I2C would be neat, for example.

That means the standard avrdude won’t do. Besides, after having coded various types of ISP sketches recently, I realized that the upload mechanism is really quite simple. If all you need is STK500 compatibility (as used by several ISP programmers and by the Arduino boot loader itself), then avrdude is overkill.

So here’s a demo “rig” for JeeMon which does the same thing as avrdude, i.e. upload a sketch over a serial port:

Screen Shot 2010 06 27 at 23.19.45

That removes the need to compile and install avrdude. Better still, this should work as is on every platform supported by JeeMon.

(Note: the above code is now part of JeeMon, but the source code can also be found here on the web)

Onwards!

Update – the above code has been integrated into JeeMon as new Upload rig – with dudeLoader renamed to “stk500” and readIntelHex now called “readHexFile”. Here’s a new demo “application.tcl” using this:

Screen Shot 2010 06 28 at 01.46.48

Now works with the Arduino boot loader as well as with the Flash Board ISP programmer (add “19200” arg).

(Reminder: the Jee Labs shop will be closed from July 14th through August 14th)