Computing stuff tied to the physical world

JeeNode USB problems

In Hardware on Nov 7, 2011 at 00:01

A new problem has surfaced with the JeeNode USB v3: it turns out that the most recent units have accidentally been assembled and shipped with a 3.0V regulator instead of the required 3.3V regulator.

This is a problem for two reasons:

  • all attached devices will run at 3.0V instead of 3.3V
  • the ATmega328 runs further out of spec at 16 MHz

The second (older) problem is that the original OptiBoot loader had a bug in combination with the FTDI interface as used on JeeNodes. This has affected a number of people over the past months, but was resolved before the summer by switching to a modified version of OptiBoot on all JeeNode USB’s and JeeLinks (the fix was to change the watchdog timeout from 500ms to 1s). This problem appears to only matter on Windows systems.

Still, these past few weeks, I was surprised to see yet more problem reports on the forum about “not being able to upload a new sketch” – which is how the OptiBoot bug manifested itself.

My hunch is that this new 3.0V regulator mixup can lead to the same problem, but for a different reason: when running at 16 MHz from 3.0V, the ATmega is running out of spec, or in other words: it’s being over-clocked.

Over-clocking works fine, mostly. But not always. It may depend on chip fabrication, operating temperatures, and maybe even on what the chip is doing.

In short: there are probably still a few dozen JeeNode USB’s out there which are not performing properly due to either the OptiBoot bug, or the wrong regulator (probably not both, because the OptiBoot bug was solved before the regulator mixup happened).

How to diagnose these problems

If you have a JeeNode USB which isn’t working as expected, you can take these steps:

  • try to upload a new sketch to it (set the board type to “Arduino Uno”)
  • if that doesn’t work, and you’re certain that you did it right, you can return it

If you can upload sketches (best chances are on Mac and Linux), then you can perform an additional test to check the regulator voltage, by uploading the following sketch:

Screen Shot 2011 11 06 at 21 22 03

This only works on the JeeNode USB, which has a voltage divider connected to analog pin 6.

Now open the serial console and watch the values, reported once a second:

  • with a good 3.3V regulator, they will be about 4200
  • with an incorrect 3.0V regulator, they’ll be ≈ 4600

The 4600 value comes from making an incorrect calculation, due to the assumption that the total input range is 6600 mV, whereas with a 3.0V regulator it is actually 6000 mV. If you change the 6600 to 6000 on a bad unit, it’ll again report about 4200, which is indeed the voltage on the PWR pin when powered from the USB bus through the LiPo charger (even when no LiPo battery is attached).

If you have the wrong regulator, you can of course also return it and I’ll repair it and send it back ASAP. In that case, please return it to the address listed on this page and include your own address so I can send it back.

So what happened?

Well, these units are being assembled in small quantities for me by a third party. Somewhere, someone messed up and placed the wrong chips on the board. These are SMD chips, so after such a mixup happens it’s pretty unlikely that anyone would spot it from the tiny lettering on the regulator. During programming, the power is supplied by the ISP programmer, i.e. the intended 3.3V. And during testing they will probably all have passed because the ATmega’s do tend to work correctly, even when over-clocked to 16 MHz @ 3.0V.

Sooo… I / we messed up, and delivered flawed units. For which I am really sorry, and for which I sincerely apologize. It shouldn’t have happened, but it did. Mr. Murphy decided to drop by, clearly.

I find this deeply embarrassing. I know how frustrating it is (who doesn’t?) when dealing with products which don’t work as expected – as promised, in fact. This post is obviously about damage control. Doing what is needed to address the issue. Not just for now, but also to make sure it won’t happen again. I’ve got some ideas on that, which I hope will have an effect – an invisible one, in that such mishaps will not happen again. Or much less.

Because, let’s face it: mistakes happen.

For a normal business, the way to deal with mishaps like these, is no doubt to improve the QC/QA procedures, install a sense of more accountability and stricter procedures, and basically “improve the system”. This is a slow and costly process. And the way to handle it is to scale up and amortize the expenses. Economies of scale.

But JeeLabs is not a normal business in this respect. Scale is not my goal, sustainability is. Which means that quality and durability is just as important to me as to anyone, but there are no others to “install a sense of more accountability” into. Like any business, I try to pick my suppliers with care, and work with them so we all perform as well as we can, but that’s about it. Problems must be solved at JeeLabs’ current scale of operation.

Maybe it can’t be done, maybe it can. But IMO it’s worth trying, because more small business-like “shops” are being started every day. So why not try and figure out how to produce stuff – good stuff – on a smaller scale?

Anyway. If you’ve got a flawed JeeNode USB, and want to have it fixed ASAP, please return it. New ones will be ready in three weeks, so if you’d rather use it and receive the replacement first, please let me know by email and I’ll send a replacement out once available. If so, there’s no need to send it back before the new one arrives.

Again, my apologies for the mixup. Products are fun – and producing quality products is hard.

  1. Jc, Indeed, a difficult challenge to design robust QA in a small scale environment. Perhaps the final check philosophy can be tuned somewhat. A basic functional check is needed on 100% of the units to catch individual failures (gross component or assembly errors) but a subtle systematic error like this needs an additional approach. A useful technique is in fact to explore the margin envelop on a representative sample. Varying a stress parameter (low supply voltage, high clock rate, temperature extreme) may weed out an additional failure and trigger suspicion about the batch. For example, running a loop back test with low supply rail while ‘cold shocked’ in the freezer. The additional time for a small sample is not too onerous and I’ve seen it detect many hard to find issues, including out of spec chips and hairline trace cracks. The short cut approach in the field was drop the supply rail and a squirt of “freezer”, but not very eco-friendly depending on the cooling agent used. Keeping QA/QC costs, time and effort in balance is tough – having a “no questions” replacement policy with a loyal user-base to catch the third sigma is perhaps part of the equation also

  2. Oh well, these things happen. Especially when you’re working with components the size of a grain of rice. Even if you know what you are looking for, reading the numbers on SMD components is a challenge at the best of times.

    BTW, there is an easier way to check the regulator assuming you have a multimeter and the JeeNode USB is not in a box.

    First provide your JN USB with a good supply of power. USB or some good batteries. Check the voltage between GND and PWR on one of the ports. Hopefully you have something above 3.4v here.

    Now once you have a good supply, check the voltage between GND and +. You should see 3.3v (or allowing for tolerance, something very close. Certainly within 0.1v).

    If you see 3.0v then you have the wrong regulator.

  3. For my Jeenode USB I had to change in the Arduino Uno section in boards.txt upload.protocol from stk500 to arduino, and also add the arduino programmer section to avrdude.conf (it existed in my system supplied /etc/avrdude.conf, but not in the one I got with arduino-0022). Is this an unrelated problem?

    • This is unrelated. Not sure what’s going on – sounds like an Arduino issue. Are you uploading via an ISP programmer instead of the normal serial USB mechanism?

  4. Ok. I use the normal serial USB. I’m running Ubunutu 10.04, I figured it had some old tools or something (I didn’t try Windows/newer Ubuntu since I got it working).

Comments are closed.