Computing stuff tied to the physical world

Linux real-time hickups

In Hardware, Software on Sep 24, 2012 at 00:01

To give you an idea why a simple little Linux board (the Carambola on OpenWrt in this case) is not a convenient platform for real-time use, let me show you some snapshots.

I’ve been trying to toggle two I/O pins in a burst to get some data out. The current experiment uses a little C program which does ioctl(fd, GPIO_SET, ...) and ioctl(fd, GPIO_CLEAR, ...) calls to toggle the I/O pins. After the usual mixups, stupid mistakes, messed up wires, etc, I get to see this:

SCR18

A clean burst on two I/O pins, right? Not so fast… here’s what happens some of the time:

SCR19

For 10 ms, during that activity, Linux decides that it wants to do something else and suspends the task! (and the system is idling 85% of the time)

If you compare the hex string at the bottom of these two screenshots, you can see that the toggling is working out just fine: both are sending out 96 bytes of data, and the first few bytes are identical.

The situation can be improved by running the time-critical processes with a higher priority (nice --20 cmd...), but there can still be occasional glitches.

Without kernel support, Linux cannot do real-time control of a couple of I/O pins!

  1. Linux has never been deterministic like you seek, its not a micro-controller. Looking in RTAI or other real-time approaches.

  2. Is this with the PREEMPT_RT patchset?

  3. There IS kernel support. Did you set the scheduling policy of that process to (soft) realtime? See “man schedtool”. You might need to apt-get install it first.

    • Alas, might not be an easy option with OpenWrt (“schedtool” is not a standard package). But good to know for other Linux’es, thx.

  4. Xenomai looks very interesting, didn’t realize it was up on the R-Pi already. If I understand the webpage, it’s showing 14 usec latencies. Not at all bad for a Linux box, but of course a simple micro can still improve on that. Right now I’m using a Parallax Propeller at 100 MHz as my real-time front end to a R-Pi. It’s the easiest/cheapest way I’ve found to get 10 ns resolution timing measurements.

  5. Remember to set the scheduler parameters and priority. Nice is not the right tool here.

  6. Even setting priorities and scheduling won’t completely save you. If, for example, you are on a network and a packet comes in..the resulting interrupt with fire the driver off and you’ll get a timing hiccup. Even the clock interrupts will give you minor timing hiccus as their ISR runs.

  7. Embedded Linux boxes are great but why not add a small, dedicated Atmel chip for the real-time issues? Then use DFU to update the Atmel chip as needed.

    That being said, small loadable kernel modules are relatively easy to create and maintain.

    • Yes. Note that I’m not proposing to use Linux for everything, just pointing out trade-offs.

  8. Introducing (in my dreams) the Jeepi … a shield board that sits on top of a Pi, connected by the GPIO header, and equipped with the usual Jee niceness of a 328, RFM12B, standard Jee ports and maybe an RTC as well? Considering how much Pi there is out there looking for a use, it could be very popular!

    • I have toyed with the idea of building a 3.3v powered device very similar to how you describe…

      Then the Gertboard was released, and it rather took the enthusiasm out of it for me. The Gertboard doesn’t have the RFM12B, but I doubt there is much preventing you attaching one bar 5v to 3.3v level conversion.

  9. “Got any more details…?” I have some code showing what I’m doing with the R-Pi and Propeller (which unfortunately this margin is too small to contain :-). Will post a link soon.

  10. I worked with Concurrent Computer’s real time unix years ago and found similar dead times, even though it had lots of kernel support for real time, the full POSIX Realtime extensions, etc. Turns out the time gap was due to disk sync activity, where everything was stopped for milliseconds while it did its thing. To fix, I had to suspend disk syncing while real-time programs were running, which could be dangerous. That was easy to do, but you had to remember to re-enable it.

  11. Here is a post including the Propeller code for a 5-channel timing analyzer, comparing 5 1-PPS input signals with 10 ns resolution. http://www.raspberrypi.org/phpBB3/viewtopic.php?f=41&t=18335&p=181077#p181077

  12. In my humble opinion, for a linux-like (actually POSIX) hard real-time system, you should look at QNX.

    This RTOS is microkernel-based (you can (re)load any driver without rebooting) and has a very very small footprint. AFAIR, back to 1996 or so, they won great prizes with their full featured RTOS with windowed GUI, web server and web browser packaged on a single 1,44 floppy disk.

    Today, QNX code went Open Source and you can freely download and use their development environment for non commercial developments. That’s great for hobbyists like us.

    And it runs on many architectures as well as on x86.

    • I used QNX years ago for some very demanding real time work, and it was really great. As Fred mentions, writing a real (not user-space) driver was very convenient, since they are separate processes that can be created and destroyed without rebooting. GUI development was also a strong point as I recall. To use it I had to relearn some standard linux techniques and conventions, but in return I had a much more deterministic system than most others.

    • For Linux to do anything hard real-time, the PREEMPT_RT kernel branch is required, and the proper calls to enable POSIX real-time scheduling for processes/threads. It is then up to proper device driver and proper device implementation to reach deterministic low latency. Even then, expect ~20-150 microseconds response times depending on the exact hardware and device drivers. See https://www.osadl.org/Continuous-latency-monitoring.qa-farm-monitoring.0.html

      QNX is much better suited if you really need hard real-time, on the other hand, using a CPLD, FPGA, microcontroller on the side might often be the best trade-off.

      Fred, you mention “Today, QNX code went open source”, however from what I remember this was years ago and some parts (micro kernel)where closed again after the Harman Kardon take-over?

  13. You’re right, Leon, I was timeshifted. QNX in not open source any more and, according to some aficionados, has never truly been.

    Nevertheless, there is an academic and non-commercial license that is “free and are perpetual for non-commercial developers”. Cheers !

Comments are closed.