4 ms·

Not much to tell. I was looking at maybe using an RTOS to reduce VoIP jitter. Picked RTEMS on a generic x86 PC, there wasn't a driver for the NICs we were using

Not much to tell. I was looking at maybe using an RTOS to reduce VoIP jitter. Picked RTEMS on a generic x86 PC, there wasn't a driver for the NICs we were using, so wrote a driver using a FreeBSD driver (for an unrelated chip) as a base template and a Linux driver for a related chip as a kind of spec. Don't remember if had to do the top-end or just the bottom-end for the NIC. The driver worked fine on RTEMS, but the UDP stack was basically straight from FreeBSD and was definitely not real-time. Wound up not being much of an improvement, it wasn't worth rewriting the entire UDP stack or introducing RT to my non-RT team, so dropped the idea. Did find a performance bug in the core I don't remember much about (unfortunate choice of operations that flushed caches or something and realllly slowed down... something). Submitted the half-a-dozen line patch upstream that Joel Sherrill pulled without drama, even though my test case was overly complicated and opaque.

elcritch 4y ago
What limited the networking stack in terms of RT? I'm having difficulty imagining where the real time blockers would be, but curious as thats a big motivator for me.
- mgsouth 4y ago
  We wanted to handle 10's K of VoIP channels, on multiple processors, with each channel sending a UDP packet every 20 mSec +/- a couple of mSec. The network stack was just the standard FreeBSD "box" and had no provision for that kind of clocking or time-to-process guarentees. Our application could call send() on a fixed schedule, but couldn't control when the packet would actually pop out. The process time would vary because of things such as global locks for buffers, varying code paths (do we ARP?), and varying amounts of competition from other processes. The only way to get reasonably consistent times was to way over-provision the system so it was loafing. We were already doing that with Solaris, so not worth it. What we needed was something like Linux's scheduling queue disciplines [0]; it would have allowed us to say "here's a packet, schedule it for 10 mSec from now", and we'd be good as long as the vast majority of packets took less than 10 msec to wend through the stack. But that didn't exist at that point. [0] https://tsn.readthedocs.io/qdiscs.html https://tsn.readthedocs.io/qdiscs.html
okl 4y ago
In my experience, RTEMS works great if your scheduling/timing is in the ms-range but if you need precise timing in the us-range, especially on smaller MCUs, then you will run into lots of trouble with jitter from different sources, for example because certain timers are checked sporadically, messing with your cache. I don't know if the situation has improved since then.
- adrian_b 4y ago
  I believe that what you say is correct when you use RTEMS with one of the existing BSPs, with no changes. Because RTEMS is open-source and also because its internal structure is many orders of magnitude easier to understand than that of something like Linux, or even of some *BSD, extreme customization can be done with relatively modest work. When you do not need them, it is easy to remove completely the implementations of many parts of the API and of many classes of kernel objects, to obtain a much leaner RTOS kernel. Any causes of jitter from the kernel or device drivers (if they occur in higher priority tasks, as the lower priority tasks should not be able to interfere) should be relatively easy to identify and patch. I have used once RTEMS for an application with many channels of synchronous multiplexed telephony, with very tough jitter requirements, and there have been no problems, but it was not RTEMS out-of-the-box, it was customized for the application, i.e. with about half of the kernel removed, a new interrupt handler added, and with custom device drivers.

elcritch 4y ago

What limited the networking stack in terms of RT? I'm having difficulty imagining where the real time blockers would be, but curious as thats a big motivator for me.

mgsouth 4y ago
We wanted to handle 10's K of VoIP channels, on multiple processors, with each channel sending a UDP packet every 20 mSec +/- a couple of mSec. The network stack was just the standard FreeBSD "box" and had no provision for that kind of clocking or time-to-process guarentees. Our application could call send() on a fixed schedule, but couldn't control when the packet would actually pop out. The process time would vary because of things such as global locks for buffers, varying code paths (do we ARP?), and varying amounts of competition from other processes. The only way to get reasonably consistent times was to way over-provision the system so it was loafing. We were already doing that with Solaris, so not worth it. What we needed was something like Linux's scheduling queue disciplines [0]; it would have allowed us to say "here's a packet, schedule it for 10 mSec from now", and we'd be good as long as the vast majority of packets took less than 10 msec to wend through the stack. But that didn't exist at that point. [0] https://tsn.readthedocs.io/qdiscs.html https://tsn.readthedocs.io/qdiscs.html

okl 4y ago

In my experience, RTEMS works great if your scheduling/timing is in the ms-range but if you need precise timing in the us-range, especially on smaller MCUs, then you will run into lots of trouble with jitter from different sources, for example because certain timers are checked sporadically, messing with your cache. I don't know if the situation has improved since then.

adrian_b 4y ago
I believe that what you say is correct when you use RTEMS with one of the existing BSPs, with no changes. Because RTEMS is open-source and also because its internal structure is many orders of magnitude easier to understand than that of something like Linux, or even of some *BSD, extreme customization can be done with relatively modest work. When you do not need them, it is easy to remove completely the implementations of many parts of the API and of many classes of kernel objects, to obtain a much leaner RTOS kernel. Any causes of jitter from the kernel or device drivers (if they occur in higher priority tasks, as the lower priority tasks should not be able to interfere) should be relatively easy to identify and patch. I have used once RTEMS for an application with many channels of synchronous multiplexed telephony, with very tough jitter requirements, and there have been no problems, but it was not RTEMS out-of-the-box, it was customized for the application, i.e. with about half of the kernel removed, a new interrupt handler added, and with custom device drivers.