6 ms·

The BeagleBone's I/O pins: inside the software stack that makes them work

63 points by dwaxe 10y ago

corysama 10y ago
TLDR for old C hackers: The pins are memory-mapped registers at fixed physical memory addresses. You can use mmap to map the physical address to your process's virtual address space. From there, you can just read and write the array. Note that you will run into hardware timing issues if you try to do this too fast. > By using mmap and toggling the register directly from a C++ program, the GPIO can be toggled at about 2.8 MHz, much faster than the device driver approach.[3] If you think this approach will solve your problems, think again. As before, there are jitter and multi-millisecond dropouts if there is any other load on the system. There is a system file abstraction in place. But, it seems like a lot more work for a 20x overhead vs manipulating a struct. The tiny example program at the bottom is what more my style. http://www.righto.com/2016/08/the-beaglebones-io-pins-inside-software.html#ref3 http://www.righto.com/2016/08/the-beaglebones-io-pins-inside... I read through this article rather quickly. I'd be happy to hear if I got anything wrong.
- kens 10y ago
  The interesting thing about the BeagleBone is there are two microcontrollers inside the processor chip that you can use if you want to do anything that's genuinely real-time. (I'm in the middle of using these to emulate an Ethernet for the Alto restoration. There's a big learning curve to use them.) As far as the file system abstraction, I'm not sure if it's clever or over-engineered. It seems like a whole lot of infrastructure to simply flip a bit. And the BeagleBone device trees - I can see why they did that, but it seems like there must be a better way.
  - revelation 10y ago
    Do you use the MII interface for the PRU-ICSS to do the Ethernet? I can't really understand why they don't have even a one-page documentation on that stuff, but I've found code to interface that in parts of their industrial SDK.
    - kens 10y ago
      I'm using bit-banging since the Alto's Ethernet is only 3 megabits/sec so the PRU (the microcontroller inside the processor chip) is fast enough. I'm pretty sure that using the MII (Media-Independent Interface, i.e. the Ethernet controller inside the PRU inside the chip) would just cause me more problems.
  - Something1234 10y ago
    Did you ever find any good tutorials on using the beaglebone's micro controllers?
- cnvogel 10y ago
  mmap'ing /dev/mem and directly poking at registers is fine as long as you know what you are doing and one of the best ways to explore and play with something not yet "properly" supported... ;-) But of course for this you'll have to map whole memory pages (there's no smaller granularity in providing direct memory access to registers), and this typically includes all registers of (here:) the GPIO peripheral. Only toggling outputs via the set/clear registers has little potential for damage, but anything requiring a read-modify-write will race against anything the kernel might want to do with GPIO pins in this bank. And only a particular boards' configuration can tell you if there's anything important on any other pin. A serial bus might for example want to change the direction of a pin, a keyboard might enable/disable interrupts on pin-changes, there's SD-card insertion, control of a WiFi cards' transmit functionality (rfkill), power to USB busses, detection of USB OTG cables. A good compromise, if you really have to toggle pins fast, might be to first reserve and configure a GPIOs functionality via sysfs, and then only use the mmap() for the actual input/output. Or write a proper kernel module. There are inline functions/macros (for in-kernel-use) that basically do the single-register-write.
- nbadg 10y ago
  > By using mmap and toggling the register directly from a C++ program, the GPIO can be toggled at about 2.8 MHz, much faster than the device driver approach. I mean, it doesn't need to be C++. I was getting pretty close to that speed with the same strategy (mmap the registers directly) with a python library I wrote. It's not exactly pretty, and was the first semi-major python project I was working on, but it was fairly functional.[1] If I remember right (it was quite a while ago), I was routinely seeing speeds around 1MHz. BUT you'll occasionally see hiccups of as much as tenths of a second, because Linux is decidedly not real-time. I ran into a pretty tall brick wall when I started trying to do pin muxing[2] using the same strategy. Going pretty far down this path lead me to the conclusion that the best strategy for boards like BB/RazPi is to run them on an exokernel and then have per-language libraries that access the registers, with the exokernel[3] enforcing memory access rules. You could even use that library on multiple platforms by declaring the memory mapping for the registers as, for example, a json schema. All of the systems on a chip that I looked at datasheets for while I was working on that project (which was admittedly an extremely limited sample size) go through the same strategy of fixed physical memory addresses for peripheral pins. [1] https://github.com/Badg/hwiopy https://github.com/Badg/hwiopy [2] Pinmuxing is the process of switching functions for a given pin. Basically, SoC's are so small that the chip designers will put more than one thing, say USB + GPIO + HDMI, on the same pin. Muxing -- which is ultimately also controlled by a fixed physical register -- switches between modes. Muxing is only available from code running in protected mode, and besides, Linux wets the bed if you try to do this at runtime. That may sound silly -- why would I want to switch between USB and HDMI at runtime -- until you either think of other examples (for example, GPIO vs analog in), or realize how much of a pain in the ass it makes it to do a bunch of operating system modifications just to reconfigure your pinout. Basically, it makes development an enormous pain in the ass. [3] Exokernels are really cool, basically the lowest level of abstraction you could put in a kernel: it promises not to let you fuck up the hardware, and provides some very basic abstractions, but otherwise just gives you free reign. Very much the "consenting adults" strategy. https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel
  - amboar 10y ago
    I don't know how recently you've played with pinmux on Linux, but my understanding is the pinctrl subsystem should handle runtime muxing without wetting the bed. What issues did you run into?
fake-name 10y ago
The BeagleBone suffers from the same exact issues the Raspberry Pi does, namely that there are only two real ways to run the peripherals: The horribly, horribly slow filesystem interface, and "Open /dev/mem RW! YOLO! Run Apache as root for your crappy internet of shit project!". There are basically no actual sane drivers for doing anything fast. At one point, I spent a while looking at writing my own driver, but apparently I'm not a good enough hacker to write kernel drivers.
- snops 10y ago
  Except the BeagleBone has PRUs, which are two 32 bit microcontrollers on the same silicon that can also access the GPIO peripherals. They share memory with the main processor but can be exclusively dedicated to your needs, and so peripheral access can be very fast indeed. Here is someone who has used them to turn the BeagleBone into a 100MHz Logic Analyser: http://theembeddedkitchen.net/beaglelogic-building-a-logic-analyzer-with-the-prus-part-1/449 http://theembeddedkitchen.net/beaglelogic-building-a-logic-a...
- amboar 10y ago
  If your running relatively recent kernels on the board there's now a character device interface for gpios: https://patchwork.ozlabs.org/patch/580307/ https://patchwork.ozlabs.org/patch/580307/
- bsder 10y ago
  Did you try this reference: https://groups.google.com/forum/#!topic/beagleboard/dyuax5415dc https://groups.google.com/forum/#!topic/beagleboard/dyuax541... I found it quite invaluable for understanding how to write a kernel driver for things like the RaspberryPi or the BeagleBone. The fact that so many things about embedded hardware are so poorly documented is a real problem. Unfortunately, there just aren't that many of us hacking these at a kernel level.
revelation 10y ago
Hilarious, GHz+ processors and they can toggle a GPIO pin at 2.8MHz with millisecond jitter (so, not really 2.8MHz). This is what it has come to. Honestly, the BeagleBone (or the TI AMx series) has got it right: modern processors with their layers upon layers of caching and modern operating systems are utterly useless for this task. The deterministic 200MHz PRU-ICSS processors however are perfectly suited to do it. Don't bother fighting the OS and 30 years of CPU evolution when a PRU can toggle at 5ns intervals.
- bsder 10y ago
  > Hilarious, GHz+ processors and they can toggle a GPIO pin at 2.8MHz with millisecond jitter (so, not really 2.8MHz). Um, allow me to flip your statement around "GHz+ processors so CHEAP that you can use them to toggle a GPIO pin." However, that's not generally how I use a BeagleBone. The BeagleBone provides the user interface (USB, Ethernet, etc.) while a cheap cape provides the hardware specific part. BeagleBone's have absolutely proliferated around me once the started being able to run mainline Debian instead of weird things like Angstrom. That having been said--I have bit-banged the bone. The annoyance isn't the speed so much but you have to be in Supervisor mode to change the direction of the pin. That's really annoying and is a limitation of the TI Sitara SoC. The RaspberryPi does not seem to have that limitation. So, if you're gonna bit-bang, an RPi is probably a better choice. As for why it's so slow, there are two parts. 1) mmap() marks the writes as strongly-ordered and slows things down You can actually get to 12.5MHz. https://groups.google.com/forum/#!category-topic/beagleboard/xuHRlDL19NI https://groups.google.com/forum/#!category-topic/beagleboard... 2) The system seems to have internal packetized transfer buses that gate the speed The bus frequency determines the ultimate speed. This is no different from the much smaller microcontrollers, except that the smaller microcontrollers tend to run their bus frequencies at the same speed as the internal frequency since they are running much slower.
  - revelation 10y ago
    The PRUs have no bus, their GPIO is in the register file and memory is SRAM. They do have access to the beefy ARMs bus, which makes them infinitely more useful than a cape. More often than not a cape will not solve the problem because quelle surprise, there is then just no low latency reliable path to control the cape from the ARM. So you've just shifted the issue for a 100x increase in complexity.
ambrop7 10y ago
It's sad how the software stacks have gotten to this point where toggling a pin is so horribly inefficient. And there's no good reason it has to be. - Instead of filesystem operations there could be ioctl, which is still safe from the OS perspective (yes there is overhead compared to direct register manipulation but much less than putting strings into the /proc filesystem). - With a real-time kernel combined with understanding of the scheduling and relevant concepts (e.g lock memory), it should be possible to achieve much better latencies, possibly on the order of tens of microseconds. I'm not sure how much variance there invariably will be due to processor cache or other sources but I suspect it's too little for most people to care. - Finally, does anyone have an idea about the possibility of handling interrupts in user space? I don't mean "second-level" (bottom-half) handlers, but the raw interrupt handlers, without any overhead of scheduler or work queue in front of your code. Sure there are more things to be careful about like making sure you're acknowledging an interrupt but that's a given. Doesn't it mostly boil down to the kernel setting up appropriate context like virtual memory within the ISR? Do CPU designs prevent doing this? Saying to "leave the real-time things to microcontrollers" is basically giving up. Hardware should be as general-purpose as possible and this also applies to being able to do real-time things. I'm not saying the PRUs are not useful, but they are a very awkward solution for probably most real-time programming problems that are not currently easily solvable with just the main CPU. These things use a custom assembly language and there isn't even a C compiler for them! I would be much happier with an integrated ARM microcontrollers.
- revelation 10y ago
  There is a C compiler for the PRUs, but frankly you're missing the point if you're asking for that, the whole reason of their existence is to manage processes that require extremely deterministic timing. You might update your C compiler, find something is compiled differently and now all the timings are off. They are not so much meant as microcontrollers, they are for things you would otherwise have to do on an FPGA to get the timing reliably correct. Tens of microseconds just don't cut it. Some areas such as realtime ethernet protocols require nanosecond-accurate packet insertion. The summary of why things are the way they are is that modern operating systems and CPUs value throughput over everything.

63 points by dwaxe 10y ago

TLDR for old C hackers: The pins are memory-mapped registers at fixed physical memory addresses. You can use mmap to map the physical address to your process's virtual address space. From there, you can just read and write the array. Note that you will run into hardware timing issues if you try to do this too fast. > By using mmap and toggling the register directly from a C++ program, the GPIO can be toggled at about 2.8 MHz, much faster than the device driver approach.[3] If you think this approach will solve your problems, think again. As before, there are jitter and multi-millisecond dropouts if there is any other load on the system. There is a system file abstraction in place. But, it seems like a lot more work for a 20x overhead vs manipulating a struct. The tiny example program at the bottom is what more my style. http://www.righto.com/2016/08/the-beaglebones-io-pins-inside-software.html#ref3 http://www.righto.com/2016/08/the-beaglebones-io-pins-inside... I read through this article rather quickly. I'd be happy to hear if I got anything wrong.

kens 10y ago
The interesting thing about the BeagleBone is there are two microcontrollers inside the processor chip that you can use if you want to do anything that's genuinely real-time. (I'm in the middle of using these to emulate an Ethernet for the Alto restoration. There's a big learning curve to use them.) As far as the file system abstraction, I'm not sure if it's clever or over-engineered. It seems like a whole lot of infrastructure to simply flip a bit. And the BeagleBone device trees - I can see why they did that, but it seems like there must be a better way.
- revelation 10y ago
  Do you use the MII interface for the PRU-ICSS to do the Ethernet? I can't really understand why they don't have even a one-page documentation on that stuff, but I've found code to interface that in parts of their industrial SDK.
  - kens 10y ago
    I'm using bit-banging since the Alto's Ethernet is only 3 megabits/sec so the PRU (the microcontroller inside the processor chip) is fast enough. I'm pretty sure that using the MII (Media-Independent Interface, i.e. the Ethernet controller inside the PRU inside the chip) would just cause me more problems.
- Something1234 10y ago
  Did you ever find any good tutorials on using the beaglebone's micro controllers?
cnvogel 10y ago
mmap'ing /dev/mem and directly poking at registers is fine as long as you know what you are doing and one of the best ways to explore and play with something not yet "properly" supported... ;-) But of course for this you'll have to map whole memory pages (there's no smaller granularity in providing direct memory access to registers), and this typically includes all registers of (here:) the GPIO peripheral. Only toggling outputs via the set/clear registers has little potential for damage, but anything requiring a read-modify-write will race against anything the kernel might want to do with GPIO pins in this bank. And only a particular boards' configuration can tell you if there's anything important on any other pin. A serial bus might for example want to change the direction of a pin, a keyboard might enable/disable interrupts on pin-changes, there's SD-card insertion, control of a WiFi cards' transmit functionality (rfkill), power to USB busses, detection of USB OTG cables. A good compromise, if you really have to toggle pins fast, might be to first reserve and configure a GPIOs functionality via sysfs, and then only use the mmap() for the actual input/output. Or write a proper kernel module. There are inline functions/macros (for in-kernel-use) that basically do the single-register-write.
nbadg 10y ago
> By using mmap and toggling the register directly from a C++ program, the GPIO can be toggled at about 2.8 MHz, much faster than the device driver approach. I mean, it doesn't need to be C++. I was getting pretty close to that speed with the same strategy (mmap the registers directly) with a python library I wrote. It's not exactly pretty, and was the first semi-major python project I was working on, but it was fairly functional.[1] If I remember right (it was quite a while ago), I was routinely seeing speeds around 1MHz. BUT you'll occasionally see hiccups of as much as tenths of a second, because Linux is decidedly not real-time. I ran into a pretty tall brick wall when I started trying to do pin muxing[2] using the same strategy. Going pretty far down this path lead me to the conclusion that the best strategy for boards like BB/RazPi is to run them on an exokernel and then have per-language libraries that access the registers, with the exokernel[3] enforcing memory access rules. You could even use that library on multiple platforms by declaring the memory mapping for the registers as, for example, a json schema. All of the systems on a chip that I looked at datasheets for while I was working on that project (which was admittedly an extremely limited sample size) go through the same strategy of fixed physical memory addresses for peripheral pins. [1] https://github.com/Badg/hwiopy https://github.com/Badg/hwiopy [2] Pinmuxing is the process of switching functions for a given pin. Basically, SoC's are so small that the chip designers will put more than one thing, say USB + GPIO + HDMI, on the same pin. Muxing -- which is ultimately also controlled by a fixed physical register -- switches between modes. Muxing is only available from code running in protected mode, and besides, Linux wets the bed if you try to do this at runtime. That may sound silly -- why would I want to switch between USB and HDMI at runtime -- until you either think of other examples (for example, GPIO vs analog in), or realize how much of a pain in the ass it makes it to do a bunch of operating system modifications just to reconfigure your pinout. Basically, it makes development an enormous pain in the ass. [3] Exokernels are really cool, basically the lowest level of abstraction you could put in a kernel: it promises not to let you fuck up the hardware, and provides some very basic abstractions, but otherwise just gives you free reign. Very much the "consenting adults" strategy. https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel
- amboar 10y ago
  I don't know how recently you've played with pinmux on Linux, but my understanding is the pinctrl subsystem should handle runtime muxing without wetting the bed. What issues did you run into?

fake-name 10y ago

The BeagleBone suffers from the same exact issues the Raspberry Pi does, namely that there are only two real ways to run the peripherals: The horribly, horribly slow filesystem interface, and "Open /dev/mem RW! YOLO! Run Apache as root for your crappy internet of shit project!". There are basically no actual sane drivers for doing anything fast. At one point, I spent a while looking at writing my own driver, but apparently I'm not a good enough hacker to write kernel drivers.

snops 10y ago
Except the BeagleBone has PRUs, which are two 32 bit microcontrollers on the same silicon that can also access the GPIO peripherals. They share memory with the main processor but can be exclusively dedicated to your needs, and so peripheral access can be very fast indeed. Here is someone who has used them to turn the BeagleBone into a 100MHz Logic Analyser: http://theembeddedkitchen.net/beaglelogic-building-a-logic-analyzer-with-the-prus-part-1/449 http://theembeddedkitchen.net/beaglelogic-building-a-logic-a...
amboar 10y ago
If your running relatively recent kernels on the board there's now a character device interface for gpios: https://patchwork.ozlabs.org/patch/580307/ https://patchwork.ozlabs.org/patch/580307/
bsder 10y ago
Did you try this reference: https://groups.google.com/forum/#!topic/beagleboard/dyuax5415dc https://groups.google.com/forum/#!topic/beagleboard/dyuax541... I found it quite invaluable for understanding how to write a kernel driver for things like the RaspberryPi or the BeagleBone. The fact that so many things about embedded hardware are so poorly documented is a real problem. Unfortunately, there just aren't that many of us hacking these at a kernel level.

revelation 10y ago

Hilarious, GHz+ processors and they can toggle a GPIO pin at 2.8MHz with millisecond jitter (so, not really 2.8MHz). This is what it has come to. Honestly, the BeagleBone (or the TI AMx series) has got it right: modern processors with their layers upon layers of caching and modern operating systems are utterly useless for this task. The deterministic 200MHz PRU-ICSS processors however are perfectly suited to do it. Don't bother fighting the OS and 30 years of CPU evolution when a PRU can toggle at 5ns intervals.

bsder 10y ago
> Hilarious, GHz+ processors and they can toggle a GPIO pin at 2.8MHz with millisecond jitter (so, not really 2.8MHz). Um, allow me to flip your statement around "GHz+ processors so CHEAP that you can use them to toggle a GPIO pin." However, that's not generally how I use a BeagleBone. The BeagleBone provides the user interface (USB, Ethernet, etc.) while a cheap cape provides the hardware specific part. BeagleBone's have absolutely proliferated around me once the started being able to run mainline Debian instead of weird things like Angstrom. That having been said--I have bit-banged the bone. The annoyance isn't the speed so much but you have to be in Supervisor mode to change the direction of the pin. That's really annoying and is a limitation of the TI Sitara SoC. The RaspberryPi does not seem to have that limitation. So, if you're gonna bit-bang, an RPi is probably a better choice. As for why it's so slow, there are two parts. 1) mmap() marks the writes as strongly-ordered and slows things down You can actually get to 12.5MHz. https://groups.google.com/forum/#!category-topic/beagleboard/xuHRlDL19NI https://groups.google.com/forum/#!category-topic/beagleboard... 2) The system seems to have internal packetized transfer buses that gate the speed The bus frequency determines the ultimate speed. This is no different from the much smaller microcontrollers, except that the smaller microcontrollers tend to run their bus frequencies at the same speed as the internal frequency since they are running much slower.
- revelation 10y ago
  The PRUs have no bus, their GPIO is in the register file and memory is SRAM. They do have access to the beefy ARMs bus, which makes them infinitely more useful than a cape. More often than not a cape will not solve the problem because quelle surprise, there is then just no low latency reliable path to control the cape from the ARM. So you've just shifted the issue for a 100x increase in complexity.

ambrop7 10y ago

It's sad how the software stacks have gotten to this point where toggling a pin is so horribly inefficient. And there's no good reason it has to be. - Instead of filesystem operations there could be ioctl, which is still safe from the OS perspective (yes there is overhead compared to direct register manipulation but much less than putting strings into the /proc filesystem). - With a real-time kernel combined with understanding of the scheduling and relevant concepts (e.g lock memory), it should be possible to achieve much better latencies, possibly on the order of tens of microseconds. I'm not sure how much variance there invariably will be due to processor cache or other sources but I suspect it's too little for most people to care. - Finally, does anyone have an idea about the possibility of handling interrupts in user space? I don't mean "second-level" (bottom-half) handlers, but the raw interrupt handlers, without any overhead of scheduler or work queue in front of your code. Sure there are more things to be careful about like making sure you're acknowledging an interrupt but that's a given. Doesn't it mostly boil down to the kernel setting up appropriate context like virtual memory within the ISR? Do CPU designs prevent doing this? Saying to "leave the real-time things to microcontrollers" is basically giving up. Hardware should be as general-purpose as possible and this also applies to being able to do real-time things. I'm not saying the PRUs are not useful, but they are a very awkward solution for probably most real-time programming problems that are not currently easily solvable with just the main CPU. These things use a custom assembly language and there isn't even a C compiler for them! I would be much happier with an integrated ARM microcontrollers.

revelation 10y ago
There is a C compiler for the PRUs, but frankly you're missing the point if you're asking for that, the whole reason of their existence is to manage processes that require extremely deterministic timing. You might update your C compiler, find something is compiled differently and now all the timings are off. They are not so much meant as microcontrollers, they are for things you would otherwise have to do on an FPGA to get the timing reliably correct. Tens of microseconds just don't cut it. Some areas such as realtime ethernet protocols require nanosecond-accurate packet insertion. The summary of why things are the way they are is that modern operating systems and CPUs value throughput over everything.