7 ms·

The problem with airgap test equipment (I work on this kinda of stuff professionally). Is often the redevelopment of real time software for lab and calibration

The problem with airgap test equipment (I work on this kinda of stuff professionally). Is often the redevelopment of real time software for lab and calibration equipment is very expensive. Its easy to think, "Why don't they use [New Hotness Software ]?" Which on the surface seems to be a good idea. Until you absolutely need sub-millisecond precision I/O, then you kinda start to cry when you realize how hard precise timing in computers is. If you use lab equipment in say Linux, BSD, OSX, Windows. Your using a time shared, not Read-Time OS. So your I/O events aren't when the event happens, but when the scheduler is damn well ready to let you know the event happened. The easiest example is a timing equipment I was using to count digital pulses form a quartz crystal. In a 'modern' secure OS I couldn't really get below 0.1% margin of error. Which wasn't acceptable for the equipment which wasn't low enough for our uses. I fall back to an older insecure real-time platform and bumped up to 0.005%. Security is great. I attend security conferences in my spare time and try to stay up to date on the topic. The main problem is when you get into most this computing everything isn't running glibc and win32. Hacking isn't very easy unless you know the system to start with.

RyanZAG 13y ago
Wouldn't it have been better to just stick a specialized piece of hardware/fpga to read directly from the crystal, buffer it, and pass it on? In fact that's what the timing equipment usually does for you. There's no reason to give up security when a few minor changes in architecture will give you performance, security and ease of maintenance and interconnectability.
- valarauca1 13y ago
  Your acting like the read/write time to the hardware/FPGA is negligible which it isn't. When you access hardware on say a PCI bus (which you would in this scenario). Your call to the PCI bus does not take place WHEN you call for it to take place. You call the Kernel, which calls the scheduler, which calls the hardware manager, which calls the driver, which finally processes your request. Now your request is processed all this is dumped and something else runs while the processor waits to hear back from the PCI bus with the response because this takes ages in processor time. Finally an interrupt arrives, is made sense of, the appropriate driver is called, then it gives your information back to your process and you back on your merry little way. :.:.: The problem is when you called OS to start this long chain of events the real world didn't stop. Your real time module is still counting the 32,600,000 pulses per second. This is where you'll get errors. Because often its easy to think things in a computer happen instantly, or so blindingly fast you don't care what happens, order or speed. The situation you described is what originally gave me 0.1% error. Eventually I switched to a more aggressive tact of polling asynchronously in a separate thread and when a process called for the time responding with the latest received time. This got me down to 0.07% error. Still not acceptable. :.:.: Its nice to be starry eyed and think there is no reason to give up security. But sometimes secure software can't do what you need it to do. The more crap you put between you and the metal the more time it'll take to execute. This is logically provable. If you take process A and B. Both are the optimal way to do something. There is no faster way to do this task. Therefore one can assume A and B's execution time are equal. Yet B operates in a secure sand-boxed environment with a time sharing OS. Therefore B's true execution is B+C+D. We know A = B, but for A = (B+C+D) C and D must be Zero, which they can never be in the real world.
  - RyanZAG 13y ago
    I get what you're saying about how it's definitely going to be slower, but I don't understand why the read/write time to the hardware matters if the hardware is buffering the last 100 timings or so. What I'm trying to point out is that you don't have to make your entire system down to the keyboard real time - you only need to make the tiny piece that is doing the physical process real time (with some very simple logic gates that can run far faster than any general purpose CPU), and then sending the results over the PCI/whatever bus in batches later to be processed.
    - valarauca1 13y ago
      The only real time component of the software stack is the kernel. If you want another real time module your screwed because you need to run it in kernel space, but if you have a kernel you can't. Or you run a real time OS which may have problems because they aren't developed with security but IO timing in mind. It's a fundamental flaw of time shared OS's. :.:.: Second security works in a simple way. Cost to secure vs money lost. Lab equipment is expensive. The loss of an entire calibration bench could run into the $250,000 to $1million and beyond range. But redeveloping and entire OS to do this? Your talking about spending 20 to 100x MORE on security then your losses. That's idiotic at best.
      - LukeShu 13y ago
        > The only real time component of the software stack is > the kernel. If you want another real time module your > screwed because you need to run it in kernel space, but > if you have a kernel you can't. I think you're confused about what RyanZAG is saying. If I'm reading correctly, he's saying "don't run the real-time stuff on the CPU." Have that stuff run on a much simpler piece of hardware that is real-time, and runs the real-time code, then have the non-real-time userland on the CPU talk to it in not-real-time. To take your example: > When you access hardware on say a PCI bus (which you would in > this scenario). Your call to the PCI bus does not take place > WHEN you call for it to take place. You call the Kernel, which > calls the scheduler, which calls the hardware manager, which > calls the driver, which finally processes your request. Design the PCI card to have it's own, smaller CPU (or FPGA, or whatever), that does the real-time interaction with the "32,600,000 pulses per second." Don't have the real-time bits depend in any way with the code running on the CPU. Have it buffer the data. The, when the PCI card is accessed by the userland program on the CPU, it dumps the buffer onto the PCI bus. The userland would obviously have be fast enough that the buffer doesn't fill up, but that speed is much less than "real time". You can then work with the data in the userland, running in non-real time.
im3w1l 13y ago
It would be so nice if programs could request a dedicated core for these kinds of things.
- valarauca1 13y ago
  Even that wouldn't work. Their are a lot of parts of the CPU and all of them work together. RAM access, Cache access, Interrupts, and Memory management is handled globally not on a pre-core basis. You'd run into the problem of needing multiple north bridges (do we even have those anymore or are those on chip now?) which you couldn't have. You have to build an entire OS with real time I/O at heart. They do exist, some are secure (Blackberry's platform) but they aren't deployed to the test industry. The most likely version is licensing fees, nobody writings Data Acquisitions for them.
- MertsA 13y ago
  Under Linux you can. You can boot the kernel with the maxcpus argument set to 1 or the number of cores you want the Linux kernel to use and then on a quad core machine you have 3 cores available all the time and setup to where Linux won't ever need to handle interrupts on that core. Then you just start your application and set the affinity to that unused core. You can extend this further by doing things like mmaping in 4GB of memory through the kernels hugepage support and lock the physical to virtual address map so the kernel can't touch your physical block of ram you just allocated. Then you can do things like talk directly to a pci device like a network card and set up DMA directly from a NIC into a buffer in your applications memory. All of this is done completely in userspace but you get all the performance benefits of implementing everything like it was running in Ring 0 and the kernel is not involved in anything apart from the initial setup and teardown. You can build an extremely high performance application basically running on bare metal but with the Linux kernel still running on a different core to handle anything that doesn't directly involve your application and there wouldn't need to be any syscalls between the two to service some request.
  - valarauca1 13y ago
    What I'm confused about. I'll likely just have to play around with this feature at some point. I knew about APIC, but not completely sandboxing cores. Is if you have bare metal operations do you still have access to kernel functionality ala stdio and libc libraries? Normally when you hit bare metal your on your own. I'm just wondering because the idea of writing my own threading, and memory management libraries excites me to no end </sarcasm>. Also if you can call these functions like your in userland then do they block until execution has completed on the other 'kernel' cores? Also if you creating P-Threads elsewhere but not managing their execution on the 'non-kernel' core what happens? >userspace but you get all the performance benefits of implementing everything like it was running in Ring 0 Can you give any literature on this? these terms are contradictory.

RyanZAG 13y ago

Wouldn't it have been better to just stick a specialized piece of hardware/fpga to read directly from the crystal, buffer it, and pass it on? In fact that's what the timing equipment usually does for you. There's no reason to give up security when a few minor changes in architecture will give you performance, security and ease of maintenance and interconnectability.

valarauca1 13y ago
Your acting like the read/write time to the hardware/FPGA is negligible which it isn't. When you access hardware on say a PCI bus (which you would in this scenario). Your call to the PCI bus does not take place WHEN you call for it to take place. You call the Kernel, which calls the scheduler, which calls the hardware manager, which calls the driver, which finally processes your request. Now your request is processed all this is dumped and something else runs while the processor waits to hear back from the PCI bus with the response because this takes ages in processor time. Finally an interrupt arrives, is made sense of, the appropriate driver is called, then it gives your information back to your process and you back on your merry little way. :.:.: The problem is when you called OS to start this long chain of events the real world didn't stop. Your real time module is still counting the 32,600,000 pulses per second. This is where you'll get errors. Because often its easy to think things in a computer happen instantly, or so blindingly fast you don't care what happens, order or speed. The situation you described is what originally gave me 0.1% error. Eventually I switched to a more aggressive tact of polling asynchronously in a separate thread and when a process called for the time responding with the latest received time. This got me down to 0.07% error. Still not acceptable. :.:.: Its nice to be starry eyed and think there is no reason to give up security. But sometimes secure software can't do what you need it to do. The more crap you put between you and the metal the more time it'll take to execute. This is logically provable. If you take process A and B. Both are the optimal way to do something. There is no faster way to do this task. Therefore one can assume A and B's execution time are equal. Yet B operates in a secure sand-boxed environment with a time sharing OS. Therefore B's true execution is B+C+D. We know A = B, but for A = (B+C+D) C and D must be Zero, which they can never be in the real world.
- RyanZAG 13y ago
  I get what you're saying about how it's definitely going to be slower, but I don't understand why the read/write time to the hardware matters if the hardware is buffering the last 100 timings or so. What I'm trying to point out is that you don't have to make your entire system down to the keyboard real time - you only need to make the tiny piece that is doing the physical process real time (with some very simple logic gates that can run far faster than any general purpose CPU), and then sending the results over the PCI/whatever bus in batches later to be processed.
  - valarauca1 13y ago
    The only real time component of the software stack is the kernel. If you want another real time module your screwed because you need to run it in kernel space, but if you have a kernel you can't. Or you run a real time OS which may have problems because they aren't developed with security but IO timing in mind. It's a fundamental flaw of time shared OS's. :.:.: Second security works in a simple way. Cost to secure vs money lost. Lab equipment is expensive. The loss of an entire calibration bench could run into the $250,000 to $1million and beyond range. But redeveloping and entire OS to do this? Your talking about spending 20 to 100x MORE on security then your losses. That's idiotic at best.
    - LukeShu 13y ago
      > The only real time component of the software stack is > the kernel. If you want another real time module your > screwed because you need to run it in kernel space, but > if you have a kernel you can't. I think you're confused about what RyanZAG is saying. If I'm reading correctly, he's saying "don't run the real-time stuff on the CPU." Have that stuff run on a much simpler piece of hardware that is real-time, and runs the real-time code, then have the non-real-time userland on the CPU talk to it in not-real-time. To take your example: > When you access hardware on say a PCI bus (which you would in > this scenario). Your call to the PCI bus does not take place > WHEN you call for it to take place. You call the Kernel, which > calls the scheduler, which calls the hardware manager, which > calls the driver, which finally processes your request. Design the PCI card to have it's own, smaller CPU (or FPGA, or whatever), that does the real-time interaction with the "32,600,000 pulses per second." Don't have the real-time bits depend in any way with the code running on the CPU. Have it buffer the data. The, when the PCI card is accessed by the userland program on the CPU, it dumps the buffer onto the PCI bus. The userland would obviously have be fast enough that the buffer doesn't fill up, but that speed is much less than "real time". You can then work with the data in the userland, running in non-real time.

im3w1l 13y ago

It would be so nice if programs could request a dedicated core for these kinds of things.

valarauca1 13y ago
Even that wouldn't work. Their are a lot of parts of the CPU and all of them work together. RAM access, Cache access, Interrupts, and Memory management is handled globally not on a pre-core basis. You'd run into the problem of needing multiple north bridges (do we even have those anymore or are those on chip now?) which you couldn't have. You have to build an entire OS with real time I/O at heart. They do exist, some are secure (Blackberry's platform) but they aren't deployed to the test industry. The most likely version is licensing fees, nobody writings Data Acquisitions for them.
MertsA 13y ago
Under Linux you can. You can boot the kernel with the maxcpus argument set to 1 or the number of cores you want the Linux kernel to use and then on a quad core machine you have 3 cores available all the time and setup to where Linux won't ever need to handle interrupts on that core. Then you just start your application and set the affinity to that unused core. You can extend this further by doing things like mmaping in 4GB of memory through the kernels hugepage support and lock the physical to virtual address map so the kernel can't touch your physical block of ram you just allocated. Then you can do things like talk directly to a pci device like a network card and set up DMA directly from a NIC into a buffer in your applications memory. All of this is done completely in userspace but you get all the performance benefits of implementing everything like it was running in Ring 0 and the kernel is not involved in anything apart from the initial setup and teardown. You can build an extremely high performance application basically running on bare metal but with the Linux kernel still running on a different core to handle anything that doesn't directly involve your application and there wouldn't need to be any syscalls between the two to service some request.
- valarauca1 13y ago
  What I'm confused about. I'll likely just have to play around with this feature at some point. I knew about APIC, but not completely sandboxing cores. Is if you have bare metal operations do you still have access to kernel functionality ala stdio and libc libraries? Normally when you hit bare metal your on your own. I'm just wondering because the idea of writing my own threading, and memory management libraries excites me to no end </sarcasm>. Also if you can call these functions like your in userland then do they block until execution has completed on the other 'kernel' cores? Also if you creating P-Threads elsewhere but not managing their execution on the 'non-kernel' core what happens? >userspace but you get all the performance benefits of implementing everything like it was running in Ring 0 Can you give any literature on this? these terms are contradictory.