4 ms·

What's the problem that you're trying to solve? OOE, branch prediction, prefetching, are to allow a CPU to execute multiple instructions per cycle on a GHz clo

by YZF 10y ago

What's the problem that you're trying to solve? OOE, branch prediction, prefetching, are to allow a CPU to execute multiple instructions per cycle on a GHz clock. I.e. within a nanosecond. Memory latency all the way out to DRAM is more than enough to do things in microseconds and the caches are big enough to get you a lot of concurrency within that. It's true that when you're coding over a modern OS in a high level language you can't get e.g. a GPIO pin to update in a microsecond. But that's life. Use a real-time OS you can do better. Get rid of the OS and you'll do even better. Use a different CPU or an FGPA and you may do better. The various techniques used to get a CPU to execute 20 instructions in a nanosecond aren't the limiting factor in that stack. EDIT: Once upon a time there was a UART called the 8250. You could either poll it or if you wanted to do something else concurrently you'd use interrupts. Only interrupts have to do a context switch (in this case pushing a few things onto the stack) so they cost a little more. Once serial rates became high enough interrupts started consuming more and more cycles. So they added a buffer, and we got the 16550 (or one of its cousins). Now the thing would only interrupt above a certain watermark and we can cut the number of interrupts by 10 so we can get higher throughput at the expense of higher latency. Same thing happens with network interfaces. You can drive a 10Gbps network because a lot of stuff is buffered and/or hardware offloaded. It doesn't make sense to bitbang 10G Ethernet out of a general purpose CPU. These sorts of tradeoffs in different parts of the system are unavoidable but yet the tools to solve problems down to GHz clocks are out there both for software and hardware.

caf 10y ago
The problem being posed here is: we have a thread that is computing something, and as part of that it needs to make a request that will be satisfied with millisecond-range latency (it might be an RDMA request for a memory page on another machine in the same data centre, or a maybe a largish vector of data to be processed on a GPU, etc.). If we suspend the thread at the OS level and re-schedule it when the result is available (as is done when waiting for, eg, data from a local disk) then the latency of the rescheduling swamps that of the actual operation. If we spin and wait for the result, we quickly run out of instruction-level parallelism and end up underutilising our CPUs.
- YZF 10y ago
  In the general case what you would do is one of these: 1. Increase the size of the data you're operating on so you get more compute per context switch. I.e. you amortize the latency over more data or in other words you trade off latency for throughput. If you're dealing with your local disk e.g. you fread() 100MB instead of 1MB. 2. Rearrange your software. If you have many threads each reading small chunks and operating on them you rearrange the data and your threads such that you perhaps have some threads doing I/O while other threads doing computation. Other rearrangements include moving some of your code into the kernel or alternatively moving pieces from the kernel into user mode to reduce the "depth" of the stack. (RTOS or getting rid of the OS are more extreme examples of "rearranging") 3. Use different hardware, e.g. historically DSP vs. general purpose CPUs. What I'm trying to say, apparently not very clearly, is that I see this as an engineering problem, not a fundamental problem, and an old problem, not a new problem. It's hard to talk about this though without a concrete example. Instruction level parallelism is in nanoseconds, not microseconds, a 2GHz CPU can execute 10000-ish ops across all its execution units in a microsecond. So that's not a bottleneck in processing things that are on the order of microseconds. As long as what you're switching every microsecond is "small" you can perform many concurrent tasks at microsecond blocks. If you need to flush all your caches and remap all your memory then that will be a problem. But caches have a certain latency because they are very close to the CPU and they are sized under similar constraints. This is where it's up to us to engineer the right solution under the constraints. If you need microsecond latency on your processing, let's say you want to do real-time processing on 1000 video conference channels and you never want to introduce more than 1ms delay, then you need to engineer your system differently than a system that doesn't have that sort of requirement. That's always been the case. If the expectation is that you'll run 1000 Linux processes that read and write the video from disk and this will just somehow work then that's not going to happen. It seems we're trying to "backsplain" why something that wasn't designed to be a real time system isn't behaving like a real time system. Linux is a general purpose OS built for certain tasks and the CPUs we tend to run it on are also built for certain tasks. But there are other OSes and other CPUs (or FPGA or ASICs or GPUs). Not to say there isn't a lot of flexibility in Linux on x86 for different situations.
  - 10y ago

by YZF 10y ago

caf 10y ago

The problem being posed here is: we have a thread that is computing something, and as part of that it needs to make a request that will be satisfied with millisecond-range latency (it might be an RDMA request for a memory page on another machine in the same data centre, or a maybe a largish vector of data to be processed on a GPU, etc.). If we suspend the thread at the OS level and re-schedule it when the result is available (as is done when waiting for, eg, data from a local disk) then the latency of the rescheduling swamps that of the actual operation. If we spin and wait for the result, we quickly run out of instruction-level parallelism and end up underutilising our CPUs.

YZF 10y ago
In the general case what you would do is one of these: 1. Increase the size of the data you're operating on so you get more compute per context switch. I.e. you amortize the latency over more data or in other words you trade off latency for throughput. If you're dealing with your local disk e.g. you fread() 100MB instead of 1MB. 2. Rearrange your software. If you have many threads each reading small chunks and operating on them you rearrange the data and your threads such that you perhaps have some threads doing I/O while other threads doing computation. Other rearrangements include moving some of your code into the kernel or alternatively moving pieces from the kernel into user mode to reduce the "depth" of the stack. (RTOS or getting rid of the OS are more extreme examples of "rearranging") 3. Use different hardware, e.g. historically DSP vs. general purpose CPUs. What I'm trying to say, apparently not very clearly, is that I see this as an engineering problem, not a fundamental problem, and an old problem, not a new problem. It's hard to talk about this though without a concrete example. Instruction level parallelism is in nanoseconds, not microseconds, a 2GHz CPU can execute 10000-ish ops across all its execution units in a microsecond. So that's not a bottleneck in processing things that are on the order of microseconds. As long as what you're switching every microsecond is "small" you can perform many concurrent tasks at microsecond blocks. If you need to flush all your caches and remap all your memory then that will be a problem. But caches have a certain latency because they are very close to the CPU and they are sized under similar constraints. This is where it's up to us to engineer the right solution under the constraints. If you need microsecond latency on your processing, let's say you want to do real-time processing on 1000 video conference channels and you never want to introduce more than 1ms delay, then you need to engineer your system differently than a system that doesn't have that sort of requirement. That's always been the case. If the expectation is that you'll run 1000 Linux processes that read and write the video from disk and this will just somehow work then that's not going to happen. It seems we're trying to "backsplain" why something that wasn't designed to be a real time system isn't behaving like a real time system. Linux is a general purpose OS built for certain tasks and the CPUs we tend to run it on are also built for certain tasks. But there are other OSes and other CPUs (or FPGA or ASICs or GPUs). Not to say there isn't a lot of flexibility in Linux on x86 for different situations.
- 10y ago