5 ms·

In the general case what you would do is one of these: 1. Increase the size of the data you're operating on so you get more compute per context switch. I.e. y

by YZF 10y ago

In the general case what you would do is one of these: 1. Increase the size of the data you're operating on so you get more compute per context switch. I.e. you amortize the latency over more data or in other words you trade off latency for throughput. If you're dealing with your local disk e.g. you fread() 100MB instead of 1MB. 2. Rearrange your software. If you have many threads each reading small chunks and operating on them you rearrange the data and your threads such that you perhaps have some threads doing I/O while other threads doing computation. Other rearrangements include moving some of your code into the kernel or alternatively moving pieces from the kernel into user mode to reduce the "depth" of the stack. (RTOS or getting rid of the OS are more extreme examples of "rearranging") 3. Use different hardware, e.g. historically DSP vs. general purpose CPUs. What I'm trying to say, apparently not very clearly, is that I see this as an engineering problem, not a fundamental problem, and an old problem, not a new problem. It's hard to talk about this though without a concrete example. Instruction level parallelism is in nanoseconds, not microseconds, a 2GHz CPU can execute 10000-ish ops across all its execution units in a microsecond. So that's not a bottleneck in processing things that are on the order of microseconds. As long as what you're switching every microsecond is "small" you can perform many concurrent tasks at microsecond blocks. If you need to flush all your caches and remap all your memory then that will be a problem. But caches have a certain latency because they are very close to the CPU and they are sized under similar constraints. This is where it's up to us to engineer the right solution under the constraints. If you need microsecond latency on your processing, let's say you want to do real-time processing on 1000 video conference channels and you never want to introduce more than 1ms delay, then you need to engineer your system differently than a system that doesn't have that sort of requirement. That's always been the case. If the expectation is that you'll run 1000 Linux processes that read and write the video from disk and this will just somehow work then that's not going to happen. It seems we're trying to "backsplain" why something that wasn't designed to be a real time system isn't behaving like a real time system. Linux is a general purpose OS built for certain tasks and the CPUs we tend to run it on are also built for certain tasks. But there are other OSes and other CPUs (or FPGA or ASICs or GPUs). Not to say there isn't a lot of flexibility in Linux on x86 for different situations.

caf 10y ago
They mention point 1 in the article - essentially this is what the HPC crowd are doing with their static environment that's easier to batch, but they're saying it doesn't work in the Google-type dynamic environment because they have a lots of heterogeneous threads doing slightly different things for different users. Instruction level parallelism is in nanoseconds, not microseconds, a 2GHz CPU can execute 10000-ish ops across all its execution units in a microsecond. So that's not a bottleneck in processing things that are on the order of microseconds. It's not about the CPU being a bottleneck at all. It's about not having your CPU either sit idle for that time in which it could have done 10,000-ish ops, or spend that amount of time running scheduler code that is pure overhead. The idea is millions of threads (across your whole compute environment, not one machine!) that are each doing compute/usec-latency-io/compute/usec-latency-io/... and scheduling those to make efficient use of your CPU resources. Coupling each usec-latency-io request with a heavyweight reschedule that's on the same order of magnitude is clearly not efficient, and neither is spinning for that time in which your CPU could have been doing a quantum of useful work. It seems we're trying to "backsplain" why something that wasn't designed to be a real time system isn't behaving like a real time system. This is going down the wrong path - the issue at hand isn't about real-time latency at all. The reason rescheduling to wait for this kind of IO isn't a good solution isn't because of latency introduced, it's because you might end up wasting 50% or more of your CPU time just running scheduler code - it's purely efficiency. Use different hardware, e.g. historically DSP vs. general purpose CPUs. This is also covered in the article, the whole idea is for this to be workable by mere-mortal programmers on general purpose platforms.
- YZF 10y ago
  Let's talk actual examples. I've read the article. Can you give me a concrete example of the "attack of the killer microseconds"? Something that doesn't boil down to "the article said so" or "we analyzed Google's data center and we found we're running poorly written software on it?". Something that is fundamental. I've worked on software where the amount of context switching was so high that it consumed a significant amount of cycles (on hypervisors this used to be a bigger problem btw, and maybe still is) and we analyzed the problem and fixed it reducing the number of context switches by a factor of 10. The bottom line there was poorly designed software. I understand the author wants to have his cake and eat it or in his words "code simplicity and programmer productivity". He wants some magical architectures where software engineers don't have to worry about tradeoffs, the OS or the hardware. They should be able to write 500 lines Python and the whole thing will run in a single cycle and do I/O and compute at the same time. They should be able to write 1000's of threads with a working set of gigabytes and not worry about the size of the L1 cache. They should be able to write a 10GB file 1 byte at a time. That's unlikely to happen. In the meantime specific engineering problems can be solved with specific techniques. The author's conclusion: System designers can no longer ignore efficient support for microsecond-scale I/O, as the most useful new warehouse-scale computing technologies start running at that time scale. Today's hardware and system software make an inadequate platform, particularly given support for synchronous programming models is deemed critical for software productivity. Micro-second scale I/O isn't new. Low level I/O devices always operated at these time scales. Writes to disks with memory buffers. Sequential disk reads. Networking. Video cards. From a software perspective very low overhead concurrency goes back to coroutines in the 1950's through various forms of lightweight threads. The ASICs inside your typical rack-top switch are running software that is absolutely capable of handling this sort of workload efficiently. I think what is a relatively new phenomena is that there are a lot of developers who don't understand anything about the underlying hardware. The other thing that has happened is that systems have become more complex. It's not easy for someone to go from "wasted cycles" in the data center to the design issues that are the cause of these inefficiencies. People have tried many different permutations of software and hardware architecture and I think it's unlikely that they completely missed the magical one that lets you have your cake and eat it. The primary reason being that the constraints are physical ones. Can we do things a little more cleverly? Sure, we do that all the time. I keep coming back to real-time OSes not because this is necessarily a hard real-time situation but because real-time OSes have forever had very lightweight threading, the ability to handle interrupts in lower layers of the stack, zero copy network stacks etc. They are an off-the-shelf solution for some of the problems discussed. Obviously (or not so obviously) there is a price to pay for giving up a lot of the "luxuries" of high level OSes but mere-mortals are absolutely capable of working in this environment. If it's not clear ;) I didn't like this article. EDIT: Another IMO interesting observation is that a lot of the techniques for dealing with storage I/O are from times where CPUs were a lot slower. For the longest time ~10ms seek times were relatively constant as CPUs got faster by orders of magnitude. So in some sense a microsecond latency SSD on a 3GHz CPU is already benefiting from a lot of work that went into shaving cycles from the millisecond I/O on the 8Mhz CPU of past.

by YZF 10y ago

caf 10y ago

They mention point 1 in the article - essentially this is what the HPC crowd are doing with their static environment that's easier to batch, but they're saying it doesn't work in the Google-type dynamic environment because they have a lots of heterogeneous threads doing slightly different things for different users. Instruction level parallelism is in nanoseconds, not microseconds, a 2GHz CPU can execute 10000-ish ops across all its execution units in a microsecond. So that's not a bottleneck in processing things that are on the order of microseconds. It's not about the CPU being a bottleneck at all. It's about not having your CPU either sit idle for that time in which it could have done 10,000-ish ops, or spend that amount of time running scheduler code that is pure overhead. The idea is millions of threads (across your whole compute environment, not one machine!) that are each doing compute/usec-latency-io/compute/usec-latency-io/... and scheduling those to make efficient use of your CPU resources. Coupling each usec-latency-io request with a heavyweight reschedule that's on the same order of magnitude is clearly not efficient, and neither is spinning for that time in which your CPU could have been doing a quantum of useful work. It seems we're trying to "backsplain" why something that wasn't designed to be a real time system isn't behaving like a real time system. This is going down the wrong path - the issue at hand isn't about real-time latency at all. The reason rescheduling to wait for this kind of IO isn't a good solution isn't because of latency introduced, it's because you might end up wasting 50% or more of your CPU time just running scheduler code - it's purely efficiency. Use different hardware, e.g. historically DSP vs. general purpose CPUs. This is also covered in the article, the whole idea is for this to be workable by mere-mortal programmers on general purpose platforms.

YZF 10y ago
Let's talk actual examples. I've read the article. Can you give me a concrete example of the "attack of the killer microseconds"? Something that doesn't boil down to "the article said so" or "we analyzed Google's data center and we found we're running poorly written software on it?". Something that is fundamental. I've worked on software where the amount of context switching was so high that it consumed a significant amount of cycles (on hypervisors this used to be a bigger problem btw, and maybe still is) and we analyzed the problem and fixed it reducing the number of context switches by a factor of 10. The bottom line there was poorly designed software. I understand the author wants to have his cake and eat it or in his words "code simplicity and programmer productivity". He wants some magical architectures where software engineers don't have to worry about tradeoffs, the OS or the hardware. They should be able to write 500 lines Python and the whole thing will run in a single cycle and do I/O and compute at the same time. They should be able to write 1000's of threads with a working set of gigabytes and not worry about the size of the L1 cache. They should be able to write a 10GB file 1 byte at a time. That's unlikely to happen. In the meantime specific engineering problems can be solved with specific techniques. The author's conclusion: System designers can no longer ignore efficient support for microsecond-scale I/O, as the most useful new warehouse-scale computing technologies start running at that time scale. Today's hardware and system software make an inadequate platform, particularly given support for synchronous programming models is deemed critical for software productivity. Micro-second scale I/O isn't new. Low level I/O devices always operated at these time scales. Writes to disks with memory buffers. Sequential disk reads. Networking. Video cards. From a software perspective very low overhead concurrency goes back to coroutines in the 1950's through various forms of lightweight threads. The ASICs inside your typical rack-top switch are running software that is absolutely capable of handling this sort of workload efficiently. I think what is a relatively new phenomena is that there are a lot of developers who don't understand anything about the underlying hardware. The other thing that has happened is that systems have become more complex. It's not easy for someone to go from "wasted cycles" in the data center to the design issues that are the cause of these inefficiencies. People have tried many different permutations of software and hardware architecture and I think it's unlikely that they completely missed the magical one that lets you have your cake and eat it. The primary reason being that the constraints are physical ones. Can we do things a little more cleverly? Sure, we do that all the time. I keep coming back to real-time OSes not because this is necessarily a hard real-time situation but because real-time OSes have forever had very lightweight threading, the ability to handle interrupts in lower layers of the stack, zero copy network stacks etc. They are an off-the-shelf solution for some of the problems discussed. Obviously (or not so obviously) there is a price to pay for giving up a lot of the "luxuries" of high level OSes but mere-mortals are absolutely capable of working in this environment. If it's not clear ;) I didn't like this article. EDIT: Another IMO interesting observation is that a lot of the techniques for dealing with storage I/O are from times where CPUs were a lot slower. For the longest time ~10ms seek times were relatively constant as CPUs got faster by orders of magnitude. So in some sense a microsecond latency SSD on a 3GHz CPU is already benefiting from a lot of work that went into shaving cycles from the millisecond I/O on the 8Mhz CPU of past.