8 ms·

What does I/O bound mean? (2008)

79 points by higerordermap 6y ago

kyuudou 6y ago
So true about not coding for performance "because the disks are slow". Almost all disk i/o, especially big-iron, has cache in front of it. Maybe in the case of cloud storage - what does I/O bound mean then? It should still conceivably be the same IMO.
zamadatix 6y ago
(2008)
- mywittyname 6y ago
  I'd say this is more relevant now than in 2008. With SSDs and 100Gbit network connections, it's a lot easier to saturate a CPU with data. And a lot of cloud providers now recommend colocating computing such that it's as close to the data as possible.
  - Rafuino 6y ago
    This is what's really interesting to me right now. A 1 TB 980 Pro SSD (I know, not a "data center" SSD, but it's the most recent Gen4 SSD I've seen announced) can handle ~300MB/s [1] of random writes at realistic queue depths up to 32. Previously with 10 Gbit (1.25GB) connection, 4 of those SSDs could handle that incoming write traffic, which is no sweat and can actually physically fit on a server. But as you go to 100 Gbit (12.5GB) of incoming random writes to a storage server, something's gotta give. At this level of traffic, you'd need 41 of those SSDs to be able to handle that incoming traffic without a write buffer on the server. This is a simplification of a very complex topic, but it's what I/O bound means to me, at least. [1] https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/5 https://www.anandtech.com/show/16087/the-samsung-980-pro-pci...
    - zamadatix 6y ago
      Once you factor the storage latency to/through the server the IOPS at realistic queue depths fall by an enormous factor. This is usually balanced by putting tons of servers on it at which point 41 NVMe drives isn't really a lot - will usually fit in a single rack mount storage enclosure.
      - Rafuino 6y ago
        The issue though is that each server itself is starting to have 100 Gbit network connections, not just the rack itself.
        zamadatix 6y ago
        Bandwidth isn't the problem, latency is. Each server could have 400 terabit interfaces and you'll still have garbage iops compared to local pcie on a laptop for reasonable depth.
        Rafuino 6y ago
        Can you help me understand why latency kills IOPS? There's lots of work showing NVMe over TCP bringing similar performance and latency from a remote server as compared to a local SSD. Here's an example: https://www.micron.com/about/blog/2020/march/nvme-over-tcp-proof-of-concept https://www.micron.com/about/blog/2020/march/nvme-over-tcp-p...
    - wtallis 6y ago
      Note that the numbers presented in that review are for single-threaded benchmarks, when the current best practices for a server application that really has a lot of write traffic would be to split the work over multiple threads, and preferably use io_uring for both network and disk IO. Even on those consumer-grade SSDs, that would allow for much higher throughput than what's presented in that review—though most of those consumer-grade SSDs wouldn't be able to sustain such write performance for very long. Right now, a lot of software that wasn't written specifically with SSDs in mind is effectively bottlenecked on the CPU side with system call overhead, or is limited by the SSD's latency while using only a fraction of its potential throughput because the drive is only given one or a few requests to work on at a time. If you do write your software with fast storage in mind, and have the fastest SSD money can't quite buy, the results can be very different: https://lore.kernel.org/io-uring/4af91b50-4a9c-8a16-9470-a51430bd7733@kernel.dk/T/#u https://lore.kernel.org/io-uring/4af91b50-4a9c-8a16-9470-a51...
  - zamadatix 6y ago
    It's both easier and harder. Easier in that raw storage bandwidth has increased faster than single core performance. Harder in that to access that bandwidth you have to optimize that latency for that bandwidth has not decreased, in fact in cloud it's increased. I.e. cloud storage can be extremely fast, but only if it's multi stream storage. Similarly cloud compute can be fast, but only if it's multi thread CPU. But the main point of (2008) is that a lot of the links are broken and tools mentioned outdated. The general concept "you can probably optimize your storage access" is still true though but usually any such concept is, it's the details around current caveats that are interesting.
  - fulafel 6y ago
    Interestingly for most people the network is just as slow today as 10 years ago beacuse 10/100 GBe has stalled in the datacenters and the trickling down that used to happen seems to have stopped. (Gigabit ethernet was 10 years old already in 2008)
- 082349872349872 6y ago
  Go back to the 1960s and people were already talking about i/o vs cpu bound. The units for measuring both have changed but the tradeoff is evergreen. Supercomputer: device used to turn compute-bound problems into i/o bound ones. Mainframe: device used to turn i/o-bound problems into compute bound ones.
btrask 6y ago
Here is a recent HN discussion on this topic: https://news.ycombinator.com/item?id=24519786 https://news.ycombinator.com/item?id=24519786 My current answer is... it's complicated.
staticassertion 6y ago
I was only commenting on this on HN just the other day. When most people say IO bound what they really mean is "There's a hot CPU but it's across a network" ie: "I wrote really inefficient SQL queries, therefor I'm IO bound, therefor I don't have to care about CPU" - and the process pushes further and further downstream as every service talking to one of these "IO bound" services also becomes "IO bound".
- crgwbr 6y ago
  This is obviously sometimes the case. But more often I’ve seen IO bound apps spending all their time on network roundtrip latency. I.E. not a few poorly performing SQL queries, but a thousand queries which all take a millisecond or two.
  - staticassertion 6y ago
    Totally. I've seen similar things. I've also seen thread contention (such as on a connection pool) that can look a lot like a slow database query/ an "IO bound" workflow. I think profiling is just really hard and lots of code tends to be very inefficient at actually performing IO.
    - charrondev 6y ago
      I spent my last 2 weeks optimizing a decade old SaaS to handle massive traffic spikes from one of our biggest customers. We had other customers serving similar amounts of traffic, but with smaller data sets. - Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes. - The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more). - now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request. - Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load. - While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse. - I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack. - the bottleneck was clearly still the DB. - at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM. - the database was still the bottleneck. In the end the thing that fixed it? A simple OPTIMIZE TABLE. Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue. If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.
brendangregg 6y ago
Please stop using bonnie (I know, it's a 2008 post, but in case it inspires anyone). It was useful at the time, but written in a different era, and nowadays produces misleading results on multi-threaded I/O stacks (along with other problems). I spoke to Tim about this issue back at Sun, and he did write about it. Today I usually use fio for disk/FS benchmarks, as it's frequently updated by Jens Axboe, who is also responsible for the modern Linux multiqueue block I/O implementation.
- Rafuino 6y ago
  fio is definitely the way to go! Are you looking at what Jens has done with io_uring as well for async IO?
  - brendangregg 6y ago
    Yes, io_uring is great, and not just for I/O: it's a generic asynchronous interface for other kernel calls as well.
Negitivefrags 6y ago
In order for something to be "bound" by something else there must be an element of concurrency. People often forget this and I think it confuses the definition. At this point people say "bound by X" when what they really mean is "most of the time is spent doing X". If you spend 5 seconds doing I/O and then afterwards 1 second doing CPU heavy work, you are not I/O bound. You can tell this because making the CPU work take 0.5 seconds less time still saves you 0.5 seconds. It's not a bound at all. In order for this to be I/O bound you would need to do the CPU heavy work during the I/O. Once the task is done that way, any optimisation on the CPU heavy work has precisely zero effect on runtime. This is how you know it's bound. I think the reason that web programmers have become confused by this particularly is because on a web server there is almost always inter-request concurrency. You run a bunch of workers on the same CPU and now your waiting-for-db time is concurrent with the doing-cpu-work time for a different request. The system as a whole can be considered I/O bound here because no mater how much faster you make the CPU, the number of requests per second you can process doesn't go up and so it seems like you might as well not care about how fast the cpu-work part is. And that's all well and good, but it's cold comfort for any individual request that still has to wait the full amount of time for a request to be served. You could absolutely improve the situation for any given request by improving the speed of the CPU or optimising the code it runs.
- HeavenFox 6y ago
  While correct, I think that's a bit too pedantic. For example, if you have a program that spend 5 seconds doing I/O and 0.05 second in CPU, for all intents and purposes it's I/O bound. Given the fast CPUs in today's world, this kind of workload is pretty common.
  - wtallis 6y ago
    It's probably better to look at things proportionally, rather than in terms of absolute time saved. And it's definitely more useful to consider terms like "IO bound" and "CPU bound" as relative and approximate, rather than unattainable absolutes. Using the original numbers of 5 seconds IO, 1 second CPU: If doubling the speed of the CPU portion only makes the overall process 8% faster, it's not meaningfully CPU bound. But doubling the IO speed would reduce the total job time by 41%, which is a pretty big speedup overall. Using your example of 5 seconds IO and 0.05 seconds CPU: doubling the CPU speed gives a 0.5% overall reduction in job time, and doubling the IO performance gives a 49.5% overall reduction. So this example can be meaningfully said to be more IO bound than the first example.
deleted 6y ago
[deleted]
vinay_ys 6y ago
tl;dr: In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources more cheaply until cpu becomes the bottleneck). Hence cpu optimizations matter today more than it did in 2008. A program's execution time is a function of cpu performance, memory performance, disk performance or network performance. Performance here is a complex phenomenon – usually we measure performance in terms of 'latency' and 'throughput' of the operations we are executing – and its characteristics vary by attributes like – sequential vs random, block sizes, simple vs complex instructions, instruction offloads, queue depths, scheduling queues, context/mode switching etc. Imagine a naive execution of a program where all operations (cpu, memory, disk, network) are sequentially scheduled and executed. We can make two observations about such a program: 1. This program's execution time can be sped up by upgrading to a faster cpu or memory or disk or network – which of these we should upgrade first depends on what operation the program spends most of its execution time vs what costs less to upgrade. 2. While the program is executing on one of the four resources, the other resources are idling. In other words resource utilization will be less. #1 happens every few months/years as the hardware becomes faster for the same price. #2 is addressed by modern cpus, compilers and operating system schedulers through mechanisms like pipelining, parallelizing, prefetching, offloading etc. – to increase the overall utilization of all the resources while the program is executing. These techniques turn this naive program execution into a complex program execution. The automated optimization of the naive program in this way is not perfect/complete. A programmer will have to adjust the program to utilize the idling resources on a computer. This is the performance optimization work. Even after doing this, the program execution will be constrained by one of the 4 resources. In this situation we can say its execution time is bound by that resource's performance. Theoretically, if that resource performance were to become faster (through hardware upgrade), then the program will no longer by bound by that particular resource and it will now be bound by another resource. By definition, while the program execution time is bound by one resource, the other resources are under-utilized. In designing hardware+software systems, a purist objective is to ensure least resource underutilization occurs while ensuring the program's performance objective is met. Since resources cost differently, the under-utilization has to be weighed by its cost. This resource optimization at a datacenter level takes on a different dimension – a common mistake I have seen is to provision $5000 servers (where majority cost component is cpu and memory) and skimp on the network bandwidth between the servers. To build a full-clos non-blocking inter-server network at a reasonable enterprise scale, it would cost less than $300 per 10GE port. I have seen people save $150 per port and deploy an over-subscribed network that results in indeterministic network performance and cost much more wastage in the utilization of those $5000 servers. Another common mistake I have seen is to provision too much RAM (expensive in purchase cost as well as running cost - power/cooling) while not provisioning enough high-speed SSDs. In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources more cheaply until cpu becomes the bottleneck). Hence cpu optimizations matter today more than it did in 2008.

79 points by higerordermap 6y ago

So true about not coding for performance "because the disks are slow". Almost all disk i/o, especially big-iron, has cache in front of it. Maybe in the case of cloud storage - what does I/O bound mean then? It should still conceivably be the same IMO.

zamadatix 6y ago

(2008)

mywittyname 6y ago
I'd say this is more relevant now than in 2008. With SSDs and 100Gbit network connections, it's a lot easier to saturate a CPU with data. And a lot of cloud providers now recommend colocating computing such that it's as close to the data as possible.
- Rafuino 6y ago
  This is what's really interesting to me right now. A 1 TB 980 Pro SSD (I know, not a "data center" SSD, but it's the most recent Gen4 SSD I've seen announced) can handle ~300MB/s [1] of random writes at realistic queue depths up to 32. Previously with 10 Gbit (1.25GB) connection, 4 of those SSDs could handle that incoming write traffic, which is no sweat and can actually physically fit on a server. But as you go to 100 Gbit (12.5GB) of incoming random writes to a storage server, something's gotta give. At this level of traffic, you'd need 41 of those SSDs to be able to handle that incoming traffic without a write buffer on the server. This is a simplification of a very complex topic, but it's what I/O bound means to me, at least. [1] https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/5 https://www.anandtech.com/show/16087/the-samsung-980-pro-pci...
  - zamadatix 6y ago
    Once you factor the storage latency to/through the server the IOPS at realistic queue depths fall by an enormous factor. This is usually balanced by putting tons of servers on it at which point 41 NVMe drives isn't really a lot - will usually fit in a single rack mount storage enclosure.
    - Rafuino 6y ago
      The issue though is that each server itself is starting to have 100 Gbit network connections, not just the rack itself.
      - zamadatix 6y ago
        Bandwidth isn't the problem, latency is. Each server could have 400 terabit interfaces and you'll still have garbage iops compared to local pcie on a laptop for reasonable depth.
        Rafuino 6y ago
        Can you help me understand why latency kills IOPS? There's lots of work showing NVMe over TCP bringing similar performance and latency from a remote server as compared to a local SSD. Here's an example: https://www.micron.com/about/blog/2020/march/nvme-over-tcp-proof-of-concept https://www.micron.com/about/blog/2020/march/nvme-over-tcp-p...
  - wtallis 6y ago
    Note that the numbers presented in that review are for single-threaded benchmarks, when the current best practices for a server application that really has a lot of write traffic would be to split the work over multiple threads, and preferably use io_uring for both network and disk IO. Even on those consumer-grade SSDs, that would allow for much higher throughput than what's presented in that review—though most of those consumer-grade SSDs wouldn't be able to sustain such write performance for very long. Right now, a lot of software that wasn't written specifically with SSDs in mind is effectively bottlenecked on the CPU side with system call overhead, or is limited by the SSD's latency while using only a fraction of its potential throughput because the drive is only given one or a few requests to work on at a time. If you do write your software with fast storage in mind, and have the fastest SSD money can't quite buy, the results can be very different: https://lore.kernel.org/io-uring/4af91b50-4a9c-8a16-9470-a51430bd7733@kernel.dk/T/#u https://lore.kernel.org/io-uring/4af91b50-4a9c-8a16-9470-a51...
- zamadatix 6y ago
  It's both easier and harder. Easier in that raw storage bandwidth has increased faster than single core performance. Harder in that to access that bandwidth you have to optimize that latency for that bandwidth has not decreased, in fact in cloud it's increased. I.e. cloud storage can be extremely fast, but only if it's multi stream storage. Similarly cloud compute can be fast, but only if it's multi thread CPU. But the main point of (2008) is that a lot of the links are broken and tools mentioned outdated. The general concept "you can probably optimize your storage access" is still true though but usually any such concept is, it's the details around current caveats that are interesting.
- fulafel 6y ago
  Interestingly for most people the network is just as slow today as 10 years ago beacuse 10/100 GBe has stalled in the datacenters and the trickling down that used to happen seems to have stopped. (Gigabit ethernet was 10 years old already in 2008)
082349872349872 6y ago
Go back to the 1960s and people were already talking about i/o vs cpu bound. The units for measuring both have changed but the tradeoff is evergreen. Supercomputer: device used to turn compute-bound problems into i/o bound ones. Mainframe: device used to turn i/o-bound problems into compute bound ones.

btrask 6y ago

Here is a recent HN discussion on this topic: https://news.ycombinator.com/item?id=24519786 https://news.ycombinator.com/item?id=24519786 My current answer is... it's complicated.

staticassertion 6y ago

I was only commenting on this on HN just the other day. When most people say IO bound what they really mean is "There's a hot CPU but it's across a network" ie: "I wrote really inefficient SQL queries, therefor I'm IO bound, therefor I don't have to care about CPU" - and the process pushes further and further downstream as every service talking to one of these "IO bound" services also becomes "IO bound".

crgwbr 6y ago
This is obviously sometimes the case. But more often I’ve seen IO bound apps spending all their time on network roundtrip latency. I.E. not a few poorly performing SQL queries, but a thousand queries which all take a millisecond or two.
- staticassertion 6y ago
  Totally. I've seen similar things. I've also seen thread contention (such as on a connection pool) that can look a lot like a slow database query/ an "IO bound" workflow. I think profiling is just really hard and lots of code tends to be very inefficient at actually performing IO.
  - charrondev 6y ago
    I spent my last 2 weeks optimizing a decade old SaaS to handle massive traffic spikes from one of our biggest customers. We had other customers serving similar amounts of traffic, but with smaller data sets. - Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes. - The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more). - now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request. - Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load. - While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse. - I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack. - the bottleneck was clearly still the DB. - at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM. - the database was still the bottleneck. In the end the thing that fixed it? A simple OPTIMIZE TABLE. Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue. If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.

brendangregg 6y ago

Please stop using bonnie (I know, it's a 2008 post, but in case it inspires anyone). It was useful at the time, but written in a different era, and nowadays produces misleading results on multi-threaded I/O stacks (along with other problems). I spoke to Tim about this issue back at Sun, and he did write about it. Today I usually use fio for disk/FS benchmarks, as it's frequently updated by Jens Axboe, who is also responsible for the modern Linux multiqueue block I/O implementation.

Rafuino 6y ago
fio is definitely the way to go! Are you looking at what Jens has done with io_uring as well for async IO?
- brendangregg 6y ago
  Yes, io_uring is great, and not just for I/O: it's a generic asynchronous interface for other kernel calls as well.

Negitivefrags 6y ago

In order for something to be "bound" by something else there must be an element of concurrency. People often forget this and I think it confuses the definition. At this point people say "bound by X" when what they really mean is "most of the time is spent doing X". If you spend 5 seconds doing I/O and then afterwards 1 second doing CPU heavy work, you are not I/O bound. You can tell this because making the CPU work take 0.5 seconds less time still saves you 0.5 seconds. It's not a bound at all. In order for this to be I/O bound you would need to do the CPU heavy work during the I/O. Once the task is done that way, any optimisation on the CPU heavy work has precisely zero effect on runtime. This is how you know it's bound. I think the reason that web programmers have become confused by this particularly is because on a web server there is almost always inter-request concurrency. You run a bunch of workers on the same CPU and now your waiting-for-db time is concurrent with the doing-cpu-work time for a different request. The system as a whole can be considered I/O bound here because no mater how much faster you make the CPU, the number of requests per second you can process doesn't go up and so it seems like you might as well not care about how fast the cpu-work part is. And that's all well and good, but it's cold comfort for any individual request that still has to wait the full amount of time for a request to be served. You could absolutely improve the situation for any given request by improving the speed of the CPU or optimising the code it runs.

HeavenFox 6y ago
While correct, I think that's a bit too pedantic. For example, if you have a program that spend 5 seconds doing I/O and 0.05 second in CPU, for all intents and purposes it's I/O bound. Given the fast CPUs in today's world, this kind of workload is pretty common.
- wtallis 6y ago
  It's probably better to look at things proportionally, rather than in terms of absolute time saved. And it's definitely more useful to consider terms like "IO bound" and "CPU bound" as relative and approximate, rather than unattainable absolutes. Using the original numbers of 5 seconds IO, 1 second CPU: If doubling the speed of the CPU portion only makes the overall process 8% faster, it's not meaningfully CPU bound. But doubling the IO speed would reduce the total job time by 41%, which is a pretty big speedup overall. Using your example of 5 seconds IO and 0.05 seconds CPU: doubling the CPU speed gives a 0.5% overall reduction in job time, and doubling the IO performance gives a 49.5% overall reduction. So this example can be meaningfully said to be more IO bound than the first example.

deleted 6y ago

[deleted]

vinay_ys 6y ago

tl;dr: In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources more cheaply until cpu becomes the bottleneck). Hence cpu optimizations matter today more than it did in 2008. A program's execution time is a function of cpu performance, memory performance, disk performance or network performance. Performance here is a complex phenomenon – usually we measure performance in terms of 'latency' and 'throughput' of the operations we are executing – and its characteristics vary by attributes like – sequential vs random, block sizes, simple vs complex instructions, instruction offloads, queue depths, scheduling queues, context/mode switching etc. Imagine a naive execution of a program where all operations (cpu, memory, disk, network) are sequentially scheduled and executed. We can make two observations about such a program: 1. This program's execution time can be sped up by upgrading to a faster cpu or memory or disk or network – which of these we should upgrade first depends on what operation the program spends most of its execution time vs what costs less to upgrade. 2. While the program is executing on one of the four resources, the other resources are idling. In other words resource utilization will be less. #1 happens every few months/years as the hardware becomes faster for the same price. #2 is addressed by modern cpus, compilers and operating system schedulers through mechanisms like pipelining, parallelizing, prefetching, offloading etc. – to increase the overall utilization of all the resources while the program is executing. These techniques turn this naive program execution into a complex program execution. The automated optimization of the naive program in this way is not perfect/complete. A programmer will have to adjust the program to utilize the idling resources on a computer. This is the performance optimization work. Even after doing this, the program execution will be constrained by one of the 4 resources. In this situation we can say its execution time is bound by that resource's performance. Theoretically, if that resource performance were to become faster (through hardware upgrade), then the program will no longer by bound by that particular resource and it will now be bound by another resource. By definition, while the program execution time is bound by one resource, the other resources are under-utilized. In designing hardware+software systems, a purist objective is to ensure least resource underutilization occurs while ensuring the program's performance objective is met. Since resources cost differently, the under-utilization has to be weighed by its cost. This resource optimization at a datacenter level takes on a different dimension – a common mistake I have seen is to provision $5000 servers (where majority cost component is cpu and memory) and skimp on the network bandwidth between the servers. To build a full-clos non-blocking inter-server network at a reasonable enterprise scale, it would cost less than $300 per 10GE port. I have seen people save $150 per port and deploy an over-subscribed network that results in indeterministic network performance and cost much more wastage in the utilization of those $5000 servers. Another common mistake I have seen is to provision too much RAM (expensive in purchase cost as well as running cost - power/cooling) while not provisioning enough high-speed SSDs. In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources more cheaply until cpu becomes the bottleneck). Hence cpu optimizations matter today more than it did in 2008.