7 ms·

I was only commenting on this on HN just the other day. When most people say IO bound what they really mean is "There's a hot CPU but it's across a network" ie:

I was only commenting on this on HN just the other day. When most people say IO bound what they really mean is "There's a hot CPU but it's across a network" ie: "I wrote really inefficient SQL queries, therefor I'm IO bound, therefor I don't have to care about CPU" - and the process pushes further and further downstream as every service talking to one of these "IO bound" services also becomes "IO bound".

crgwbr 6y ago
This is obviously sometimes the case. But more often I’ve seen IO bound apps spending all their time on network roundtrip latency. I.E. not a few poorly performing SQL queries, but a thousand queries which all take a millisecond or two.
- staticassertion 6y ago
  Totally. I've seen similar things. I've also seen thread contention (such as on a connection pool) that can look a lot like a slow database query/ an "IO bound" workflow. I think profiling is just really hard and lots of code tends to be very inefficient at actually performing IO.
  - charrondev 6y ago
    I spent my last 2 weeks optimizing a decade old SaaS to handle massive traffic spikes from one of our biggest customers. We had other customers serving similar amounts of traffic, but with smaller data sets. - Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes. - The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more). - now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request. - Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load. - While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse. - I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack. - the bottleneck was clearly still the DB. - at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM. - the database was still the bottleneck. In the end the thing that fixed it? A simple OPTIMIZE TABLE. Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue. If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.
  - unilynx 6y ago
    These are hinted by high system% usage when your system is busy (ie higher than say 10%). If it looks cpu bound but spends a lot of time in the kernel thread switching or synchronization (eg mutexes) is happening too much.
- magicalhippo 6y ago
  And worth noting that this can be missed during development if you have a good network to the server but customer is using a not so great WiFi network. We had this at work, where one customer complained some operation was very slow, taking around 30 minutes. Couldn't pin it down, copied their database to my machine and it took only a couple of minutes. A bit of digging and I found that in this case this module caused a few million of fairly trivial SQL statements to be executed. Each took less than a millisecond to execute locally, but round-trip time over WiFi can be 10 milliseconds or more. So suddenly 2 minutes becomes over 20 minutes. I asked how the client connected to the LAN, and it was indeed via WiFi. As a quick fix we got the customer to use a network cable, which did indeed reduce the running time to a few minutes. The proper fix was to a bit of caching.
  - chousuke 6y ago
    > And worth noting that this can be missed during development if you have a good network It can be even worse if development is done using a local database, possibly on super fast local SSD. The latencies can be orders of magnitude lower, hiding performance issues that would be obvious even with only a millisecond of additional latency. I've seen many vendors claim that they only "support" monolithic single-machine setups (sometimes even virtualization has "unacceptable overhead") when it's blatantly obvious that the application is just written with the assumption that database latency is approximately zero.
btrask 6y ago
Hey Static, thanks for the thought-provoking dicussion the other day! Let me ask you this: let's say you're downstream of a very slow server that is outside of your control. You need to access it, and there's nothing you can do to speed it up. Are you IO bound in this case?
- staticassertion 6y ago
  Honestly, this is a reasonable question. I'm not really sure, and I think this is a gap in terminology (or even tooling), as I think we both sort of stumbled into the other day. I want to say no? If I were to reframe the question as a slow function that I had to call, I wouldn't call it IO bound, so I think not - the fact that there's IO in between the two components is more of a confounding factor in profiling than an actual performance limitation.
  - btrask 6y ago
    Good answer :)
  - kbenson 6y ago
    I've definitely seen it referred to as IO bound when an upstream server just can't get your input fast enough, whether that's because you've maxed out the medium (e.g. you're bound by a 10/100/1000 Mbit link), or bound by how fast the disks can serve up bits, to downstream it's generally encountered as the same thing, the ability to process IO faster than it's available.
- GauntletWizard 6y ago
  In this case, I call it “dependency bound”. Anything not in your direct control is a dependency, and if that’s your limiting factor, call it such.
- hinkley 6y ago
  If you're on a group hike/bike/ride/run, the group goes at the speed of the slowest person. It may be that you didn't sleep well last night, or that your stomach is bugging you, but the reason it's going to take an extra hour today is because Tim has a blister. Whatever is holding up that server is what's holding up the entire train of communication. If that is where $5000 will fix the problem, don't talk about any other problems, you'll just confuse management. The next place to stop if that doesn't work is, if you can't fix "Tim's foot" can you offload some things "he" is dealing with? Dump some of his load, work steal, etc. But these are just mitigations.
klysm 6y ago
I have usually heard it in reference to a serialized process that’s spending most of its time waiting for hard drive / file system requests to go through.
Volundr 6y ago
I wouldn't call this IO bound, and I'd argue anyone calling it IO bound is wrong, and needs coaching. If your waiting on a query to execute your bound on whatever the query is bound on (IO/Memory/CPU). Also I'm IO bound, so don't care is a weird response. Doesn't matter were I'm bound, if it needs to go faster, I need to examine what I can do to remove that bottleneck. What I'm bound on just gives me a clue where most of my energy tuning/troubleshooting goes, not shrug and forget the problem. But maybe I'm just lucky to have always worked with good people.
- staticassertion 6y ago
  I feel like you're agreeing with me? edit: To be clear, when I say "most people" I mean "many people I've talked to, especially with regards to excuses for not caring about their service performance", and that they are incorrect.
  - Volundr 6y ago
    Yeah we definitely aren't disagreeing. I think my comment was mostly driven by surprise that you've apparently encountered a large number of people who just throw up their hands at a slow database. I've worked with some people who really frustrated me ("this operator isn't doing what I expect, and I don't see what's wrong, must be a bug in sql server") but even they'd try to tune a slow query. But I that might be more an artifact of working in a shop where the devs owned the database bottom to top (no dba's, and the sysadmins certainly don't know sql). I forget that for a lot of devs writing sql might not be a daily occurrence.
    - staticassertion 6y ago
      Oh, gotcha, yeah maybe I'm just lucky like that haha
- lotyrin 6y ago
  Sure, my holistic service is possibly bound by compute in some fashion on the SQL host, but what is claimed is that the process/host that is issuing those queries to the db isn't compute bound -- it spends the majority of its time waiting for input (from the db).
eternalban 6y ago
I/O means precisely that: input / output. If I/O interactions with a remote peer are (for whatever reason) stalling work in the local node, as far as the local node is concerned, it is I/O bound. If you own both local and remote peer, you then instrument the remote peer. In your hypothetical scenario, CPU saturation would finger the query provided, in which case you are back to correcting that code in the I/O bound node. So, your calling/local node was I/O bound, and your remote/server node was CPU bound. Sometimes this very same I/O bound pattern -- low cpu utilization on the client node due to blocked threads waiting on I/O from remote peers -- is simply due to insufficient resources and requires changes in capacity planning. Extending your hypothetical, let's assume as the system in question (post- query optimization) is used over time, the extent in data to be processed grows, with the subsequent result that the optimizied query is now saturating the CPUs while supporting the same number of clients as before. Do you still insist on asserting that processing conditions of a remote peer allow for mis-characterizing the processing condition on the local nodes as "CPU bound"?
- staticassertion 6y ago
  Uh, I guess somehow I wasn't clear that I agree that the situation I mentioned is not IO bound but is in fact a mistake I see a lot.
munchbunny 6y ago
I don’t think I’ve ever encountered someone calling a slow query bottleneck “I/O bound” unless the query itself was I/O bound. I usually just see it explained as “bottlenecked by a slow query”.
- snoshy 6y ago
  Not to justify the abuse of the term, but here, the implication is that since the query is running on a database, and the database is accessible only over a network, that it is blocked on network I/O.
eru 6y ago
It depends on where you draw the lines around your boxes. The whole system together is CPU bound. But the individual part might be 'IO bound'.
- TickleSteve 6y ago
  Exactly this. The individual subsystems can be I/O bound on other subsystems whereas considered as an entire system, it could be CPU-bound. So, yes, one subsystem "blocking" on network I/O definitely is I/O bound regardless of the nature of that I/O, be it database-related or not, it's still I/O. The system at the other end (the db subsystem) may also be I/O (or CPU) bound, but that is distinct from the clients bottleneck.

crgwbr 6y ago

This is obviously sometimes the case. But more often I’ve seen IO bound apps spending all their time on network roundtrip latency. I.E. not a few poorly performing SQL queries, but a thousand queries which all take a millisecond or two.

staticassertion 6y ago
Totally. I've seen similar things. I've also seen thread contention (such as on a connection pool) that can look a lot like a slow database query/ an "IO bound" workflow. I think profiling is just really hard and lots of code tends to be very inefficient at actually performing IO.
- charrondev 6y ago
  I spent my last 2 weeks optimizing a decade old SaaS to handle massive traffic spikes from one of our biggest customers. We had other customers serving similar amounts of traffic, but with smaller data sets. - Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes. - The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more). - now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request. - Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load. - While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse. - I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack. - the bottleneck was clearly still the DB. - at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM. - the database was still the bottleneck. In the end the thing that fixed it? A simple OPTIMIZE TABLE. Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue. If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.
- unilynx 6y ago
  These are hinted by high system% usage when your system is busy (ie higher than say 10%). If it looks cpu bound but spends a lot of time in the kernel thread switching or synchronization (eg mutexes) is happening too much.
magicalhippo 6y ago
And worth noting that this can be missed during development if you have a good network to the server but customer is using a not so great WiFi network. We had this at work, where one customer complained some operation was very slow, taking around 30 minutes. Couldn't pin it down, copied their database to my machine and it took only a couple of minutes. A bit of digging and I found that in this case this module caused a few million of fairly trivial SQL statements to be executed. Each took less than a millisecond to execute locally, but round-trip time over WiFi can be 10 milliseconds or more. So suddenly 2 minutes becomes over 20 minutes. I asked how the client connected to the LAN, and it was indeed via WiFi. As a quick fix we got the customer to use a network cable, which did indeed reduce the running time to a few minutes. The proper fix was to a bit of caching.
- chousuke 6y ago
  > And worth noting that this can be missed during development if you have a good network It can be even worse if development is done using a local database, possibly on super fast local SSD. The latencies can be orders of magnitude lower, hiding performance issues that would be obvious even with only a millisecond of additional latency. I've seen many vendors claim that they only "support" monolithic single-machine setups (sometimes even virtualization has "unacceptable overhead") when it's blatantly obvious that the application is just written with the assumption that database latency is approximately zero.

btrask 6y ago

Hey Static, thanks for the thought-provoking dicussion the other day! Let me ask you this: let's say you're downstream of a very slow server that is outside of your control. You need to access it, and there's nothing you can do to speed it up. Are you IO bound in this case?

staticassertion 6y ago
Honestly, this is a reasonable question. I'm not really sure, and I think this is a gap in terminology (or even tooling), as I think we both sort of stumbled into the other day. I want to say no? If I were to reframe the question as a slow function that I had to call, I wouldn't call it IO bound, so I think not - the fact that there's IO in between the two components is more of a confounding factor in profiling than an actual performance limitation.
- btrask 6y ago
  Good answer :)
- kbenson 6y ago
  I've definitely seen it referred to as IO bound when an upstream server just can't get your input fast enough, whether that's because you've maxed out the medium (e.g. you're bound by a 10/100/1000 Mbit link), or bound by how fast the disks can serve up bits, to downstream it's generally encountered as the same thing, the ability to process IO faster than it's available.
GauntletWizard 6y ago
In this case, I call it “dependency bound”. Anything not in your direct control is a dependency, and if that’s your limiting factor, call it such.
hinkley 6y ago
If you're on a group hike/bike/ride/run, the group goes at the speed of the slowest person. It may be that you didn't sleep well last night, or that your stomach is bugging you, but the reason it's going to take an extra hour today is because Tim has a blister. Whatever is holding up that server is what's holding up the entire train of communication. If that is where $5000 will fix the problem, don't talk about any other problems, you'll just confuse management. The next place to stop if that doesn't work is, if you can't fix "Tim's foot" can you offload some things "he" is dealing with? Dump some of his load, work steal, etc. But these are just mitigations.

klysm 6y ago

I have usually heard it in reference to a serialized process that’s spending most of its time waiting for hard drive / file system requests to go through.

Volundr 6y ago

I wouldn't call this IO bound, and I'd argue anyone calling it IO bound is wrong, and needs coaching. If your waiting on a query to execute your bound on whatever the query is bound on (IO/Memory/CPU). Also I'm IO bound, so don't care is a weird response. Doesn't matter were I'm bound, if it needs to go faster, I need to examine what I can do to remove that bottleneck. What I'm bound on just gives me a clue where most of my energy tuning/troubleshooting goes, not shrug and forget the problem. But maybe I'm just lucky to have always worked with good people.

staticassertion 6y ago
I feel like you're agreeing with me? edit: To be clear, when I say "most people" I mean "many people I've talked to, especially with regards to excuses for not caring about their service performance", and that they are incorrect.
- Volundr 6y ago
  Yeah we definitely aren't disagreeing. I think my comment was mostly driven by surprise that you've apparently encountered a large number of people who just throw up their hands at a slow database. I've worked with some people who really frustrated me ("this operator isn't doing what I expect, and I don't see what's wrong, must be a bug in sql server") but even they'd try to tune a slow query. But I that might be more an artifact of working in a shop where the devs owned the database bottom to top (no dba's, and the sysadmins certainly don't know sql). I forget that for a lot of devs writing sql might not be a daily occurrence.
  - staticassertion 6y ago
    Oh, gotcha, yeah maybe I'm just lucky like that haha
lotyrin 6y ago
Sure, my holistic service is possibly bound by compute in some fashion on the SQL host, but what is claimed is that the process/host that is issuing those queries to the db isn't compute bound -- it spends the majority of its time waiting for input (from the db).

eternalban 6y ago

I/O means precisely that: input / output. If I/O interactions with a remote peer are (for whatever reason) stalling work in the local node, as far as the local node is concerned, it is I/O bound. If you own both local and remote peer, you then instrument the remote peer. In your hypothetical scenario, CPU saturation would finger the query provided, in which case you are back to correcting that code in the I/O bound node. So, your calling/local node was I/O bound, and your remote/server node was CPU bound. Sometimes this very same I/O bound pattern -- low cpu utilization on the client node due to blocked threads waiting on I/O from remote peers -- is simply due to insufficient resources and requires changes in capacity planning. Extending your hypothetical, let's assume as the system in question (post- query optimization) is used over time, the extent in data to be processed grows, with the subsequent result that the optimizied query is now saturating the CPUs while supporting the same number of clients as before. Do you still insist on asserting that processing conditions of a remote peer allow for mis-characterizing the processing condition on the local nodes as "CPU bound"?

staticassertion 6y ago
Uh, I guess somehow I wasn't clear that I agree that the situation I mentioned is not IO bound but is in fact a mistake I see a lot.

munchbunny 6y ago

I don’t think I’ve ever encountered someone calling a slow query bottleneck “I/O bound” unless the query itself was I/O bound. I usually just see it explained as “bottlenecked by a slow query”.

snoshy 6y ago
Not to justify the abuse of the term, but here, the implication is that since the query is running on a database, and the database is accessible only over a network, that it is blocked on network I/O.

eru 6y ago

It depends on where you draw the lines around your boxes. The whole system together is CPU bound. But the individual part might be 'IO bound'.

TickleSteve 6y ago
Exactly this. The individual subsystems can be I/O bound on other subsystems whereas considered as an entire system, it could be CPU-bound. So, yes, one subsystem "blocking" on network I/O definitely is I/O bound regardless of the nature of that I/O, be it database-related or not, it's still I/O. The system at the other end (the db subsystem) may also be I/O (or CPU) bound, but that is distinct from the clients bottleneck.