4 ms·

Network Performance: Bandwidth is only part of the problem (2005)

27 points by unmole 9y ago

flukus 9y ago
Developers are generally oblivious to network latency. It's not just a LAN vs WAN thing either, the LAN will bottleneck if you have to cross the wire 1000 times to full fill a request. n+1 queries aren't necessarily slow because the database has to do disk reads, they're slow because they have to go back and forth across the LAN thousands of times. There are seriously a lot of places out there that think a 10 second wait to load a table with 20 items is perfectly normal. A common anti-pattern I see in the c# world is a pointless service layer (in a separate process) combined with n+1 crud data lookups. All the service layer brings to the table is a lot of serialization (which can be relatively expensive) and a doubling of network latency. What's even worse is the commonly implemented solution: caching. The database get's unfairly blamed and a caching layer is added, of course this just adds another layer of network latency, does little for best case performance (data in cache) and worsens the worst case performance. Rant over. Obligatory latency numbers every programmer should know: https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html https://people.eecs.berkeley.edu/~rcs/research/interactive_l...
- WatchDog 9y ago
  I too have seen many occurrences of the dreaded application level JOIN. Not to mention some of the craziness that happens when a "NoSQL" DB was chosen, when you really needed a relational database.
  - flukus 9y ago
    I think these application level joins (great description, I'm stealing that in future) are responsible for half the instances NoSql databases (probably more) because relational ones weren't "scaleable" enough.
- hinkley 9y ago
  > and a doubling of network latency. Worse than that. Middleman services are almost never streaming. So they have to read the entire response, parse it, reformulate it and send it back out. Any IO and CPU bottlenecks are seen by the downstream server as additional latency.
danellis 9y ago
Do we have to have all these scathing headlines telling us what we do and don't know? I mean, I've been doing this a while. I do know a thing or two, and I'm hardly the only one. I'm surprised ACM stoops to this level of unprofessional clickbait.
- pmoriarty 9y ago
  Honestly, I don't care if the title is clickbaity or not. What I care about is whether the content is worth reading. If it is, I'm happy if the authors entice me to read it, using whatever means necessary (including by using clickbaity titles). If the article itself was fluff, then I'd be upset if I wasted my time reading it. But titles? Who cares? In fact, I'd argue it's more harmful to have boring, non-clickbaity titles on interesting/informative articles than it is to have clickbaity titles on fluff. At least with the latter, it only takes a few seconds of my time to scan through it and see it's fluff. With the former, I may never read the article and completely miss out because I never gave it a chance, which is a waste.
- zkms 9y ago
  I don't care about the clickbait, if it gets someone to realise that in most cases, latency is at least as important a figure of merit than bandwidth or packet loss, that article does its job. Long, uncontrolled queues that live at fast-slow transition points and induce pathological latency is a very generic issue and is certainly not unique to IPv4 packets getting bufferbloated at CPEs or whatnot. It's always tempting to use cheap memory to make huge buffers to try to increase utilisation (and thus bandwidth) or decrease queuing loss. Any practitioner who works in any role (designer, programmer, operator/administrator) with packet-switched networks should have some familiarity with this sort of problem -- even if it's just at a "some queuing is good because it increases link utilisation but if queues are too big and are allowed to stay full then latency goes to hell and the network breaks" level.
- hinkley 9y ago
  Do you never find yourself with the urge to scold people for not knowing all of these things? Wouldn't it be more productive to send them a link?
peterwwillis 9y ago
Some people, when confronted with a networking problem, think "I know, I'll make my own protocol." Now they have two problems.
- greglindahl 9y ago
  The 3rd time I tried to re-invent TCP/IP for a particular situation, I got it right.
jrauser 9y ago
I wrote a talk about this some years ago, "TCP and The Lower Bound of Web Performance." Here's a relatively recent video: https://www.youtube.com/watch?v=C8orjQLacTo https://www.youtube.com/watch?v=C8orjQLacTo
sliken 9y ago
I'm aware of gridftp and the HPC version of ssh/scp that helps maximize bandwidth over high BDP links. Any suggestions for others?

27 points by unmole 9y ago

Developers are generally oblivious to network latency. It's not just a LAN vs WAN thing either, the LAN will bottleneck if you have to cross the wire 1000 times to full fill a request. n+1 queries aren't necessarily slow because the database has to do disk reads, they're slow because they have to go back and forth across the LAN thousands of times. There are seriously a lot of places out there that think a 10 second wait to load a table with 20 items is perfectly normal. A common anti-pattern I see in the c# world is a pointless service layer (in a separate process) combined with n+1 crud data lookups. All the service layer brings to the table is a lot of serialization (which can be relatively expensive) and a doubling of network latency. What's even worse is the commonly implemented solution: caching. The database get's unfairly blamed and a caching layer is added, of course this just adds another layer of network latency, does little for best case performance (data in cache) and worsens the worst case performance. Rant over. Obligatory latency numbers every programmer should know: https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html https://people.eecs.berkeley.edu/~rcs/research/interactive_l...

WatchDog 9y ago
I too have seen many occurrences of the dreaded application level JOIN. Not to mention some of the craziness that happens when a "NoSQL" DB was chosen, when you really needed a relational database.
- flukus 9y ago
  I think these application level joins (great description, I'm stealing that in future) are responsible for half the instances NoSql databases (probably more) because relational ones weren't "scaleable" enough.
hinkley 9y ago
> and a doubling of network latency. Worse than that. Middleman services are almost never streaming. So they have to read the entire response, parse it, reformulate it and send it back out. Any IO and CPU bottlenecks are seen by the downstream server as additional latency.

danellis 9y ago

Do we have to have all these scathing headlines telling us what we do and don't know? I mean, I've been doing this a while. I do know a thing or two, and I'm hardly the only one. I'm surprised ACM stoops to this level of unprofessional clickbait.

pmoriarty 9y ago
Honestly, I don't care if the title is clickbaity or not. What I care about is whether the content is worth reading. If it is, I'm happy if the authors entice me to read it, using whatever means necessary (including by using clickbaity titles). If the article itself was fluff, then I'd be upset if I wasted my time reading it. But titles? Who cares? In fact, I'd argue it's more harmful to have boring, non-clickbaity titles on interesting/informative articles than it is to have clickbaity titles on fluff. At least with the latter, it only takes a few seconds of my time to scan through it and see it's fluff. With the former, I may never read the article and completely miss out because I never gave it a chance, which is a waste.
zkms 9y ago
I don't care about the clickbait, if it gets someone to realise that in most cases, latency is at least as important a figure of merit than bandwidth or packet loss, that article does its job. Long, uncontrolled queues that live at fast-slow transition points and induce pathological latency is a very generic issue and is certainly not unique to IPv4 packets getting bufferbloated at CPEs or whatnot. It's always tempting to use cheap memory to make huge buffers to try to increase utilisation (and thus bandwidth) or decrease queuing loss. Any practitioner who works in any role (designer, programmer, operator/administrator) with packet-switched networks should have some familiarity with this sort of problem -- even if it's just at a "some queuing is good because it increases link utilisation but if queues are too big and are allowed to stay full then latency goes to hell and the network breaks" level.
hinkley 9y ago
Do you never find yourself with the urge to scold people for not knowing all of these things? Wouldn't it be more productive to send them a link?

peterwwillis 9y ago

Some people, when confronted with a networking problem, think "I know, I'll make my own protocol." Now they have two problems.

greglindahl 9y ago
The 3rd time I tried to re-invent TCP/IP for a particular situation, I got it right.

jrauser 9y ago

I wrote a talk about this some years ago, "TCP and The Lower Bound of Web Performance." Here's a relatively recent video: https://www.youtube.com/watch?v=C8orjQLacTo https://www.youtube.com/watch?v=C8orjQLacTo

sliken 9y ago

I'm aware of gridftp and the HPC version of ssh/scp that helps maximize bandwidth over high BDP links. Any suggestions for others?