6 ms·

There have been a spate of articles lately that sum up to one fact: out of the box, all Linux distributions are tuned for nothing in particular. With some conf

There have been a spate of articles lately that sum up to one fact: out of the box, all Linux distributions are tuned for nothing in particular. With some configs you can reduce your latency by an order of magnitude, increase your MySQL ops by an order of magnitude, etc. This is one reason why I, as an application-level programmer, am glad to work at a company big enough to have someone else dealing with the platform on my behalf. I don't want to spend ten years slowly discovering traps like default window size tuned for T1 speed, default min_rto tuned for interplanetary traffic, not enough PIDs to support a 96-core machine, etc.

deleted 11y ago
[deleted]
fulafel 11y ago
Nitpick: Maybe you are thinking of Windows wrt buffer sizes? Linux TCP out of the box has window scaling enabled since 2004 or so. "sysctl net.ipv4.tcp_rmem" tells me max buffer size is 6+ MB on my old 4GB laptop, which should be good to saturate 10 a gigabit pipe with up to 50 ms RTT... Mininum retransmission tuning might get you something on your own network if it's congested but would probably be bad to crank down for general internet usage, given things like bufferbloat and large delays caused by wireless L2 retransmissions.
acd 11y ago
Linux TCP window tuning is in fact off, agree with thrownaway. It's by the way a trick that the CDN providers use since they tune their stacks to increase the download speed. But if you could as well do the same tuning yourself.
superuser2 11y ago
How can one learn to do this? Are there good books/classes on the topic? Learning from mentors seems pretty vulnerable to cargo culting/bad science but I don't know where to even start.
- j1e 11y ago
  http://www.brendangregg.com/ http://www.brendangregg.com/
rdtsc 11y ago
> all Linux distributions are tuned for nothing in particular. Which is good, because often tuning means making trade-offs. A lot of those things like polling and interrupt shuffling have side effects and might for example negatively impact other scenarios -- presumably not everyone out there is running single client-server ping-pong with small UDP packets. Yes a lot of parameters have retained default values from back in the day when system characteristics were different. Switching defaults drastically might break existing code that relied on them. Perhaps those are better updated during major distro version upgrades (RHEL 6->7 for ex). In general though, I would say Linux is tuned for average throughput not latency. Sometimes they go hand in hand, sometimes they don't. Over the years there have been patches and forks related to operating on low latency environments (stock trading, media processing, hardware control, graphics in desktop environment). Those range from parameters tweaks in /proc to kernel patches, to almost full kernel bypass via things like DPDK for Ethernet for example.
- thrownaway2424 11y ago
  Sure, they are all tradeoffs of some kind or another. It's crazy to think that one person deploying a linux box on AWS would be aware of them all, though. For example what is the default file-max on Ubuntu? Why?
  - rdtsc 11y ago
    Well you have different use cases and you have a large panel of nobs and switches you can adjust. You need someone who knows what the knobs do and what effect they have. So perhaps there is a use case for wizard like configuration tool for intermediate users. It can ask you questions like -- how many clients do you have? How many concurrent connections. Would you rather have higher average throughput (say for downloading files) or lower packet latency (say for audio processing). Would you rather use your memory for a file cache for for your application data. And so on. Next step up -- a self tuning system. It monitors your usage pattern then decides what settings to use. Monitors again, and adjusts a bit and so on.
    - gshx 11y ago
billyhoffman 11y ago
I agree, and Linux's default state is a matter I raise when people talk to me about how awesome DevOps and Full Stack developers are. They may be able to launch a VM and get a web stack up and running with yum, but that's really not reasonable for production usage
- exelius 11y ago
  The way we work this is that DevOps builds the systems, and the performance testers tune the configs based on their testing. Performance testing is really not the right word for what they do; it's really performance engineering. Regardless of whether you use DevOps or not, if scalability is an issue, someone needs to be doing performance testing and sizing to have a rough idea of how many users per node you can support. It's part of the QA cycle; and that doesn't go away with DevOps.
exelius 11y ago
Yeah, you forgot the default file descriptor limits that are an order of magnitude too low for running any kind of modern web service. At least it doesn't require a patch and recompile anymore like it used to back in the 90s when I was running big IRC servers.
scurvy 11y ago
Isn't this what tuned is for? http://servicesblog.redhat.com/2012/04/16/tuning-your-system-with-tuned/ http://servicesblog.redhat.com/2012/04/16/tuning-your-system...

deleted 11y ago

[deleted]

fulafel 11y ago

Nitpick: Maybe you are thinking of Windows wrt buffer sizes? Linux TCP out of the box has window scaling enabled since 2004 or so. "sysctl net.ipv4.tcp_rmem" tells me max buffer size is 6+ MB on my old 4GB laptop, which should be good to saturate 10 a gigabit pipe with up to 50 ms RTT... Mininum retransmission tuning might get you something on your own network if it's congested but would probably be bad to crank down for general internet usage, given things like bufferbloat and large delays caused by wireless L2 retransmissions.

acd 11y ago

Linux TCP window tuning is in fact off, agree with thrownaway. It's by the way a trick that the CDN providers use since they tune their stacks to increase the download speed. But if you could as well do the same tuning yourself.

superuser2 11y ago

How can one learn to do this? Are there good books/classes on the topic? Learning from mentors seems pretty vulnerable to cargo culting/bad science but I don't know where to even start.

j1e 11y ago
http://www.brendangregg.com/ http://www.brendangregg.com/

rdtsc 11y ago

> all Linux distributions are tuned for nothing in particular. Which is good, because often tuning means making trade-offs. A lot of those things like polling and interrupt shuffling have side effects and might for example negatively impact other scenarios -- presumably not everyone out there is running single client-server ping-pong with small UDP packets. Yes a lot of parameters have retained default values from back in the day when system characteristics were different. Switching defaults drastically might break existing code that relied on them. Perhaps those are better updated during major distro version upgrades (RHEL 6->7 for ex). In general though, I would say Linux is tuned for average throughput not latency. Sometimes they go hand in hand, sometimes they don't. Over the years there have been patches and forks related to operating on low latency environments (stock trading, media processing, hardware control, graphics in desktop environment). Those range from parameters tweaks in /proc to kernel patches, to almost full kernel bypass via things like DPDK for Ethernet for example.

thrownaway2424 11y ago
Sure, they are all tradeoffs of some kind or another. It's crazy to think that one person deploying a linux box on AWS would be aware of them all, though. For example what is the default file-max on Ubuntu? Why?
- rdtsc 11y ago
  Well you have different use cases and you have a large panel of nobs and switches you can adjust. You need someone who knows what the knobs do and what effect they have. So perhaps there is a use case for wizard like configuration tool for intermediate users. It can ask you questions like -- how many clients do you have? How many concurrent connections. Would you rather have higher average throughput (say for downloading files) or lower packet latency (say for audio processing). Would you rather use your memory for a file cache for for your application data. And so on. Next step up -- a self tuning system. It monitors your usage pattern then decides what settings to use. Monitors again, and adjusts a bit and so on.
  - gshx 11y ago

billyhoffman 11y ago

I agree, and Linux's default state is a matter I raise when people talk to me about how awesome DevOps and Full Stack developers are. They may be able to launch a VM and get a web stack up and running with yum, but that's really not reasonable for production usage

exelius 11y ago
The way we work this is that DevOps builds the systems, and the performance testers tune the configs based on their testing. Performance testing is really not the right word for what they do; it's really performance engineering. Regardless of whether you use DevOps or not, if scalability is an issue, someone needs to be doing performance testing and sizing to have a rough idea of how many users per node you can support. It's part of the QA cycle; and that doesn't go away with DevOps.

exelius 11y ago

Yeah, you forgot the default file descriptor limits that are an order of magnitude too low for running any kind of modern web service. At least it doesn't require a patch and recompile anymore like it used to back in the 90s when I was running big IRC servers.

scurvy 11y ago

Isn't this what tuned is for? http://servicesblog.redhat.com/2012/04/16/tuning-your-system-with-tuned/ http://servicesblog.redhat.com/2012/04/16/tuning-your-system...