3 ms·

Systems tuning, but it's an area that really got me to learn what I have over the years. I realize I'm a bit biased. It's amazing the number of production iss

Systems tuning, but it's an area that really got me to learn what I have over the years. I realize I'm a bit biased. It's amazing the number of production issues I've fixed with simple sysctl parameters. The Linux kernel defaults don't fit very well in this huge-cluster/scaling out world

bombcar 4y ago
One thing we're strongly missing is the kernel being able to notify that a sysctl parameter may help. Sometimes it does things like "clock changed, is all hell breaking loose" but it could do much more when queues back up, etc.
- bravetraveler 4y ago
  Absolutely! So much of my 'bag of tricks' is having been around the block so many times that I can sniff these things out. It's a bad way for things to be!
rufius 4y ago
In general, our industry lacks a sense of "mechanical sympathy"[0]. Whether it's understanding what knobs are tunable in an operating system or using streams for reading data to keep a consistent memory profile, we often don't dig deep. Ditto for folks understanding database systems. I think in very big systems, you see folks dig deep, but it's not encouraged with smaller systems. I think there's a lot of inefficiently run services - not from a price perspective but from a "what could I really get out of 3 compute nodes and a well designed database schema." [0]: https://mechanical-sympathy.blogspot.com/ https://mechanical-sympathy.blogspot.com/
- bravetraveler 4y ago
  I appreciate you introducing me to this concept -- it's something I've thought a lot about, but could never verbalize
  - rufius 4y ago
    I think about it a lot. A portion of my job has been either optimizing big systems or building very minimally invasive software that has to process a lot of data. It’s akin to tuning a sports car. You can slap a big engine in a car, but the difference of a few seconds off of a lap is thousands of man-hours in optimization and tuning.

bombcar 4y ago

One thing we're strongly missing is the kernel being able to notify that a sysctl parameter may help. Sometimes it does things like "clock changed, is all hell breaking loose" but it could do much more when queues back up, etc.

bravetraveler 4y ago
Absolutely! So much of my 'bag of tricks' is having been around the block so many times that I can sniff these things out. It's a bad way for things to be!

rufius 4y ago

In general, our industry lacks a sense of "mechanical sympathy"[0]. Whether it's understanding what knobs are tunable in an operating system or using streams for reading data to keep a consistent memory profile, we often don't dig deep. Ditto for folks understanding database systems. I think in very big systems, you see folks dig deep, but it's not encouraged with smaller systems. I think there's a lot of inefficiently run services - not from a price perspective but from a "what could I really get out of 3 compute nodes and a well designed database schema." [0]: https://mechanical-sympathy.blogspot.com/ https://mechanical-sympathy.blogspot.com/

bravetraveler 4y ago
I appreciate you introducing me to this concept -- it's something I've thought a lot about, but could never verbalize
- rufius 4y ago
  I think about it a lot. A portion of my job has been either optimizing big systems or building very minimally invasive software that has to process a lot of data. It’s akin to tuning a sports car. You can slap a big engine in a car, but the difference of a few seconds off of a lap is thousands of man-hours in optimization and tuning.