3 ms·
Tight feedback loops are great for software development, but terrible for systems administration. In my career, I regularly have these conversations: "We did A
by jiggawatts 5y ago
Tight feedback loops are great for software development, but terrible for systems administration. In my career, I regularly have these conversations:
"We did A, it was slow, then we did B, and it was fast." == "Congratulations, you discovered that the cache works, not that B is faster than A."
"We ran this test on AWS image size such-and-such, and it performed 5% better than the competing product that we tested previously." == "That's just noisy neighbours on a multi-tenant platform, you've learned nothing."
"We ran a test and it succeeded, hence it is now working!" == "The system fails nine out of ten times randomly, you just stopped when you had your first success. Try it again."
This ability to spin up a test environment quickly and run it through a scenario can be intoxicating. People try all sorts of random stuff, see a "win", and then call it a day without really doing the legwork to prove that they achieved something meaningful.
- k__ 5y agoDo you have good resources on how to do "the legwork"?
- jiggawatts 5y agoThis guy has had a few talks at CppCon over the years, and covers the basics of how to attempt benchmarking with actual computer science instead of myth and voodoo rituals: https://www.youtube.com/watch?v=koTf7u0v41o https://www.youtube.com/watch?v=koTf7u0v41o In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :) Simple methods I use and can recommend are: - Ideally, collect data from production, where the change behaves exactly as it would in production because it is production! - Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc... - Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc... - Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance. - Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph. - Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph above can help, but only if your input size goes waaaay past the biggest cache sizes in the system. As a real-world example: I recently reduced the size of a database platform by a few vCPUs to save on licensing costs, but increased the per-core performance and simultaneously did some network tuning. Some benchmarks showed the same performance, others showed an improvement. Both were extremely noisy before and after the change. I would like to take credit and say that I improved performance while saving money. However, the scientifically honest result is that any performance change is statistically insignificant. The cost saving is significant, because it has very little variance. That's good enough, and I don't have to make a statement about performance that the numbers can't really back up.
- lanstin 5y agoI would add always (for network servers) run “open” loads tests with each request being independent of the fates of the other requests, probably from a Poisson distribution (generated by having the delays be pulled from exponential random distribution ( https://en.wikipedia.org/wiki/Exponential_distribution https://en.wikipedia.org/wiki/Exponential_distribution ). So many load tools send a request then wait for the reply then send the next. That is basically useless for understanding variance under spiky load and for assessing when a system is below its capacity how stable it is. For real world performance, the variance matters a lot. The code will make some things be cached some not (so the data variability in the test load should follow the same sort of power law that your real traffic does), and even if you get 10% faster at the median if the variance goes up the quality is lower.
- k__ 5y agoThe tools I used had a parallel requests option.
- josho 5y agoWe focus far too much time teaching calculus and trigonometry in school. Applied statistics would be far more relevant for nearly everyone.
- sobriquet9 5y agoIt's not easy to learn statistics without calculus.
- yjftsjthsd-h 5y agoBasic level doesn't need calc
- marcosdumay 5y agoAny part of statistics is completely dependent on calculus. You don't get to choose how features are distributed over the set of possible samples. It's always some scary looking function without exact solutions. But you don't need to teach people all the fundamentals of calculus before they can apply it. Definitions and a small set of pre-made answers are enough to start using it and focusing on the statistics. When (and if) those people decide to learn it more in depth, they can go back to calculus fundamentals, and will have an easier time doing so, because they will know why those things exist.
- ozim 5y agoI guess that parent poster was thinking about probability and mixed it up with statistics. If yes, then I agree with him. People should have more probability lessons at school than trigonometry or calculus.
- sobriquet9 5y agoBasic probability (coin flipping, events over trials) by itself is just as useless as basic trigonometry. It only becomes useful as you move from simple combinatorics to continuous distributions, at which point you need differentiation and integration to switch between CDF and PDF. Much in the same way trigonometry becomes useful in physics once you need to deal with vectors and waves.