7 ms·
This kind of thing doesn't surprise me at all - a surprising number of developers miss those types of things, and one of the things a lot of people miss seems t
by vidarh 5y ago
This kind of thing doesn't surprise me at all - a surprising number of developers miss those types of things, and one of the things a lot of people miss seems to be checking for situations where you end up with excessive amount of calls doing little work, for some reason.
E.g. at one point (long time ago), MySQL's C client library would call read() for 4 bytes to read a length indicator, and then read exactly the number of bytes indicated by the protocol, which led to a lot of time spent context-switching vs. reading into a larger user-space buffer and copying out of that.
On Linux simple, plain "strace" remains one of my favourite first steps to find performance problems in part because these things are near immediately apparent if you strace a process because the really pathological cases often ends up dominating the output so much that you'll spot them right away.
Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood.
A similar example of testing with too small datasets and/or on a large enough machine to not spot what becomes immediately obvious if you trace execution was how a type 1 font reference library made available by Adobe (no idea if they wrote it or if it was from another source originally) which would exhibit another typical pathological behaviour and call malloc() hundreds of times on loading a font to allocate 4 bytes at the time instead of allocating larger buffers. (strace won't catch malloc(), but ltrace does)) To avoid messing too much with the code, we replaced the calls to malloc() with calls to a simple arena allocator and load speed shot through the roof and memory usage dropped massively.
Too few people trace execution of their code. I know that because if most developers did, issues like the above would get caught much sooner, and would be rarer in published code.
- formerly_proven 5y ago> Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood. When you add e.g. Python's standard library as a ZIP to the Python search path, the thing will open/read/read/read/close that file approximately a gazillion times on startup. That's OK on Linux/Unix, where that is fairly cheap. Guess which OS doesn't like that pattern at all?