4 ms·

Adding test coverage might indeed destroy their performance, in that they might expose where their system's bottlenecks are and realize that they've been focusi

Adding test coverage might indeed destroy their performance, in that they might expose where their system's bottlenecks are and realize that they've been focusing on the wrong things. When it comes to performance optimizations, folks' intuitions are almost always wrong. It's important to actually test and measure stuff like that. I think it's true in general that modern CPUs are insanely fast at executing code within a single process, and context switches are orders of magnitude slower due to their overhead. So, monolithic systems are going to be intrinsically "fast" compared to systems that pass messages through an operating system. They seemingly made a good decision for their project there, whether or not they made that decision rationally. It's possible they invested a lot in infrastructure for runtime performance profiling and monitoring and therefore know exactly what pieces of code to optimize. Reading the article, though, they're seemingly sweating the overhead of polymorphic function calls. The overhead of instrumenting the code would likely be higher than the overhead of polymorphism in a language like C#, so if their code was actually that performance sensitive, they wouldn't have it instrumented in production. If they did have it instrumented, they would likely have realized that their code wouldn't be bottlenecked by a little bit of function call indirection. Also, all the talented folk I know that stress the importance of runtime instrumentation also stress the importance of regression testing, so I suspect it would be pretty rare for a team to value one but not the other.

vp8989 6y ago
It's hilarious that you are skeptical of the competence of the SO core devs. You should do a little further reading, there is lots of content available online from the core devs who actually "do the work" there (ie, not the people who wrote this fluffy blog post).
- andrewem 6y ago
  FYI, the first listed author of this post has the job title Principal Software Developer, which I assume makes that person is one of the people who write the code. Edit: added missing “is”
  - vp8989 6y ago
    Fair enough, thanks for the correction. I guess the tone of the post doesn't convey that they know what they are doing to some people, although it should be obvious that they didn't just sleepwalk into having one of the highest "power to weight" ratio web apps on the Internet.
- sgtnoodle 6y ago
  I'm basing my opinion on reading the article. The claims that were made are consistent with what I've experienced working with folk with relatively limited software engineering skill. I don't think that's an unfair or mean thing to say. The whole premise of the article is to try and convince me that not following best practices can be a good thing. To me, the article seemingly reads as defensive, as if the author is actually trying to convince themselves or their coworkers that they aren't to blame for an accumulated decade of technical debt. In reality, there's no blame to assign when tackling tech debt. Real projects are messy. Folk defer work for short term gains at the cost to their future self. Growing tech debt is just an eventual engineering problem to tackle like anything else. Best practices are distilled wisdom intended to keep those sorts of problems from growing out of control. Re: the competency of the SO core devs, they very well could and probably are solid within their domain. Not knowing anything about them though, I think it's possible that they're a bit siloed off in their knowledge. If a lot of their core team were there from the beginning of the project and were relatively inexperienced at the time, they may not have ever invested their time into developing test tooling. In some ways it's very impressive that they're able to keep that large of a project going with so little testing infrastructure.
im3w1l 6y ago
I think you need both. Intuition most of the time to prevent the problem of a little bit of slow everywhere adding up, and then profiling to make sure there is no huge thing hiding in the blindspot of your intuition.
pjscott 6y ago
I'd like to take this opportunity to encourage people to learn about sampling profilers. What they do is periodically record the program counter and call stack of a process, then aggregate these to get a picture of where the program was spending its time. With a low enough sampling frequency they can have negligible impact on the speed of the software being profiled, they require no changes to production code, and they operate with fine enough granularity that they can easily tell you if (for example) a particular polymorphic function call is adding significantly to the runtime of a hot loop. Often all you need to do to find out what the real bottlenecks are is to point one of these profilers at a running program and look through the results. Super low effort, potentially big payoff.

vp8989 6y ago

It's hilarious that you are skeptical of the competence of the SO core devs. You should do a little further reading, there is lots of content available online from the core devs who actually "do the work" there (ie, not the people who wrote this fluffy blog post).

andrewem 6y ago
FYI, the first listed author of this post has the job title Principal Software Developer, which I assume makes that person is one of the people who write the code. Edit: added missing “is”
- vp8989 6y ago
  Fair enough, thanks for the correction. I guess the tone of the post doesn't convey that they know what they are doing to some people, although it should be obvious that they didn't just sleepwalk into having one of the highest "power to weight" ratio web apps on the Internet.
sgtnoodle 6y ago
I'm basing my opinion on reading the article. The claims that were made are consistent with what I've experienced working with folk with relatively limited software engineering skill. I don't think that's an unfair or mean thing to say. The whole premise of the article is to try and convince me that not following best practices can be a good thing. To me, the article seemingly reads as defensive, as if the author is actually trying to convince themselves or their coworkers that they aren't to blame for an accumulated decade of technical debt. In reality, there's no blame to assign when tackling tech debt. Real projects are messy. Folk defer work for short term gains at the cost to their future self. Growing tech debt is just an eventual engineering problem to tackle like anything else. Best practices are distilled wisdom intended to keep those sorts of problems from growing out of control. Re: the competency of the SO core devs, they very well could and probably are solid within their domain. Not knowing anything about them though, I think it's possible that they're a bit siloed off in their knowledge. If a lot of their core team were there from the beginning of the project and were relatively inexperienced at the time, they may not have ever invested their time into developing test tooling. In some ways it's very impressive that they're able to keep that large of a project going with so little testing infrastructure.

im3w1l 6y ago

I think you need both. Intuition most of the time to prevent the problem of a little bit of slow everywhere adding up, and then profiling to make sure there is no huge thing hiding in the blindspot of your intuition.

pjscott 6y ago

I'd like to take this opportunity to encourage people to learn about sampling profilers. What they do is periodically record the program counter and call stack of a process, then aggregate these to get a picture of where the program was spending its time. With a low enough sampling frequency they can have negligible impact on the speed of the software being profiled, they require no changes to production code, and they operate with fine enough granularity that they can easily tell you if (for example) a particular polymorphic function call is adding significantly to the runtime of a hot loop. Often all you need to do to find out what the real bottlenecks are is to point one of these profilers at a running program and look through the results. Super low effort, potentially big payoff.