19 ms·
I get a bit annoyed by this sort of post, blanket statements like: "With our code base, you won’t be able to easily do test driven development or similar pract
by ed_elliott_asc 6y ago
I get a bit annoyed by this sort of post, blanket statements like:
"With our code base, you won’t be able to easily do test driven development or similar practices that the industry seems to love."
Why does the industry seem to love testing? Does the industry love testing? Does this mean you do or don't love testing?
It also feels like they wrote this blog post to show off: we write our c# code as if it was c code because we are better than all of you.
The fact is that every application is different, they chose to use a language that back in 2000's wasn't designed for performance and so they paid the price of using it for performance and didn't want to pay the scale out price.
Is their codebase testable, yes completely. Will adding unit tests to their code destroy their performance? no, there are always ways to make it testable and fast. Maybe in a few years we will see a "We thought testing was impossible but we made it work and also keep our codebase fast".
- sgtnoodle 6y agoAdding test coverage might indeed destroy their performance, in that they might expose where their system's bottlenecks are and realize that they've been focusing on the wrong things. When it comes to performance optimizations, folks' intuitions are almost always wrong. It's important to actually test and measure stuff like that. I think it's true in general that modern CPUs are insanely fast at executing code within a single process, and context switches are orders of magnitude slower due to their overhead. So, monolithic systems are going to be intrinsically "fast" compared to systems that pass messages through an operating system. They seemingly made a good decision for their project there, whether or not they made that decision rationally. It's possible they invested a lot in infrastructure for runtime performance profiling and monitoring and therefore know exactly what pieces of code to optimize. Reading the article, though, they're seemingly sweating the overhead of polymorphic function calls. The overhead of instrumenting the code would likely be higher than the overhead of polymorphism in a language like C#, so if their code was actually that performance sensitive, they wouldn't have it instrumented in production. If they did have it instrumented, they would likely have realized that their code wouldn't be bottlenecked by a little bit of function call indirection. Also, all the talented folk I know that stress the importance of runtime instrumentation also stress the importance of regression testing, so I suspect it would be pretty rare for a team to value one but not the other.
- vp8989 6y agoIt's hilarious that you are skeptical of the competence of the SO core devs. You should do a little further reading, there is lots of content available online from the core devs who actually "do the work" there (ie, not the people who wrote this fluffy blog post).
- andrewem 6y agoFYI, the first listed author of this post has the job title Principal Software Developer, which I assume makes that person is one of the people who write the code. Edit: added missing “is”
- vp8989 6y agoFair enough, thanks for the correction. I guess the tone of the post doesn't convey that they know what they are doing to some people, although it should be obvious that they didn't just sleepwalk into having one of the highest "power to weight" ratio web apps on the Internet.
- sgtnoodle 6y agoI'm basing my opinion on reading the article. The claims that were made are consistent with what I've experienced working with folk with relatively limited software engineering skill. I don't think that's an unfair or mean thing to say. The whole premise of the article is to try and convince me that not following best practices can be a good thing. To me, the article seemingly reads as defensive, as if the author is actually trying to convince themselves or their coworkers that they aren't to blame for an accumulated decade of technical debt. In reality, there's no blame to assign when tackling tech debt. Real projects are messy. Folk defer work for short term gains at the cost to their future self. Growing tech debt is just an eventual engineering problem to tackle like anything else. Best practices are distilled wisdom intended to keep those sorts of problems from growing out of control. Re: the competency of the SO core devs, they very well could and probably are solid within their domain. Not knowing anything about them though, I think it's possible that they're a bit siloed off in their knowledge. If a lot of their core team were there from the beginning of the project and were relatively inexperienced at the time, they may not have ever invested their time into developing test tooling. In some ways it's very impressive that they're able to keep that large of a project going with so little testing infrastructure.
- arcturus17 6y agoI also found the "or similar practices that the industry seems to love" coming across as a bit vague and even arrogant. On the other hand, I think that they have earned bragging rights - we're talking about a site that has massive scale and reach, and has a reputation for being a very efficient system. I appreciate that people like them share their views, even if they are contrarian and a touch arrogant. > they chose to use a language that back in 2000's wasn't designed for performance and so they paid the price of using it for performance and didn't want to pay the scale out price. In terms of more performant languages at the time what could have they picked? C++? It sounds like they would've lost a lot of the conveniences that a framework like .net gave them. Isn't the fact that they avoided paying the scale out price remarkable?
- chovybizzass 6y agothey seem to love it...non-committal.
- orclev 6y agoIt's important to remember Joel Spolsky, one of the founders of stackoverflow is an old school Microsoft guy that managed the Excel team before starting his own company. He has always been a bit of a Microsoft fanboy, and used to advocate a lot for VB as a serious language back before it got swallowed by .Net and turned into a less powerful syntax for C#. The fact they chose C# isn't surprising at all given that. It actually would have been far more surprising to see them pick anything but C#. I think the previous poster was a bit off the mark though. The language performance wasn't really the issue there, rather it's the fact that they picked a language that at the time really only ran on Windows, and as a consequence they were forced into running their web servers on Windows. That choice then forces them to scale up rather than out since each instance has license costs attached to it. For most companies running on Linux, it's trivial to scale out since your only costs are the compute cost (or the hardware cost in a non-cloud model), where as it tends to be far more expensive to scale up as more powerful hardware tends more towards geometric increases rather than linear. These days the choice of C# wouldn't be such a big issue as .Net core can easily run on Linux servers, but back in the 2000s using C# was putting a pretty big albatross around your neck by way of Windows licenses.
- mcv 6y agoLines like that[0] sound like the goal is to please the industry, without questioning why the industry loves these things. It's good to question established best-practices, but it's also good to understand why they are best-practices. [0] "With our code base, you won’t be able to easily do test driven development or similar practices that the industry seems to love."
- klingon78 6y agoSomething not mentioned but is a much bigger problem than performance for most is that if you write too many tests, you may be unable to easily upgrade/update to newer versions without a significant amount of work to update all of the tests. You’re right to call out performance as a reason for testing. Unfortunately, though, it’s usually not that easy. A problem not found in testing may be the result of inexperience or lack of foresight in design, such as the assumption that another adapter service in front of that enterprise cloud service that only exists in production isn’t a bottleneck, or that your code that had 100% unit test coverage turned out to leave nasty half-updated data due to non-transactional services and was deadlock-prone with memory leaks.
- stonemetal12 6y agoIsn't that a bit of a red herring? Either I have automated tests that will take time to update every once in a while, or I have manual regression tests that I have to run for every code change. The automated test path seems to be the much less time intensive path, depending on how often you make big changes. On the other hand I really believe 100% unit test coverage is a bit of a waste of time. Unit tests should be reserved for algorithmic\calculative code, more mundane code logging\carting data from a to b type code should be covered by automated black box testing.
- klingon78 6y agoI meant upgrading application frameworks, OS, etc., so it’s not a red herring. Those can inflict a lot of changes if you have a lot of tests affected.
- stonemetal12 6y agoThus automated test time + automated test maintenance is usually less than manual test time + manual test maintenance. Mostly because execution time for automated tests is so much shorter than manual execution time that the higher cost of automated test maintenance is recouped and then some. The only way for framework\OS changes to impact that is if they happen often enough to make automated test maintenance "weigh" more than all the time saved by no longer having to manually run your test suite.
- Cthulhu_ 6y agoThey're really not selling their codebase or practices to me to be honest. If their system is built like that it might have enough performance to run a top x site on only a handful of bare metal servers, but the article also implies there's a lot of "here be dragons" code - don't touch this, you are not clever enough to test this, and since we don't have tests, you might break it by touching it. Which is just bad practice on the one hand, and elitist on the other. I for one am thankful I don't have to work with those constraints, and I can adopt a better mindset - make it work, make it pretty, make it fast, for example. I think you cannot optimize a system if you cannot read and understand it. Also note I explicitly said "system", not "codebase"; at a big enough scale, you can't look at performance as a factor of code, but of architecture and patterns. You need to be able to get figures from specific systems in the whole architecture, not individual lines of code. Make the architecture easy to analyze and optimize.
- yowlingcat 6y ago> Which is just bad practice on the one hand, and elitist on the other. I got exactly the same impression. Making a strawman out of the rest of the market and special-casing yourself always sets off a red flag. It indicates the possibility of a culture could be out of touch with reality, that chooses to rationalize its poor decision rather than facing reality. I wouldn't work at a firm with this kind of culture, it's a liability.
- helsinkiandrew 6y ago> the article also implies there's a lot of "here be dragons" code Agreed, fine if enough of the core developers are around, but if the company merged, the app needed to be extended or altered significantly, or some of the underlying technology stack is deprecated/becomes unsupported keeping things working will be very expensive.
- ed_elliott_asc 6y agoI think that they are in the situation where it can't be altered significantly which ironically is probably a good thing for stack overflow - they don't have reams of features.
- Veuxdo 6y agoIs there not a whole industry devoted to selling developers on automated testing?
- thrower123 6y agoYou seem to be searching out a reason to be offended by this rather bland, vanilla blog post that simply explains what they did.
- gambler 6y ago>Why does the industry seem to love testing? Because testing coverage is a typical way modern software engineers and managers show off an otherwise mediocre codebase. >because we are better than all of you. Well, at least they have something tangible to show for it. TDD elitism is much more ridiculous, because (as I've seen over and over) having high test coverage does not mean the code is bug-free or even properly modularized. If you want to brag about your tests, don't shot me your coverage, show me how you have 10 times less bug reports than the next guy.
- bluGill 6y ago> show me how you have 10 times less bug reports than the next guy. You can't actually do that because there are too many other differences. However good tests do in the long run tend to result in high quality code. If nothing else because sometimes the obvious fix for one bug breaks something seemingly unrelated where the obvious fix is to undo the first first. I've seen that go on for 4 rounds over 8 years in one company with different contractors doing the work every time (only by luck one of the few full time employees happened to review the code twice and think to check version control to see the history of the obvious fix). Tests are more expensive to get an equivalent quality version one out the door in general. (if you have a 15 year development cycle maybe not). However by the time you are on version 3 you save yourself a lot of retesting the same thing again to get quality. Though the most obvious affect of tests is as you say: a metric that management can brag about. Also one of the few metrics where gaming the metric for more isn't entirely bad (though it can be a waste of money!)
- shadowgovt 6y agoThe industry loves testing because the industry (big handwave, encompassing a huge pile of heterogeneous software development houses) is in the business of risk management and scaling practices. If SO needs to double its engineering base tomorrow, can they? A well-tested infrastructure means you can trust engineers less deeply familiar with the system to make improvements without breaking pieces they aren't familiar with. The tests provide a bit of a behavior-expectations guarantee (not foolproof, as there's good language theory indicating why unit testing can't be sufficient for proof of correctness, but better than nothing). Without it, you have less safety net if junior programmers change component A failing to realize that component F deeply removed from it was assuming some behaviors of A. One piece of this post worth noting: > Currently, we’re trying to change this. We’re actively trying to write more tests and make our code more testable. It’s an engineering goal we aim to achieve, but the changes needed are significant. ... and that means if they have a reason to massively scale-up tomorrow, their codebase isn't starting in a good place to make that change. They've made a first-to-market risk tradeoff that they may have to pay for if they need to add a bunch of new engineers right now and have them operating independently by Friday.
- dustinmoris 6y ago100% agreed. This was a very poor article. They've basically gone with an overly expensive Microsoft stack which meant that from the beginning they were fighting the wrong fight. Instead of thinking about scale for best performance, they had to think about best cost for minimum scale. Furthermore they decided to bastardise their C# code to some fake low level untestable C dragon code which makes matters even worse. They have created a huge mess which is probably extremely hard to maintain and too precious to touch for a performance gain which probably still only scratches on the surface of what a true low level language could have achieved. The best practice would have been to build the parts which matter most in a language which allows them to be blazingly fast and write nice testable and easy to understand code. Something like Rust or Go. But no, they stuck with C# for god knows what reason (maybe financial Microsoft incentives who knows?) and had to throw away every sensible coding practise in order to make it work. This is literally a blog post describing how not to run a large scale website.
- shalmanese 6y ago> But no, they stuck with C# for god knows what reason Mainly because Rust (2010) and Go (2009) weren't around when SO was first created (2008).
- kcartlidge 6y ago> The best practice would have been to build the parts which matter most in a language which allows them to be blazingly fast and write nice testable and easy to understand code. Something like Rust or Go. But no, they stuck with C# for god knows what reason Whilst for work I've done approaching two decades of C#, for my own stuff I have largely used Go for years now so I'm far from an MS apologist, but that impression of C# is wrong. To be fair, the article leads partly to your conclusion so that's understandable. C# with .Net Core, almost by design and by default, massively encourages "nice testable and easy to understand code" - much more so than most languages out there. And unless the codebase is a hacked together monstrosity of the type that can be created using any ecosystem, .Net Core and C# are massively performant (pre-Core .Net was just about up there with some competitors but Core is in another league over the old Framework). It's a shame that it doesn't have the mindshare outside of enterprise that it deserves, as C# on .Net Core basically gives you (practically, not syntactically) type-safe Node in terms of async/await, Go-like performance, Java-like power, and Ruby-like simplicity. Though to be honest it can take a fair while to advance from "hello world" given the atrocious naming and versioning mistakes Microsoft have made and how that impacts documentation and research.