17 ms·

Best practices can slow your application down

195 points by kettunen 6y ago

phendrenad2 6y ago
This should be obvious. Writing well-structured, decoupled, modular code can make you lose track of performance concerns. And since no one likes to write things more than once, so there's only "one shot" at writing the application, people tend to err on the side of "well-structured", because hey, who's gonna fore you for following best practices?
- deleted 6y ago
  [deleted]
- beny23 6y ago
  I’d argue that getting performance in a coupled unstructured mess (I’m not saying that’s what SO made) is going to be more difficult.
- mytailorisrich 6y ago
  On the other hand, trying to optimise too early leads is wasteful and leads to headache down the line. Writing "well-structured, decoupled, modular code" is a best design practice for a good reason. It what leads to best results overall, where "best results" means code that is easier overall to test, maintain, modify/expand, and optimise as needed. Regarding optimisation, something related to keep in mind is that optimisation at system level (which is often the desired end result) is not the sum of independent local optimisations.
nsoonhui 6y ago
Duplicate of [https://news.ycombinator.com/item?id=26381640 https://news.ycombinator.com/item?id=26381640}]
t0r0nat0r 6y ago
Clearly adding further constraints can only worsen performance. I would code according to best practices except for cases where it’s possible to infer them implying tight bottlenecks. Then profile the code and if circumventing best practices results in a meaningful performance increase, do that.
sgtnoodle 6y ago
The article makes the excuse that static (singleton) interfaces are inherently simpler and less resource intensive enough for it to be worth skipping regression testing (because testing that style of code is hard.) Reading between the lines, it sounds like maybe they know they've dug themselves into a very expensive hole to dig out of. I've jumped head first into codebases that made the same excuse, and to me it seems severely limiting in terms of long term productivity. There are definitely benefits to simple, "static" singleton interfaces, and such interfaces aren't inherently incompatible with mocking and dependency injection. It does mean you likely need a layer of indirection somewhere, but if you're sweating the overhead of a single vtable lookup or explicit function pointer call on code that's internally accessing a non-trivially sized data structure, how are you even able to measure it? The problem is that a handful of foundational static interfaces never got unit tested, and all the other code calls directly into them, so nobody anywhere wrote any unit tests because they couldn't (without touching foundational code that's unsafe to modify for lack of unit tests.) The first step in the right direction is to fix the foundation. This is terrifying for the folk that have been around a while and cemented in their assumptions about risk because they've tiptoed around modifying portions of the codebase for years. Luckily, a big benefit of static singleton interfaces is that it's very easy to modify them with O(n) developer time (where n is the number of references in the codebase). So, you just have to buckle down and get your hands dirty. You're pretty much guaranteed to find at least one latent bug in any old piece of code that you unit test, and so folk start to see the merits of the test coverage and it becomes easier to prioritize refactoring more and more ancient and scary code. After you've done it to half a dozen or so disjoint pieces of code, something magical happens. Suddenly, the vast majority of the codebase becomes easily unit testable.
- brown9-2 6y ago
  The post also presents a false choice between what’s known as “the singleton pattern” versus just allocating a single instance of the object and passing it around where needed: > We don’t write a lot of these because our code doesn’t follow standard decoupling practices; while those principles make for easy to maintain code for a team, they add extra steps during runtime, and allocate more memory. It’s not much on any given transaction, but over thousands per second, it adds up. Things like polymorphism and dependency injection have been replaced with static fields and service locators. “dependency injection” doesn’t require constantly allocating new instances of the things your classes need.
syastrov 6y ago
I’m just thinking about how much CO2 emission was spared by optimizing performance instead of scaling horizontally. It would be nice to see the effect on the environment as part of the decision-making process for deciding whether to optimize versus scale. I think developers have an innate desire to fix performance issues as optimization produces a better understanding of the code and the problem and developers would like to solve the problem in the best possible way. And it helps to know that the problem won’t bite again later.
waf 6y ago
It sounds like the team is interested in adding more automated tests, but are blocked by static singletons, which have high performance but also high coupling, resulting in poor testability. I'm sure they've heard of test libraries like MS Fakes and Pose; I wonder if these libraries would let them maintain high performance, and only introduce the required layer of indirection during testing?
grumple 6y ago
These sound like some poor excuses to me. The idea that you can have either best practices or performance is a false dichotomy. Testing can be used to help track down performance issues. It can help you write code faster and prevent breakages. It should not slow you down. Best practices inside a codebase should not sacrifice performance. If you have to do that to conform to some structural ideal, you don’t do it. I would love to see this supposedly untestable code. Stackoverflow may contain lots of great dev knowledge, but it’s also a weird site that discourages participation and entrenches old answers. I am unsurprised that the devs themselves are a proverbial old dog that can’t learn new tricks.
- alkonaut 6y ago
  > The idea that you can have either best practices or performance is a false dichotomy. Testing can be used to help track down performance issues. It can help you write code faster and prevent breakages. It should not slow you down. It orks the other way too. Benchmarking can show that your bottleneck is virtcalls/boxing where you have used an interface field instead of a static instance to a concrete dependency. The argument isn't "performance vs. tests" it's really "performance vs modularization". You can always move the tests upwards until you test the whole monolith. This isn't a common tradeoff but I also wouldn't dismiss it as something that never occurs and "best practices" wrt. modularization and unit testing always enables similar performance. It doesn't.
ed_elliott_asc 6y ago
I get a bit annoyed by this sort of post, blanket statements like: "With our code base, you won’t be able to easily do test driven development or similar practices that the industry seems to love." Why does the industry seem to love testing? Does the industry love testing? Does this mean you do or don't love testing? It also feels like they wrote this blog post to show off: we write our c# code as if it was c code because we are better than all of you. The fact is that every application is different, they chose to use a language that back in 2000's wasn't designed for performance and so they paid the price of using it for performance and didn't want to pay the scale out price. Is their codebase testable, yes completely. Will adding unit tests to their code destroy their performance? no, there are always ways to make it testable and fast. Maybe in a few years we will see a "We thought testing was impossible but we made it work and also keep our codebase fast".
- sgtnoodle 6y ago
  Adding test coverage might indeed destroy their performance, in that they might expose where their system's bottlenecks are and realize that they've been focusing on the wrong things. When it comes to performance optimizations, folks' intuitions are almost always wrong. It's important to actually test and measure stuff like that. I think it's true in general that modern CPUs are insanely fast at executing code within a single process, and context switches are orders of magnitude slower due to their overhead. So, monolithic systems are going to be intrinsically "fast" compared to systems that pass messages through an operating system. They seemingly made a good decision for their project there, whether or not they made that decision rationally. It's possible they invested a lot in infrastructure for runtime performance profiling and monitoring and therefore know exactly what pieces of code to optimize. Reading the article, though, they're seemingly sweating the overhead of polymorphic function calls. The overhead of instrumenting the code would likely be higher than the overhead of polymorphism in a language like C#, so if their code was actually that performance sensitive, they wouldn't have it instrumented in production. If they did have it instrumented, they would likely have realized that their code wouldn't be bottlenecked by a little bit of function call indirection. Also, all the talented folk I know that stress the importance of runtime instrumentation also stress the importance of regression testing, so I suspect it would be pretty rare for a team to value one but not the other.
ctdonath 6y ago
I find anyone using the term “best practices” usually doesn’t understand them.
hergin 6y ago
I think the problem with this article and their approach will be more visible when super star developers start to leave the company. If they are all the same people from the very beginning, yes, throw lots of practices out of the window because people know the codebase very well. But once they are out, there will be a mountain of tech-debt. They even try to make it testable.
beny23 6y ago
In my opinion, the idea that you can have well-tested code OR performance is tosh. If you don’t have good testing and do it regularly how can you refactor your application for speed if you’re not going to be sure that you’re going to introduce a regression? And I think the stackoverflow team know it: > Currently, we’re trying to change this. We’re actively trying to write more tests and make our code more testable. It’s an engineering goal we aim to achieve, but the changes needed are significant. This makes it sound like a lofty goal but I think it just sounds more like. “Oh crap, we’re sitting on a mountain of tech debt” No I know you can’t always expect to arrive at a project and find the sunny uplands of TDD and continuous delivery but if you do get to “optimised for speed” code (to me that’s a synonym for tech debt that’s slowing down delivery) all you can do is slowly add what should have been done in the first place (tests).
- hinkley 6y ago
  Very, very few people know how to make cheap tests. I have one guy who writes fixtures that couples all of his tests. When I need to add functionality, first I have to replace his fixtures with basic stubs, just so I can add another bit of functionality. And we have two people who copy his coding style. Probably to fit in. I think, based on my own experiences and working with people learning to write tests, that 'simple' tests often feel more like the pejorative meaning of the word. They get uncomfortable, like you're debasing yourself to write such boring code. Surely I can write more interesting tests. Yes, you can write more interesting tests if your goal is for nobody to ever touch your code. Mine, mine, mine. The best tests, I know where my new code went wrong simply by reading the test output (not the test). If I refactor I should be able to maintain that coverage even if the order or structure of things gets completely switched around. Complex tests begat complex tests. If your old test is 2-3 lines of code, I just delete it and write a new one that does the same thing.
  - beny23 6y ago
    Oh absolutely, keeping things simple should never be restricted to just production code. Testing code should be simple too. The idea that tests inhibits refactoring I find problematic though. Good tests enable refactoring. If the tests are too complex for anyone to change the code, the the code is too complex. Any complex system should be composed of simple components. Now this applies to monoliths as well as microservices. I agree it is easy to fall into the trap of thinking that tests are boring though as devs become more experienced there should be less of that thinking, hopefully.
choeger 6y ago
The argument against Tests sounds like a tooling problem to me. Why shouldn't a compiler be able to optimize the modular approach to the same code that a monolithic approach yields?
- nitwit005 6y ago
  As long as dynamically loading code at runtime is allowed, proving that there will only ever be one implementation of an interface isn't going to be possible. That makes fully optimizing away the cost of the indirection difficult.
- tsimionescu 6y ago
  Why is that relevant? The fact is that it doesn't (there are good reasons for it, but that is another matter). You can't work with the ideal tools, you can only work with the tools that actually exist. So, with the actual tools, this sacrifice must be made. And now do answer your actual question, the reason is that the languages we use are just not expressive enough, and the compiler doesn't have enough time. If you are calling an interface which could be either the actual code or the mock code, the actual code can't be inlined, and that alone usually discards dozens of other optimizations. You can't specify in a language something like 'assume this implementation in optimized builds', so we're stuck. There are ways around this, to be fair. In C#, you could ale all of your code generic, such that there could be two versions of your class - one with a real dependency, MyClass<RealDependency1, RealDependency2,...>, and one with mocked dependencies, MyClass<MockedDependency1, MockedDependency2,...>. Not sure if the C# compiler or JIT actually knows how to use this during optimization, but in principle it could. With C++ templates this would definitely allow optimization. With Java's extremely advanced JIT, you wouldn't actually need this code change, the compiler will be able to do it for you after a few thousand runs through the code, when it notices a single version is being called.
  - choeger 6y ago
    It is relevant because they chose to abandon best practices because of a lack of tooling. A badly chosen language constraining your development methods is like a tail wagging the dog. There are languages and, more importantly, implementations out there that allow this kind of design. Just think of SML/NJ as an example for the principle. If performance is important and C# doesn't allow unit testing performant code then switch to C++ or Rust. It is staggering how many people seem to see the choice of language as an absolutely unchangeable decision. Rewriting even millions of lines of js or python code is definitely doable, even incrementally, and if your organization does not deem it worth the effort then maybe performance is not your most important concern.
alexf95 6y ago
Fully agree with the article. > "Best practices, not required practices" Is a thing mentioned from the article that I think many people forget. It also not only applies to programming or coding but rather to most things in life. Just because someone says it is best doesnt mean it is for you in your situation.
- DarkWiiPlayer 6y ago
  Yea, I really like that quote because it perfectly expresses how I feel about "best practices": Use them when you're unsure, but ditch them if you know something else will work better.
acd 6y ago
Somewhere down the line, developers will start to realize that single applications running in the same server memory and CPU cache will run much faster than Micro services applications that makes api calls over the network. Depending on your organization scale of course, but for many smaller shops, there is a significant performance overhead of going over the network. Ie a Single binary running Rust/Go/Java/NodeJs vs distributed Microservices applications.
- scarmig 6y ago
  Within the same data center, a network call doesn't need to have significant performance overhead; it can be on the order of a read from main memory. Still overhead, but for many SLOs it's irrelevant. The inherent cost of microservices is conceptual and organizational complexity, not runtime performance (though you can certainly build slow microservices, you can also build slow monoliths).
  - tleb_ 6y ago
    You might get similar throughput but the latency isn't comparable.
- vergessenmir 6y ago
  I'd like to point out that the article does mention that their system is not single process. They have a database, caching layer and web servers at the very least which could be run on a single box. The number of API calls is a matter of degree when compared with a microservices architecture. A microservices approach does not mean that they should run on different hosts and an early design decision should be that the entire architecture can run on a single host and that the scale out is a feature of the system. It's worrying that this assumption is often made because it does lead to the maintenance and velocity issues that teams encounter when naively adopting microservices.
- pbowyer 6y ago
  I like RDS because I don't have to think about backups, replication etc but if I colocate the app server (web server, application code) and the database server on the same machine I see 1-2 orders of magnitude performance improvement in database-intensive apps. Replication is the same as usual; have another machine setup to replicate onto. The restriction is you can't fan out hundreds of application servers and half a dozen database servers if you need that - but I don't need to. I haven't done this with clients but I'm doing it with my own (cost-sensitive) side projects and appreciate removing the internal network latency.
kodah 6y ago
When I was an SRE-SE I despised the term "best practices". Teams would come in telling me all about the new best practice with serverless, Kubernetes, secrets storage, or whatever. It was almost always like someone took a magnificent full Lego set and snapped off a wall and was like, "Hey, try this, it was useful to us." If it's not a protocol or standard recognized by one of the IEEE, IETF, or ISO then it's a tutorial with a strong opinion. Nothing wrong about that, but just know you're often reading a strong opinion, not a standard. "Best practices" almost always come out of some business or project explaining what tested best with their circumstances. That alone tells you everything you need to know. Invest in a testing process which exposes the most critical parts of your application/infrastructure, attempt to overwhelm or exploit them, and you'll find out what your best practices need to be real quick. Those best practices may work at other companies but not nearly as well as if they invested in their own testing processes that are as or more rigorous than yours.
- koirapoika 6y ago
  I agree. Lately, I try to refrain from using the term 'best practices' in favor of 'good practices', 'common practices' or even 'my practices' for several reasons: - True best practices mainly refer to standards, and this list is very slowly updated, unlike the number of false ones; - It's often used for marketing purposes, which works well, but doesn't always imply quality; - Since this spreads fast, we often have things at the places where they do not fit, but squeezed into, because it's the best, so we must have it; - Forms a wrong mindset to rely upon someone else's experience instead of learning and gaining own understanding. In the end, we have tons of inapplicable or controversial information to filter, which doesn't make things better, in my opinion. Sharing is great but we should do it responsibly. It's also a bit funny to watch how the best practices suddenly become the worst. Edit: formatting
- marcosdumay 6y ago
  Standards are for interoperability. If you try to get best practices out of them, you are set for a world of pain. But I do agree, the hype of the day isn't a best practice either. And just the term "best practice" without any qualifier or context is already a sign of incompetence.
- 6y ago
harg 6y ago
One thing that stood out to me was they wouldn't need necessarily need to optimize for performance if they didn't choose a stack whose costs increase massively as you scale. In many applications things like having a testable, easy-to-read, well-structured codebase can trump performance, and it's often worth sacrificing a bit of performance if it means it's easier to refactor and build new features.
- martinald 6y ago
  I don't know why they went for SQL Server, .NET has had good support for other databases even back to 2009. I imagine that was a huge amount of their licensing cost. With .NET Core running so well on Linux now they would have no Windows Server licenses to worry about too. The first production version of .NET core was released in 2016. Even back in 2009 it was possible to run web apps on mono on linux too, I know a lot of shops around that era that used windows for more complex parts of the app but then used mono for 'simple' high volume parts to save $$$ on windows licensing.
  - jve 6y ago
    Because that was what the main developer and founder Jeff Atwood was familiar with. And he really liked the stack. > Atwood: We are not personally going to be language agnostic, because we need to actually build the site. And in terms of people actually working on it, Joel's in an advisory role, I'm gonna be writing code, and then a friend of mine, Jarrod, I'll be working very closely with. So it's sort of like 1.5 developers, so I need to actually get things done. In order to do that, I'm gonna fall back on what I know, and what I know is essentially ASP.NET. So ASP.NET is gonna be the platform. And I actually really like ASP.NET. Episode 003 April 29, 2008, from the StackOverflow podcast. https://stackoverflow.fogbugz.com/default.asp?W56 https://stackoverflow.fogbugz.com/default.asp?W56 Analysis from 2008 about Scaling Up / Out. Electricity also is something you can't ignore. https://blog.codinghorror.com/scaling-up-vs-scaling-out-hidden-costs/ https://blog.codinghorror.com/scaling-up-vs-scaling-out-hidd... Ah, sorry, this is a discussion about SQL Server being expensive rather than .NET stack.
lifeisstillgood 6y ago
I have to admit I would far rather work on a conceptually simple codebase, that has been kept fast and light with minimal testing, than to work on a conceptually muddled codebase with 100% testing. Performance is a feature, and so is being able to reason through a codebase. Being able to fit all the pieces in your head is a feature, but not so snappy.
- DarkWiiPlayer 6y ago
  > Being able to fit all the pieces in your head is a feature, but not so snappy. How about "architectural sanity is a feature"? It kind of isn't though. Clients don't care about qualities of your code that they can observe. They notice if your code is fast, but don't care it it's easy or hard to reason about. If anything, I'd say it's a bit of a meta-feature, because code-quality is directly linked to many other properties that people do care about (speed, new features, quick bugfixes, etc.)
  - lifeisstillgood 6y ago
    yes, but I think we live in a world of increasing software literacy - where a novel or a manifesto that is easy to read will beat out those that are not. Similar for software - the good software will include 'makes sense in the readers head'
raverbashing 6y ago
If someone says to me "something is a best practice" my question is: according to whom? Because 95% of "Best Practices" around are just somebody's opinion disguised as a rule. "Oh TDD is a best practice now" only if it's to sell more books. Yes I love unit testing, but to do it how it's the "best practice" please Do you have an objective metric that's improved by your "best practice"? SO itself is a great example. They're still on IIS+C# and it works for them. What if they didn't stop chasing the new fad or "best practice", like Kubernetes
thrower123 6y ago
To be fair, many of the ASP.NET best practices are completely insane.
arkh 6y ago
TDD does not mean "small unit tests at the method level in code". It means you code your test cases before you implement the functionalities. Those tests can (and IMO should) be behavior tests: then you can rip out all your code and replace it with whatever you like, even off-the-shelf software and your test suite will still run. The excuse of those kind of tests being slow or hard to setup mean only one thing: we have to work on their tooling.
- rochacon 6y ago
  I share the same opinion. Its been a long time since I read Kent Beck's book on TDD, but I don't recall him saying TDD = unit tests. When I see people using these as synonyms it feels to me they don't quite understood what TDD means and just jumped into this conclusion of "unit tests that check every method/line of code you write". Testing behavior from the user perspective should be the goal, these are the kind of tests that bring most value in the long run.
  - arkh 6y ago
    > conclusion of "unit tests that check every method/line of code you write". I think a way to smell you're too far gone on this way is when you write tests only to get code coverage. With behavior tests, code coverage should become a tool to show where your dead code is so you can remove it.
nromiun 6y ago
I am not saying that you have to test every single piece of your code, but IMO sooner or later you will be forced to do so. Weather it is through the type checker (static or gradual), unit tests or manual tests is up to you. Unit tests just make this a little easier. You can replicate unit tests every time you commit (testing for bugs) without writing them down.
vp8989 6y ago
I think the industry should re-evaluate whether unit testing ought to still be the dominant testing paradigm. With Docker and things like WireMock I can fake/mock anything downstream of me in a fast, isolated and deterministic way. Both locally and in CI. There is so much accidental complexity cruft and ceremony involved with the writing of unit tests and making things unit testable. In a lot of cases, the "coverage" is an illusion and the tests add more friction than there really ought to be.
nspattak 6y ago
I think that this is more of a marketing post than a technical post. That said, I really wish people do not promote bad practices, they are already too well established :)
bxrxdx 6y ago
So, if this article actually showed me how writing modular testable code slowed down their application, and how they fixed it, that would be really interesting.
jayd16 6y ago
What are we supposed to do with this article with no code and no benchmarks? It reads like baseless justification for bad code.
michaelcampbell 6y ago
Like in anything else when someone says "best" the only response should be "best for what?"
viseztrance 6y ago
I've seen this pattern more often than not, but as the person joining the company later and having to fix other people's mess who either left or moved into management.
hnedeotes 6y ago
> .NET - so fast you can't do dependency injection
datavirtue 6y ago
This is almost obvious for a company made up of senior, capable developers. Hopefully no one else takes this as advice because things are already bad enough without giving people license to forgo testing or design for loose coupling.
1penny42cents 6y ago
> it’s no silver bullet: your software is not going to crash and burn if you don’t write your tests first, and the presence of tests alone does not mean you won’t have maintainability issues This is not what tests are for. Tests rarely catch bugs for the first version of their code. They often catch bugs in modifications to that code, and any bugs that aren't caught can be added as a test. Tests are like checkpoints in video games. They guarantee that you don't regress past a point. Over time you build up so many checkpoints that your code is hard to break without a test failing. It's an evolutionary process.
- tcgv 6y ago
  Agree. Also, as a consequence in the long run writing unit tests / automated tests is the only way for maintaining productivity in a complex project, as every required code change to an existent component poses a great risk, and without automation it will be necessary countless hours of manual testing. Shameless plug: I while ago I wrote on the topic of how projects should be structured with the goal of automated testing in mind [1] [1] https://thomasvilhena.com/2020/04/on-the-architecture-for-unit-testing https://thomasvilhena.com/2020/04/on-the-architecture-for-un...
1penny42cents 6y ago
They noted in the post that they had a fixed number of servers and those servers had fixed performance characteristics. A lot of what they say applies specifically to that situation. The problem is that they make broader claims about the validity of best practices (design, OOP, unit testing). Those best practices apply just fine to the status quo, where you can scale out easily and maintainability is more important than performance.
scott00 6y ago
I don't understand the claim that replacing "polymorphism and dependency injection" with "static fields and service locators" reduces allocations. I would expect that getting rid of dependency injection would allow you to convert a virtual method call to a direct method call, and replace a constructor argument with a static field reference, but I don't see the connection to allocation. Anybody understand this?
socialdemocrat 6y ago
Don’t fully understand the argument here. One the one hand they claim they built a successful solution with minor problems without test driven development or a lot of unit tests. On the other hand they claim to really like tests and have it as a goal to do more of them. But this is a contradiction. If they successfully built a large site with minimal testing then why exactly do they need all these unit tests? If they where successful without test driven development then why do they need TDD? To be cool? It sure what to make of this. The argument seems inconsistent. If I had major success without best practices I would have proclaimed loudly that those practices are BS. But these guys seem to want it but ways.
- hnedeotes 6y ago
  I don't mean this as insulting, but let's think this through, with the exception of the scale SO is a forum (a QA where the OP post is a question - a topic in a forum - and the following posts are answers). It does have many details that were built throughout time (moderators, different exchanges, job posts, etc) that can significantly increase the complexity of it all when thought as a single platform, but it's nothing ground-breaking per se, even in 2008 (again, not saying that the scale and amount of exchanges and keeping all that working together non-stop is trivial or not worthy of note by itself). But you can imagine that, perhaps, the slow down in what they can do with all these platforms - most developments seem to be things in separate bolted on to the main QA idea, others take a long time to see the light of day - is that it has become quite difficult/slow to move without breaking half of it? That would explain why they want to move towards writing code in a more testable manner. I don't even need to experience their scale to know what the pains are of trying to refactor or add functionality to an existing code base that is untested and where I was the one writing the code - ALONE - it has happened to me, with my perfect code writing abilities and clear pure thought more than I would like to admit. And the same is true when we talk about mocks and etc. Yeah, if you don't think in using them from the beginning it becomes usually really problematic to implement tests and mocks after the fact. You don't need to write them but your code needs at least to know they're going to be used and written in a way that does - the problem with this approach is that you really need to know what you're doing to write it in a way that is testable without writing the tests, so my takeaway from my own endeavours is, write the tests as they'll show you exactly if something is well or not written, because when you start digging deep into 3 nested layers and building complex mocks of responses to mock a signup you know something is wrong. If you don't do it from the beginning guess what, you would have to have studied whatever you're integrating with and written that mock (even if mentally) to write that code ANYWAY, with the difference being that now all that work you did went to the bin because it only exists the execution path. What you learned from it is nowhere to be used and provide small guarantees. If you don't write them, once it's all working and working correctly, you're in a world of pain, and you're probably going to be overtaken at some point by something new that sees what you're doing and do it better (because in the end it's not rocket science) - the ones who can ignore this and can punt all the way usually have some deep pockets. > If they where successful without test driven development then why do they need TDD? They were also funded in total up till now with $153M, one year after being beta they had a $6M funding. I think these kind of posts do a disservice. I know I said to myself "you don't need that, look at X doing without it". Sometimes I look at the industry and think we pat ourselves way too much on the back when we shouldn't. As an example that isn't web based. I used to work with Adobe Suite, CS5/6 - as an aside the current experience for me is an awful thing, so awful that I went to GIMP instead - but forgetting that, for instance they had and still have a batch utility (Image Processor) to do bulk transform pipelines on collections of images through scripts recording your actions. It had and still has many small stupid things but it mostly worked well. At some point, they decided to do a better one and touted it as the replacement so they did. Result, it's way slower to operate and, guess what, it can only work on an image at a time. What? It does better compression of gifs for instance, but it can't do batch processing. There's tickets open for years asking for the batch processing. Not done. Probably won't ever be. The previous processor also didn't get the benefits of this new compressing algorithm. What? How can it be that switching the used algorithm on the old one or adding the ability to do multiple images is more than a one man month work?

195 points by kettunen 6y ago

This should be obvious. Writing well-structured, decoupled, modular code can make you lose track of performance concerns. And since no one likes to write things more than once, so there's only "one shot" at writing the application, people tend to err on the side of "well-structured", because hey, who's gonna fore you for following best practices?

deleted 6y ago
[deleted]
beny23 6y ago
I’d argue that getting performance in a coupled unstructured mess (I’m not saying that’s what SO made) is going to be more difficult.
mytailorisrich 6y ago
On the other hand, trying to optimise too early leads is wasteful and leads to headache down the line. Writing "well-structured, decoupled, modular code" is a best design practice for a good reason. It what leads to best results overall, where "best results" means code that is easier overall to test, maintain, modify/expand, and optimise as needed. Regarding optimisation, something related to keep in mind is that optimisation at system level (which is often the desired end result) is not the sum of independent local optimisations.

nsoonhui 6y ago

Duplicate of [https://news.ycombinator.com/item?id=26381640 https://news.ycombinator.com/item?id=26381640}]

t0r0nat0r 6y ago

Clearly adding further constraints can only worsen performance. I would code according to best practices except for cases where it’s possible to infer them implying tight bottlenecks. Then profile the code and if circumventing best practices results in a meaningful performance increase, do that.

sgtnoodle 6y ago

The article makes the excuse that static (singleton) interfaces are inherently simpler and less resource intensive enough for it to be worth skipping regression testing (because testing that style of code is hard.) Reading between the lines, it sounds like maybe they know they've dug themselves into a very expensive hole to dig out of. I've jumped head first into codebases that made the same excuse, and to me it seems severely limiting in terms of long term productivity. There are definitely benefits to simple, "static" singleton interfaces, and such interfaces aren't inherently incompatible with mocking and dependency injection. It does mean you likely need a layer of indirection somewhere, but if you're sweating the overhead of a single vtable lookup or explicit function pointer call on code that's internally accessing a non-trivially sized data structure, how are you even able to measure it? The problem is that a handful of foundational static interfaces never got unit tested, and all the other code calls directly into them, so nobody anywhere wrote any unit tests because they couldn't (without touching foundational code that's unsafe to modify for lack of unit tests.) The first step in the right direction is to fix the foundation. This is terrifying for the folk that have been around a while and cemented in their assumptions about risk because they've tiptoed around modifying portions of the codebase for years. Luckily, a big benefit of static singleton interfaces is that it's very easy to modify them with O(n) developer time (where n is the number of references in the codebase). So, you just have to buckle down and get your hands dirty. You're pretty much guaranteed to find at least one latent bug in any old piece of code that you unit test, and so folk start to see the merits of the test coverage and it becomes easier to prioritize refactoring more and more ancient and scary code. After you've done it to half a dozen or so disjoint pieces of code, something magical happens. Suddenly, the vast majority of the codebase becomes easily unit testable.

brown9-2 6y ago
The post also presents a false choice between what’s known as “the singleton pattern” versus just allocating a single instance of the object and passing it around where needed: > We don’t write a lot of these because our code doesn’t follow standard decoupling practices; while those principles make for easy to maintain code for a team, they add extra steps during runtime, and allocate more memory. It’s not much on any given transaction, but over thousands per second, it adds up. Things like polymorphism and dependency injection have been replaced with static fields and service locators. “dependency injection” doesn’t require constantly allocating new instances of the things your classes need.

syastrov 6y ago

I’m just thinking about how much CO2 emission was spared by optimizing performance instead of scaling horizontally. It would be nice to see the effect on the environment as part of the decision-making process for deciding whether to optimize versus scale. I think developers have an innate desire to fix performance issues as optimization produces a better understanding of the code and the problem and developers would like to solve the problem in the best possible way. And it helps to know that the problem won’t bite again later.

waf 6y ago

It sounds like the team is interested in adding more automated tests, but are blocked by static singletons, which have high performance but also high coupling, resulting in poor testability. I'm sure they've heard of test libraries like MS Fakes and Pose; I wonder if these libraries would let them maintain high performance, and only introduce the required layer of indirection during testing?

grumple 6y ago

These sound like some poor excuses to me. The idea that you can have either best practices or performance is a false dichotomy. Testing can be used to help track down performance issues. It can help you write code faster and prevent breakages. It should not slow you down. Best practices inside a codebase should not sacrifice performance. If you have to do that to conform to some structural ideal, you don’t do it. I would love to see this supposedly untestable code. Stackoverflow may contain lots of great dev knowledge, but it’s also a weird site that discourages participation and entrenches old answers. I am unsurprised that the devs themselves are a proverbial old dog that can’t learn new tricks.

alkonaut 6y ago
> The idea that you can have either best practices or performance is a false dichotomy. Testing can be used to help track down performance issues. It can help you write code faster and prevent breakages. It should not slow you down. It orks the other way too. Benchmarking can show that your bottleneck is virtcalls/boxing where you have used an interface field instead of a static instance to a concrete dependency. The argument isn't "performance vs. tests" it's really "performance vs modularization". You can always move the tests upwards until you test the whole monolith. This isn't a common tradeoff but I also wouldn't dismiss it as something that never occurs and "best practices" wrt. modularization and unit testing always enables similar performance. It doesn't.

ed_elliott_asc 6y ago

I get a bit annoyed by this sort of post, blanket statements like: "With our code base, you won’t be able to easily do test driven development or similar practices that the industry seems to love." Why does the industry seem to love testing? Does the industry love testing? Does this mean you do or don't love testing? It also feels like they wrote this blog post to show off: we write our c# code as if it was c code because we are better than all of you. The fact is that every application is different, they chose to use a language that back in 2000's wasn't designed for performance and so they paid the price of using it for performance and didn't want to pay the scale out price. Is their codebase testable, yes completely. Will adding unit tests to their code destroy their performance? no, there are always ways to make it testable and fast. Maybe in a few years we will see a "We thought testing was impossible but we made it work and also keep our codebase fast".

sgtnoodle 6y ago
Adding test coverage might indeed destroy their performance, in that they might expose where their system's bottlenecks are and realize that they've been focusing on the wrong things. When it comes to performance optimizations, folks' intuitions are almost always wrong. It's important to actually test and measure stuff like that. I think it's true in general that modern CPUs are insanely fast at executing code within a single process, and context switches are orders of magnitude slower due to their overhead. So, monolithic systems are going to be intrinsically "fast" compared to systems that pass messages through an operating system. They seemingly made a good decision for their project there, whether or not they made that decision rationally. It's possible they invested a lot in infrastructure for runtime performance profiling and monitoring and therefore know exactly what pieces of code to optimize. Reading the article, though, they're seemingly sweating the overhead of polymorphic function calls. The overhead of instrumenting the code would likely be higher than the overhead of polymorphism in a language like C#, so if their code was actually that performance sensitive, they wouldn't have it instrumented in production. If they did have it instrumented, they would likely have realized that their code wouldn't be bottlenecked by a little bit of function call indirection. Also, all the talented folk I know that stress the importance of runtime instrumentation also stress the importance of regression testing, so I suspect it would be pretty rare for a team to value one but not the other.

ctdonath 6y ago

I find anyone using the term “best practices” usually doesn’t understand them.

hergin 6y ago

I think the problem with this article and their approach will be more visible when super star developers start to leave the company. If they are all the same people from the very beginning, yes, throw lots of practices out of the window because people know the codebase very well. But once they are out, there will be a mountain of tech-debt. They even try to make it testable.

beny23 6y ago

In my opinion, the idea that you can have well-tested code OR performance is tosh. If you don’t have good testing and do it regularly how can you refactor your application for speed if you’re not going to be sure that you’re going to introduce a regression? And I think the stackoverflow team know it: > Currently, we’re trying to change this. We’re actively trying to write more tests and make our code more testable. It’s an engineering goal we aim to achieve, but the changes needed are significant. This makes it sound like a lofty goal but I think it just sounds more like. “Oh crap, we’re sitting on a mountain of tech debt” No I know you can’t always expect to arrive at a project and find the sunny uplands of TDD and continuous delivery but if you do get to “optimised for speed” code (to me that’s a synonym for tech debt that’s slowing down delivery) all you can do is slowly add what should have been done in the first place (tests).

hinkley 6y ago
Very, very few people know how to make cheap tests. I have one guy who writes fixtures that couples all of his tests. When I need to add functionality, first I have to replace his fixtures with basic stubs, just so I can add another bit of functionality. And we have two people who copy his coding style. Probably to fit in. I think, based on my own experiences and working with people learning to write tests, that 'simple' tests often feel more like the pejorative meaning of the word. They get uncomfortable, like you're debasing yourself to write such boring code. Surely I can write more interesting tests. Yes, you can write more interesting tests if your goal is for nobody to ever touch your code. Mine, mine, mine. The best tests, I know where my new code went wrong simply by reading the test output (not the test). If I refactor I should be able to maintain that coverage even if the order or structure of things gets completely switched around. Complex tests begat complex tests. If your old test is 2-3 lines of code, I just delete it and write a new one that does the same thing.
- beny23 6y ago
  Oh absolutely, keeping things simple should never be restricted to just production code. Testing code should be simple too. The idea that tests inhibits refactoring I find problematic though. Good tests enable refactoring. If the tests are too complex for anyone to change the code, the the code is too complex. Any complex system should be composed of simple components. Now this applies to monoliths as well as microservices. I agree it is easy to fall into the trap of thinking that tests are boring though as devs become more experienced there should be less of that thinking, hopefully.

choeger 6y ago

The argument against Tests sounds like a tooling problem to me. Why shouldn't a compiler be able to optimize the modular approach to the same code that a monolithic approach yields?

nitwit005 6y ago
As long as dynamically loading code at runtime is allowed, proving that there will only ever be one implementation of an interface isn't going to be possible. That makes fully optimizing away the cost of the indirection difficult.
tsimionescu 6y ago
Why is that relevant? The fact is that it doesn't (there are good reasons for it, but that is another matter). You can't work with the ideal tools, you can only work with the tools that actually exist. So, with the actual tools, this sacrifice must be made. And now do answer your actual question, the reason is that the languages we use are just not expressive enough, and the compiler doesn't have enough time. If you are calling an interface which could be either the actual code or the mock code, the actual code can't be inlined, and that alone usually discards dozens of other optimizations. You can't specify in a language something like 'assume this implementation in optimized builds', so we're stuck. There are ways around this, to be fair. In C#, you could ale all of your code generic, such that there could be two versions of your class - one with a real dependency, MyClass<RealDependency1, RealDependency2,...>, and one with mocked dependencies, MyClass<MockedDependency1, MockedDependency2,...>. Not sure if the C# compiler or JIT actually knows how to use this during optimization, but in principle it could. With C++ templates this would definitely allow optimization. With Java's extremely advanced JIT, you wouldn't actually need this code change, the compiler will be able to do it for you after a few thousand runs through the code, when it notices a single version is being called.
- choeger 6y ago
  It is relevant because they chose to abandon best practices because of a lack of tooling. A badly chosen language constraining your development methods is like a tail wagging the dog. There are languages and, more importantly, implementations out there that allow this kind of design. Just think of SML/NJ as an example for the principle. If performance is important and C# doesn't allow unit testing performant code then switch to C++ or Rust. It is staggering how many people seem to see the choice of language as an absolutely unchangeable decision. Rewriting even millions of lines of js or python code is definitely doable, even incrementally, and if your organization does not deem it worth the effort then maybe performance is not your most important concern.

alexf95 6y ago

Fully agree with the article. > "Best practices, not required practices" Is a thing mentioned from the article that I think many people forget. It also not only applies to programming or coding but rather to most things in life. Just because someone says it is best doesnt mean it is for you in your situation.

DarkWiiPlayer 6y ago
Yea, I really like that quote because it perfectly expresses how I feel about "best practices": Use them when you're unsure, but ditch them if you know something else will work better.

acd 6y ago

Somewhere down the line, developers will start to realize that single applications running in the same server memory and CPU cache will run much faster than Micro services applications that makes api calls over the network. Depending on your organization scale of course, but for many smaller shops, there is a significant performance overhead of going over the network. Ie a Single binary running Rust/Go/Java/NodeJs vs distributed Microservices applications.

scarmig 6y ago
Within the same data center, a network call doesn't need to have significant performance overhead; it can be on the order of a read from main memory. Still overhead, but for many SLOs it's irrelevant. The inherent cost of microservices is conceptual and organizational complexity, not runtime performance (though you can certainly build slow microservices, you can also build slow monoliths).
- tleb_ 6y ago
  You might get similar throughput but the latency isn't comparable.
vergessenmir 6y ago
I'd like to point out that the article does mention that their system is not single process. They have a database, caching layer and web servers at the very least which could be run on a single box. The number of API calls is a matter of degree when compared with a microservices architecture. A microservices approach does not mean that they should run on different hosts and an early design decision should be that the entire architecture can run on a single host and that the scale out is a feature of the system. It's worrying that this assumption is often made because it does lead to the maintenance and velocity issues that teams encounter when naively adopting microservices.
pbowyer 6y ago
I like RDS because I don't have to think about backups, replication etc but if I colocate the app server (web server, application code) and the database server on the same machine I see 1-2 orders of magnitude performance improvement in database-intensive apps. Replication is the same as usual; have another machine setup to replicate onto. The restriction is you can't fan out hundreds of application servers and half a dozen database servers if you need that - but I don't need to. I haven't done this with clients but I'm doing it with my own (cost-sensitive) side projects and appreciate removing the internal network latency.

kodah 6y ago

When I was an SRE-SE I despised the term "best practices". Teams would come in telling me all about the new best practice with serverless, Kubernetes, secrets storage, or whatever. It was almost always like someone took a magnificent full Lego set and snapped off a wall and was like, "Hey, try this, it was useful to us." If it's not a protocol or standard recognized by one of the IEEE, IETF, or ISO then it's a tutorial with a strong opinion. Nothing wrong about that, but just know you're often reading a strong opinion, not a standard. "Best practices" almost always come out of some business or project explaining what tested best with their circumstances. That alone tells you everything you need to know. Invest in a testing process which exposes the most critical parts of your application/infrastructure, attempt to overwhelm or exploit them, and you'll find out what your best practices need to be real quick. Those best practices may work at other companies but not nearly as well as if they invested in their own testing processes that are as or more rigorous than yours.

koirapoika 6y ago
I agree. Lately, I try to refrain from using the term 'best practices' in favor of 'good practices', 'common practices' or even 'my practices' for several reasons: - True best practices mainly refer to standards, and this list is very slowly updated, unlike the number of false ones; - It's often used for marketing purposes, which works well, but doesn't always imply quality; - Since this spreads fast, we often have things at the places where they do not fit, but squeezed into, because it's the best, so we must have it; - Forms a wrong mindset to rely upon someone else's experience instead of learning and gaining own understanding. In the end, we have tons of inapplicable or controversial information to filter, which doesn't make things better, in my opinion. Sharing is great but we should do it responsibly. It's also a bit funny to watch how the best practices suddenly become the worst. Edit: formatting
marcosdumay 6y ago
Standards are for interoperability. If you try to get best practices out of them, you are set for a world of pain. But I do agree, the hype of the day isn't a best practice either. And just the term "best practice" without any qualifier or context is already a sign of incompetence.
6y ago

harg 6y ago

One thing that stood out to me was they wouldn't need necessarily need to optimize for performance if they didn't choose a stack whose costs increase massively as you scale. In many applications things like having a testable, easy-to-read, well-structured codebase can trump performance, and it's often worth sacrificing a bit of performance if it means it's easier to refactor and build new features.

martinald 6y ago
I don't know why they went for SQL Server, .NET has had good support for other databases even back to 2009. I imagine that was a huge amount of their licensing cost. With .NET Core running so well on Linux now they would have no Windows Server licenses to worry about too. The first production version of .NET core was released in 2016. Even back in 2009 it was possible to run web apps on mono on linux too, I know a lot of shops around that era that used windows for more complex parts of the app but then used mono for 'simple' high volume parts to save $$$ on windows licensing.
- jve 6y ago
  Because that was what the main developer and founder Jeff Atwood was familiar with. And he really liked the stack. > Atwood: We are not personally going to be language agnostic, because we need to actually build the site. And in terms of people actually working on it, Joel's in an advisory role, I'm gonna be writing code, and then a friend of mine, Jarrod, I'll be working very closely with. So it's sort of like 1.5 developers, so I need to actually get things done. In order to do that, I'm gonna fall back on what I know, and what I know is essentially ASP.NET. So ASP.NET is gonna be the platform. And I actually really like ASP.NET. Episode 003 April 29, 2008, from the StackOverflow podcast. https://stackoverflow.fogbugz.com/default.asp?W56 https://stackoverflow.fogbugz.com/default.asp?W56 Analysis from 2008 about Scaling Up / Out. Electricity also is something you can't ignore. https://blog.codinghorror.com/scaling-up-vs-scaling-out-hidden-costs/ https://blog.codinghorror.com/scaling-up-vs-scaling-out-hidd... Ah, sorry, this is a discussion about SQL Server being expensive rather than .NET stack.

lifeisstillgood 6y ago

I have to admit I would far rather work on a conceptually simple codebase, that has been kept fast and light with minimal testing, than to work on a conceptually muddled codebase with 100% testing. Performance is a feature, and so is being able to reason through a codebase. Being able to fit all the pieces in your head is a feature, but not so snappy.

DarkWiiPlayer 6y ago
> Being able to fit all the pieces in your head is a feature, but not so snappy. How about "architectural sanity is a feature"? It kind of isn't though. Clients don't care about qualities of your code that they can observe. They notice if your code is fast, but don't care it it's easy or hard to reason about. If anything, I'd say it's a bit of a meta-feature, because code-quality is directly linked to many other properties that people do care about (speed, new features, quick bugfixes, etc.)
- lifeisstillgood 6y ago
  yes, but I think we live in a world of increasing software literacy - where a novel or a manifesto that is easy to read will beat out those that are not. Similar for software - the good software will include 'makes sense in the readers head'

raverbashing 6y ago

If someone says to me "something is a best practice" my question is: according to whom? Because 95% of "Best Practices" around are just somebody's opinion disguised as a rule. "Oh TDD is a best practice now" only if it's to sell more books. Yes I love unit testing, but to do it how it's the "best practice" please Do you have an objective metric that's improved by your "best practice"? SO itself is a great example. They're still on IIS+C# and it works for them. What if they didn't stop chasing the new fad or "best practice", like Kubernetes

thrower123 6y ago

To be fair, many of the ASP.NET best practices are completely insane.

arkh 6y ago

TDD does not mean "small unit tests at the method level in code". It means you code your test cases before you implement the functionalities. Those tests can (and IMO should) be behavior tests: then you can rip out all your code and replace it with whatever you like, even off-the-shelf software and your test suite will still run. The excuse of those kind of tests being slow or hard to setup mean only one thing: we have to work on their tooling.

rochacon 6y ago
I share the same opinion. Its been a long time since I read Kent Beck's book on TDD, but I don't recall him saying TDD = unit tests. When I see people using these as synonyms it feels to me they don't quite understood what TDD means and just jumped into this conclusion of "unit tests that check every method/line of code you write". Testing behavior from the user perspective should be the goal, these are the kind of tests that bring most value in the long run.
- arkh 6y ago
  > conclusion of "unit tests that check every method/line of code you write". I think a way to smell you're too far gone on this way is when you write tests only to get code coverage. With behavior tests, code coverage should become a tool to show where your dead code is so you can remove it.

nromiun 6y ago

I am not saying that you have to test every single piece of your code, but IMO sooner or later you will be forced to do so. Weather it is through the type checker (static or gradual), unit tests or manual tests is up to you. Unit tests just make this a little easier. You can replicate unit tests every time you commit (testing for bugs) without writing them down.

vp8989 6y ago

I think the industry should re-evaluate whether unit testing ought to still be the dominant testing paradigm. With Docker and things like WireMock I can fake/mock anything downstream of me in a fast, isolated and deterministic way. Both locally and in CI. There is so much accidental complexity cruft and ceremony involved with the writing of unit tests and making things unit testable. In a lot of cases, the "coverage" is an illusion and the tests add more friction than there really ought to be.

nspattak 6y ago

I think that this is more of a marketing post than a technical post. That said, I really wish people do not promote bad practices, they are already too well established :)

bxrxdx 6y ago

So, if this article actually showed me how writing modular testable code slowed down their application, and how they fixed it, that would be really interesting.

jayd16 6y ago

What are we supposed to do with this article with no code and no benchmarks? It reads like baseless justification for bad code.

michaelcampbell 6y ago

Like in anything else when someone says "best" the only response should be "best for what?"

viseztrance 6y ago

I've seen this pattern more often than not, but as the person joining the company later and having to fix other people's mess who either left or moved into management.

hnedeotes 6y ago

> .NET - so fast you can't do dependency injection

datavirtue 6y ago

This is almost obvious for a company made up of senior, capable developers. Hopefully no one else takes this as advice because things are already bad enough without giving people license to forgo testing or design for loose coupling.

1penny42cents 6y ago

> it’s no silver bullet: your software is not going to crash and burn if you don’t write your tests first, and the presence of tests alone does not mean you won’t have maintainability issues This is not what tests are for. Tests rarely catch bugs for the first version of their code. They often catch bugs in modifications to that code, and any bugs that aren't caught can be added as a test. Tests are like checkpoints in video games. They guarantee that you don't regress past a point. Over time you build up so many checkpoints that your code is hard to break without a test failing. It's an evolutionary process.

tcgv 6y ago
Agree. Also, as a consequence in the long run writing unit tests / automated tests is the only way for maintaining productivity in a complex project, as every required code change to an existent component poses a great risk, and without automation it will be necessary countless hours of manual testing. Shameless plug: I while ago I wrote on the topic of how projects should be structured with the goal of automated testing in mind [1] [1] https://thomasvilhena.com/2020/04/on-the-architecture-for-unit-testing https://thomasvilhena.com/2020/04/on-the-architecture-for-un...

1penny42cents 6y ago

They noted in the post that they had a fixed number of servers and those servers had fixed performance characteristics. A lot of what they say applies specifically to that situation. The problem is that they make broader claims about the validity of best practices (design, OOP, unit testing). Those best practices apply just fine to the status quo, where you can scale out easily and maintainability is more important than performance.

scott00 6y ago

I don't understand the claim that replacing "polymorphism and dependency injection" with "static fields and service locators" reduces allocations. I would expect that getting rid of dependency injection would allow you to convert a virtual method call to a direct method call, and replace a constructor argument with a static field reference, but I don't see the connection to allocation. Anybody understand this?

socialdemocrat 6y ago

Don’t fully understand the argument here. One the one hand they claim they built a successful solution with minor problems without test driven development or a lot of unit tests. On the other hand they claim to really like tests and have it as a goal to do more of them. But this is a contradiction. If they successfully built a large site with minimal testing then why exactly do they need all these unit tests? If they where successful without test driven development then why do they need TDD? To be cool? It sure what to make of this. The argument seems inconsistent. If I had major success without best practices I would have proclaimed loudly that those practices are BS. But these guys seem to want it but ways.

hnedeotes 6y ago
I don't mean this as insulting, but let's think this through, with the exception of the scale SO is a forum (a QA where the OP post is a question - a topic in a forum - and the following posts are answers). It does have many details that were built throughout time (moderators, different exchanges, job posts, etc) that can significantly increase the complexity of it all when thought as a single platform, but it's nothing ground-breaking per se, even in 2008 (again, not saying that the scale and amount of exchanges and keeping all that working together non-stop is trivial or not worthy of note by itself). But you can imagine that, perhaps, the slow down in what they can do with all these platforms - most developments seem to be things in separate bolted on to the main QA idea, others take a long time to see the light of day - is that it has become quite difficult/slow to move without breaking half of it? That would explain why they want to move towards writing code in a more testable manner. I don't even need to experience their scale to know what the pains are of trying to refactor or add functionality to an existing code base that is untested and where I was the one writing the code - ALONE - it has happened to me, with my perfect code writing abilities and clear pure thought more than I would like to admit. And the same is true when we talk about mocks and etc. Yeah, if you don't think in using them from the beginning it becomes usually really problematic to implement tests and mocks after the fact. You don't need to write them but your code needs at least to know they're going to be used and written in a way that does - the problem with this approach is that you really need to know what you're doing to write it in a way that is testable without writing the tests, so my takeaway from my own endeavours is, write the tests as they'll show you exactly if something is well or not written, because when you start digging deep into 3 nested layers and building complex mocks of responses to mock a signup you know something is wrong. If you don't do it from the beginning guess what, you would have to have studied whatever you're integrating with and written that mock (even if mentally) to write that code ANYWAY, with the difference being that now all that work you did went to the bin because it only exists the execution path. What you learned from it is nowhere to be used and provide small guarantees. If you don't write them, once it's all working and working correctly, you're in a world of pain, and you're probably going to be overtaken at some point by something new that sees what you're doing and do it better (because in the end it's not rocket science) - the ones who can ignore this and can punt all the way usually have some deep pockets. > If they where successful without test driven development then why do they need TDD? They were also funded in total up till now with $153M, one year after being beta they had a $6M funding. I think these kind of posts do a disservice. I know I said to myself "you don't need that, look at X doing without it". Sometimes I look at the industry and think we pat ourselves way too much on the back when we shouldn't. As an example that isn't web based. I used to work with Adobe Suite, CS5/6 - as an aside the current experience for me is an awful thing, so awful that I went to GIMP instead - but forgetting that, for instance they had and still have a batch utility (Image Processor) to do bulk transform pipelines on collections of images through scripts recording your actions. It had and still has many small stupid things but it mostly worked well. At some point, they decided to do a better one and touted it as the replacement so they did. Result, it's way slower to operate and, guess what, it can only work on an image at a time. What? It does better compression of gifs for instance, but it can't do batch processing. There's tickets open for years asking for the batch processing. Not done. Probably won't ever be. The previous processor also didn't get the benefits of this new compressing algorithm. What? How can it be that switching the used algorithm on the old one or adding the ability to do multiple images is more than a one man month work?