48 ms·

TDD from the Factorio Team

457 points by sorahn 5y ago

tobyhinloopen 5y ago
I wonder how common TDD is in game development land, especially when you’re using things like Unity or Unreal. I feel like testing your behaviors is pretty hard, and even if you unit-test your behaviors, there’s still integration tests. I only write games as a hobby and never use TDD, even if I’d like to, since the tooling is just either poorly documented or too slow, or both. Usually this ends with me being frustrated with the slow development cycle and pushes me towards more unconventional methods of developing games in Javascript using Mocha to run the tests directly in the browser.
- exdsq 5y ago
  Anecdotal but I think it's pretty rare - my friend worked as a game developer for Epic and didn't know what TDD (or SQL for that matter) actually meant.
  - tobyhinloopen 5y ago
    Given how common it is for bugs to reappear in Fortnite, I’m pretty confident their testing suite is either incomplete or not present at all
- mschuster91 5y ago
  Testing? In games? There is no such thing on a wide scale, not anymore since the cost of distributing patches essentially became a small budget line for a CDN. Modern games are notorious for using the first, most loyal customers as beta testers (hello Fallout 76...). The reason is two-fold... while you absolutely can test some parts of the engine (e.g. collision detection, networking) you can't really "test" stuff that needs a human eye to see if it's working as intended (anything that's rendered) or involves randomness (e.g. fire, fog, water, opponent spawning, loot). That means you have to hire lots of skilled (!) humans, provide them with expensive rigs, and give them time. Which is incredibly expensive.
  - gmueckl 5y ago
    Just to give you some perspective: my current employer's main product isn't called a game, but it has an engine at its core that is a game engine in all aspect except its name. And we test the sh*t out of it. We have thousands of automated and very sensitive tests on that stuff. Our test suite goes as far as testing for pixel perfect output. And this involves stuff that is "random". It took us some effort to be random in a perfectly reproduceable way, but we got there. Game QA is more involved than that, of course. Content needs to go through a signoff and QA process that involves humans (we do that, too).
  - kempbellt 5y ago
    This is definitely true for some games, and Early Access is increasingly popular, but for others, QA and testing is definitely a part of the development process. I interviewed at a game company a few years ago where one of my daily tasks would be to spend an hour just playing the game and seeing if I spot any bugs. I didn't end up taking the job so I didn't see how involved their actual code testing process was, but it was apparent that they actually cared a bit about quality control.
- deleted 5y ago
  [deleted]
- Danieru 5y ago
  Factorio is perhaps the only game I am aware of using TDD. Lots of engine teams use extensive automated testing. Only Factorio is applying it to game logic of any major game I know.
  - Glowbox 5y ago
    https://twitter.com/playartifact/status/1051964775658217473?lang=en https://twitter.com/playartifact/status/1051964775658217473?... Artifact did it too (which is no longer being worked on though).
  - mywittyname 5y ago
    Factorio seems especially well suited to TDD. The core gameplay loop involves automating away manual tasks. So I have to imagine that the test cases leverage blueprints a lot, i.e., create a blueprint for a feature, pipe resources into it via conveyor belt, test output rates on conveyor belts.
- CodeGlitch 5y ago
  I was in the industry from early 2000 to early 2010. It was only towards the end that Unit Testing was a thing. At the start we didn't even do code reviews or use a sensible source-control system. Yeah it was a painful experience, but I survived.
- mewse 5y ago
  In twenty years in game development I have never worked on a game which had real unit tests or even integration tests. I’ve seen engines which used them, but not games. The rationale was that it was just too hard, which always felt like a cop-out to me. I would dearly love to have more automated tests in the game I’m working on now, but I’ve never seen a model of it working well which I could copy, and part of me suspects that it’d be a huge investment of time to figure it out entirely on my own when I’m already vastly overworked as it is. If anybody has references to indepth case studies of making game engines more friendly toward automated tests, I’d be super interested to see whether there were lessons I could apply toward my own situation!
  - dsego 5y ago
    Not TDD but here is a talk about automated testing at Croteam Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk https://www.youtube.com/watch?v=YGIvWT-NBHk
  - tarcon 5y ago
    Interesting. I always assumed that to get the balancing feel right, a game would have to run huge parameterized test-suits to make sure win/lose results to user inputs are in balance.
    - beckingz 5y ago
      Good luck doing this for a huge modern AAA game.
      - mywittyname 5y ago
        I've always assumed they did this for games like LoL. Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours. They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.
- ashtonkem 5y ago
  I feel like game dev is probably an area where TDD has some value. My team writes distributed systems, which drastically reduces the value of a TDD approach. There’s only so far you can take the technique with a database backed api before it just becomes absurd.
- Thaxll 5y ago
  Video game client don't have tests.
- jayd16 5y ago
  So there's a few reasons tests aren't as ubiquitous in games as they are in non-game dev. You need a large QA team (relative to your team size) to test for fun anyway. The game is constantly getting tested and bugs will get logged. The marginal benefit of automated tests is less than other places because of this. Games have no specification. You have almost no idea of even the genre of game you'll end up with at the end of the dev cycle unless you're making a sequel that has to fit into a mold. Sure you can write tests as you go along and test that enemies with negative health die. The next day someone will suggest "what if they stay alive for a period of time and then explode!" The definition of correct is constantly changing. Tests ossify functionality. It makes it harder to change things because at the very least you need to also change the test. If you're just changing tests whenever you want to suit your new desires then its hard to build trust in the tests. Games don't need to be correct. They just need to be fun. This also decreases the marginal benefit of tests compared to other industries. That said, it would be natural to unit test some data structure or some well defined system. Also, once your game is done, a la Factorio, you can go back and write tests for some refactor because you know the full design specs.
  - indeedmug 5y ago
    I don't know if "correct" and "fun" are at odds with each other. There are very famous examples of games failing at the start because of bugs like Cyberpunk. There is a point where the game is too broken to enjoy. You want games where the correct behavior is the fun behavior. (However, there are counter examples like Goat Simulator.) To be fair, I don't pretend to know how CDProject developed their games. They might already have testing and the timetable was the problem.
harryf 5y ago
Came here hoping they’d turned Factorio into a tool of creating tests in other codebaes. Like literal gamification of work.
- dgb23 5y ago
  You might have looked at Flow Based Programming? It has certain characteristics that align with a game like Factorio or Oxygen Not Included etc. such as visual programming, backpressure, common interfaces, local retention etc. I can imagine this being applied to distributed/cloud computing as a way to reason about high level interactions and perhaps functional/integrated testing.
  - ashtonkem 5y ago
    I’ve done a bit in my home automation; it’s no replacement for a scripting language.
dgb23 5y ago
Two interesting takeaways: > This is the beautiful thing about having a company that isn't on the stock market. Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that. Luckily, we don't have any shareholders, and we understand the vital importance of this investment in the long run. Not only in the project, but also in our skill and knowledge, so we do better next time. This is reassuring the notion of what I think actually matters, what the real essence is of developing a product, may that be a piece of art and entertainment (like here) or a productivity tool etc. There are creators and there are consumers. We split them up by developers, designers, domain experts and so on, but what matters is that all the other participants, especially those who can exert power traditionally are not part of the essence and if not being careful and responsible, can easily add complexity and limitations that are entirely accidental and can even be harmful. This reminds me of the agile manifesto, modern UX approaches and other processes that are driven by creators, but are often and very unfortunately being bent over backwards to fit into hierarchical power structures. > TDD actually is the constant fast swithing between extending the tests and making them pass continously. So as you write tests, you write code to satisfy them basically at the same time. This allows you to instantly test what you write, and mainly use tests as specifiation of what the code should acctually do, which guides the thought process to make you think about where you are headed to, and to write code that is more structured and testable from the very beginning. The important notion here is that TDD is not about tests and correctness, but about development. It continuously checks assumptions and explores the surrounding code, state and data until a sufficient solution is found. If we squint a little we can see how closely related TDD with REPL Driven Development is. In essence it is the same thing and even has similar results, where the tests or REPL code can be left as an artifact for further, likely historical understanding. We know now that neither is sufficient for a high degree of correctness, but they are certainly useful for understanding and development.
- rpastuszak 5y ago
  > The important notion here is that TDD is not about tests and correctness, but about development. Yup, writing tests helps me sleep at night. TDD helps me manage my mental resources and iterate. (another reason is communication--we code for our colleagues first, then for the machine: https://sonnet.io/posts/code-sober-debug-drunk/ https://sonnet.io/posts/code-sober-debug-drunk/) IIRC smalltalk had a workflow where you'd debug and write your program at the same time. You'd just reach a path that has not been implemented yet, break, implement it and continue.
kevmo314 5y ago
> Which is a big improvement already, as adding and maintaining the new logic only requires you to look at one place instead of several, and it makes it generally more readable and less prone to errors. It's interesting to think about the other HN thread discussion about comments vs one-time-call function abstractions in this light: https://news.ycombinator.com/item?id=27546135 https://news.ycombinator.com/item?id=27546135 I'm a big fan of "put code in one place" too. It was the biggest factor that convinced me that JSX was a great idea compared to separating the templating logic out.
- adflux 5y ago
  Agreed, which is why I love vue components so much
achairapart 5y ago
Warning: This page almost crashed my browser (FireFox on MacOS) and put my CPU on fire.
- Metacelsus 5y ago
  I'm also using Firefox on Mac and had no issues. I'm blocking their Javascript though.
  - kllrnohj 5y ago
    There doesn't seem to be any meaningful JS on the page. A single google analytics script and a tiny[0] little toy script for doing a silly animation when you click on the #rocket element. Possibly the google analytics is doing something heavy (although it doesn't look to be when spot checking with a profiler), but there's otherwise nothing JS that runs continuously. 0: https://factorio.com/static/js/factorio.js https://factorio.com/static/js/factorio.js
- Smaug123 5y ago
  Likewise (Firefox 89.0.1 on macOS 10.14.6) I had to close the page once I'd scrolled down to the layout of the various building interfaces; ended up just reading the HTML.
- Diggsey 5y ago
  Strange. I'm using firefox on windows and didn't notice any problems.
  - Aachen 5y ago
    Firefox @ Linux, also no problems (and a crappy cpu at that). I've noticed some of their very-gif-heavy posts slowing down this laptop before, but not this post.
- DizzyDoo 5y ago
  My poor 2015 MacBook Air with FireFox went full 100% CPU on this page, I think it's the gifs. I think Factorio itself actually runs better on this laptop than that Factorio blog post does.
truncate 5y ago
> (1) no new features for a quarter or two, refactor the code, learn new methodologies etc > (2) This allows you to instantly test what you write, and mainly use tests as specification > (3) the problem comes when you break something and a lot of tests start to fail suddenly My favorites. Don't expect to give away entire quarter, but at-least sometime would definitely be nice. All three so fundamental, and often ignored. In my experience, you get these right, it makes developer life so much easier. As someone earlier mentioned in thread, TDD is kind of like REPL driven development. I think, one immediate benefit of companies focusing on good code is that engineers can aim for much more ambitious projects, and they can be more brave with the codebase. Instead we often end up with 100 over-engineered components with no well defined/enforced contracts, and a set of monolithic tests which runs the entire stack to test the most basic case.
- sidlls 5y ago
  On the other hand, with TDD we often end up with code that has been butchered in the name of "testability," and which is both less efficient and more complex than necessary.
  - fendy3002 5y ago
    Which IMO, a bad practice. Too much interface, abstraction, mocking do not reflect real process. I find it often break when integrated with services.
  - leprechaun1066 5y ago
    This usually happens when the developers in this situation are focusing (or are being forced to focus) on the tests over focusing on the solution to the actual problem in the product. TDD is just a development methodology which is a means to an end, not the goal.
    - sidlls 5y ago
      Development methodologies exist to solve product development problems, not the product problems. In that sense, an organization that adopts TDD necessarily focuses on the tests (as part of the development process), by definition. The problem is, TDD is a poor development methodology.
IMTDb 5y ago
> Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that > now there are 9 programmers Companies on the stock market don't have "9 programmers". They have a lot of teams of 9 programmers. So while it's true that it would probably completely be impossible for a stock market company to completely freeze for a quarter or two, individual teams can still do that. If the factorio team grows to tens of programmers (it probably won't and probably shouldn't), I would be very surprised if they find the need - and if they manage to - freeze all teams together for a big refactoring round. I am also unsure that it would be the right approach. That observation holds wether they go public or stay private.
- Aditya_Garg 5y ago
  Okay imagine you are a small startup backed with VC money. The same situation can still arise.
  - meesles 5y ago
    Except that as the author of the article says, they don't owe investors any explanations. VC money will demand results which you cannot just ignore for a quarter.
ramblerman 5y ago
I'd be curious to hear what kovarex thinks in 2-3 months. TDD is often sold as a fix-all solution, which is incredibly appealing to mgmt and quite fun for most programmers as a new paradigm, allowing for quick adoption. It also has its uses, especially in the enterprise space where requirements aren't often clear. But I don't know many good programmers that truly stick to the dogma after the honeymoon period. It becomes just another tool in your toolset. Uncle bob is a salesman, not a "craftsman".
- ashtonkem 5y ago
  Red/Green is a good technique for fixing bugs and extending existing functionality.
  - koonsolo 5y ago
    What is the chance of a fixed bug getting broken again? As it turned out in our analytics over 10 years: very very low. So the effort you put into writing a test for a bug, has most likely a negative return on investment. That time could be better spend somewhere else.
    - RHSeeger 5y ago
      You're ignoring various parts of the equation, though. - Writing a test for a bug is part of understanding the bug, much like rubber ducking, it helps you make sure you know exactly what causes the bug. - The better you are at writing tests, the less time it takes you to do so. - Having tests for the code means the code is testable. In general (but not always), code being testable means it's better code (more readable, less complex, etc). - Having tests for code means you have to worry less about breaking it when making changes. This allows work to proceed faster. So it's not just "did adding this test prevent this one bug from re-occurring", it's "did adding tests improve our development overall". The later is far more likely to be true than the former.
    - duncan-donuts 5y ago
      Maybe for you and your org though. This isn’t universal. I’ve worked somewhere that would use the phrase “whack-a-mole bugs” because teams would start thrashing and break each others’ shit over and over. There’s a ton of stuff we could have done — communicate (even just a tiny bit), less silos, communicate!, write better code, write tests. When your problems are much more fundamental, like teams just flat out don’t talk, writing the test is an easier investment.
happyweasel 5y ago
You can TDD as much as you want to once the initial game mechanics are in place and a gameprotoype shows enough promise to be realized until completion. Because then most of the core stuff/ideas/principles won't wildly change and won't be thrown away.. The core mechanics are in place. But would TDD help you reach that stage? I guess it is simply too much overhead. So yeah, this is TDD after the fact ;-). I still love TDD :)
- chii 5y ago
  > But would TDD help you reach that stage? if your game has a lot of interactions, and you want to make sure that your changes are not causing unintended interactions, tests like these would help a lot during development.
  - marcosdumay 5y ago
    There is always a comment with that claim on a TDD thread. Just to clarify, writing tests is not TDD. There are plenty of ways you can have a codebase full of tests, TDD is only one. But anyway, I doubt tests help at all in the prototype phase (by any procedure you want to get them). My guess is that they are incredibly harmful.
- meheleventyone 5y ago
  > You can TDD as much as you want to once the initial game mechanics are in place and a gameprotoype shows enough promise to be realized until completion. Because then most of the core stuff/ideas/principles won't wildly change and won't be thrown away. If only game development worked this way!
ashtonkem 5y ago
Honestly, I think the factorio team probably now knows more than Uncle Bob does, based on their blog posts.
Keeper93 5y ago
Kinda sad that they give this guy a platform.
- danielatc 5y ago
  I've noticed that as well and got a reply from the creator on reddit. And it's been a rather disappointing one... "Take the cancel culture mentaility and shove it up your ass." https://www.reddit.com/r/factorio/comments/o2ly6f/friday_facts_366_the_only_way_to_go_fast_is_to_go/h273tim/ https://www.reddit.com/r/factorio/comments/o2ly6f/friday_fac...
  - oauea 5y ago
    He's completely right. The behavior of this Uncle Bob person on social media (or anywhere outside of the linked videos) has absolutely nothing to do with the linked videos. Please stop derailing these discussions by trying to insert drama where there is none.
    - ashtonkem 5y ago
      Yeah, thinking that Uncle Bob’s extracurricular activities are off topic doesn’t excuse flaming your own fans about it. That will always make things worse, and lead a lot of people to reasonably believe that the real problem is that he agrees with Uncle Bob on these things. Especially when Kovarex ignored complaints about Uncle Bob’s programming advice to focus in on the culture war instead.
      - oauea 5y ago
        Not even close to what happened. Some dramaqueens came by and started whining about unrelated points about someone he referenced in his article. Understandably, he is annoyed that these people are now personally attacking both this person and Kovarex himself. American cancel culture is incredibly toxic.
  - Sr_developer 5y ago
    Well, tbh you kinda of deserved it. Although uncle Bob's methodology can be highly suspect and not worth 1% of its hype the article you linked (https://techexplained.substack.com/p/tech-bullshit-explained-uncle-bob https://techexplained.substack.com/p/tech-bullshit-explained...) was not a technical criticism , it was something which at best seems written by a tumblerina.
Aardwolf 5y ago
I used to follow friday facts until it stopped being weekly. I'm glad that every friday fact now gets posted on hacker news, that serves as my notification for new ones :) I also misread TDD as TTD (related to the trains in factorio) first
- depaya 5y ago
  The trains are my favorite part of Factorio. I would love "TTD but in Factorio"
tgtweak 5y ago
Damn I misread TTD and got excited that they were building it into factorio... Great article though, was not dissapointed.
- dvgt 5y ago
  Exactly the same thing happened to me.
swiley 5y ago
Factorio convinced me that some people still write good closed commercial games. I wish the best for the authors and hope they don't stop any time soon.
nanis 5y ago
This is a neat article. I do have comments about testing in general though. IME most developer do not understand each test has four possible outcomes: * Code is good and test passes * Code is bad and test fails These are the only two possible outcomes developers focus on: When I ask what they should do if a test that used to pass now fails, they always tell me stories about how to debug the code under test. There are two additional possibilities in test: * Code is bad yet test passes (false negative) * Code is good yet test fails (false positive) Again, IME, most people do not look at the test again once it passes for the first time. As a result, tests which are themselves code, become the largest untested part of the code base. You get these thousands and thousands of lines of untested code yet you have 100% code coverage. Some of my blog posts on testing: * Deception in tests considered harmful https://www.nu42.com/2017/02/deception-in-tests-harmful.html https://www.nu42.com/2017/02/deception-in-tests-harmful.html * Know what you are testing: The case of the test for median in Boost.Accumulators C++ Library <https://www.nu42.com/2016/12/cpp-boost-median-test.html https://www.nu42.com/2016/12/cpp-boost-median-test.html> * Who is testing the tests? https://www.nu42.com/2015/05/who-is-testing-the-tests.html https://www.nu42.com/2015/05/who-is-testing-the-tests.html * Slashing one's feet with tests, or, how to fix 2,950 test failures in one fell swoop https://www.nu42.com/2015/08/fix-2950-test-failures.html https://www.nu42.com/2015/08/fix-2950-test-failures.html
blindmute 5y ago
I'm not sure I understand why they're committing to such a long term refactor for a game which has already reached the tail end of its sales curve. As far as I know there are no internal monetization schemes in Factorio, and I really doubt further updates will boost sales anywhere near enough to justify the dev salaries.
- shepherdjerred 5y ago
  They're planning to release a paid expansion.
- colonwqbang 5y ago
  I'm also surprised. Factorio feels like a finished game. It's more polished than most games I've played. Maybe the devs haven't found a worthy new project yet?
  - naikrovek 5y ago
    They're working on an expansion for Factorio, and this refactor may have something to do with that. Maybe they just want to leave their code in good shape so they (or someone else) can come back to it at a later time and pick it up relatively quickly.
    - kllrnohj 5y ago
      Expansion packs typically depend on the base game, so improving the base engine is likely directly contributing to the work on the expansion. The expansion is likely changing things about the base game, and they'd want tests to assert both with & without the expansion are still working as intended.
- robryan 5y ago
  As he says, they don't have external shareholders that are demanding the everything they do maxmise profit.
  - 5y ago
bluGill 5y ago
> Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that. It is called restructuring and big companies do it all the time. Investors allow it, though they are rightly suspicious - sometimes it is good, but often it is change for the sake of change and not change for better.
ineedasername 5y ago
Every time a Factorio thread makes it to HN I feel like a fully recovered meth addict who suddenly has their old dealer knocking at their door: -- Dealer: "Hey? You there? I got some meth for you." Me: "Go away! I don't want any!" Dealer: "Oh now don't say that. You remember how good it is? I know you do." Me: "I can't, I can't afford it, the price is too high." Dealer: "What? Come on, it's free! You already paid for it!. Me: "I'll lose my job, I can't, just go away!" Dealer: "Your JOB? This IS your job. Open the damn door, THE FACTORY MUST GROW" Me: ::cowers in the closet chanting please leave please leave please leave:: -- Also it was never good. It was more like a mind virus. Like the sort of problem or project at work you can't stop thinking about until it's done. Only with Factorio, it's never done. Never. My best defense against it are other video games I can stop & start when needed. Or booting up my VPN connection and picking a work task from my back log until the cravings go away.
- LegitGandalf 5y ago
  I always say, never start with a zero vacation balance!
- hatsuseno 5y ago
  Factorio is just personal project for people who are too tired after work to actually do one. Like me. It scratches the itch do "build" something, even if it's only an outlet and not intrinsically productive.
AwaAwa 5y ago
While I'm ambivalent on TDD, there seems to be an attempt at cancellation brewing for his invocation of Uncle Bob.

457 points by sorahn 5y ago

I wonder how common TDD is in game development land, especially when you’re using things like Unity or Unreal. I feel like testing your behaviors is pretty hard, and even if you unit-test your behaviors, there’s still integration tests. I only write games as a hobby and never use TDD, even if I’d like to, since the tooling is just either poorly documented or too slow, or both. Usually this ends with me being frustrated with the slow development cycle and pushes me towards more unconventional methods of developing games in Javascript using Mocha to run the tests directly in the browser.

exdsq 5y ago
Anecdotal but I think it's pretty rare - my friend worked as a game developer for Epic and didn't know what TDD (or SQL for that matter) actually meant.
- tobyhinloopen 5y ago
  Given how common it is for bugs to reappear in Fortnite, I’m pretty confident their testing suite is either incomplete or not present at all
mschuster91 5y ago
Testing? In games? There is no such thing on a wide scale, not anymore since the cost of distributing patches essentially became a small budget line for a CDN. Modern games are notorious for using the first, most loyal customers as beta testers (hello Fallout 76...). The reason is two-fold... while you absolutely can test some parts of the engine (e.g. collision detection, networking) you can't really "test" stuff that needs a human eye to see if it's working as intended (anything that's rendered) or involves randomness (e.g. fire, fog, water, opponent spawning, loot). That means you have to hire lots of skilled (!) humans, provide them with expensive rigs, and give them time. Which is incredibly expensive.
- gmueckl 5y ago
  Just to give you some perspective: my current employer's main product isn't called a game, but it has an engine at its core that is a game engine in all aspect except its name. And we test the sh*t out of it. We have thousands of automated and very sensitive tests on that stuff. Our test suite goes as far as testing for pixel perfect output. And this involves stuff that is "random". It took us some effort to be random in a perfectly reproduceable way, but we got there. Game QA is more involved than that, of course. Content needs to go through a signoff and QA process that involves humans (we do that, too).
- kempbellt 5y ago
  This is definitely true for some games, and Early Access is increasingly popular, but for others, QA and testing is definitely a part of the development process. I interviewed at a game company a few years ago where one of my daily tasks would be to spend an hour just playing the game and seeing if I spot any bugs. I didn't end up taking the job so I didn't see how involved their actual code testing process was, but it was apparent that they actually cared a bit about quality control.
deleted 5y ago
[deleted]
Danieru 5y ago
Factorio is perhaps the only game I am aware of using TDD. Lots of engine teams use extensive automated testing. Only Factorio is applying it to game logic of any major game I know.
- Glowbox 5y ago
  https://twitter.com/playartifact/status/1051964775658217473?lang=en https://twitter.com/playartifact/status/1051964775658217473?... Artifact did it too (which is no longer being worked on though).
- mywittyname 5y ago
  Factorio seems especially well suited to TDD. The core gameplay loop involves automating away manual tasks. So I have to imagine that the test cases leverage blueprints a lot, i.e., create a blueprint for a feature, pipe resources into it via conveyor belt, test output rates on conveyor belts.
CodeGlitch 5y ago
I was in the industry from early 2000 to early 2010. It was only towards the end that Unit Testing was a thing. At the start we didn't even do code reviews or use a sensible source-control system. Yeah it was a painful experience, but I survived.
mewse 5y ago
In twenty years in game development I have never worked on a game which had real unit tests or even integration tests. I’ve seen engines which used them, but not games. The rationale was that it was just too hard, which always felt like a cop-out to me. I would dearly love to have more automated tests in the game I’m working on now, but I’ve never seen a model of it working well which I could copy, and part of me suspects that it’d be a huge investment of time to figure it out entirely on my own when I’m already vastly overworked as it is. If anybody has references to indepth case studies of making game engines more friendly toward automated tests, I’d be super interested to see whether there were lessons I could apply toward my own situation!
- dsego 5y ago
  Not TDD but here is a talk about automated testing at Croteam Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk https://www.youtube.com/watch?v=YGIvWT-NBHk
- tarcon 5y ago
  Interesting. I always assumed that to get the balancing feel right, a game would have to run huge parameterized test-suits to make sure win/lose results to user inputs are in balance.
  - beckingz 5y ago
    Good luck doing this for a huge modern AAA game.
    - mywittyname 5y ago
      I've always assumed they did this for games like LoL. Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours. They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.
ashtonkem 5y ago
I feel like game dev is probably an area where TDD has some value. My team writes distributed systems, which drastically reduces the value of a TDD approach. There’s only so far you can take the technique with a database backed api before it just becomes absurd.
Thaxll 5y ago
Video game client don't have tests.
jayd16 5y ago
So there's a few reasons tests aren't as ubiquitous in games as they are in non-game dev. You need a large QA team (relative to your team size) to test for fun anyway. The game is constantly getting tested and bugs will get logged. The marginal benefit of automated tests is less than other places because of this. Games have no specification. You have almost no idea of even the genre of game you'll end up with at the end of the dev cycle unless you're making a sequel that has to fit into a mold. Sure you can write tests as you go along and test that enemies with negative health die. The next day someone will suggest "what if they stay alive for a period of time and then explode!" The definition of correct is constantly changing. Tests ossify functionality. It makes it harder to change things because at the very least you need to also change the test. If you're just changing tests whenever you want to suit your new desires then its hard to build trust in the tests. Games don't need to be correct. They just need to be fun. This also decreases the marginal benefit of tests compared to other industries. That said, it would be natural to unit test some data structure or some well defined system. Also, once your game is done, a la Factorio, you can go back and write tests for some refactor because you know the full design specs.
- indeedmug 5y ago
  I don't know if "correct" and "fun" are at odds with each other. There are very famous examples of games failing at the start because of bugs like Cyberpunk. There is a point where the game is too broken to enjoy. You want games where the correct behavior is the fun behavior. (However, there are counter examples like Goat Simulator.) To be fair, I don't pretend to know how CDProject developed their games. They might already have testing and the timetable was the problem.

harryf 5y ago

Came here hoping they’d turned Factorio into a tool of creating tests in other codebaes. Like literal gamification of work.

dgb23 5y ago
You might have looked at Flow Based Programming? It has certain characteristics that align with a game like Factorio or Oxygen Not Included etc. such as visual programming, backpressure, common interfaces, local retention etc. I can imagine this being applied to distributed/cloud computing as a way to reason about high level interactions and perhaps functional/integrated testing.
- ashtonkem 5y ago
  I’ve done a bit in my home automation; it’s no replacement for a scripting language.

dgb23 5y ago

Two interesting takeaways: > This is the beautiful thing about having a company that isn't on the stock market. Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that. Luckily, we don't have any shareholders, and we understand the vital importance of this investment in the long run. Not only in the project, but also in our skill and knowledge, so we do better next time. This is reassuring the notion of what I think actually matters, what the real essence is of developing a product, may that be a piece of art and entertainment (like here) or a productivity tool etc. There are creators and there are consumers. We split them up by developers, designers, domain experts and so on, but what matters is that all the other participants, especially those who can exert power traditionally are not part of the essence and if not being careful and responsible, can easily add complexity and limitations that are entirely accidental and can even be harmful. This reminds me of the agile manifesto, modern UX approaches and other processes that are driven by creators, but are often and very unfortunately being bent over backwards to fit into hierarchical power structures. > TDD actually is the constant fast swithing between extending the tests and making them pass continously. So as you write tests, you write code to satisfy them basically at the same time. This allows you to instantly test what you write, and mainly use tests as specifiation of what the code should acctually do, which guides the thought process to make you think about where you are headed to, and to write code that is more structured and testable from the very beginning. The important notion here is that TDD is not about tests and correctness, but about development. It continuously checks assumptions and explores the surrounding code, state and data until a sufficient solution is found. If we squint a little we can see how closely related TDD with REPL Driven Development is. In essence it is the same thing and even has similar results, where the tests or REPL code can be left as an artifact for further, likely historical understanding. We know now that neither is sufficient for a high degree of correctness, but they are certainly useful for understanding and development.

rpastuszak 5y ago
> The important notion here is that TDD is not about tests and correctness, but about development. Yup, writing tests helps me sleep at night. TDD helps me manage my mental resources and iterate. (another reason is communication--we code for our colleagues first, then for the machine: https://sonnet.io/posts/code-sober-debug-drunk/ https://sonnet.io/posts/code-sober-debug-drunk/) IIRC smalltalk had a workflow where you'd debug and write your program at the same time. You'd just reach a path that has not been implemented yet, break, implement it and continue.

kevmo314 5y ago

> Which is a big improvement already, as adding and maintaining the new logic only requires you to look at one place instead of several, and it makes it generally more readable and less prone to errors. It's interesting to think about the other HN thread discussion about comments vs one-time-call function abstractions in this light: https://news.ycombinator.com/item?id=27546135 https://news.ycombinator.com/item?id=27546135 I'm a big fan of "put code in one place" too. It was the biggest factor that convinced me that JSX was a great idea compared to separating the templating logic out.

adflux 5y ago
Agreed, which is why I love vue components so much

achairapart 5y ago

Warning: This page almost crashed my browser (FireFox on MacOS) and put my CPU on fire.

Metacelsus 5y ago
I'm also using Firefox on Mac and had no issues. I'm blocking their Javascript though.
- kllrnohj 5y ago
  There doesn't seem to be any meaningful JS on the page. A single google analytics script and a tiny[0] little toy script for doing a silly animation when you click on the #rocket element. Possibly the google analytics is doing something heavy (although it doesn't look to be when spot checking with a profiler), but there's otherwise nothing JS that runs continuously. 0: https://factorio.com/static/js/factorio.js https://factorio.com/static/js/factorio.js
Smaug123 5y ago
Likewise (Firefox 89.0.1 on macOS 10.14.6) I had to close the page once I'd scrolled down to the layout of the various building interfaces; ended up just reading the HTML.
Diggsey 5y ago
Strange. I'm using firefox on windows and didn't notice any problems.
- Aachen 5y ago
  Firefox @ Linux, also no problems (and a crappy cpu at that). I've noticed some of their very-gif-heavy posts slowing down this laptop before, but not this post.
DizzyDoo 5y ago
My poor 2015 MacBook Air with FireFox went full 100% CPU on this page, I think it's the gifs. I think Factorio itself actually runs better on this laptop than that Factorio blog post does.

truncate 5y ago

> (1) no new features for a quarter or two, refactor the code, learn new methodologies etc > (2) This allows you to instantly test what you write, and mainly use tests as specification > (3) the problem comes when you break something and a lot of tests start to fail suddenly My favorites. Don't expect to give away entire quarter, but at-least sometime would definitely be nice. All three so fundamental, and often ignored. In my experience, you get these right, it makes developer life so much easier. As someone earlier mentioned in thread, TDD is kind of like REPL driven development. I think, one immediate benefit of companies focusing on good code is that engineers can aim for much more ambitious projects, and they can be more brave with the codebase. Instead we often end up with 100 over-engineered components with no well defined/enforced contracts, and a set of monolithic tests which runs the entire stack to test the most basic case.

sidlls 5y ago
On the other hand, with TDD we often end up with code that has been butchered in the name of "testability," and which is both less efficient and more complex than necessary.
- fendy3002 5y ago
  Which IMO, a bad practice. Too much interface, abstraction, mocking do not reflect real process. I find it often break when integrated with services.
- leprechaun1066 5y ago
  This usually happens when the developers in this situation are focusing (or are being forced to focus) on the tests over focusing on the solution to the actual problem in the product. TDD is just a development methodology which is a means to an end, not the goal.
  - sidlls 5y ago
    Development methodologies exist to solve product development problems, not the product problems. In that sense, an organization that adopts TDD necessarily focuses on the tests (as part of the development process), by definition. The problem is, TDD is a poor development methodology.

IMTDb 5y ago

> Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that > now there are 9 programmers Companies on the stock market don't have "9 programmers". They have a lot of teams of 9 programmers. So while it's true that it would probably completely be impossible for a stock market company to completely freeze for a quarter or two, individual teams can still do that. If the factorio team grows to tens of programmers (it probably won't and probably shouldn't), I would be very surprised if they find the need - and if they manage to - freeze all teams together for a big refactoring round. I am also unsure that it would be the right approach. That observation holds wether they go public or stay private.

Aditya_Garg 5y ago
Okay imagine you are a small startup backed with VC money. The same situation can still arise.
- meesles 5y ago
  Except that as the author of the article says, they don't owe investors any explanations. VC money will demand results which you cannot just ignore for a quarter.

ramblerman 5y ago

I'd be curious to hear what kovarex thinks in 2-3 months. TDD is often sold as a fix-all solution, which is incredibly appealing to mgmt and quite fun for most programmers as a new paradigm, allowing for quick adoption. It also has its uses, especially in the enterprise space where requirements aren't often clear. But I don't know many good programmers that truly stick to the dogma after the honeymoon period. It becomes just another tool in your toolset. Uncle bob is a salesman, not a "craftsman".

ashtonkem 5y ago
Red/Green is a good technique for fixing bugs and extending existing functionality.
- koonsolo 5y ago
  What is the chance of a fixed bug getting broken again? As it turned out in our analytics over 10 years: very very low. So the effort you put into writing a test for a bug, has most likely a negative return on investment. That time could be better spend somewhere else.
  - RHSeeger 5y ago
    You're ignoring various parts of the equation, though. - Writing a test for a bug is part of understanding the bug, much like rubber ducking, it helps you make sure you know exactly what causes the bug. - The better you are at writing tests, the less time it takes you to do so. - Having tests for the code means the code is testable. In general (but not always), code being testable means it's better code (more readable, less complex, etc). - Having tests for code means you have to worry less about breaking it when making changes. This allows work to proceed faster. So it's not just "did adding this test prevent this one bug from re-occurring", it's "did adding tests improve our development overall". The later is far more likely to be true than the former.
  - duncan-donuts 5y ago
    Maybe for you and your org though. This isn’t universal. I’ve worked somewhere that would use the phrase “whack-a-mole bugs” because teams would start thrashing and break each others’ shit over and over. There’s a ton of stuff we could have done — communicate (even just a tiny bit), less silos, communicate!, write better code, write tests. When your problems are much more fundamental, like teams just flat out don’t talk, writing the test is an easier investment.

happyweasel 5y ago

You can TDD as much as you want to once the initial game mechanics are in place and a gameprotoype shows enough promise to be realized until completion. Because then most of the core stuff/ideas/principles won't wildly change and won't be thrown away.. The core mechanics are in place. But would TDD help you reach that stage? I guess it is simply too much overhead. So yeah, this is TDD after the fact ;-). I still love TDD :)

chii 5y ago
> But would TDD help you reach that stage? if your game has a lot of interactions, and you want to make sure that your changes are not causing unintended interactions, tests like these would help a lot during development.
- marcosdumay 5y ago
  There is always a comment with that claim on a TDD thread. Just to clarify, writing tests is not TDD. There are plenty of ways you can have a codebase full of tests, TDD is only one. But anyway, I doubt tests help at all in the prototype phase (by any procedure you want to get them). My guess is that they are incredibly harmful.
meheleventyone 5y ago
> You can TDD as much as you want to once the initial game mechanics are in place and a gameprotoype shows enough promise to be realized until completion. Because then most of the core stuff/ideas/principles won't wildly change and won't be thrown away. If only game development worked this way!

ashtonkem 5y ago

Honestly, I think the factorio team probably now knows more than Uncle Bob does, based on their blog posts.

Keeper93 5y ago

Kinda sad that they give this guy a platform.

danielatc 5y ago
I've noticed that as well and got a reply from the creator on reddit. And it's been a rather disappointing one... "Take the cancel culture mentaility and shove it up your ass." https://www.reddit.com/r/factorio/comments/o2ly6f/friday_facts_366_the_only_way_to_go_fast_is_to_go/h273tim/ https://www.reddit.com/r/factorio/comments/o2ly6f/friday_fac...
- oauea 5y ago
  He's completely right. The behavior of this Uncle Bob person on social media (or anywhere outside of the linked videos) has absolutely nothing to do with the linked videos. Please stop derailing these discussions by trying to insert drama where there is none.
  - ashtonkem 5y ago
    Yeah, thinking that Uncle Bob’s extracurricular activities are off topic doesn’t excuse flaming your own fans about it. That will always make things worse, and lead a lot of people to reasonably believe that the real problem is that he agrees with Uncle Bob on these things. Especially when Kovarex ignored complaints about Uncle Bob’s programming advice to focus in on the culture war instead.
    - oauea 5y ago
      Not even close to what happened. Some dramaqueens came by and started whining about unrelated points about someone he referenced in his article. Understandably, he is annoyed that these people are now personally attacking both this person and Kovarex himself. American cancel culture is incredibly toxic.
- Sr_developer 5y ago
  Well, tbh you kinda of deserved it. Although uncle Bob's methodology can be highly suspect and not worth 1% of its hype the article you linked (https://techexplained.substack.com/p/tech-bullshit-explained-uncle-bob https://techexplained.substack.com/p/tech-bullshit-explained...) was not a technical criticism , it was something which at best seems written by a tumblerina.

Aardwolf 5y ago

I used to follow friday facts until it stopped being weekly. I'm glad that every friday fact now gets posted on hacker news, that serves as my notification for new ones :) I also misread TDD as TTD (related to the trains in factorio) first

depaya 5y ago
The trains are my favorite part of Factorio. I would love "TTD but in Factorio"

tgtweak 5y ago

Damn I misread TTD and got excited that they were building it into factorio... Great article though, was not dissapointed.

dvgt 5y ago
Exactly the same thing happened to me.

swiley 5y ago

Factorio convinced me that some people still write good closed commercial games. I wish the best for the authors and hope they don't stop any time soon.

nanis 5y ago

This is a neat article. I do have comments about testing in general though. IME most developer do not understand each test has four possible outcomes: * Code is good and test passes * Code is bad and test fails These are the only two possible outcomes developers focus on: When I ask what they should do if a test that used to pass now fails, they always tell me stories about how to debug the code under test. There are two additional possibilities in test: * Code is bad yet test passes (false negative) * Code is good yet test fails (false positive) Again, IME, most people do not look at the test again once it passes for the first time. As a result, tests which are themselves code, become the largest untested part of the code base. You get these thousands and thousands of lines of untested code yet you have 100% code coverage. Some of my blog posts on testing: * Deception in tests considered harmful https://www.nu42.com/2017/02/deception-in-tests-harmful.html https://www.nu42.com/2017/02/deception-in-tests-harmful.html * Know what you are testing: The case of the test for median in Boost.Accumulators C++ Library <https://www.nu42.com/2016/12/cpp-boost-median-test.html https://www.nu42.com/2016/12/cpp-boost-median-test.html> * Who is testing the tests? https://www.nu42.com/2015/05/who-is-testing-the-tests.html https://www.nu42.com/2015/05/who-is-testing-the-tests.html * Slashing one's feet with tests, or, how to fix 2,950 test failures in one fell swoop https://www.nu42.com/2015/08/fix-2950-test-failures.html https://www.nu42.com/2015/08/fix-2950-test-failures.html

blindmute 5y ago

I'm not sure I understand why they're committing to such a long term refactor for a game which has already reached the tail end of its sales curve. As far as I know there are no internal monetization schemes in Factorio, and I really doubt further updates will boost sales anywhere near enough to justify the dev salaries.

shepherdjerred 5y ago
They're planning to release a paid expansion.
colonwqbang 5y ago
I'm also surprised. Factorio feels like a finished game. It's more polished than most games I've played. Maybe the devs haven't found a worthy new project yet?
- naikrovek 5y ago
  They're working on an expansion for Factorio, and this refactor may have something to do with that. Maybe they just want to leave their code in good shape so they (or someone else) can come back to it at a later time and pick it up relatively quickly.
  - kllrnohj 5y ago
    Expansion packs typically depend on the base game, so improving the base engine is likely directly contributing to the work on the expansion. The expansion is likely changing things about the base game, and they'd want tests to assert both with & without the expansion are still working as intended.
robryan 5y ago
As he says, they don't have external shareholders that are demanding the everything they do maxmise profit.
- 5y ago

bluGill 5y ago

ineedasername 5y ago

Every time a Factorio thread makes it to HN I feel like a fully recovered meth addict who suddenly has their old dealer knocking at their door: -- Dealer: "Hey? You there? I got some meth for you." Me: "Go away! I don't want any!" Dealer: "Oh now don't say that. You remember how good it is? I know you do." Me: "I can't, I can't afford it, the price is too high." Dealer: "What? Come on, it's free! You already paid for it!. Me: "I'll lose my job, I can't, just go away!" Dealer: "Your JOB? This IS your job. Open the damn door, THE FACTORY MUST GROW" Me: ::cowers in the closet chanting please leave please leave please leave:: -- Also it was never good. It was more like a mind virus. Like the sort of problem or project at work you can't stop thinking about until it's done. Only with Factorio, it's never done. Never. My best defense against it are other video games I can stop & start when needed. Or booting up my VPN connection and picking a work task from my back log until the cravings go away.

LegitGandalf 5y ago
I always say, never start with a zero vacation balance!
hatsuseno 5y ago
Factorio is just personal project for people who are too tired after work to actually do one. Like me. It scratches the itch do "build" something, even if it's only an outlet and not intrinsically productive.

AwaAwa 5y ago

While I'm ambivalent on TDD, there seems to be an attempt at cancellation brewing for his invocation of Uncle Bob.