3 ms·

This approach works well for apps of a certain size and lifespan. Too many full-stack tests can easily lead to non-parallelized build times of an hour or so ma

This approach works well for apps of a certain size and lifespan. Too many full-stack tests can easily lead to non-parallelized build times of an hour or so magnitude. Apps that end up in this state are much harder to change even with parallelized builds. You may be waiting 20 minutes for feedback to know with confidence if a large refactor (or even small one) broke something. Conversely, apps that start off with tons of unit tests and fully TDD may never make it to market. There's lots of perfect software making nobody any money, and tons of low-quality software making lots. As with anything, it's a balance. Yes start off prototyping, if it's just you. Only do capybara specs. But be really sure that you have a good strategy for scaling up development and the test suite. You have to have people really in tune with the idea of not testing, because they know when to test/not test. And not just building a mentality of we don't do much (isolated) testing here. Likelihood is they are not testing because they don't know how to, or don't see the benefits or are under too much pressure. It's a dangerous line to walk if you are trying to get away with minimum testing and trusting everyone else also understands when you need to and how you need to start having more test coverage. Plus retro-fitting unit tests is normally a nightmare, because the devs never were forced to write uncoupled software.

pmarreck 11y ago
The original article is quite... unwise, I'll say carefully, and possibly, so are parts of your response, and I will argue why. The problem is that everyone is thinking about tests wrong. Tests should be considered an integral part of coding. Tests are a scientific experiment that your theory about the world (your code) is in fact correct in the real world (your test of it). If you have no tests, you are basically running wild with theories, and will likely stumble at any moment (introduce a bug). Tests also cause you (as a natural side-effect) to write better code, because easily-unit-testable code IS better (more modular, more focused, more maintainable, fewer-side-effects, fewer-dependencies) code. Do we let scientists theorize wildly, without showing their data? What's the programming form of "evidence"? Your test suite passing. The next problem is this perception that testing from the get-go "slows product release time." I say bullshit. Testing is a labor-saving device, by which I mean the few extra minutes it takes is more than made up for by a lack of hours of debugging later on. Note that I emphasize UNIT tests here. I am not saying integration/full-stack tests are unnecessary- but really, the ratio of unit tests to integration tests in an ideal test suite is probably something like 95%/5%, not the typical 40%/60% we likely see today. And if you start a new Rails codebase TODAY, the test suite should be set up to run in parallel, right off the bat. NOT because it will keep your test suite running in 1/8 the time it normally would (which it may), but because it will uncover nasty concurrency bugs fairly immediately. From the OP: > One day, I was writing a controller spec to make sure that calling the “index” method with a “get” request would return a 200 status code when I realized how absurd it was. It is not absurd. There are things that will break your index, and they will throw a fail here. What IS absurd is that such a test, coupled with likely hundreds of other Rails model tests, are hitting (read: testing and retesting and unnecessarily re-retesting) the ORM, the database connection and the database, when the database should be mocked or stubbed out in most tests (although TBH, you may discover fewer concurrency bugs if you do... so this depends) > Only do capybara specs. First of all, Capybara is terrible (read: slow and single-threaded) and only necessary if you have a lot of frontend JS code that you are not testing in some other frontend fashion. Asserting on HTTP responses (coupled with testing functional-style JS code via something like Node.js) is FAR better (and faster). Second of all, if you ONLY do integration tests at first, then 1) your code will be crap, because of my very first argument above, 2) you are literally betting against the future of your work. In other words, you are incentivizing abandoning your work at some point in the future, when it will be good that you didn't bother with too much testing upfront. This is the only situation where you will not regret not unit-testing from the get-go. Do you really want to make that bet? > Plus retro-fitting unit tests is normally a nightmare, because the devs never were forced to write uncoupled software. EXACTLY. So do it upfront, or GTFO.

pmarreck 11y ago

The original article is quite... unwise, I'll say carefully, and possibly, so are parts of your response, and I will argue why. The problem is that everyone is thinking about tests wrong. Tests should be considered an integral part of coding. Tests are a scientific experiment that your theory about the world (your code) is in fact correct in the real world (your test of it). If you have no tests, you are basically running wild with theories, and will likely stumble at any moment (introduce a bug). Tests also cause you (as a natural side-effect) to write better code, because easily-unit-testable code IS better (more modular, more focused, more maintainable, fewer-side-effects, fewer-dependencies) code. Do we let scientists theorize wildly, without showing their data? What's the programming form of "evidence"? Your test suite passing. The next problem is this perception that testing from the get-go "slows product release time." I say bullshit. Testing is a labor-saving device, by which I mean the few extra minutes it takes is more than made up for by a lack of hours of debugging later on. Note that I emphasize UNIT tests here. I am not saying integration/full-stack tests are unnecessary- but really, the ratio of unit tests to integration tests in an ideal test suite is probably something like 95%/5%, not the typical 40%/60% we likely see today. And if you start a new Rails codebase TODAY, the test suite should be set up to run in parallel, right off the bat. NOT because it will keep your test suite running in 1/8 the time it normally would (which it may), but because it will uncover nasty concurrency bugs fairly immediately. From the OP: > One day, I was writing a controller spec to make sure that calling the “index” method with a “get” request would return a 200 status code when I realized how absurd it was. It is not absurd. There are things that will break your index, and they will throw a fail here. What IS absurd is that such a test, coupled with likely hundreds of other Rails model tests, are hitting (read: testing and retesting and unnecessarily re-retesting) the ORM, the database connection and the database, when the database should be mocked or stubbed out in most tests (although TBH, you may discover fewer concurrency bugs if you do... so this depends) > Only do capybara specs. First of all, Capybara is terrible (read: slow and single-threaded) and only necessary if you have a lot of frontend JS code that you are not testing in some other frontend fashion. Asserting on HTTP responses (coupled with testing functional-style JS code via something like Node.js) is FAR better (and faster). Second of all, if you ONLY do integration tests at first, then 1) your code will be crap, because of my very first argument above, 2) you are literally betting against the future of your work. In other words, you are incentivizing abandoning your work at some point in the future, when it will be good that you didn't bother with too much testing upfront. This is the only situation where you will not regret not unit-testing from the get-go. Do you really want to make that bet? > Plus retro-fitting unit tests is normally a nightmare, because the devs never were forced to write uncoupled software. EXACTLY. So do it upfront, or GTFO.