12 ms·

> tests are great in the long run, but can slow you down in the short run. This is just silly, especially given you mention freezing releases and having an ent

> tests are great in the long run, but can slow you down in the short run. This is just silly, especially given you mention freezing releases and having an entire QA team go over the product. It would be much faster to have no QA team, no freezing of releases and instead have a robust test infrastructure. It’s fine that this set of rules works for your company, you like them and are familiar with them, but that’s all this list is.

threeseed 4y ago
The context is that of an early stage startup. At that point most products are basic enough for developers to test the product themselves. And if the codebase is undergoing lots of churn and heavy refactoring then tests unquestionably will slow you down. Maybe it is better to only begin automated tests when you reach product-market-fit.
- sbdncuvh 4y ago
  Why is everyone in this thread forgetting what tests actually do? Tests force you to write better code. Code that is easier to change. Code that is more robust. You don't need 100% coverage or perfect tests to gain this benefit. A good test suite enables you to make large refractors safely and quickly. The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests. Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started. Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively. Tests speed up development velocity and improve your codebase. Just don't get stupid about it and you won't negatively impact startup performance.
  - osigurdson 4y ago
    I generally feel the highest value tests are at the lowest level (code with few dependencies) as well as end-to-end integration tests. It gets a little murkier in middle layers since alternate implementations of dependencies (aka mocks) need to be created, or one has to deal with complex setup of dependencies or in some cases re-testing the same logic that has already been tested at a lower level. I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
    - sbdncuvh 4y ago
      I hardly ever do unit testing. It's almost always integration testing with code that uses real implementations against a testing db. This is only possible because my code is decoupled in the right places to enable this sort of no mock style testing. I don't strive for 100% coverage and I hardly ever do TDD unless I'm working on something hard to solve. Tests give me confidence and results in less time repeating the feedback loop. I might have a slower "coding speed" but I have fewer regressions during refactoring. Less bugs to fix while iterating. And less back and forth with stakeholders to ship something. Overall I'm way faster with tests than without.
  - josiahpeters 4y ago
    This is the advice I give to junior developers: 1. Feel free to write tests that help you develop your thought or solution. Not all of those should be checked in 2. Ask yourself: “What test do you need to write in order to protect this code from your future self next year when you have forgotten everything about this?”
- deleted 4y ago
  [deleted]
- datavirtue 4y ago
  Omg...do not like where this conversation is going. Handing people untested software is ridiculous. At that point we might as well have cooks dip thier hands in raw chicken before making your salad or encourage machinists to dawn long flowing scarves. People who don't test don't test because they don't know how. They are hacks.
  - sokoloff 4y ago
    Nit: it’s “don” not dawn. Also hopefully interesting, the opposite of “don” is “doff”.
    - WesolyKubeczek 4y ago
      Because those are shorthands for “do on” and “do off”. Then again, don and doff are regular verbs — donned, doffed.
  - jfim 4y ago
    > People who don't test don't test because they don't know how. They are hacks. Or maybe they're just doing a different trade off. Do you unit test all of your shell scripts and jupyter notebooks? If not, why is that? Is it that some code is either obvious enough or doesn't warrant having tests for it? There's a spectrum of testing, from zero tests (which is fine for throwaway code), to testing a few key components and having end to end tests for the rest, to having 100% branch coverage. One should write enough tests that they're confident about the quality of the software that they're shipping, and no more, in my opinion.
  - deleted 4y ago
    [deleted]
- SkyPuncher 4y ago
  I work at an early stage startup. We evaluated testing vs velocity. Our conclusion was almost universally: * BE tests are easy to write, catch lots of important bugs, and generally speed up development. * FE tests are painful to write, rarely catch meaningful issues, and don't really save any effort - devs still need to click through a UI to make it's correct. * E2E tests were even more painful to write. Likely caught more than FE tests, but not enough to justify their even more insane overhead. ---- Typescript does wonders for FE test coverage. The bugs we do run into tend to be silly oversights (empty arrays, FE <> BE type mismatches, etc). Annoying, but quick fixes. Not worth a 30% overhead on every feature to save 5% occassionally.
solarmist 4y ago
I'll give you a case where this is true if you're prototyping something new and need to keep changing fundamental things like the data model. I have nearly 100% unit test coverage for a couple of the base libraries I've created, but yeah, writing tests was slowing me down significantly for the rest. Once I have something where chunks of the application aren't being scrapped every other week, I'll start adding unit tests to it too.
spyspy 4y ago
I think the author is just saying not to have draconian testing requirements for every PR like some companies do. But really, I can see it. Firstly, a lot of engineers suck are writing tests. White box unit tests with interfaces and mocking everywhere tightly couple code blocks together. Integration tests are nice but take a lot of effort to get right while also keeping them performant. When (not if, remember we're talking about an early stage company) the codebase and/or the database schema changes dramatically it will often break the tests which is just another hangup.
- trhway 4y ago
  > Integration tests are nice but take a lot of effort to get right while also keeping them performant. In my pretty long experience it is a product issue, not tests'. And that is a kind of issue which sets the tone for that product development for years ahead. >the codebase and/or the database schema changes dramatically it will often break the tests That shouldn't break integration tests [much].
BurningFrog 4y ago
A good test suite is a huge project asset! It also requires considerable skill to build and maintain, and not every team has that... ...in which case you might have a less good and useful test suite.
neeh0 4y ago
> This is just silly, I think it is silly to use this one sentence without a context and start arguing. The whole paragraph in the article makes sense, gives context and advice is reasonable.
presentation 4y ago
A reasonable middle ground is to have e2e smoke tests just so that you have the confidence that the most business critical flows are ok even as code changes. Doesn’t require that much effort but 80/20 rule style gets you big bang for your buck.
rubyist5eva 4y ago
In my experience, tests speed you up in the short-run too. I've run this experiment myself at my own company within different teams. Code that lacks tests has the most bugs, the most QA returns, the most churn, etc....we end up spending (read: WASTING) much more time and money fixing broken code that has no tests than any other code in our entire codebase.
datavirtue 4y ago
Writing tests does not slow me down. It reveals bugs immediately and generally helps me get quality software out faster. Time is gained.
danielvaughn 4y ago
In my experience, during the early days of a product, things change so rapidly that tests are extremely difficult to maintain. On the other hand, if you're not going to have tests, you need to be able to respond to, diagnose, and deploy fixes for bugs within 30 minutes.
wefarrell 4y ago
Yeah this is by no means one size fits all. The appropriate mix of QA and automated testing really depends on what kind of product you're building but I can't think of any use cases where you want no automation whatsoever. Something that's missing from this list that I find extremely useful: An unstable environment with actual users doing real things with it. They could be internal, they could be advisors, they could be customers who are also investors or are willing to be testers for a discount. That works really well if your code is designed to fail loudly (validations, assertions, etc...) and you have a culture of fixing issues immediately.
djbusby 4y ago
This is basically the "load up on tech debt early" play. If you grow you (presumably) have money to fix the debt, if you fail (at business) the code debt is zero (dollars)
- convolvatron 4y ago
  the problem with this play is that the time never really arrives. and bad architecture begets more bad architecture until you're left trying to sell a rewrite, which nobody thinks is a good idea...or sell 1-2 yrs of painstaking incremental rewrite with additional test development. the latter is the only hope, but practically no one is willing to commit to it now when I talk to people about making their product be something that development can be done on again - I try to figure out how serious they are. almost never.
zeroq 4y ago
tl;dr figure out what you want before committing all your resources. It might be just me and a terrible string of bad luck bad I've seen so many projects that were kickstarted preemptively due to company politics or just management wanting to secure additional funding. By the time (usually between 3 to 9 months) business stakeholders came to an agreement on what they actually wanted the team developed so much technical debt that it's virtually impossible to untangle from all bad decisions for the rest of the development cycle. That's why I'm a huge advocate of starting simple, with a monolithic approach, to tread the water and have enough time to figure out the true objectives of the project first.
inopinatus 4y ago
The best versions of this I've seen have QA folks embedded in the service/product team to own the release pipeline, and their job isn’t performing tests, but writing automated ones, directly pairing with developers, analysts, SREs to do so. It's the antithesis of the all-too-common scenario of QA being isolated as a separate team. The primary goal is obviously to produce useful and targeted feedback as early as possible, but more strategically it elevates the status of QA, making it a gateway job to developer, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and a reframing helps to flatten the pecking order and improve access to the talent pool. Before we all went remote work, I'd always recommend adjacent seating for quality folks, to exploit the natural proximity bias. Alas, I'm still not sure what that looks like today.
SkyPuncher 4y ago
Our team actually dropped our FE testing. After doing it for a year, we basically came to the conclusion it wasn't catching bugs we cared about while costing us loads of time. We still test on the BE. On the FE, we noticed: * Most of the bugs were in UX/UI that are very hard to capture in a test suite. * CSS/UI bugs are better handled with a good UI library. We have an _amazing_ UI dev that's built a really stable UI library for the team. * It took forever to write FE tests. * It took even longer to run FE tests. * Even well-written FE tests miss a lot of subtle issues. * We rarely had "random" bugs. UI bugs were almost always in the code we changed. Easily detect with quick click testing.
- Gigachad 4y ago
  I have looked in to front end unit tests a bunch and never seen any way to do them that makes sense. Jest and the lot just don’t seem that useful. What _does_ work is front end integration tests with cypress/selenium. You just write some basic tests that click through the flows and see you got to the end. This catches issues all the time where the api response or something has changed unexpectedly and the UI gets stuck half way on an error. It’s also super simple, it’s 10 likes of looking up elements by ID and running .click(). They hardly ever break from normal UI changes and prove that the app is mostly working as intended. Switching to TypeScript is also a big one. It’s almost like half a test system given to you for free.
  - suzzer99 4y ago
    I agree on front end unit tests. Also on the usefulness of selenium tests. The only problem is they're flaky and slow as hell. But still worth it imo.
  - abracadabra_ 4y ago
    I appreciate your position but I could not disagree with you more. React and other frameworks allow for components to be unit tested very easily, with good control over the inputs. This gives confidence in each component which can then be added to with a (very small) set of end to end tests. I would also state that "10 likes of looking up elements by ID and running .click()" is a big part of why e2e tests are flaky and hard to maintain. Good e2e tests require good engineering.
kqr 4y ago
Verfying functionality is not even the primary way tests speed things up, in my experience. What really speeds development up is high quality code, and code written to be testable is usually of higher quality than code that is not. In a similar manner, putting legacy code under test tends to remove a bunch of bugs; not because the tests find them, but because when I refactor the code to be testable, the bugs in the code become much more obvious.

threeseed 4y ago

The context is that of an early stage startup. At that point most products are basic enough for developers to test the product themselves. And if the codebase is undergoing lots of churn and heavy refactoring then tests unquestionably will slow you down. Maybe it is better to only begin automated tests when you reach product-market-fit.

sbdncuvh 4y ago
Why is everyone in this thread forgetting what tests actually do? Tests force you to write better code. Code that is easier to change. Code that is more robust. You don't need 100% coverage or perfect tests to gain this benefit. A good test suite enables you to make large refractors safely and quickly. The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests. Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started. Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively. Tests speed up development velocity and improve your codebase. Just don't get stupid about it and you won't negatively impact startup performance.
- osigurdson 4y ago
  I generally feel the highest value tests are at the lowest level (code with few dependencies) as well as end-to-end integration tests. It gets a little murkier in middle layers since alternate implementations of dependencies (aka mocks) need to be created, or one has to deal with complex setup of dependencies or in some cases re-testing the same logic that has already been tested at a lower level. I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
  - sbdncuvh 4y ago
    I hardly ever do unit testing. It's almost always integration testing with code that uses real implementations against a testing db. This is only possible because my code is decoupled in the right places to enable this sort of no mock style testing. I don't strive for 100% coverage and I hardly ever do TDD unless I'm working on something hard to solve. Tests give me confidence and results in less time repeating the feedback loop. I might have a slower "coding speed" but I have fewer regressions during refactoring. Less bugs to fix while iterating. And less back and forth with stakeholders to ship something. Overall I'm way faster with tests than without.
- josiahpeters 4y ago
  This is the advice I give to junior developers: 1. Feel free to write tests that help you develop your thought or solution. Not all of those should be checked in 2. Ask yourself: “What test do you need to write in order to protect this code from your future self next year when you have forgotten everything about this?”
deleted 4y ago
[deleted]
datavirtue 4y ago
Omg...do not like where this conversation is going. Handing people untested software is ridiculous. At that point we might as well have cooks dip thier hands in raw chicken before making your salad or encourage machinists to dawn long flowing scarves. People who don't test don't test because they don't know how. They are hacks.
- sokoloff 4y ago
  Nit: it’s “don” not dawn. Also hopefully interesting, the opposite of “don” is “doff”.
  - WesolyKubeczek 4y ago
    Because those are shorthands for “do on” and “do off”. Then again, don and doff are regular verbs — donned, doffed.
- jfim 4y ago
  > People who don't test don't test because they don't know how. They are hacks. Or maybe they're just doing a different trade off. Do you unit test all of your shell scripts and jupyter notebooks? If not, why is that? Is it that some code is either obvious enough or doesn't warrant having tests for it? There's a spectrum of testing, from zero tests (which is fine for throwaway code), to testing a few key components and having end to end tests for the rest, to having 100% branch coverage. One should write enough tests that they're confident about the quality of the software that they're shipping, and no more, in my opinion.
- deleted 4y ago
  [deleted]
SkyPuncher 4y ago
I work at an early stage startup. We evaluated testing vs velocity. Our conclusion was almost universally: * BE tests are easy to write, catch lots of important bugs, and generally speed up development. * FE tests are painful to write, rarely catch meaningful issues, and don't really save any effort - devs still need to click through a UI to make it's correct. * E2E tests were even more painful to write. Likely caught more than FE tests, but not enough to justify their even more insane overhead. ---- Typescript does wonders for FE test coverage. The bugs we do run into tend to be silly oversights (empty arrays, FE <> BE type mismatches, etc). Annoying, but quick fixes. Not worth a 30% overhead on every feature to save 5% occassionally.

solarmist 4y ago

I'll give you a case where this is true if you're prototyping something new and need to keep changing fundamental things like the data model. I have nearly 100% unit test coverage for a couple of the base libraries I've created, but yeah, writing tests was slowing me down significantly for the rest. Once I have something where chunks of the application aren't being scrapped every other week, I'll start adding unit tests to it too.

spyspy 4y ago

I think the author is just saying not to have draconian testing requirements for every PR like some companies do. But really, I can see it. Firstly, a lot of engineers suck are writing tests. White box unit tests with interfaces and mocking everywhere tightly couple code blocks together. Integration tests are nice but take a lot of effort to get right while also keeping them performant. When (not if, remember we're talking about an early stage company) the codebase and/or the database schema changes dramatically it will often break the tests which is just another hangup.

trhway 4y ago
> Integration tests are nice but take a lot of effort to get right while also keeping them performant. In my pretty long experience it is a product issue, not tests'. And that is a kind of issue which sets the tone for that product development for years ahead. >the codebase and/or the database schema changes dramatically it will often break the tests That shouldn't break integration tests [much].

BurningFrog 4y ago

A good test suite is a huge project asset! It also requires considerable skill to build and maintain, and not every team has that... ...in which case you might have a less good and useful test suite.

neeh0 4y ago

> This is just silly, I think it is silly to use this one sentence without a context and start arguing. The whole paragraph in the article makes sense, gives context and advice is reasonable.

presentation 4y ago

A reasonable middle ground is to have e2e smoke tests just so that you have the confidence that the most business critical flows are ok even as code changes. Doesn’t require that much effort but 80/20 rule style gets you big bang for your buck.

rubyist5eva 4y ago

In my experience, tests speed you up in the short-run too. I've run this experiment myself at my own company within different teams. Code that lacks tests has the most bugs, the most QA returns, the most churn, etc....we end up spending (read: WASTING) much more time and money fixing broken code that has no tests than any other code in our entire codebase.

datavirtue 4y ago

Writing tests does not slow me down. It reveals bugs immediately and generally helps me get quality software out faster. Time is gained.

danielvaughn 4y ago

In my experience, during the early days of a product, things change so rapidly that tests are extremely difficult to maintain. On the other hand, if you're not going to have tests, you need to be able to respond to, diagnose, and deploy fixes for bugs within 30 minutes.

wefarrell 4y ago

Yeah this is by no means one size fits all. The appropriate mix of QA and automated testing really depends on what kind of product you're building but I can't think of any use cases where you want no automation whatsoever. Something that's missing from this list that I find extremely useful: An unstable environment with actual users doing real things with it. They could be internal, they could be advisors, they could be customers who are also investors or are willing to be testers for a discount. That works really well if your code is designed to fail loudly (validations, assertions, etc...) and you have a culture of fixing issues immediately.

djbusby 4y ago

This is basically the "load up on tech debt early" play. If you grow you (presumably) have money to fix the debt, if you fail (at business) the code debt is zero (dollars)

convolvatron 4y ago
the problem with this play is that the time never really arrives. and bad architecture begets more bad architecture until you're left trying to sell a rewrite, which nobody thinks is a good idea...or sell 1-2 yrs of painstaking incremental rewrite with additional test development. the latter is the only hope, but practically no one is willing to commit to it now when I talk to people about making their product be something that development can be done on again - I try to figure out how serious they are. almost never.

zeroq 4y ago

tl;dr figure out what you want before committing all your resources. It might be just me and a terrible string of bad luck bad I've seen so many projects that were kickstarted preemptively due to company politics or just management wanting to secure additional funding. By the time (usually between 3 to 9 months) business stakeholders came to an agreement on what they actually wanted the team developed so much technical debt that it's virtually impossible to untangle from all bad decisions for the rest of the development cycle. That's why I'm a huge advocate of starting simple, with a monolithic approach, to tread the water and have enough time to figure out the true objectives of the project first.

inopinatus 4y ago

The best versions of this I've seen have QA folks embedded in the service/product team to own the release pipeline, and their job isn’t performing tests, but writing automated ones, directly pairing with developers, analysts, SREs to do so. It's the antithesis of the all-too-common scenario of QA being isolated as a separate team. The primary goal is obviously to produce useful and targeted feedback as early as possible, but more strategically it elevates the status of QA, making it a gateway job to developer, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and a reframing helps to flatten the pecking order and improve access to the talent pool. Before we all went remote work, I'd always recommend adjacent seating for quality folks, to exploit the natural proximity bias. Alas, I'm still not sure what that looks like today.

SkyPuncher 4y ago

Our team actually dropped our FE testing. After doing it for a year, we basically came to the conclusion it wasn't catching bugs we cared about while costing us loads of time. We still test on the BE. On the FE, we noticed: * Most of the bugs were in UX/UI that are very hard to capture in a test suite. * CSS/UI bugs are better handled with a good UI library. We have an _amazing_ UI dev that's built a really stable UI library for the team. * It took forever to write FE tests. * It took even longer to run FE tests. * Even well-written FE tests miss a lot of subtle issues. * We rarely had "random" bugs. UI bugs were almost always in the code we changed. Easily detect with quick click testing.

Gigachad 4y ago
I have looked in to front end unit tests a bunch and never seen any way to do them that makes sense. Jest and the lot just don’t seem that useful. What _does_ work is front end integration tests with cypress/selenium. You just write some basic tests that click through the flows and see you got to the end. This catches issues all the time where the api response or something has changed unexpectedly and the UI gets stuck half way on an error. It’s also super simple, it’s 10 likes of looking up elements by ID and running .click(). They hardly ever break from normal UI changes and prove that the app is mostly working as intended. Switching to TypeScript is also a big one. It’s almost like half a test system given to you for free.
- suzzer99 4y ago
  I agree on front end unit tests. Also on the usefulness of selenium tests. The only problem is they're flaky and slow as hell. But still worth it imo.
- abracadabra_ 4y ago
  I appreciate your position but I could not disagree with you more. React and other frameworks allow for components to be unit tested very easily, with good control over the inputs. This gives confidence in each component which can then be added to with a (very small) set of end to end tests. I would also state that "10 likes of looking up elements by ID and running .click()" is a big part of why e2e tests are flaky and hard to maintain. Good e2e tests require good engineering.

kqr 4y ago

Verfying functionality is not even the primary way tests speed things up, in my experience. What really speeds development up is high quality code, and code written to be testable is usually of higher quality than code that is not. In a similar manner, putting legacy code under test tends to remove a bunch of bugs; not because the tests find them, but because when I refactor the code to be testable, the bugs in the code become much more obvious.