3 ms·

Interesting post! We use a monorepo at Mito [1], it’s open source if you want to check it out [2]. We solved the test flakiness issue by getting rid of all of

by narush 5y ago

Interesting post! We use a monorepo at Mito [1], it’s open source if you want to check it out [2]. We solved the test flakiness issue by getting rid of all of our frontend tests, which means we spend more time manually testing our product. We’re happy with the tradeoff because we like being forced to use our product. It’s something a velocity focused team can forget to do… at least something we can forget. Merge races aren’t really a huge issue for us. They happen sometimes, we catch them, and we fix them. Putting thought into making them easier to fix isn’t something that makes sense at our scale. That’s being said, we’re a tiny team of 3 first time founders - so the above choices make a lot more sense for in our context than at stripe :-) A reminder that you need not design your systems how the big companies do! Read their informative blog posts, and then design them for your goals and constraints! There’s a whole world of solutions to problems you don’t have when your small, and it can be very easy to feel like you need to adopt all of them. Hint: you don’t. [1] https://trymito.io https://trymito.io [2] https://GitHub.com/mito-ds/monorepo https://GitHub.com/mito-ds/monorepo

latchkey 5y ago
> We solved the test flakiness issue by getting rid of all of our frontend tests, which means we spend more time manually testing our product. This is a terrible idea. Getting rid of testing doesn't make the problem (flakiness) go away, it just moves it to another part of workflow (manually testing). So, now instead of having tests, you have more humans (or because you're a small company, instead of building features which grow the company, you spend time manually testing and fixing regressions that your end users embarrassingly find for you). Never mind that it just doesn't scale as your codebase increases. Even a basic level of (snapshot) testing react components is not difficult. I wrote a library [1] that intentionally has a ton of tests and over the years, I've been able to do new releases that upgrade to new versions of the underlying dependencies, without worry. Seriously, write tests for the frontend. [1] https://github.com/lookfirst/mui-rff https://github.com/lookfirst/mui-rff
- sanitycheck 5y ago
  I think frontend tests are the reason that every f*king app I use these days seems to crash regularly. Nobody is manually testing, all the (flawed) tests are passing, "yay, we're done, push the damn thing out".
  - volker48 5y ago
    Classic garbage in garbage out. If all the tests are bad then how could they possible help anything. That doesn't mean unit tests or other automated tests are worthless though.
- dgb23 5y ago
  You can't avoid manually testing the frontend anyways. In my experience there is a small part that benefits from automated tests but the vast majority of frontend code needs to be looked at and interacted with by a human being, preferably people who are really good testers. I work with someone who does this _really_ well, he finds stuff (very quickly) that your typical user/client just wouldn't - and all the other stuff as well. We sometimes joke about how cool it would be if we were able to encode this in automated tests somehow. A good tester knows their users and knows the tech well enough to find anything that could break that communication between them. Some of that you can encode in acceptance tests - after the fact. But the tricky things you just can't. You have to interact with it, discuss it, look at it from different angles and try to break it - play.
  - latchkey 5y ago
    This is why I use frontend error tracking tools (Rollbar, Sentry, LogRocket, take your pick) to notify me when there are edgecase errors. There are too many combinations of browsers and settings to really rely on any sort of manual frontend testing. It is definitely a case of letting your users test for you, but if you've got notification of thrown exceptions, then now you have a chance of fixing things. Getting rid of frontend unit testing is a huge mistake though. I don't blame people for doing it. Testing frontend is poorly documented and misunderstood. It took me hours of research to even figure out how to do it myself. That said, I did it. It works. I've proven it works over many years of doing frontend development. Don't use initial difficulty as an excuse to not do it.
  - ridaj 5y ago
    There's a difference between regression testing and exploratory testing. For regression testing, stuff that gets repeated every release is hard to catch by human eye, especially as releases get more and more frequent. Automating rote regression testing just to ensure that what works yesterday still works today is a great way to ensure that the manual testing is about to focus on exploratory testing only
jkaptur 5y ago
> We solved the test flakiness issue by getting rid of all of our frontend tests Getting rid of the whole swath of tests seems extreme, but I am absolutely a proponent of outright deleting flaky tests. If a test has a reputation for being flaky, if people roll their eyes and sigh when it fails, and especially if anyone ever ignores it and submits/merges anyway, then that test has negative value. "Maybe it will catch something one day" is not a strong enough argument for keeping it around forever. The person making that argument should be arguing to spend the time/effort to make the test not flaky, because tests that people don't trust are just useless baggage. I think this becomes more true as a product gets bigger and more complicated, because it's more likely that people are changing something they don't fully understand and thus have more of a need to be able to trust the tests.

by narush 5y ago

latchkey 5y ago

> We solved the test flakiness issue by getting rid of all of our frontend tests, which means we spend more time manually testing our product. This is a terrible idea. Getting rid of testing doesn't make the problem (flakiness) go away, it just moves it to another part of workflow (manually testing). So, now instead of having tests, you have more humans (or because you're a small company, instead of building features which grow the company, you spend time manually testing and fixing regressions that your end users embarrassingly find for you). Never mind that it just doesn't scale as your codebase increases. Even a basic level of (snapshot) testing react components is not difficult. I wrote a library [1] that intentionally has a ton of tests and over the years, I've been able to do new releases that upgrade to new versions of the underlying dependencies, without worry. Seriously, write tests for the frontend. [1] https://github.com/lookfirst/mui-rff https://github.com/lookfirst/mui-rff

sanitycheck 5y ago
I think frontend tests are the reason that every f*king app I use these days seems to crash regularly. Nobody is manually testing, all the (flawed) tests are passing, "yay, we're done, push the damn thing out".
- volker48 5y ago
  Classic garbage in garbage out. If all the tests are bad then how could they possible help anything. That doesn't mean unit tests or other automated tests are worthless though.
dgb23 5y ago
You can't avoid manually testing the frontend anyways. In my experience there is a small part that benefits from automated tests but the vast majority of frontend code needs to be looked at and interacted with by a human being, preferably people who are really good testers. I work with someone who does this _really_ well, he finds stuff (very quickly) that your typical user/client just wouldn't - and all the other stuff as well. We sometimes joke about how cool it would be if we were able to encode this in automated tests somehow. A good tester knows their users and knows the tech well enough to find anything that could break that communication between them. Some of that you can encode in acceptance tests - after the fact. But the tricky things you just can't. You have to interact with it, discuss it, look at it from different angles and try to break it - play.
- latchkey 5y ago
  This is why I use frontend error tracking tools (Rollbar, Sentry, LogRocket, take your pick) to notify me when there are edgecase errors. There are too many combinations of browsers and settings to really rely on any sort of manual frontend testing. It is definitely a case of letting your users test for you, but if you've got notification of thrown exceptions, then now you have a chance of fixing things. Getting rid of frontend unit testing is a huge mistake though. I don't blame people for doing it. Testing frontend is poorly documented and misunderstood. It took me hours of research to even figure out how to do it myself. That said, I did it. It works. I've proven it works over many years of doing frontend development. Don't use initial difficulty as an excuse to not do it.
- ridaj 5y ago
  There's a difference between regression testing and exploratory testing. For regression testing, stuff that gets repeated every release is hard to catch by human eye, especially as releases get more and more frequent. Automating rote regression testing just to ensure that what works yesterday still works today is a great way to ensure that the manual testing is about to focus on exploratory testing only

jkaptur 5y ago

> We solved the test flakiness issue by getting rid of all of our frontend tests Getting rid of the whole swath of tests seems extreme, but I am absolutely a proponent of outright deleting flaky tests. If a test has a reputation for being flaky, if people roll their eyes and sigh when it fails, and especially if anyone ever ignores it and submits/merges anyway, then that test has negative value. "Maybe it will catch something one day" is not a strong enough argument for keeping it around forever. The person making that argument should be arguing to spend the time/effort to make the test not flaky, because tests that people don't trust are just useless baggage. I think this becomes more true as a product gets bigger and more complicated, because it's more likely that people are changing something they don't fully understand and thus have more of a need to be able to trust the tests.