6 ms·

Why we shift testing left: A software dev cycle that doesn’t scale

31 points by serverlessmom 2y ago

Kinrany 2y ago
What does this have to do with microservices? The article is a mess.
- bluGill 2y ago
  It is very easy with microservices to create something that nobody can figure out how to start everything needed for a full system test on their development machine. This is one of the downsides of microservices and if you use them you should be mitigating this somehow. (I don't work in a microservices environment and I can think of a dozen options - I'm sure someone who does can think of more as well as tell me why some of my ideas don't work)
  - serverlessmom 2y ago
    ^^ this right here, I want to quote this in my next blog post :)
- geodel 2y ago
  Huh, it has lot to do with Micro service. Now we have so many moving parts in form of micro services that dev, qa, UAT testing does not mean much because each team is different timeline for release and final solution will have same version all across is not guaranteed. I see this issue all the time.
dpflan 2y ago
I'm interested in others' experiences with "shifting left": if you are in the process of, are currently doing, are thinking about whether you should, or have and decided to "shift right" again, do you mind sharing here?
- wetpaws 2y ago
  [dead]
dpflan 2y ago
This is part 2 of this article: https://thenewstack.io/why-shift-testing-left-part-2-qa-does-more-after-devs-run-tests/ https://thenewstack.io/why-shift-testing-left-part-2-qa-does...
taneq 2y ago
I'm confused by this idea that 'testing' is a single thing that you can 'shift'. Testing is part of each step of envisaging, specifying, designing, implementing, shipping, supporting, updating, maintaining (and I'm sure I've missed a few) a product.
- yuliyp 2y ago
  obviously there's some amount of validation that happens at each step. The question is just how much you do when, and how that affects when issues get discovered.
MeetingsBrowser 2y ago
> There’s a cynical read of “shift left” that is “make people do more work but pay them the same.” I haven't heard of this. I thought shifting left was the new hotness. I was under the impression that finding and fix mistakes as soon as possible is considered best practice. > Shifting Testing Left Is a Return of an Older, Better System I don't really understand. I thought the "old way" was to wait until everything was done to check if it works, and the "new way" was to add tests for individual components as you go, making sure nothing breaks as new features are added.
- taneq 2y ago
  And just to demonstrate the universality of the human condition... just yesterday I was having a rant about (a) everything being always underbudgeted because the cheapest quote usually wins, so everyone underquotes hoping the client changes their mind and the resulting variation(s) swing the needle into profitability, and (b) that there's also a tendency to create artificial urgency (unrealistic deadlines etc.) in order to try and extract greater productivity from your workers. Hanlon's Razor would tend to discredit (b) but I've seen it in action too many times to fully accept this...
- NikolaNovak 2y ago
  >>I thought the "old way" was to wait until everything was done to check if it works, and the "new way" was to add tests for individual components as you go, making sure nothing breaks as new features are added. Things tend to be more cyclical. 30 years ago, we did unit tests where we continually test individual components, then system tests were you test if your system works as a whole, then integration tests where you check how your system works with others. We had dedicated environments for these. And this was in the big-corporation, bog-standard, boring, commercial-off-the-shelf standard enterprise resource planning software. Yes, we had waterfall / predictive project management model, but I'm not sure where this notion comes in that before right now, this exciting brilliant moment in time, nobody in IT knew at all what they were doing and built entire bridges for years and waited to see if they'd collapse when the first car came :-) Grumble grumble get off my lawn young kids etc etc :->
  - 2y ago
dboreham 2y ago
I'm not sure how this is different than TDD, but at the end it hits on a key point, something I've called "ROAC" (Runs On Any Computer): "there was often a way to run the whole system, or a facsimile, on your laptop with minimal friction" Any time the code you're developing can't be run without some special magic computer (that you don't have right now), special magic database (that isn't available right now), special magic workload (production load, yeah...), I predict the outcome is going to be crappy one way or another.
- dpflan 2y ago
  I went on read more articles that were linked. Another idea was having a cluster that is very close to production to which teams can deploy PRs that need to be tested (if they are related). - https://thenewstack.io/improve-developer-velocity-by-decentralizing-testing/ https://thenewstack.io/improve-developer-velocity-by-decentr...
- oldpersonintx 2y ago
  ROAC is a pipe dream (I used to believe in it also) most large cloud/distributed systems cannot be realistically reproduced in your compose/k3s/whatever environment a large cloud/distributed system must be tested on something similar in scale at best, ROAC lets you do some development remotely/offline, but is not a reliable model of reality
  - vundercind 2y ago
    Anything interesting and server-based usually depends on an entire complex data pipeline (one system for this if you’re lucky) and a dozen or more external systems with no great way to mock them that won’t itself be out-of-sync with reality often, anyway.
    - hu3 2y ago
      I once heard Facebook uses production database to develop and test new feature because anything else was unfeasable. Even a smaller copy of it. I'd love to have a confirmation from someone at Meta. edit: found an amazing discussion about their release process here: https://news.ycombinator.com/item?id=30899362 https://news.ycombinator.com/item?id=30899362
kelsey98765431 2y ago
The author poses a very real and valid question but posits no solution and frames this as a binary. The question is when to test. There is benefit and tradeoff when you decide when and how to test your application. Sometimes you simply cannot test beforehand, such as when you are integrating with a live system with a cost of replicating the environment very very high due to this being a managed solution contract with a large cough IBM cough vendor. In these cases often the question is when to mock test. Then when to test. But this is still just how and when. With a solo developer, it makes plenty of sense to write the feature then test it. Writing it may be the way to actually discover the behavior you want. With a team this can still be the case. In a larger team you can pass this to QA engineers to take these rapid prototypes and find the issues with them for a different type of developer to fix bugs in. The reverse is the same. You can discover your feature behavior by experimenting with tests until the desired behavior is asserted to be occuring through things like type validation and object patching for instrumentation, or whatever profiling or debugging tools and test suites you like. The hard answer is there is no one answer. Just like iterative development urges us to do our best to keep up by iterating on our features, we should iterate on our tests. Do not worship coverage, use it as a tool. Do not fear testing, it's just code afterall so you can actually make a full application that is just a series of tests (experiments) that run and if everything has passed, well you just wrote the same thing as a bunch of conditional checks in your library code. Do what works for you, your team, the project you are on and where you are at. If youre struggling add some more tests. If you have so many tests failing you dont know what to do, write some more code and string your tests together with new functions. It gives a nice way to say ok, this is the next step of my iteration. Some like to start by testing some like to finish. I do both. Whatever you do, put love into it and if thats super sturdy prod code that has no tests and will never fail because you are the safest data handler in the world with the most flexible business logic in the world, go for it - but just remember if you someday later forget how things work and dont want to read documentation or experiment with the live system - you can always write tests to remind you how it works. Tests are a form of documentation. They may not prove what you think they prove, but they document a fact about your code that is objective as determined by the code itself. This is a wonderful power, use it however you wish.
- serverlessmom 2y ago
moomin 2y ago
I feel like this is a case of one of those "movements" that is just "Yes, under certain circumstances this might be a good idea; and under others, it won't be". Consider that tests are literally executable specifications of how the system or component should behave (a formulation I associate with Dan North, but I'm sure the idea is far from original). Now the question becomes "should you write a spec every time before you start implementation"? It seems obvious that the answer to that rather depends on how well you actually understand the problem space you're trying to solve. And if all the work you're doing is well understood before you start, I suggest finding somewhere your manager you with the hard stuff.
- eyelidlessness 2y ago
  > It seems obvious that the answer to that rather depends on how well you actually understand the problem space you're trying to solve. An advocate for TDD (as in “red > green > refactor”) might say that writing the spec upfront is an excellent way to understand the problem space, or at least to help achieve that understanding. At least when I’ve been fortunate enough to work on projects where such a TDD approach is possible, I’ve found that to be the case. I’ve also often found it harder to gain that understanding when testing lags development.
ssfrr 2y ago
This is the first time I’ve heard the term “shifting left”. Maybe I missed it, but is this the same as TDD?
- Jgrubb 2y ago
  "Shift left" is a relatively hot, relatively new catchall bizterm that typically makes one visualize pushing a workload "upstream" from its current place in the flow. In this context, I think TDD is close to what it means.
  - drewcoo 2y ago
    > "Shift left" is a relatively hot, relatively new catchall bizterm that typically makes one visualize pushing a workload "upstream" from its current place in the flow. I've been shifting left (and using those words) for over two decades. Nothing new here. > In this context, I think TDD is close to what it means. Nope. It's only close in that both have to do with testing. TDD is you writing unit tests. Shifting left is about parallelizing someone else's work in the rest of the test pyramid. If QA can write their integration/system/contract/other tests as you're writing your code, you can have working tests for new features ideally as soon as the feature is runnable/testable instead of QA waiting to "see what the dev code does" to write against it.
    - iainmerrick 2y ago
      Thank you, that helps clear things up. The original article takes a very long time to get to its points. Like the first commenter in this thread, I’d never heard the term “shifting left” before. That’s surprising for a buzzword you say has been around for decades and is hot right now, but I guess there’s a first time for everything!
jf22 2y ago
>This process then becomes more of a waterfall model where dev and QA are distinct, sequential phases. I think we overuse the term waterfall. Having a QA phase as part of a development cycle doesn't not make something "waterfall.:
- pydry 2y ago
  Not if it exists adjacent to it. It does if it habitually gates releases though. If you can't risk not having QA gating your releases then your process is probably more waterfall than you might have hoped.
  - jf22 2y ago
    See this is what I mean about the overuse. Release gating doesn't make something waterfall.
    - pydry 2y ago
      It kind of does. If you need to run your build through QA, product, upper management and regulatory approval then sorry, no matter how many standups you have - you're waterfall. If you need an approval on a pull request and some automated tests to pass to give you enough confidence to deploy then that part of your process is agile at least (even if, say, the way you get requirements is not).
      - jf22 2y ago
        I don't know what regulatory approval or standups have to do with waterfall, and I'm not sure what automated tests have to do with whether or not you are agile.
  - bluGill 2y ago
    Despite going all in on agile from day one of a new greenfield project, very high test coverage from TDDing everything, and a large suite of automated integration tests - we still have a month of QA gating the release process as bugs constantly sneak past all that process in a complex system. QA and automated tests look for different classes of problems.

31 points by serverlessmom 2y ago

Kinrany 2y ago

What does this have to do with microservices? The article is a mess.

bluGill 2y ago
It is very easy with microservices to create something that nobody can figure out how to start everything needed for a full system test on their development machine. This is one of the downsides of microservices and if you use them you should be mitigating this somehow. (I don't work in a microservices environment and I can think of a dozen options - I'm sure someone who does can think of more as well as tell me why some of my ideas don't work)
- serverlessmom 2y ago
  ^^ this right here, I want to quote this in my next blog post :)
geodel 2y ago
Huh, it has lot to do with Micro service. Now we have so many moving parts in form of micro services that dev, qa, UAT testing does not mean much because each team is different timeline for release and final solution will have same version all across is not guaranteed. I see this issue all the time.

dpflan 2y ago

I'm interested in others' experiences with "shifting left": if you are in the process of, are currently doing, are thinking about whether you should, or have and decided to "shift right" again, do you mind sharing here?

wetpaws 2y ago
[dead]

dpflan 2y ago

This is part 2 of this article: https://thenewstack.io/why-shift-testing-left-part-2-qa-does-more-after-devs-run-tests/ https://thenewstack.io/why-shift-testing-left-part-2-qa-does...

taneq 2y ago

I'm confused by this idea that 'testing' is a single thing that you can 'shift'. Testing is part of each step of envisaging, specifying, designing, implementing, shipping, supporting, updating, maintaining (and I'm sure I've missed a few) a product.

yuliyp 2y ago
obviously there's some amount of validation that happens at each step. The question is just how much you do when, and how that affects when issues get discovered.

MeetingsBrowser 2y ago

> There’s a cynical read of “shift left” that is “make people do more work but pay them the same.” I haven't heard of this. I thought shifting left was the new hotness. I was under the impression that finding and fix mistakes as soon as possible is considered best practice. > Shifting Testing Left Is a Return of an Older, Better System I don't really understand. I thought the "old way" was to wait until everything was done to check if it works, and the "new way" was to add tests for individual components as you go, making sure nothing breaks as new features are added.

taneq 2y ago
And just to demonstrate the universality of the human condition... just yesterday I was having a rant about (a) everything being always underbudgeted because the cheapest quote usually wins, so everyone underquotes hoping the client changes their mind and the resulting variation(s) swing the needle into profitability, and (b) that there's also a tendency to create artificial urgency (unrealistic deadlines etc.) in order to try and extract greater productivity from your workers. Hanlon's Razor would tend to discredit (b) but I've seen it in action too many times to fully accept this...
NikolaNovak 2y ago
>>I thought the "old way" was to wait until everything was done to check if it works, and the "new way" was to add tests for individual components as you go, making sure nothing breaks as new features are added. Things tend to be more cyclical. 30 years ago, we did unit tests where we continually test individual components, then system tests were you test if your system works as a whole, then integration tests where you check how your system works with others. We had dedicated environments for these. And this was in the big-corporation, bog-standard, boring, commercial-off-the-shelf standard enterprise resource planning software. Yes, we had waterfall / predictive project management model, but I'm not sure where this notion comes in that before right now, this exciting brilliant moment in time, nobody in IT knew at all what they were doing and built entire bridges for years and waited to see if they'd collapse when the first car came :-) Grumble grumble get off my lawn young kids etc etc :->
- 2y ago

dboreham 2y ago

I'm not sure how this is different than TDD, but at the end it hits on a key point, something I've called "ROAC" (Runs On Any Computer): "there was often a way to run the whole system, or a facsimile, on your laptop with minimal friction" Any time the code you're developing can't be run without some special magic computer (that you don't have right now), special magic database (that isn't available right now), special magic workload (production load, yeah...), I predict the outcome is going to be crappy one way or another.

dpflan 2y ago
I went on read more articles that were linked. Another idea was having a cluster that is very close to production to which teams can deploy PRs that need to be tested (if they are related). - https://thenewstack.io/improve-developer-velocity-by-decentralizing-testing/ https://thenewstack.io/improve-developer-velocity-by-decentr...
oldpersonintx 2y ago
ROAC is a pipe dream (I used to believe in it also) most large cloud/distributed systems cannot be realistically reproduced in your compose/k3s/whatever environment a large cloud/distributed system must be tested on something similar in scale at best, ROAC lets you do some development remotely/offline, but is not a reliable model of reality
- vundercind 2y ago
  Anything interesting and server-based usually depends on an entire complex data pipeline (one system for this if you’re lucky) and a dozen or more external systems with no great way to mock them that won’t itself be out-of-sync with reality often, anyway.
  - hu3 2y ago
    I once heard Facebook uses production database to develop and test new feature because anything else was unfeasable. Even a smaller copy of it. I'd love to have a confirmation from someone at Meta. edit: found an amazing discussion about their release process here: https://news.ycombinator.com/item?id=30899362 https://news.ycombinator.com/item?id=30899362

kelsey98765431 2y ago

The author poses a very real and valid question but posits no solution and frames this as a binary. The question is when to test. There is benefit and tradeoff when you decide when and how to test your application. Sometimes you simply cannot test beforehand, such as when you are integrating with a live system with a cost of replicating the environment very very high due to this being a managed solution contract with a large cough IBM cough vendor. In these cases often the question is when to mock test. Then when to test. But this is still just how and when. With a solo developer, it makes plenty of sense to write the feature then test it. Writing it may be the way to actually discover the behavior you want. With a team this can still be the case. In a larger team you can pass this to QA engineers to take these rapid prototypes and find the issues with them for a different type of developer to fix bugs in. The reverse is the same. You can discover your feature behavior by experimenting with tests until the desired behavior is asserted to be occuring through things like type validation and object patching for instrumentation, or whatever profiling or debugging tools and test suites you like. The hard answer is there is no one answer. Just like iterative development urges us to do our best to keep up by iterating on our features, we should iterate on our tests. Do not worship coverage, use it as a tool. Do not fear testing, it's just code afterall so you can actually make a full application that is just a series of tests (experiments) that run and if everything has passed, well you just wrote the same thing as a bunch of conditional checks in your library code. Do what works for you, your team, the project you are on and where you are at. If youre struggling add some more tests. If you have so many tests failing you dont know what to do, write some more code and string your tests together with new functions. It gives a nice way to say ok, this is the next step of my iteration. Some like to start by testing some like to finish. I do both. Whatever you do, put love into it and if thats super sturdy prod code that has no tests and will never fail because you are the safest data handler in the world with the most flexible business logic in the world, go for it - but just remember if you someday later forget how things work and dont want to read documentation or experiment with the live system - you can always write tests to remind you how it works. Tests are a form of documentation. They may not prove what you think they prove, but they document a fact about your code that is objective as determined by the code itself. This is a wonderful power, use it however you wish.

serverlessmom 2y ago

moomin 2y ago

I feel like this is a case of one of those "movements" that is just "Yes, under certain circumstances this might be a good idea; and under others, it won't be". Consider that tests are literally executable specifications of how the system or component should behave (a formulation I associate with Dan North, but I'm sure the idea is far from original). Now the question becomes "should you write a spec every time before you start implementation"? It seems obvious that the answer to that rather depends on how well you actually understand the problem space you're trying to solve. And if all the work you're doing is well understood before you start, I suggest finding somewhere your manager you with the hard stuff.

eyelidlessness 2y ago
> It seems obvious that the answer to that rather depends on how well you actually understand the problem space you're trying to solve. An advocate for TDD (as in “red > green > refactor”) might say that writing the spec upfront is an excellent way to understand the problem space, or at least to help achieve that understanding. At least when I’ve been fortunate enough to work on projects where such a TDD approach is possible, I’ve found that to be the case. I’ve also often found it harder to gain that understanding when testing lags development.

ssfrr 2y ago

This is the first time I’ve heard the term “shifting left”. Maybe I missed it, but is this the same as TDD?

Jgrubb 2y ago
"Shift left" is a relatively hot, relatively new catchall bizterm that typically makes one visualize pushing a workload "upstream" from its current place in the flow. In this context, I think TDD is close to what it means.
- drewcoo 2y ago
  > "Shift left" is a relatively hot, relatively new catchall bizterm that typically makes one visualize pushing a workload "upstream" from its current place in the flow. I've been shifting left (and using those words) for over two decades. Nothing new here. > In this context, I think TDD is close to what it means. Nope. It's only close in that both have to do with testing. TDD is you writing unit tests. Shifting left is about parallelizing someone else's work in the rest of the test pyramid. If QA can write their integration/system/contract/other tests as you're writing your code, you can have working tests for new features ideally as soon as the feature is runnable/testable instead of QA waiting to "see what the dev code does" to write against it.
  - iainmerrick 2y ago
    Thank you, that helps clear things up. The original article takes a very long time to get to its points. Like the first commenter in this thread, I’d never heard the term “shifting left” before. That’s surprising for a buzzword you say has been around for decades and is hot right now, but I guess there’s a first time for everything!

jf22 2y ago

>This process then becomes more of a waterfall model where dev and QA are distinct, sequential phases. I think we overuse the term waterfall. Having a QA phase as part of a development cycle doesn't not make something "waterfall.:

pydry 2y ago
Not if it exists adjacent to it. It does if it habitually gates releases though. If you can't risk not having QA gating your releases then your process is probably more waterfall than you might have hoped.
- jf22 2y ago
  See this is what I mean about the overuse. Release gating doesn't make something waterfall.
  - pydry 2y ago
    It kind of does. If you need to run your build through QA, product, upper management and regulatory approval then sorry, no matter how many standups you have - you're waterfall. If you need an approval on a pull request and some automated tests to pass to give you enough confidence to deploy then that part of your process is agile at least (even if, say, the way you get requirements is not).
    - jf22 2y ago
      I don't know what regulatory approval or standups have to do with waterfall, and I'm not sure what automated tests have to do with whether or not you are agile.
- bluGill 2y ago
  Despite going all in on agile from day one of a new greenfield project, very high test coverage from TDDing everything, and a large suite of automated integration tests - we still have a month of QA gating the release process as bugs constantly sneak past all that process in a complex system. QA and automated tests look for different classes of problems.