3 ms·

I saw pleeeeeenty of untested code at Amazon/AWS. Looking back it was almost like the most important services/code had the least amount of testing. While intern

I saw pleeeeeenty of untested code at Amazon/AWS. Looking back it was almost like the most important services/code had the least amount of testing. While internal boondoggle projects (I worked on a couple) had complicated test plans and debates about coverage metrics.

dastbe 5y ago
my take is that the overwhelming majority of services insufficiently invest in making testing easy. the services that need to grow fast due to customer demand skip the tests while the services that aren't going much of anywhere spend way too much time on testing.
DrBenCarson 5y ago
This is almost always the case. The most important services get the most attention from leaders who apply the most pressure, especially in the first ~2y of a fast-growing or high-potential product. So people skip tests.
- foobiekr 5y ago
  reality most of the real world successful projects are mostly untested because that's not actually a high ROI endeavor. it kills me to realize that mediocre code you can hack all over to do unnatural things is generally higher value in phase I than the same code done well in twice the time.
  - jessermeyer 5y ago
    This attitude is why modern software is a continuing controlled flight into terrain.
    - virtue3 5y ago
      I think the pendulum swing back is going to be designing code that is harder to make bad. Typescript is a good example of trying to fix this. Rust is even better. Deno, I think, takes things in a better direction as well. Ultimately we're going to need systems that just don't let you do "unnatural" things but still maintain a great deal of forward mobility. I don't think that's an unreasonable ask of the future.
    - topspin 5y ago
      Finger wagging has also failed to produce any solutions.
tootie 5y ago
It's gotta be a whole thing to even think about how to accurately test this kind of software. Simulating all kinds of hardware failures, network partitions, power failures, or the thousand other failure modes. Then again they get like $100B in revenue that should buy some decent unit tests.
wbsun 5y ago
Interesting. I also work for a cloud provider. My team work on both internal infrastructure as well as product features. We take testing coverage very seriously and tie the metrics to the team's perf. Any product feature must have unit tests, integration tests at each layer of the stack, staging test, production test and continuous probers in production. But our reliability is still far from satisfactory. Now with your observation at AWS, I start wondering whether the coverage effort and different types of tests really help or not...
- discodave 5y ago
  > Now with your observation at AWS, I start wondering whether the coverage effort and different types of tests really help or not... Figuring out ROI for testing is a very tricky problem. I'm glad to hear your team invests in testing. I agree it's hard to know if you're wasting money or not doing enough!

dastbe 5y ago

my take is that the overwhelming majority of services insufficiently invest in making testing easy. the services that need to grow fast due to customer demand skip the tests while the services that aren't going much of anywhere spend way too much time on testing.

DrBenCarson 5y ago

This is almost always the case. The most important services get the most attention from leaders who apply the most pressure, especially in the first ~2y of a fast-growing or high-potential product. So people skip tests.

foobiekr 5y ago
reality most of the real world successful projects are mostly untested because that's not actually a high ROI endeavor. it kills me to realize that mediocre code you can hack all over to do unnatural things is generally higher value in phase I than the same code done well in twice the time.
- jessermeyer 5y ago
  This attitude is why modern software is a continuing controlled flight into terrain.
  - virtue3 5y ago
    I think the pendulum swing back is going to be designing code that is harder to make bad. Typescript is a good example of trying to fix this. Rust is even better. Deno, I think, takes things in a better direction as well. Ultimately we're going to need systems that just don't let you do "unnatural" things but still maintain a great deal of forward mobility. I don't think that's an unreasonable ask of the future.
  - topspin 5y ago
    Finger wagging has also failed to produce any solutions.

tootie 5y ago

It's gotta be a whole thing to even think about how to accurately test this kind of software. Simulating all kinds of hardware failures, network partitions, power failures, or the thousand other failure modes. Then again they get like $100B in revenue that should buy some decent unit tests.

wbsun 5y ago

Interesting. I also work for a cloud provider. My team work on both internal infrastructure as well as product features. We take testing coverage very seriously and tie the metrics to the team's perf. Any product feature must have unit tests, integration tests at each layer of the stack, staging test, production test and continuous probers in production. But our reliability is still far from satisfactory. Now with your observation at AWS, I start wondering whether the coverage effort and different types of tests really help or not...

discodave 5y ago
> Now with your observation at AWS, I start wondering whether the coverage effort and different types of tests really help or not... Figuring out ROI for testing is a very tricky problem. I'm glad to hear your team invests in testing. I agree it's hard to know if you're wasting money or not doing enough!