5 ms·

That reads as fairly dismissive to me. Most people running serious gear have serious engineering payroll overhead. Sure, some folks are managing something fai

That reads as fairly dismissive to me. Most people running serious gear have serious engineering payroll overhead. Sure, some folks are managing something fairly static like game servers, but many of us work for bigco with two dozen products and massive teams. Even with thousands of ec2 instances we continue to move our on-prem infrastructure to the cloud because developer productivity saves us big money in the long run.

kortilla 6y ago
Does it actually save or does it just encourage waste by the developers and poor system design? I’ve heard stories of companies having each PR spin up something like 20 ec2 instances to build up a whole deployment. A CICD design like that used to be a fireable offense. Now people see that and assume it’s saving them money because it would have bottlenecked before.
- sokoloff 6y ago
  If it spins those EC2 instances up, runs for a hour of “waste” and then turns off (or the alternate cluster turns off when no longer in use), it very likely is saving money. It might be saving money in that very moment; it might be saving money invisibly when “time to patch the servers again” is a complete non-event for staff time.
- troupe 6y ago
  So I've setup processes like you described where every PR would spin up 20 ec2 instances, run a huge number of parallel testing, and then shut itself down. The savings associated with developers getting feedback on a full regression test of the system in 20 minutes vs. 400 minutes was significant.
  - kortilla 6y ago
    That’s parallelizing, which is not what I’m talking about. I’m talking about pointlessly building up huge environments that “match production”.
- jameshart 6y ago
  That’s a pretty extreme example of CI infrastructure, but okay... so what? 20 t2 mediums costs under a dollar an hour at list rates, and if you’re using a lot of AWS infrastructure can cost a lot less. How much is the confidence that that PR will not break master worth to you? How many hours of engineer code review time would it take to achieve that same level of confidence in your build that spinning up 20 ec2 instances and running regression tests wins? How much would a failed deployment into production cost you, per minute? Before you assume that it’s wasteful for any organization to throw cloud at a problem, realize there are many orgs for which a few hundred dollars of cloud compute spend per production release is an entirely reasonable trade off, not a firing offense.
  - kortilla 6y ago
    > How much is the confidence that that PR will not break master worth to you? That’s exactly the mental trap I’m talking about. Requiring 20 instances to try to mirror production instead of getting better testing in place is a sign of testing immaturity. I have significantly less confidence in the products that use this testing strategy because it really means they don’t have much in place for testing infrastructure to stub things out, inject failures, etc. And it’s not t2 mediums. It’s whatever is specced for production because we’re such good engineers that we want to test in a production like env, right? > Before you assume that it’s wasteful for any organization to throw cloud at a problem, realize there are many orgs for which a few hundred dollars of cloud compute spend per production release That’s not even close. It’s hundreds of dollars a day leading up to thousands per release with larger teams.
twic 6y ago
If you have ten thousand EC2 instances, you also have serious engineering payroll overhead.

kortilla 6y ago

Does it actually save or does it just encourage waste by the developers and poor system design? I’ve heard stories of companies having each PR spin up something like 20 ec2 instances to build up a whole deployment. A CICD design like that used to be a fireable offense. Now people see that and assume it’s saving them money because it would have bottlenecked before.

sokoloff 6y ago
If it spins those EC2 instances up, runs for a hour of “waste” and then turns off (or the alternate cluster turns off when no longer in use), it very likely is saving money. It might be saving money in that very moment; it might be saving money invisibly when “time to patch the servers again” is a complete non-event for staff time.
troupe 6y ago
So I've setup processes like you described where every PR would spin up 20 ec2 instances, run a huge number of parallel testing, and then shut itself down. The savings associated with developers getting feedback on a full regression test of the system in 20 minutes vs. 400 minutes was significant.
- kortilla 6y ago
  That’s parallelizing, which is not what I’m talking about. I’m talking about pointlessly building up huge environments that “match production”.
jameshart 6y ago
That’s a pretty extreme example of CI infrastructure, but okay... so what? 20 t2 mediums costs under a dollar an hour at list rates, and if you’re using a lot of AWS infrastructure can cost a lot less. How much is the confidence that that PR will not break master worth to you? How many hours of engineer code review time would it take to achieve that same level of confidence in your build that spinning up 20 ec2 instances and running regression tests wins? How much would a failed deployment into production cost you, per minute? Before you assume that it’s wasteful for any organization to throw cloud at a problem, realize there are many orgs for which a few hundred dollars of cloud compute spend per production release is an entirely reasonable trade off, not a firing offense.
- kortilla 6y ago
  > How much is the confidence that that PR will not break master worth to you? That’s exactly the mental trap I’m talking about. Requiring 20 instances to try to mirror production instead of getting better testing in place is a sign of testing immaturity. I have significantly less confidence in the products that use this testing strategy because it really means they don’t have much in place for testing infrastructure to stub things out, inject failures, etc. And it’s not t2 mediums. It’s whatever is specced for production because we’re such good engineers that we want to test in a production like env, right? > Before you assume that it’s wasteful for any organization to throw cloud at a problem, realize there are many orgs for which a few hundred dollars of cloud compute spend per production release That’s not even close. It’s hundreds of dollars a day leading up to thousands per release with larger teams.

twic 6y ago

If you have ten thousand EC2 instances, you also have serious engineering payroll overhead.