4 ms·

I have access to a button that I can trivially press, that a coworker recently told me generates a $50 AWS bill.

by sgtnoodle 4y ago

systemvoltage 4y ago
Recursive lambda function call?
- sgtnoodle 4y ago
  It's actually pretty cool, and the evolution of a piece of software I barfed out about 5 years ago. It's a CI job that deterministically runs thousands of simulated flights of an autonomous airplane, across a wide variety of interesting scenarios.
  - roughly 4y ago
    I’m having the same conversations with people but from the opposite side: I’d far, far prefer you click the $50 button than do anything at all in the real world, because anytime anyone at my work touches anything that isn’t a keyboard, it’s costing us a hell of a lot more than that.
    - sgtnoodle 4y ago
      Indeed. The whole point of CI and simulation is to reduce significantly more expensive manual testing. A couple months ago, folk were grumbling about a non-deterministic failure in the main build CI plan, for the N'th time without attempting to do anything about it. So, I proceeded to queue up dozens of jobs at a time, every 15 minutes for several hours, until I had several failures worth of interesting logs. I also had the interest of several other software teams, wondering why the entire CI pipeline was clogged up. It turned out someone years ago had written a cron job that would asynchronously delete every file on the build server older than a week, every hour on the hour. At some point some build artifacts got added that came from a cached copy on a remote server rather than locally built, and their modification timestamps were preserved. Any jobs that happened to be running while on the hour would have those files mysteriously disappear without a trace. Oops!
      - za3faran 4y ago
        Is the flight simulation code open source?
        sgtnoodle 4y ago
        No it is not, but you could take a look at airsim or gazebo.
paranoidrobot 4y ago
There's various levels of foot-gun clicks that can generate AWS bills many orders of magnitude larger. Reserved Instances are a really quick and easy way to blow a few million with one click. Europe (Frankfurt), 3 year, all-upfront, Windows w/ SQL Enterprise, Dedicated Tenancy, u-12tb1.112xlarge instance at $5,926,378 up-front. The discounted $225/hour rate for it disappears into the background noise. (No, I'm not willing to test it to find out if it actually works) [1] https://aws.amazon.com/ec2/pricing/reserved-instances/pricing/ https://aws.amazon.com/ec2/pricing/reserved-instances/pricin...
- aamoscodes 4y ago
  You can increase it to $6,070,809.96 by not paying upfront, or an order of magnitude more by just buying it on-demand and not checking your bill for 3 years (not recommended).
  - paranoidrobot 4y ago
    My comment was more about the 1-click-major-spend aspect. You can always terminate a normal instance - refunding a RI purchase is going to require a lot more explanation and dealing with support/finance/lawyers.
- biggc 4y ago
  Wow! I was trying to find those numbers and the highest I could get to was ~899k. Then I switched to the OS to Windows from Linux. That licensing fee is wild!
  - eb0la 4y ago
    OS Licensing is crazy. Last month I was telling a customer that they should just change their Qlikview system to something else just to avoid paying 25% of their bill to Microsoft.
- fock 4y ago
  for that price, you probably can buy 3 equivalent machines, a consulting shop to setup redundancy (this thing isn't!), install a living cabin to your colo and have someone sitting there on watch 24/7... (should have read the licensing issue - still you can buy 2 and the consultant)

I have access to a button that I can trivially press, that a coworker recently told me generates a $50 AWS bill.

systemvoltage 4y ago

Recursive lambda function call?

sgtnoodle 4y ago
It's actually pretty cool, and the evolution of a piece of software I barfed out about 5 years ago. It's a CI job that deterministically runs thousands of simulated flights of an autonomous airplane, across a wide variety of interesting scenarios.
- roughly 4y ago
  I’m having the same conversations with people but from the opposite side: I’d far, far prefer you click the $50 button than do anything at all in the real world, because anytime anyone at my work touches anything that isn’t a keyboard, it’s costing us a hell of a lot more than that.
  - sgtnoodle 4y ago
    Indeed. The whole point of CI and simulation is to reduce significantly more expensive manual testing. A couple months ago, folk were grumbling about a non-deterministic failure in the main build CI plan, for the N'th time without attempting to do anything about it. So, I proceeded to queue up dozens of jobs at a time, every 15 minutes for several hours, until I had several failures worth of interesting logs. I also had the interest of several other software teams, wondering why the entire CI pipeline was clogged up. It turned out someone years ago had written a cron job that would asynchronously delete every file on the build server older than a week, every hour on the hour. At some point some build artifacts got added that came from a cached copy on a remote server rather than locally built, and their modification timestamps were preserved. Any jobs that happened to be running while on the hour would have those files mysteriously disappear without a trace. Oops!
    - za3faran 4y ago
      Is the flight simulation code open source?
      - sgtnoodle 4y ago
        No it is not, but you could take a look at airsim or gazebo.

paranoidrobot 4y ago

There's various levels of foot-gun clicks that can generate AWS bills many orders of magnitude larger. Reserved Instances are a really quick and easy way to blow a few million with one click. Europe (Frankfurt), 3 year, all-upfront, Windows w/ SQL Enterprise, Dedicated Tenancy, u-12tb1.112xlarge instance at $5,926,378 up-front. The discounted $225/hour rate for it disappears into the background noise. (No, I'm not willing to test it to find out if it actually works) [1] https://aws.amazon.com/ec2/pricing/reserved-instances/pricing/ https://aws.amazon.com/ec2/pricing/reserved-instances/pricin...

aamoscodes 4y ago
You can increase it to $6,070,809.96 by not paying upfront, or an order of magnitude more by just buying it on-demand and not checking your bill for 3 years (not recommended).
- paranoidrobot 4y ago
  My comment was more about the 1-click-major-spend aspect. You can always terminate a normal instance - refunding a RI purchase is going to require a lot more explanation and dealing with support/finance/lawyers.
biggc 4y ago
Wow! I was trying to find those numbers and the highest I could get to was ~899k. Then I switched to the OS to Windows from Linux. That licensing fee is wild!
- eb0la 4y ago
  OS Licensing is crazy. Last month I was telling a customer that they should just change their Qlikview system to something else just to avoid paying 25% of their bill to Microsoft.
fock 4y ago
for that price, you probably can buy 3 equivalent machines, a consulting shop to setup redundancy (this thing isn't!), install a living cabin to your colo and have someone sitting there on watch 24/7... (should have read the licensing issue - still you can buy 2 and the consultant)