4 ms·
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults
by donavanm 2mo ago
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
- pudgywalsh 2mo ago"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
- AlotOfReading 2mo agoUnit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.
- gleenn 2mo agoOne of the Mars landers famously failed due to unit conversion errors from metric to standard.
- golem14 2mo agoWasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.
- blemasle 2mo agoI didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
- Dan_- 2mo agoI know you’re being snarky, but the US system is not “imperial” anyway. It’s properly “US Customary” but is often called “US Standard.”
- fc417fc802 2mo agoClaiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures. That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
- reddalo 2mo agoYou just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.
- AlotOfReading 2mo agoAll of my science classes in the rural US from middle school onward were taught entirely in metric, and it had been that way for decades prior. Obviously I'm an adult now, and metric still isn't universal.
- picofarad 2mo agoThe cost is too high. You can't have mph and kmh signs on the same road. So you gotta switch all the signs at once. Mile markers, exit numbers (are miles), distance to signs, all have to change. I did the math on a comment elsewhere. Just google how many miles of interstate there are, and quadruple it, thats not even approaching how many signs you'd need for just the interstate. Signs run $25-ish, plus installation, which can cost >$1000. At the very least the km markers will need to be installed.
- 27183 2mo agoIt's not difficult to write regression tests that catch unit mistakes.
- zapkyeskrill 2mo agoDegrees, very funny.
- cyberax 2mo agoOne of my personal rules is to ALWAYS suffix the variables and fields with unit names: `timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`. Unless the variable type already constrains it (e.g. Duration in Go).
- markus_zhang 2mo agoThis also helps downstream a lot.
- monkpit 2mo agoUntil someone comes along and changes it without updating the name, or the value comes from a remote service and they change without you changing your name… etc.
- anvuong 2mo agoMultiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha. Units and datetime will always be the bane of any professions ...
- fragmede 2mo agonames and money movement also come to mind.
- DelightOne 2mo agoAre you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?
- 01284a7e 2mo agoNo tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo https://www.youtube.com/watch?v=3fGHaVn5rGo
- akdev1l 2mo agoNot even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
- qurren 2mo agoYou overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
- tyre 2mo agoAre you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
- geodel 2mo ago"Former" seems to an important detail here.
- gleenn 2mo agoIf someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
- nurettin 2mo agoThis is why I always fail loud rather than pick a stupid default.
- crossroadsguy 2mo agoHad it been half a million dollars or something or say like a few hundred dollars?
- sscaryterry 2mo agoThis isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
- edelbitter 2mo ago[dead]
- 8note 2mo agoits so far out as to be obviously not real a smaller error by say, just one or two orders of magnitude are much more believable as a reader
- sscaryterry 2mo agoI've seen people have panic attacks over much smaller amounts. Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.
- creatonez 2mo agoNot everyone got a 10 figure bill estimates, some posts were mentioning 8 and 9 figure estimates
- shermantanktop 2mo agoBut someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions? Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
- picofarad 2mo agoYeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars. I'm stressed thinking about it, like from a theoretical standpoint.
- Twirrim 2mo agoI wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
- hotstickyballs 2mo agoCSV files are widely known to be used by the most frugal companies so of course it is.
- fragmede 2mo agoHave you seen what RDS costs‽ AWS couldn't possibly afford that!
- berkes 2mo agoI've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far. Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt. I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day. Because CSV is the "interface" that will bring your system down if not properly designed.
- anvuong 2mo agoThat was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
- dev_l1x_be 2mo agoImagine a programming language that has physical measurement unit support so this could have never happened.
- yawaramin 2mo agoLike F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/units-of-measure https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...
- dev_l1x_be 2mo agoExactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).
- QuinnyPig 2mo agoWas this the day that gp3 EBS volumes came out by any chance?
- donavanm 2mo agoNope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.
- dlenski 2mo ago> Services emit metering values that arent directly tied to prices. Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both. I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.
- merek 2mo agoWere there cases of finalized invoices and fund transfers while the error was in place? The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.
- mlinhares 2mo agoThe last thing a billing system should have is defaults, this is such an insane thing.
- thewhitetulip 2mo agoHey man, vibe coders gonna vine code. No wait. Loop enginering. We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!
- mlitwiniuk 2mo agoWhat apology letters? Never got anything, not even a reply to my support tickets.
- AjeetTester2026 2mo ago[dead]