9 ms·

AWS faulty UX cost our business $7k and they refused to do anything about it

82 points by mikeyla85 6y ago

notyourwork 6y ago
The extra dramatics in this article turn me off. Its essentially saying I'm busy and negligent. Now I'm mad big company didn't come to my rescue. > scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital. > and amid 18 hour workdays, The quantified communication with company provides no details to what was said. Maybe op was right, maybe they weren't but this sounds like whining without any evidence.
- spoonjim 6y ago
  Yeah, all that stupid bullshit isn’t AWS’s problem. Why should Amazon care about your stupid 18 hour workday?
- kvm 6y ago
  I understand AWS UX is quite bad (I have also personally been bitten by GCP UX) and would be more sympathetic with the OP if this were the cost for 1 or 2 months. If OP didn't notice it for 8-9 months, I do think it's unfair to blame the entire amount on AWS.
  - mikeyla85 6y ago
    It was 6 months. Three of them was really my bad. The other three...well, some shit was going down.
- smoldesu 6y ago
  I'm right there with you. This new generation of "medium authors" is really not sticking the landing. Almost every one of these articles could just be renamed to "My Rookie Mistake And How I'm Going To Make It Into A 1,500 Word Blog Post"
  - EdwardDiego 6y ago
    I do rather enjoy "I made a mistake" blogs when they own the mistake and discuss what they'd do differently.
- WettowelReactor 6y ago
  The line about > Messages often took a full month to get a response. Also does not ring true to my experience with AWS unless the author was not paying for any support tier. If they did not have even developer support on the account, then they got exactly what they paid for.
  - mikeyla85 6y ago
    Are you suggesting we pay for a $30/mo support plan for AWS when we planned to spend $40/mo total on the service?
ineedasername 6y ago
The little animation shows some alert thrown by AWS when the author goes through their process, but it goes by too fast for me to make out the details. I'm not going to watch it 100 times to get it all, but I do see phrases about copies and folder objects that might be relevant to this issue. Perhaps warning about the limits of the action being taken? I can't tell.
- Prolixium 6y ago
  FWIW, the first bullet mentions lifecycle rules: "This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
- idunno246 6y ago
  "This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle policy" "This action applies to all objects within the specified folders" I'd assume the first one means youll get charged once for standard and once for glacier
- tyingq 6y ago
  Here's a 1/3rd speed copy of the image: https://imgur.com/a/K1OEnhU https://imgur.com/a/K1OEnhU
- mikeyla85 6y ago
  It claims that the operation will make a copy instead of transition the objects, which as far as I understood it would have then deleted the original objects and accomplished something relatively similar but with an extra copy step.
jinkyu 6y ago
that's hideously expensive cloud storage
bearjaws 6y ago
Turn on anomaly detection in your billing, it will pay for itself within 3 months. Everyone screws up something in AWS eventually. We had a run away EFS volume that burnt through $300 in a day, luckily was caught by an alarm. Thats basically $8700 saved if accounting had caught it first.
lambda_obrien 6y ago
I'm personally curious how a 100 employee company gets almost bankrupted by $7k? This article should have been written in a less bombastic fashion, I felt at the end like it was just an advertisement for a film.
- mikeyla85 6y ago
  We're a film company with contractors, and I was raising money as we went. I also said that under slightly different circumstances (some last-minute investment) we could have been bankrupted by this. Sorry you didn't like the tone of the article.
subhro 6y ago
> After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you, even if you think you know what you’re doing. Maybe you should step back and think why people like us put our blood and sweat and it still takes years to learn how to administer infrastructure properly. Stop whining and learn to pay for skills.
- sergiotapia 6y ago
  This is true! Infrastructure != Software Development. Like an civil engineer vs architect. Both different, complimentary skills.
  - manojlds 6y ago
    What does civil engineer vs architect have to do with Infra vs Software Development?
    - turtleish 6y ago
      Is the comparison of Pastry Chef vs. a Rotisseur closer to your sensibilities? The comparison the prior comment made was something along the lines of "both work in very similar setting and industry, but each have their own skill set and expertise." Each can probably function in the the others job, but not well and probably at a greater cost.
- chillfox 6y ago
  As an experienced sysadmin who reflexively triple checks for landmines before taking any action, I feel like my days are spent operating software with shitty UIs. It's like I am that person from Star Trek who repeats what the computer says and tells it what to do. Just because the current status quo of things is this way does not mean it is great. The world would be a better place if more people were empowered to setup computer systems in a safe way.
- mikeyla85 6y ago
  Out of curiosity, what would market rate for a sysadmin run to order a snowball and sync 24TB of data to it, ensuring that it ends up properly in Deep Glacier?
apenney 6y ago
We had something similar happen with lifecycle rules, moving an entire bucket to glacier: end result was > $200k and AWS weren’t especially helpful on our attempts to refund us. We were unaware that changes via a rule still count as an api hit per object, despite all happening in the backend. I’m sure we can’t be the only people hit by this.
- viraptor 6y ago
  I almost did that. Someone double checking the plans raised that as an issue and instead we went with a lifecycle at some specific age rule which didn't try moving everything at once. And yes, it's both fairly well documented and something that can catch you by surprise at the same time.
- thitcanh 6y ago
  The reality is that Amazon makes big bucks off the complete opacity of their costs. If you offer a UI where a click costs money, you should also display price. But that's just my opinion. Maybe others like buying things from a menu without prices. DigitalOcean emailed me once when I left my single $5/month instance open by mistake. AWS would never do that even for thousands.
- mikeyla85 6y ago
  Damn, that's insane. Would it be so hard for them to redesign AWS to tell you what something's going to cost before you do it?
sergiotapia 6y ago
I've never worked with a tool that made me this scared to use it. I mean it. One wrong click and you can blow your budget, crash you entire system, the works. Over the years I've grown used to it but the feeling of uneasiness will never leave me. I get it, AWS is very powerful but it does not give me the comfy feelings of using a product like Render or DigitalOcean, where I pay for what I need and that's the end of it.
- rlayton2 6y ago
  It would be nice to get a hard-cap on AWS costs. This tutorial includes how to setup alerts for a given budget: https://aws.amazon.com/getting-started/hands-on/control-your-costs-free-tier-budgets/ https://aws.amazon.com/getting-started/hands-on/control-your... I don't believe you can "halt everything" if you go over a certain amount though. I'd be happy to be proved wrong.
  - Sebb767 6y ago
    > It would be nice to get a hard-cap on AWS costs. > I don't believe you can "halt everything" if you go over a certain amount though. You're pretty much on point. Handling a hard cut is simply not feasible for nearly all services. I.e. simply sending E-Mails in the middle of a news letter is not useful. You can have reasonable actions for some services (i.e. disallow new EC2 machines, add the end-of-month-cost for running ones to the bill), but this could break automation in strange ways (i.e. you suddenly could not spin up a large instance temporarily due to the limit). And how would you handle traffic? Disallow peaks? What about API gateway or other usage-based services? Lastly, AWS is all about scalability. Breaking so much, especially in the scalability space, for such a small feature is simply not worth it for them. So usually they're simply lenient on bills and take the occasional beating (and it's not like there would not be an equal amount of articles on "how AWS caps broke our system [and it did not warn us/the UI was bad/ ...]"). The missing hard cap is the reason I don't have an account there, but I can absolutely understand why they don't have it.
    - wildfire 6y ago
jrochkind1 6y ago
I dunno... it might be bad UX. I have definitely been annoyed with limitations to bulk edit abilities from S3 console before. But I've been in this business for a while. I'm a programmer rather than an ops person, but i still sometimes make changes like that. I am responsible for some S3 buckets of stuff whose cost is significant for our budget. I would never make a change in any software, where a failure for the change to take would cost me a noticeable part of budget, and not do even a cursory check to see if the change took. I mean, I write software, I know how it goes. Who does that without ever checking to see if it worked? I don't do anything that matters without double-checking to see if it worked like I expected, at least if I'm doing it for the first (or second, or third) time, whether via a console or a script. Cause if you work with computers, you know they don't do what you expected all the damn time. You're making a new kind of change you have never done before? You try it with a few objects before trying it on your whole bucket. You double-check if it did what you expected. Then you do it to everything. Then you still double-check to make sure it did it as expected to everything. (Even if you "read the manual" as the OP says, it's not about that at all in fact). That's how engineering works. If you don't have time to do it carefully, then sure, your chances of making mistakes or misunderstandings goes up a lot, why would you expect someone else to be responsible for your mistakes from not being careful?
- mikeyla85 6y ago
  I checked many of the files. The change took for everything I could see, but the billing was still affected.
deleted 6y ago
[deleted]
technick 6y ago
Ignorance isn't bliss I guess and if you have the budget to hire 100 people, you have the budget to hire someone who has a clue about the cloud. While the cloud is just someone else's hardware, you still pay for it like it's your own.
- mikeyla85 6y ago
  OP here. It's a film company and a lot of those 100 people are artists and contractors. I put a lot of my own money into the company and took out a PPP loan to keep us afloat. So no, we didn't have the money to hire cloud support.
fortran77 6y ago
The interface is awful. If he had said something after the first bill, I'd be 100% on his side. But he waited 9 months and kept paying. That's just too late.
ineedasername 6y ago
I don't know, maybe there's a legitimate UX issue here, but the author also said it only took them an hour of reading up on Glacier to resolve. That seems a reasonable amount of effort to have put upfront. It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges. I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I don't expect them to step in like that for a problem sustained over the course of months.
- mattashii 6y ago
  > It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges. This. I manage a 15k/month AWS account, and if the expected expenditures of any day are > 15$ off the normal, I try to find out what happened. Especially after changing stuff in S3 and EC2; as with our usage patterns they should have predictable billing. > I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I've once reserved instances in the wrong region, but (luckily) found this out the same day, and got it refunded the day after. Would have been the equivalent of a year of salary down the drain otherwise.
jtchang 6y ago
Okay that sucks and I understand where the author is coming from. There are definitely some rough edges in the AWS interface. I do believe as a large corporation Amazon does have some responsibility to waive some of the charge. In this case the intent was not to store things in S3 and just leave it there (especially if there was no access). Amazon has traditionally been good about waiving charges for account compromises and things along that nature. However you do have to keep up on it since they bill you every month. If you're running a company part of the responsibility is at least checking in on your financials every so often. Right when you see a charge for $700+ when you expect $50 is a red flag. At that point if he had contacted Amazon I'd be surprised they didn't waive the charge. I'd be curious if Amazon was willing to waive any part of the charges at all.
- mikeyla85 6y ago
  After 8 months and many messages, they didn't waive a single dollar.
jtsiskin 6y ago
In their gif, the big alert right at the top, in the very first bullet point, says it will make a copy of the object. They intentionally scrolled down quickly in the gif to hide this. Is the argument that this should have been a pop-up that required you to click “ok” instead? "This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
- gzer0 6y ago
  Here's a screen capture of the warning sign for anyone that wants to read it: https://i.imgur.com/Qjiu8F0.png https://i.imgur.com/Qjiu8F0.png
  - dankwizard 6y ago
    thank you i appreciated it
- ineedasername 6y ago
  Yes, I don't think that failing to read a notification at the top of the page about what you're doing constitutes a significant UX problem. I also review my bills every month, which would have mitigated this issue significantly.
  - Sebb767 6y ago
    > I also review my bills every month, which would have mitigated this issue significantly. A cursory look would've been enough, really. If this would have been 20$ or so, I would not fault anyone for missing it. But 800$ missing every month should definitely get some attention, especially when these are amounts endangering your ability to stay afloat, as the author seems to argue.
  - duckfang 6y ago
    And hourly billing alerts would also help. Admittedly the feature that Azure has that AWS doesn't is "kill everything when exceed X dollars". Not so great for prod, but superb for dev. Please realize that when you hand devs with AWS keys, you're assigning an unlimited liability to your company with whatever accident a dev might do. (Accidents happens, and quite often when experimenting.)
userbinator 6y ago
I've never used AWS before so you can consider this a "fresh perspective". In defense of AWS, the warning starts with "This action creates a copy of the object". In defense of the author, "Edit storage class" is a pretty misleading name for the operation (how about "Copy with new storage class?"), and perhaps these "warnings" show up under other more routine operations frequently enough --- and unimportantly-enough to cause warning-fatigue. However, given the nature of cloud services in general (they have a strong $$$ motivation to nickel-and-dime you for every little thing they can) and especially AWS' reputation for doing that, I likely would not have made this mistake. Then again, I'm also someone who carefully reads all available information --- especially whenever it's something I'm paying for.
- mikeyla85 6y ago
  As far as I understand it, the operation creates a copy and deletes the original, which should succeed in reducing costs. What it doesn't do is delete incomplete multipart uploads and it may not operate recursively.
fastball 6y ago
I honestly can't even imagine not checking my AWS bill for 8 months.
jiggawatts 6y ago
A lot of people have this mental model that The Cloud is Magic™. They say things like this with a straight face: - The cloud is always cheaper than on-prem! - The cloud has no downtime, unlike on-prem! Recently, I came across a large government department that had moved a reasonable volume of stuff into Azure. Think 7-figure annual bills. They didn't like that they were overspending, so they complained to Microsoft, essentially demanding that they "make good" on their promise to make the cloud cheaper by cutting the cost on resources. This would have required an 80% discount, maybe higher. It's an absurd thing to even ask, but ask they did. Microsoft said no. Meanwhile, I looked at their tenant, and the stuff that goes on there is amateur hour: Daily uncompressed database backups kept forever in premium zone redundant storage. Dozens of misconfigurations like that just burning money. The worst part of it is that the "Azure Advisor" tab on the portal literally calls this kind of stuff out, recommending fixes worth hundreds of thousands of dollars a year, many with a one button "Quick Fix!" that you can press for an instant discount. Nope. No. They're not going to bother reading things, or checking the Cost Management portal every now and then. They'd rather complain to their Microsoft rep every few months in a huffy tone. "You still haven't cut my costs! Why haven't you!? I thought that's what you people did!" They're busy people and don't have the time to waste cutting half a million dollars of spend here or there with arduous tasks like button pressing.
- zbuf 6y ago
  > A lot of people have this mental model that The Cloud is Magic™. An consultant specialising in AWS once tried to convince me that moving our large scale computation into the cloud would reduce the environmental impact to nil. Amazing what wild theories people will come up with when it's hidden behind a curtain.
  - userbinator 6y ago
    Amazing what wild theories people will come up with when it's hidden behind a curtain. More like "...when it helps Amazon make more $$$." Which is basically what everyone who has "drunk the Cloud-Aid" will be doing, not only for AWS but other cloud services.
banana_giraffe 6y ago
I use AWS daily. It takes a few days to figure out those warnings are mostly there because some customer found out the hard way what they mean. You ignore them at your own peril. And seriously: > This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule. I hate the AWS Console UI for a bunch of reasons, but short of making you type that sentence before you proceed, I'm not sure what AWS can do here.
- jholloway7 6y ago
  Could they calculate and show an estimate of the monthly charges based on current objects in the bucket before and after the change in storage class? OP may still have clicked through w/o heeding, but I can see that as a potential improvement to the warning message where a user would then be forced to understand why it didn't affect pricing in the way they assumed it would.
  - banana_giraffe 6y ago
    While I would love to see AWS do this for most things, in this case, it would have resulted in a UI that said something like "Increased monthly cost expected to be $50", and I'm sure they would have read that as "Expected monthly cost change to $50" But an estimated cost for many things in the console would be a welcome sanity check.
- mikeyla85 6y ago
  Creating a copy was not what resulted in the extra charge. It's a warning about a completely different thing.
villgax 6y ago
The fact that you cannot set a hard limit or preload funds for a bill cycle is key to their business
CGamesPlay 6y ago
Is S3 marketed towards nontechnical folk like this? The other comments in this thread do a good enough job at dispelling any need for sympathy to the affected party, but my question is how they ended up inside the AWS console in the first place? It looks like the guy had just enough knowledge to shoot himself in the foot.
- Sebb767 6y ago
  > Is S3 marketed towards nontechnical folk like this? Honestly, even as a quite technical person, understanding AWS is not that easy. I've been in the situation of trying to run a very small app with basically zero budget on AWS a few years ago, and the sheer amount of services I needed to involve and things that needed to look for where quite overwhelming. I can easily imagine a non-technical person getting completely lost in the interface.
- miyuru 6y ago
  S3 began as a simple service(first S in S3), but now it definitely isn't. Now there are 6 different storage classes and loads of options to manage the objects and buckets. Interestingly the latest storage class(S3 Intelligent-Tiering) was created to minimize the overhead of S3.
- mikeyla85 6y ago
  I "ended up in the AWS console" because S3 was a cheap way to store data so I decided to use their product. I've been a casual web developer, hacked together complex Javascript webapps, built Chrome extensions know my way around the terminal. Enough knowledge to be dangerous but not a professional. On AWS, I know how to manage IAM roles, set up EC2 instances, etc. etc. The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
  - CGamesPlay 6y ago
    I don't mean to undermine your skills, but by analogy: I know that sailboats are expensive, and I've sailed a few times, but I wouldn't buy a build-your-own-sailboat kit because it's cheaper and I know how to sail. I agree with you about this, certainly: > The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
throwawayboise 6y ago
Amazon makes a bajillion dollars a second. They could have made him happy and an evangelist for their customer service at zero impact to them. They just wanted to be dicks about it.
- randomchars 6y ago
  I'd agree with you if it was -3 months, but not noticing this amount money for 8 months? For all Amazon knows, this person was just out to scam them.
  - throwawayboise 6y ago
    So offer to waive half the charges. Something....
- mikeyla85 6y ago
  I was an evangelist for the company. I posted regularly about how good of a solution S3 was for filmmakers and was offering to help friends back up their 10-20 terabytes to the cloud. Not anymore.
Mave83 6y ago
just do go to AWS, it's easy
duckfang 6y ago
And folks, this is why you hire system administrators. We understand how stuff runs, is stored, backup procedures, cost expenditures and savings, and more. We also know dev work, but it's not our specialty. We also know how to manage projects but we're not project managers. We also know how to handle billing, but we're not accountants. ... But our job encompasses all of them. And we reach out when we don't know. Our job is to make the trains run on schedule, using the average amount of fuel, with nobody stressing, and the customers happy. You get rid of us.. and... $7k of overage is nothing.
- mikeyla85 6y ago
  Out of curiosity, what would market rate for a sysadmin run to order a snowball and sync 24TB of data to it, ensuring that it ends up properly in Deep Glacier?
  - duckfang 6y ago
    Since you mentioned this is for a startup for the film industry, having a permanent sysad on the roles doesn't make sense. (Rates vary from 60k-200k depending on experience, expertise, area). Instead, I'd look at contract system administration labor. It'll probably set you back 100-150$/hr with minimums of 2h. However, sysads can catch problems, set up alerting, and fix structural issues before they turn into humongous problems. It would be my recommendation to do more hours with onboarding a consultant and a full audit (not SOX or FedRAMP or the like - but to get you aware of what you're doing even if you weren't aware). From there, a plan can be made to keep you up and running with low/no impacts. And once the hard part's done, should insulate you from any oopses. And the monitoring will help catching them in the act in case there's a new type of oops. I hope I answered you adequately :)
    - mikeyla85 6y ago
      Thanks for the information! This is helpful. Given that I was pretty knowledgeable about most aspects of backing my data up to S3, it would have been pretty hard to predict where the possibility for error would have been. I can't imagine hiring someone for 2hr to come and check my work, since they really would have had to be there for the entire process. It sounds like about 2-5 days of work to get someone to set all this up, but perhaps it's more like 1 day at a minimum. So somewhere between $800-4k for that. Later in our project we were running 5 S3 buckets with about 20 stakeholders uploading and downloading, so that could have easily been another 4 days of work. $800 would have been two years of what we were planning to pay Amazon. $4-8k would have been a major big-ticket item for us (even though, of course, that's the bill we ended up stuck with.) If I were to do it over again I don't think I'd be able to justify spending on a pro. I would definitely spend on someone to audit our expenses more carefully though.
samaman 6y ago
Ah, reminds me of the time I ran through $15k worth of compute in all of 3 weeks trying to do biological modeling. What was strange was that we were being charged for higher bandwidth even though it was not said to be any more for that same instance even if the usage stayed the same. We switched to Digital Ocean after that and that problem went away and had an all you can eat set of arguably better systems (32 CPUs vs 16) for $640 a month per device. So yeah, really do your homework, keep up with the notifications, and realize that they LOVE people like you! Why do you think cloud companies are so rich? Every startup idea gets put on there and either forgotten or blows up and ends up spending more in services. So yes, keep this in mind moving forward and dont listen to people who swear by AWS for x y of a feature because chances are they dont actually know what's going on behind the hood but have had to he trained it or heard about it in tech interviews if they are a cs person.
shsbdncudx 6y ago
There’s a great book called Extreme Ownership. You really should read it.
mikeyla85 6y ago
OP here. Thanks to some of the helpful comments, it seems that the creation of a copy when I chose "Edit storage class" may have been the culprit. I didn't suspect that because in all my correspondence with Amazon, they always said the issue was incomplete multipart uploads. So I assumed, based on the dialog box called "Edit storage class", that the net result of this action would be the same files but with a different storage class. This led me to read the warning as "We will create a copy of your objects and delete the old one, so please be aware you will be charged for the cost of the copy" rather than "We will create a copy of your objects and two versions of them will exist forever." So it does seem like Amazon tried to warn people. But why on earth would a dialog box labeled "Edit storage class" not do anything of the sort? Are there actual legitimate reasons someone would want to have two versions of their objects sitting around? Say what you will about my billing awareness (and you are saying it. Feeling the hate, all, thanks) but this is not good UX. If there's any legitimate reason to duplicate objects in the name of changing the storage class, I'd love to hear it.
- kuya11 6y ago
  > This led me to read the warning as "We will create a copy of your objects and delete the old one, so please be aware you will be charged for the cost of the copy" rather than "We will create a copy of your objects and two versions of them will exist forever." First, the same warning you are referring to also says, exactly: “You can change the storage class without making a new copy of the object using a lifecycle rule.” Second, to answer your prompt, the reason why the object is duplicated during a storage class change is because S3 is an object store and not a filesystem like you may assume. Performing a modification operation on an object does not change the object itself; instead a copy is made with modifications. When changing storage class, a copy is thus made. If you had used the lifecycle rule as the warning suggested, it would have performed your assumption: make the copy and delete the original object in the background. P.S. Not trying to be abrasive :) I do though think you should considering hiring a real devops/cloud/sysadmin to manage AWS for you. If that is really not feasible for you, I suggest at the very least looking into setting up budget anomaly detection and alerting. You then might be able to have some visibility into your budget.
jrrrr 6y ago
My perception is that with AWS, instead of managing a physical data center, you have to manage a virtual one. You still need dedicated technical staff. A small business should use a backup service like Backblaze. AWS is what you use to _build_ a backup service.
- mikeyla85 6y ago
  Amazon was the only service that had storage that I could afford ($40/mo versus hundreds per month the competitors were asking.) Backblaze, last I checked, has "unlimited" storage but requires you to plug the hard drive in every three months. A number of other details made it seem like it was not a good long term storage option. I was focused on a solution that would let us store the film we spent a year working on for the next 50 years. We did this and LTO tape.
ranguna 6y ago
> I was the most technical person on our executive team, and amid 18 hour workdays This shows how incompetent this guy really is.
tga 6y ago
The minor lesson here: be careful operating heavy machinery with no cost limits -- or hire someone who knows what they're doing to do it for you. The major lesson: audit your expenses every month (or hire someone to do it for you). The AWS UX cost them $800. Faulty business administration cost them $6200 on top of that. It could have just as easily been $MAX_CREDIT_LIMIT.
tpoacher 6y ago
For all the people arguing that the presence of a box with some text spewing warnings in technicalese constitutes "UX", and that the assertion that bad UX costs money is false due to the presence of said box: I don't think you understand what UX means.
StratusBen 6y ago
I'm the co-founder of a startup named Vantage that is an alternative to the AWS console with a focus on developer experience and cost transparency and this is exactly the sort of thing we try and help people with. We're sorry that this happened and hear about it a lot. I'd highly recommend folks sign up for Vantage if for no other reason than to get our weekly cost reports which would have caught this sooner: https://www.vantage.sh/ https://www.vantage.sh/
- mikeyla85 6y ago
  Thanks, and I received your email! I'll definitely consider switching and getting other filmmakers on the platform.

82 points by mikeyla85 6y ago

The extra dramatics in this article turn me off. Its essentially saying I'm busy and negligent. Now I'm mad big company didn't come to my rescue. > scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital. > and amid 18 hour workdays, The quantified communication with company provides no details to what was said. Maybe op was right, maybe they weren't but this sounds like whining without any evidence.

spoonjim 6y ago
Yeah, all that stupid bullshit isn’t AWS’s problem. Why should Amazon care about your stupid 18 hour workday?
kvm 6y ago
I understand AWS UX is quite bad (I have also personally been bitten by GCP UX) and would be more sympathetic with the OP if this were the cost for 1 or 2 months. If OP didn't notice it for 8-9 months, I do think it's unfair to blame the entire amount on AWS.
- mikeyla85 6y ago
  It was 6 months. Three of them was really my bad. The other three...well, some shit was going down.
smoldesu 6y ago
I'm right there with you. This new generation of "medium authors" is really not sticking the landing. Almost every one of these articles could just be renamed to "My Rookie Mistake And How I'm Going To Make It Into A 1,500 Word Blog Post"
- EdwardDiego 6y ago
  I do rather enjoy "I made a mistake" blogs when they own the mistake and discuss what they'd do differently.
WettowelReactor 6y ago
The line about > Messages often took a full month to get a response. Also does not ring true to my experience with AWS unless the author was not paying for any support tier. If they did not have even developer support on the account, then they got exactly what they paid for.
- mikeyla85 6y ago
  Are you suggesting we pay for a $30/mo support plan for AWS when we planned to spend $40/mo total on the service?

ineedasername 6y ago

The little animation shows some alert thrown by AWS when the author goes through their process, but it goes by too fast for me to make out the details. I'm not going to watch it 100 times to get it all, but I do see phrases about copies and folder objects that might be relevant to this issue. Perhaps warning about the limits of the action being taken? I can't tell.

Prolixium 6y ago
FWIW, the first bullet mentions lifecycle rules: "This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
idunno246 6y ago
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle policy" "This action applies to all objects within the specified folders" I'd assume the first one means youll get charged once for standard and once for glacier
tyingq 6y ago
Here's a 1/3rd speed copy of the image: https://imgur.com/a/K1OEnhU https://imgur.com/a/K1OEnhU
mikeyla85 6y ago
It claims that the operation will make a copy instead of transition the objects, which as far as I understood it would have then deleted the original objects and accomplished something relatively similar but with an extra copy step.

jinkyu 6y ago

that's hideously expensive cloud storage

bearjaws 6y ago

Turn on anomaly detection in your billing, it will pay for itself within 3 months. Everyone screws up something in AWS eventually. We had a run away EFS volume that burnt through $300 in a day, luckily was caught by an alarm. Thats basically $8700 saved if accounting had caught it first.

lambda_obrien 6y ago

I'm personally curious how a 100 employee company gets almost bankrupted by $7k? This article should have been written in a less bombastic fashion, I felt at the end like it was just an advertisement for a film.

mikeyla85 6y ago
We're a film company with contractors, and I was raising money as we went. I also said that under slightly different circumstances (some last-minute investment) we could have been bankrupted by this. Sorry you didn't like the tone of the article.

subhro 6y ago

> After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you, even if you think you know what you’re doing. Maybe you should step back and think why people like us put our blood and sweat and it still takes years to learn how to administer infrastructure properly. Stop whining and learn to pay for skills.

sergiotapia 6y ago
This is true! Infrastructure != Software Development. Like an civil engineer vs architect. Both different, complimentary skills.
- manojlds 6y ago
  What does civil engineer vs architect have to do with Infra vs Software Development?
  - turtleish 6y ago
    Is the comparison of Pastry Chef vs. a Rotisseur closer to your sensibilities? The comparison the prior comment made was something along the lines of "both work in very similar setting and industry, but each have their own skill set and expertise." Each can probably function in the the others job, but not well and probably at a greater cost.
chillfox 6y ago
As an experienced sysadmin who reflexively triple checks for landmines before taking any action, I feel like my days are spent operating software with shitty UIs. It's like I am that person from Star Trek who repeats what the computer says and tells it what to do. Just because the current status quo of things is this way does not mean it is great. The world would be a better place if more people were empowered to setup computer systems in a safe way.
mikeyla85 6y ago
Out of curiosity, what would market rate for a sysadmin run to order a snowball and sync 24TB of data to it, ensuring that it ends up properly in Deep Glacier?

apenney 6y ago

We had something similar happen with lifecycle rules, moving an entire bucket to glacier: end result was > $200k and AWS weren’t especially helpful on our attempts to refund us. We were unaware that changes via a rule still count as an api hit per object, despite all happening in the backend. I’m sure we can’t be the only people hit by this.

viraptor 6y ago
I almost did that. Someone double checking the plans raised that as an issue and instead we went with a lifecycle at some specific age rule which didn't try moving everything at once. And yes, it's both fairly well documented and something that can catch you by surprise at the same time.
thitcanh 6y ago
The reality is that Amazon makes big bucks off the complete opacity of their costs. If you offer a UI where a click costs money, you should also display price. But that's just my opinion. Maybe others like buying things from a menu without prices. DigitalOcean emailed me once when I left my single $5/month instance open by mistake. AWS would never do that even for thousands.
mikeyla85 6y ago
Damn, that's insane. Would it be so hard for them to redesign AWS to tell you what something's going to cost before you do it?

sergiotapia 6y ago

I've never worked with a tool that made me this scared to use it. I mean it. One wrong click and you can blow your budget, crash you entire system, the works. Over the years I've grown used to it but the feeling of uneasiness will never leave me. I get it, AWS is very powerful but it does not give me the comfy feelings of using a product like Render or DigitalOcean, where I pay for what I need and that's the end of it.

rlayton2 6y ago
It would be nice to get a hard-cap on AWS costs. This tutorial includes how to setup alerts for a given budget: https://aws.amazon.com/getting-started/hands-on/control-your-costs-free-tier-budgets/ https://aws.amazon.com/getting-started/hands-on/control-your... I don't believe you can "halt everything" if you go over a certain amount though. I'd be happy to be proved wrong.
- Sebb767 6y ago
  > It would be nice to get a hard-cap on AWS costs. > I don't believe you can "halt everything" if you go over a certain amount though. You're pretty much on point. Handling a hard cut is simply not feasible for nearly all services. I.e. simply sending E-Mails in the middle of a news letter is not useful. You can have reasonable actions for some services (i.e. disallow new EC2 machines, add the end-of-month-cost for running ones to the bill), but this could break automation in strange ways (i.e. you suddenly could not spin up a large instance temporarily due to the limit). And how would you handle traffic? Disallow peaks? What about API gateway or other usage-based services? Lastly, AWS is all about scalability. Breaking so much, especially in the scalability space, for such a small feature is simply not worth it for them. So usually they're simply lenient on bills and take the occasional beating (and it's not like there would not be an equal amount of articles on "how AWS caps broke our system [and it did not warn us/the UI was bad/ ...]"). The missing hard cap is the reason I don't have an account there, but I can absolutely understand why they don't have it.
  - wildfire 6y ago

jrochkind1 6y ago

I dunno... it might be bad UX. I have definitely been annoyed with limitations to bulk edit abilities from S3 console before. But I've been in this business for a while. I'm a programmer rather than an ops person, but i still sometimes make changes like that. I am responsible for some S3 buckets of stuff whose cost is significant for our budget. I would never make a change in any software, where a failure for the change to take would cost me a noticeable part of budget, and not do even a cursory check to see if the change took. I mean, I write software, I know how it goes. Who does that without ever checking to see if it worked? I don't do anything that matters without double-checking to see if it worked like I expected, at least if I'm doing it for the first (or second, or third) time, whether via a console or a script. Cause if you work with computers, you know they don't do what you expected all the damn time. You're making a new kind of change you have never done before? You try it with a few objects before trying it on your whole bucket. You double-check if it did what you expected. Then you do it to everything. Then you still double-check to make sure it did it as expected to everything. (Even if you "read the manual" as the OP says, it's not about that at all in fact). That's how engineering works. If you don't have time to do it carefully, then sure, your chances of making mistakes or misunderstandings goes up a lot, why would you expect someone else to be responsible for your mistakes from not being careful?

mikeyla85 6y ago
I checked many of the files. The change took for everything I could see, but the billing was still affected.

deleted 6y ago

[deleted]

technick 6y ago

Ignorance isn't bliss I guess and if you have the budget to hire 100 people, you have the budget to hire someone who has a clue about the cloud. While the cloud is just someone else's hardware, you still pay for it like it's your own.

mikeyla85 6y ago
OP here. It's a film company and a lot of those 100 people are artists and contractors. I put a lot of my own money into the company and took out a PPP loan to keep us afloat. So no, we didn't have the money to hire cloud support.

fortran77 6y ago

The interface is awful. If he had said something after the first bill, I'd be 100% on his side. But he waited 9 months and kept paying. That's just too late.

ineedasername 6y ago

I don't know, maybe there's a legitimate UX issue here, but the author also said it only took them an hour of reading up on Glacier to resolve. That seems a reasonable amount of effort to have put upfront. It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges. I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I don't expect them to step in like that for a problem sustained over the course of months.

mattashii 6y ago
> It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges. This. I manage a 15k/month AWS account, and if the expected expenditures of any day are > 15$ off the normal, I try to find out what happened. Especially after changing stuff in S3 and EC2; as with our usage patterns they should have predictable billing. > I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I've once reserved instances in the wrong region, but (luckily) found this out the same day, and got it refunded the day after. Would have been the equivalent of a year of salary down the drain otherwise.

jtchang 6y ago

Okay that sucks and I understand where the author is coming from. There are definitely some rough edges in the AWS interface. I do believe as a large corporation Amazon does have some responsibility to waive some of the charge. In this case the intent was not to store things in S3 and just leave it there (especially if there was no access). Amazon has traditionally been good about waiving charges for account compromises and things along that nature. However you do have to keep up on it since they bill you every month. If you're running a company part of the responsibility is at least checking in on your financials every so often. Right when you see a charge for $700+ when you expect $50 is a red flag. At that point if he had contacted Amazon I'd be surprised they didn't waive the charge. I'd be curious if Amazon was willing to waive any part of the charges at all.

mikeyla85 6y ago
After 8 months and many messages, they didn't waive a single dollar.

jtsiskin 6y ago

In their gif, the big alert right at the top, in the very first bullet point, says it will make a copy of the object. They intentionally scrolled down quickly in the gif to hide this. Is the argument that this should have been a pop-up that required you to click “ok” instead? "This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."

gzer0 6y ago
Here's a screen capture of the warning sign for anyone that wants to read it: https://i.imgur.com/Qjiu8F0.png https://i.imgur.com/Qjiu8F0.png
- dankwizard 6y ago
  thank you i appreciated it
ineedasername 6y ago
Yes, I don't think that failing to read a notification at the top of the page about what you're doing constitutes a significant UX problem. I also review my bills every month, which would have mitigated this issue significantly.
- Sebb767 6y ago
  > I also review my bills every month, which would have mitigated this issue significantly. A cursory look would've been enough, really. If this would have been 20$ or so, I would not fault anyone for missing it. But 800$ missing every month should definitely get some attention, especially when these are amounts endangering your ability to stay afloat, as the author seems to argue.
- duckfang 6y ago
  And hourly billing alerts would also help. Admittedly the feature that Azure has that AWS doesn't is "kill everything when exceed X dollars". Not so great for prod, but superb for dev. Please realize that when you hand devs with AWS keys, you're assigning an unlimited liability to your company with whatever accident a dev might do. (Accidents happens, and quite often when experimenting.)

userbinator 6y ago

I've never used AWS before so you can consider this a "fresh perspective". In defense of AWS, the warning starts with "This action creates a copy of the object". In defense of the author, "Edit storage class" is a pretty misleading name for the operation (how about "Copy with new storage class?"), and perhaps these "warnings" show up under other more routine operations frequently enough --- and unimportantly-enough to cause warning-fatigue. However, given the nature of cloud services in general (they have a strong $$$ motivation to nickel-and-dime you for every little thing they can) and especially AWS' reputation for doing that, I likely would not have made this mistake. Then again, I'm also someone who carefully reads all available information --- especially whenever it's something I'm paying for.

mikeyla85 6y ago
As far as I understand it, the operation creates a copy and deletes the original, which should succeed in reducing costs. What it doesn't do is delete incomplete multipart uploads and it may not operate recursively.

fastball 6y ago

I honestly can't even imagine not checking my AWS bill for 8 months.

jiggawatts 6y ago

A lot of people have this mental model that The Cloud is Magic™. They say things like this with a straight face: - The cloud is always cheaper than on-prem! - The cloud has no downtime, unlike on-prem! Recently, I came across a large government department that had moved a reasonable volume of stuff into Azure. Think 7-figure annual bills. They didn't like that they were overspending, so they complained to Microsoft, essentially demanding that they "make good" on their promise to make the cloud cheaper by cutting the cost on resources. This would have required an 80% discount, maybe higher. It's an absurd thing to even ask, but ask they did. Microsoft said no. Meanwhile, I looked at their tenant, and the stuff that goes on there is amateur hour: Daily uncompressed database backups kept forever in premium zone redundant storage. Dozens of misconfigurations like that just burning money. The worst part of it is that the "Azure Advisor" tab on the portal literally calls this kind of stuff out, recommending fixes worth hundreds of thousands of dollars a year, many with a one button "Quick Fix!" that you can press for an instant discount. Nope. No. They're not going to bother reading things, or checking the Cost Management portal every now and then. They'd rather complain to their Microsoft rep every few months in a huffy tone. "You still haven't cut my costs! Why haven't you!? I thought that's what you people did!" They're busy people and don't have the time to waste cutting half a million dollars of spend here or there with arduous tasks like button pressing.

zbuf 6y ago
> A lot of people have this mental model that The Cloud is Magic™. An consultant specialising in AWS once tried to convince me that moving our large scale computation into the cloud would reduce the environmental impact to nil. Amazing what wild theories people will come up with when it's hidden behind a curtain.
- userbinator 6y ago
  Amazing what wild theories people will come up with when it's hidden behind a curtain. More like "...when it helps Amazon make more $$$." Which is basically what everyone who has "drunk the Cloud-Aid" will be doing, not only for AWS but other cloud services.

banana_giraffe 6y ago

I use AWS daily. It takes a few days to figure out those warnings are mostly there because some customer found out the hard way what they mean. You ignore them at your own peril. And seriously: > This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule. I hate the AWS Console UI for a bunch of reasons, but short of making you type that sentence before you proceed, I'm not sure what AWS can do here.

jholloway7 6y ago
Could they calculate and show an estimate of the monthly charges based on current objects in the bucket before and after the change in storage class? OP may still have clicked through w/o heeding, but I can see that as a potential improvement to the warning message where a user would then be forced to understand why it didn't affect pricing in the way they assumed it would.
- banana_giraffe 6y ago
  While I would love to see AWS do this for most things, in this case, it would have resulted in a UI that said something like "Increased monthly cost expected to be $50", and I'm sure they would have read that as "Expected monthly cost change to $50" But an estimated cost for many things in the console would be a welcome sanity check.
mikeyla85 6y ago
Creating a copy was not what resulted in the extra charge. It's a warning about a completely different thing.

villgax 6y ago

The fact that you cannot set a hard limit or preload funds for a bill cycle is key to their business

CGamesPlay 6y ago

Is S3 marketed towards nontechnical folk like this? The other comments in this thread do a good enough job at dispelling any need for sympathy to the affected party, but my question is how they ended up inside the AWS console in the first place? It looks like the guy had just enough knowledge to shoot himself in the foot.

Sebb767 6y ago
> Is S3 marketed towards nontechnical folk like this? Honestly, even as a quite technical person, understanding AWS is not that easy. I've been in the situation of trying to run a very small app with basically zero budget on AWS a few years ago, and the sheer amount of services I needed to involve and things that needed to look for where quite overwhelming. I can easily imagine a non-technical person getting completely lost in the interface.
miyuru 6y ago
S3 began as a simple service(first S in S3), but now it definitely isn't. Now there are 6 different storage classes and loads of options to manage the objects and buckets. Interestingly the latest storage class(S3 Intelligent-Tiering) was created to minimize the overhead of S3.
mikeyla85 6y ago
I "ended up in the AWS console" because S3 was a cheap way to store data so I decided to use their product. I've been a casual web developer, hacked together complex Javascript webapps, built Chrome extensions know my way around the terminal. Enough knowledge to be dangerous but not a professional. On AWS, I know how to manage IAM roles, set up EC2 instances, etc. etc. The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
- CGamesPlay 6y ago
  I don't mean to undermine your skills, but by analogy: I know that sailboats are expensive, and I've sailed a few times, but I wouldn't buy a build-your-own-sailboat kit because it's cheaper and I know how to sail. I agree with you about this, certainly: > The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.

throwawayboise 6y ago

Amazon makes a bajillion dollars a second. They could have made him happy and an evangelist for their customer service at zero impact to them. They just wanted to be dicks about it.

randomchars 6y ago
I'd agree with you if it was -3 months, but not noticing this amount money for 8 months? For all Amazon knows, this person was just out to scam them.
- throwawayboise 6y ago
  So offer to waive half the charges. Something....
mikeyla85 6y ago
I was an evangelist for the company. I posted regularly about how good of a solution S3 was for filmmakers and was offering to help friends back up their 10-20 terabytes to the cloud. Not anymore.

Mave83 6y ago

just do go to AWS, it's easy

duckfang 6y ago

And folks, this is why you hire system administrators. We understand how stuff runs, is stored, backup procedures, cost expenditures and savings, and more. We also know dev work, but it's not our specialty. We also know how to manage projects but we're not project managers. We also know how to handle billing, but we're not accountants. ... But our job encompasses all of them. And we reach out when we don't know. Our job is to make the trains run on schedule, using the average amount of fuel, with nobody stressing, and the customers happy. You get rid of us.. and... $7k of overage is nothing.

mikeyla85 6y ago
Out of curiosity, what would market rate for a sysadmin run to order a snowball and sync 24TB of data to it, ensuring that it ends up properly in Deep Glacier?
- duckfang 6y ago
  Since you mentioned this is for a startup for the film industry, having a permanent sysad on the roles doesn't make sense. (Rates vary from 60k-200k depending on experience, expertise, area). Instead, I'd look at contract system administration labor. It'll probably set you back 100-150$/hr with minimums of 2h. However, sysads can catch problems, set up alerting, and fix structural issues before they turn into humongous problems. It would be my recommendation to do more hours with onboarding a consultant and a full audit (not SOX or FedRAMP or the like - but to get you aware of what you're doing even if you weren't aware). From there, a plan can be made to keep you up and running with low/no impacts. And once the hard part's done, should insulate you from any oopses. And the monitoring will help catching them in the act in case there's a new type of oops. I hope I answered you adequately :)
  - mikeyla85 6y ago
    Thanks for the information! This is helpful. Given that I was pretty knowledgeable about most aspects of backing my data up to S3, it would have been pretty hard to predict where the possibility for error would have been. I can't imagine hiring someone for 2hr to come and check my work, since they really would have had to be there for the entire process. It sounds like about 2-5 days of work to get someone to set all this up, but perhaps it's more like 1 day at a minimum. So somewhere between $800-4k for that. Later in our project we were running 5 S3 buckets with about 20 stakeholders uploading and downloading, so that could have easily been another 4 days of work. $800 would have been two years of what we were planning to pay Amazon. $4-8k would have been a major big-ticket item for us (even though, of course, that's the bill we ended up stuck with.) If I were to do it over again I don't think I'd be able to justify spending on a pro. I would definitely spend on someone to audit our expenses more carefully though.

samaman 6y ago

Ah, reminds me of the time I ran through $15k worth of compute in all of 3 weeks trying to do biological modeling. What was strange was that we were being charged for higher bandwidth even though it was not said to be any more for that same instance even if the usage stayed the same. We switched to Digital Ocean after that and that problem went away and had an all you can eat set of arguably better systems (32 CPUs vs 16) for $640 a month per device. So yeah, really do your homework, keep up with the notifications, and realize that they LOVE people like you! Why do you think cloud companies are so rich? Every startup idea gets put on there and either forgotten or blows up and ends up spending more in services. So yes, keep this in mind moving forward and dont listen to people who swear by AWS for x y of a feature because chances are they dont actually know what's going on behind the hood but have had to he trained it or heard about it in tech interviews if they are a cs person.

shsbdncudx 6y ago

There’s a great book called Extreme Ownership. You really should read it.

mikeyla85 6y ago

OP here. Thanks to some of the helpful comments, it seems that the creation of a copy when I chose "Edit storage class" may have been the culprit. I didn't suspect that because in all my correspondence with Amazon, they always said the issue was incomplete multipart uploads. So I assumed, based on the dialog box called "Edit storage class", that the net result of this action would be the same files but with a different storage class. This led me to read the warning as "We will create a copy of your objects and delete the old one, so please be aware you will be charged for the cost of the copy" rather than "We will create a copy of your objects and two versions of them will exist forever." So it does seem like Amazon tried to warn people. But why on earth would a dialog box labeled "Edit storage class" not do anything of the sort? Are there actual legitimate reasons someone would want to have two versions of their objects sitting around? Say what you will about my billing awareness (and you are saying it. Feeling the hate, all, thanks) but this is not good UX. If there's any legitimate reason to duplicate objects in the name of changing the storage class, I'd love to hear it.

kuya11 6y ago
> This led me to read the warning as "We will create a copy of your objects and delete the old one, so please be aware you will be charged for the cost of the copy" rather than "We will create a copy of your objects and two versions of them will exist forever." First, the same warning you are referring to also says, exactly: “You can change the storage class without making a new copy of the object using a lifecycle rule.” Second, to answer your prompt, the reason why the object is duplicated during a storage class change is because S3 is an object store and not a filesystem like you may assume. Performing a modification operation on an object does not change the object itself; instead a copy is made with modifications. When changing storage class, a copy is thus made. If you had used the lifecycle rule as the warning suggested, it would have performed your assumption: make the copy and delete the original object in the background. P.S. Not trying to be abrasive :) I do though think you should considering hiring a real devops/cloud/sysadmin to manage AWS for you. If that is really not feasible for you, I suggest at the very least looking into setting up budget anomaly detection and alerting. You then might be able to have some visibility into your budget.

jrrrr 6y ago

My perception is that with AWS, instead of managing a physical data center, you have to manage a virtual one. You still need dedicated technical staff. A small business should use a backup service like Backblaze. AWS is what you use to _build_ a backup service.

mikeyla85 6y ago
Amazon was the only service that had storage that I could afford ($40/mo versus hundreds per month the competitors were asking.) Backblaze, last I checked, has "unlimited" storage but requires you to plug the hard drive in every three months. A number of other details made it seem like it was not a good long term storage option. I was focused on a solution that would let us store the film we spent a year working on for the next 50 years. We did this and LTO tape.

ranguna 6y ago

> I was the most technical person on our executive team, and amid 18 hour workdays This shows how incompetent this guy really is.

tga 6y ago

The minor lesson here: be careful operating heavy machinery with no cost limits -- or hire someone who knows what they're doing to do it for you. The major lesson: audit your expenses every month (or hire someone to do it for you). The AWS UX cost them $800. Faulty business administration cost them $6200 on top of that. It could have just as easily been $MAX_CREDIT_LIMIT.

tpoacher 6y ago

For all the people arguing that the presence of a box with some text spewing warnings in technicalese constitutes "UX", and that the assertion that bad UX costs money is false due to the presence of said box: I don't think you understand what UX means.

StratusBen 6y ago

I'm the co-founder of a startup named Vantage that is an alternative to the AWS console with a focus on developer experience and cost transparency and this is exactly the sort of thing we try and help people with. We're sorry that this happened and hear about it a lot. I'd highly recommend folks sign up for Vantage if for no other reason than to get our weekly cost reports which would have caught this sooner: https://www.vantage.sh/ https://www.vantage.sh/

mikeyla85 6y ago
Thanks, and I received your email! I'll definitely consider switching and getting other filmmakers on the platform.