15 ms·
A single developer dropped AWS costs by 90%, then disappeared
- uejfiweun 4y agoThis is why you need strict code reviews, and leaders who actually understand the technology. Any competent organization would have caught this.
- 0xbadcafebee 4y agoSo you need a developer to review a developer. Who's reviewing the developer's developer?
- coffeeboy27 4y agoAnother developer
- IYasha 4y ago"developers, developers, developers, developers!"
- mc32 4y agoIf something seems unbelievable, it probably is. How can anyone not question how a 90% reduction was possible without proof? That said it looks like the company did save money for a few months so the 50 grand was worth it although hard to maintain indefinitely and of course against ToSes and morally questionable.
- wstuartcl 4y ago"probably" but not always. I have worked on a few projects that saved 70-90% of costs for AWS -- but these all were starting from a position of auto scaling out a ton of ec2 (on demand full cost instances) to batch process huge amounts of data. In both scenarios by optimizing the batching to align tasks to pack as much as possible into (just short of hour long) hour long windows and converting to a (wide list of ec2 types to avoid contention lockout) spot instances there was easy fodder for the improvement. But yeah -- no one should see their bill change by that much without understanding exactly what was done to trigger it. Even in the best cases, there are always tradeoffs that must be known and accepted.
- poulsbohemian 4y agoAgreed with both you and parent commenter - the story comes across as almost apocryphal; a warning both that if it's too good to seem true, it probably is, and that one must always understand the work done and the tradeoffs involved (IE: don't fall asleep at the wheel)
- prottog 4y agoTo be fair, a lot of cloud deployments are not optimized for cost, and it's easy to rack up big bills without really needing to.
- imron 4y agoTo be fair, as an IT director, when you see this sort of reduction you should dig deeper and understand where the cost savings are coming from.
- daniel-cussen 4y ago[dead]
- deltree7 4y agoReducing 90% cloud bill is not out of the ordinary.
- CrimsonRain 4y agoReducing cloud costs, especially if the dev team does not have anyone experienced enough to focus on this, is often possible on massive scales. More so if you have free reign to optimize/rewrite stuff as you see fit. While not 90%, I've reduced cost by 83% in one case while increasing the ability to serve during peak times also.
- franciscop 4y ago90% savings is very easy if the system was built by a beginner. I helped a friend once who was starting and reduced his query by 99% of time, and thus quite a chunk of money, by a couple of simple changes (IIRC avoiding a JOIN he was making that he didn't actually need and adding an index).
- orwin 4y agoI mean, you can go from poorly-sized EC2 instance with autoscalinggroups constantly interrogating remote databases to full serverless with databases hosted on AWS, and 90% is an easy target.
- selimthegrim 4y agoOriginal Twitter thread in Russian is here: https://mobile.twitter.com/xanf_ua/status/1608121311078776832 https://mobile.twitter.com/xanf_ua/status/160812131107877683...
- dikaio 4y agoWonder if stealing a companies slogan ““We Shall Build Good Ships Here; At A Profit If We Can, At A Loss If We Must, But Always Good Ships.”” — Collis Potter Huntington is considered the same thing?
- joshstrange 4y agoI'm almost annoyed the SaaS company in question profited from this for those 6 months before their costs started going back up again. Any IT Director who bought into this snake oil, allowed the person access to their codebase, and accepted their changes without review should also be fired.
- al2o3cr 4y agoDupe of https://news.ycombinator.com/item?id=34281802 https://news.ycombinator.com/item?id=34281802
- thdc 4y agoYeah, I remember seeing the original with the fictional disclaimer. Wonder why it was edited. before/after (you can find a snapshot on archive.org:jan 6, 2023) > NB: Fictional story, inspired and translated from this source (https://twitter.com/xanf_ua/status/1608121311078776832 https://twitter.com/xanf_ua/status/1608121311078776832) > NB: source (https://twitter.com/xanf_ua/status/1608121311078776832 https://twitter.com/xanf_ua/status/1608121311078776832) I was planning to drop a long-winded comment on that submission about how this kind of abuse of free tier services is why we can't have nice things but decided not to.
- mtopolov 4y agoBecause the story was true...
- worldsavior 4y agoBut it's illegal. It's abusing the system.
- nottorp 4y agoThis ticks me off every time i see it. Neither Amazon, nor Google or any other tech company are allowed to make laws. It's a breach of the terms of service maybe, and maybe illegal, but whatever the terms of service say is not a law and said terms of service could be themselves illegal. They have enough power, please stop attributing them even more by your choice of terms.
- wstuartcl 4y agoIt is not that they are creating new laws, it's that the dev committed theft of services and fraud (existing well established laws) as he utilized the system with intent to receive free services that were directly forbidden in the grant from google.
- paxys 4y agoIt is most definitely illegal to buy and use stolen credentials from darknet markets (which is what the author hypothesizes happened in this case). It is also illegal to misrepresent the service you are providing to a customer.
- advisedwang 4y agoIt's likely wire fraud
- davidguetta 4y agofree credits is not money. FAANG can't just create money by saying we give you 100k credits
- advisedwang 4y agoWire fraud doesn't require that you're stealing money: > Whoever, having devised or intending to devise any scheme or artifice to defraud, or for obtaining money or *property* by means of false or fraudulent pretenses ... (18 USC 1343) Property here includes stealing services of value (in fact it even includes intangibles such as confidential info or goodwill, see Carpenter v. United States 484 US 19).
- reidjs 4y agoPretty clever scam. He had a huge list of accounts, then set up a script that took advantage of the free introductory compute AWS gives to new users. When the script ran out of the accounts to cycle through, it fell back to the old system.
- CommitSyn 4y agoThey should've bought a few more million accounts on Fiverr and updated the yaml file.
- 411111111111111 4y agoHow can you consider it a scam though? They paid 50k and saved hundreds of thousands according to this story (90+k reduced by 90% for 6 month.) Google is the only one that got scammed as far as I can tell.
- paxys 4y agoIt's a scam because (1) they did not promise a cost reduction for just a few months, and (2) what they did was illegal and/or against the terms of service of the company's cloud provider. Just like how selling you a perfect working car for a great price – which just happens to be stolen – would be a scam.
- antifa 4y agoThey thought they were getting optimizations to their AWS infrastructure, but were actually using stolen resources from GCP.
- generalizations 4y ago> A few months later costs started to rise again. Each time a GCP free-tier account did not work, data was sent back to the old AWS system. I like this part. He may have run a total scam, but he still wrote good code, dammit.
- bostonsre 4y agoThe company also still saved money in the long run, just not as much as they had hoped initially.
- paxys 4y agoThe money the company saved would have paled in comparison to legal costs had Google or some government authority decided to take action against them for buying and using stolen accounts. And no "this random developer (who we can't find) did it" isn't a valid excuse. The company is still fully liable for their code.
- kichik 4y agoAnd still saved them about half a million dollars (6*$90k - $50k).
- 4y ago
- comprev 4y agoA good story, and likely untrue although still possible. I've helped clients save huge chunks of cash over the years from AWS because the previous developer/team paid little attention to costs. Reserved instances, alternative routing of data, cheaper S3 tiers, etc. soon add up significant savings.
- jsnell 4y agoSorry, why are we believing this story? The technical details make little sense; e.g. moving a ml processing pipeline to a different cloud just by "editing a Chrome extension" doesn't seem very plausible. It isn't even internally consistent, the hallmark of a sloppy fabrication. At first the accounts were sourced from the darknet, then they were created by the Chrome extension. Creating a million Google accounts is pretty hard. And buying so many accounts from the markets would probably cost his whole $50k price. Also, it was a million accounts and the savings were about 500k. Each account spent 50 cents worth of compute out of the $300 free trial? I assume you need to give a credit card number for the free trial. I wonder how many of those were needed for this to work...
- advisedwang 4y agoYour skepticism is warranted, but just to note: > At first the accounts were sourced from the darknet, then they were created by the Chrome extension The article says they bought Google Accounts on darknet and _created GCP accounts_ from the chrome extension. Presumably "GCP Account" here means a new GCP project/billing account with free trial or using the always free usage. So it's vague and difficult to believe but not actually contradictory.
- zerocrates 4y agoIt looks like the story is just something the author of this post pulled from a Twitter thread (linked at the top). The thread seems (I'm relying on Google Translate since it's in Ukrainian/Russian), to just say there was a big list of a boatload of email addresses sourced from the person's previous work, and it pulled from those to keep registering new Google accounts (or maybe log into them as there's mentions of passwords being included, and simply sign up for GCP on them). There's also nothing about AWS vs. Google in the original thread I see, nor specific numbers on numbers of accounts, amount saved (tens/hundreds of thousands a month is as specific as it goes), etc.
- acdha 4y agoSkepticism is warranted but the original tweets weren’t SEO-d down and seem more believable. This one makes it sound like this was a list of valid Google accounts which could be signed up for GCP: https://twitter.com/xanf_ua/status/1608121336190357504?s=20&t=uUaKLLPP4rAMBdwIWJbSrw https://twitter.com/xanf_ua/status/1608121336190357504?s=20&... That makes more plausible: start with a list of Google accounts, try to use it with GCP, and cycle accounts until you find one which hasn’t exhausted the free tier. It would be very easy to believe that the 1M account number was winnowed down rapidly by the ones which had changed passwords, tripped some kind of suspicious activity alert, etc. or been used for anything else if that wasn’t the only copy of that list in existence.
- cloudking 4y agoReminds of a developer I hired through Fiverr that claimed he could achieve 99/100 Google pagespeed score for any Shopify website (and improve overall load times of Shopify). He implemented his changes, and sure enough it scored 99/100. However, the site still seemed to be running slowly when I tested it. I dug into what code he had actually changed, and there were only a few lines of JavaScript injected into the site. The JS was obfuscated, after de-obfuscating it I realized all his code did was prevent any other JavaScript from being loaded if it detected the user-agent was coming from Google pagespeed :) After some back and forth with Fiverr I was able to get refunded, but I'm sure the thousands of other customers that gave him 5 star reviews have no idea.
- 22c 4y agoIf the service was advertised as "improve your page load times" then I can see how that's misleading, but if it was to get a 99/100 page speed score as a kind of "dumb" SEO hack then maybe you got what you paid for?
- dymk 4y agoTo be fair, the value in getting a 99/100 pagespeed is SEO optimization, so sites rank higher on Google. The point is getting more traffic to the site, not a better user experience.
- antifa 4y agoBut doesn't Google penalize you for user agent based cheating?
- mcint 4y agoAh, the original poster in Ukranian* (_ua) just deleted the thread.