21 ms·

A million-dollar engineering problem

570 points by gwintrob 10y ago

officelineback 10y ago
Is this the same thing as segment.io?
- kornish 10y ago
  Yes; they changed their name to Segment at the Series A.
- deleted 10y ago
  [deleted]
nik736 10y ago
Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.
- asptimothy 10y ago
  I was curious about that myself. What's the best way to model the costs associated with maintenance overhead that comes with bare-metal, vs the savings from managing bare-metal servers using a service (like packet), co-locating your own hardware or running your own datacenter. My gut feeling is that you have to get to an extraordinary size to realize any meaningful savings, but that's primarily based on Dropbox's migration off AWS (https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...).
  - GABeech 10y ago
    That's a big giant "it depends." It depends on what your service looks like: CPU intensive, Memory Intensive, Storage intensive? (In reality some unique mix). You probably won't see a huge savings year one, as you'll be spinning up a lot of new things and have a fairly large CapEx expenditure. Now if your growth pattern is steady/predictable then you should be able to plan out your hardware buys or do a hybrid solution to handle traffic bursts. One of the nice things about running your own hardware is that there are some costs that are easier to control. Don't need new hardware? Don't need to spend on new hardware for example. You also have much more control over your environment so you are able to really optimize your code, and infrastructure so that you don't need to scale as large system wise. But, back to the question on how to model it? You just gotta dig in, and make some educated guesses about performance,test and repeat.
  - user5994461 10y ago
    It's rather simple, you count the days you are wasting on handling the low level hardware. The entry cost is £500 per day or $1000 for an engineer. 5 hours on the phone to find the hardware and agree on the order with DELL + an afternoon with customer support because they shipped the servers without hard drives + your project is delayed by an entire week because you don't have the resources => 1 day + 1 day + half a week. These things would have been 5 minutes on a cloud.
  - QuinnyPig 10y ago
    It depends upon your constraints. As a for-instance, I was asked this exact question a year or so ago about an on-prem object store instead of S3. The break-even price to go multi-region was ~15PB on paper; that included DC space, hardware, software (build vs buy was another factored discussion), and the staff to run it. That assessment was delivered with the caveat that their uptime was not going to approach S3's by any stretch of the imagination-- and their infrastructure outages weren't going to sync with "most of the internet's." It's a complex topic, and there are many hidden costs...
- rbranson 10y ago
  It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows down the decision-making process and seriously complicates experimentation. The number of people who know how to build out datacenters at the scale of thousands of machines is vanishingly small. We'd also need to replace IaaS services like ECS, ELB, ElastiCache, RDS, and DynamoDB. There are certainly off-the-shelf replacements, but we'd need to build-out the expertise within our teams to operate these systems. We're talking roughly a dozen or so engineers working full time for many, many months to get these systems in place from scratch, on top of the even larger effort to design and build out datacenters. I'd much rather plow those cycles into efforts like expanding to multiple regions and improving reliability of internal services. That's a much better return on investment for our customers. Right now we're in the sweet spot for the cloud. We're way too big to run on trivial amounts of hardware, growing at a rate that makes it difficult to stay ahead of demand in a datacenter-centric world, and too small to justify investing in a scalable hardware build-out.
  - user5994461 10y ago
    Try all the same on Google Cloud, you'll save half the money compared to AWS, probably more. Disclaimer: I am NOT affiliated to Google in any way.
  - QuinnyPig 10y ago
    > It could be if our workload was relatively stable +1. If your entire environment is steady-state, it may be worth considering a migration to bare metal. That said, you can expect: * Worse uptime * A fair bit of retraining / hiring different skillsets * A loss of engineering focus as they work on the migration instead of feature expansion It can make sense economically, but that discussion goes well past "the monthly AWS bill."
  - lafay 10y ago
    This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure: http://firstround.com/review/the-three-infrastructure-mistakes-your-company-must-not-make/ http://firstround.com/review/the-three-infrastructure-mistak...
teh 10y ago
I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)
- andreapaiola 10y ago
  If I have to give that to you it's not "savings"... :D
  - xj9 10y ago
    but you do save money every month after that!
- cheez 10y ago
  I don't think that's a bad proposition at all. If I were a business person running on AWS, I'd do it.
- officelineback 10y ago
  I'm pretty much that in my current job only with a salary. Here's why: I can make all the recommendations I want but change has to be driven by the will of the higher ups, often as high as C-level folks (CIO/CTO). So you pay me to make recommendations, not to actually save you the money, because the second part is largely out of my control. Having said that, all the cost savings initiatives I've spearheaded are on my resume and LinkedIn profile and I take great satisfaction in optimizing those environments to save the client money.
- sly010 10y ago
  The simplest one might be to convince a company to reserve 3 years worth of AWS resources and paying upfront. I am in this situation right now, but and it's a tough pill to swallow. I decided that all of my personal projects will be GCE. It is much more cost efficient already and Google will soon allow me to commit to future usage and pay my commitment as I go (Right now AWS forces you to pay upfront to get the same discount (~50%))
fourstar 10y ago
I wish I could invest in Segment. Lot of smart people over there.
- eastbayjake 10y ago
  These last two engineering blog posts[1] were great reads and they're doing good open source work. Not looking for a job but would love to meet the team if they host an event! [1] https://segment.com/blog/ui-testing-with-nightmare/ https://segment.com/blog/ui-testing-with-nightmare/
dastbe 10y ago
Reading through this, I would change "Dynamo is Amazon’s hosted version of Cassandra" to "DynamoDB is Amazon's hosted key-value store, similar to Cassandra". The former (to me) sounds like you're saying they vend a managed Cassandra.
- rbranson 10y ago
  You are totally right. This was an oversight that we didn't catch during our review process. Thanks so much for the tip, we've updated the post.
Negitivefrags 10y ago
A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service is profitable, and will stay up. It runs way faster on the single server. It also has required less maintenance after the move too. This kind of shit is everywhere. At this point simply not using AWS is a competitive advantage.
- throwawaysed 10y ago
  A lot of people are in denial that there isn't using some magic efficiency to cloud services that other datacenters don't have. Primary cost savings on cloud VM's is from overprovisioning. The more abstracted away the service is from the hardware, the more they can overprovision without customers noticing. The 50%+ profit margins have to be coming from somewhere. AWS is not made of magic, it's made from largely the same PC parts you buy on newegg
  - user5994461 10y ago
    > The 50%+ profit margins Charge more than what it costs you. That's how to make money.
    - deleted 10y ago
      [deleted]
  - Shivetya 10y ago
    well to be honest its because it sounds cool, its in the news, and so how can such proposals be wrong. Seriously the number of times I heard "the cloud" bandied about by people who have continuously failed or were "thwarted" by people who knew better is getting silly. I am sure it happens in other places but too many think it makes them look smart to suggest it. if anything its a diversion from fixing what is broke or even admitting to it within some organizations
deleted 10y ago
[deleted]
user5994461 10y ago
Addendum missing from the article: Segment charges per request, these users who run performance testing with analytics enabled should be very profitable. Business rule => You gotta optimize for your cash cows. ;)
- tejasmanohar 10y ago
  Actually, our pricing is no longer based on API calls-- https://segment.com/pricing https://segment.com/pricing.
  - user5994461 10y ago
    $10 / 1000 MTUs "Monthly Tracked User, or the sum of logged in plus anonymous users that access your product in a given month. " P.S. I understand that you have volume discounts.
jamiesonbecker 10y ago
Great writeup. The "user_id" one really hit home for me. @ Userify (ssh key management for on-prem and cloud) we currently have hotspots where some companies integrate the userify shim and put 'change_me' (or similar) in their API ID fields. Apparently, sometimes they don't always update it before putting into production... so we get lots and lots of "Change Me" attempted logins! It's not just one company, but dozens. Fortunately, we cache everything (including failures) with Redis, so the actual cost is tiny at most, but if you are not caching failures as well as successes, this can result in unexpected and really hard to track down cost spikes. (disclaimer: AWS cert SA, AWS partner) Segment's trick to detect and record when throttling, and using that as a template for "bad keys" (which presumably are manually validated as well) seems like a great idea as well, but I'd suggest first caching even failure calls on logins if possible, as that probably would have mitigated the need to ever hit dynamo. PS the name 'project benjamin' for the cost cutting efforts.. pure genius.
kornish 10y ago
The Dynamo incident highlights an important lesson when using consistently hashed distributed data stores: make sure the actual distribution of hash keys mirrors the expected distribution. (though to their credit, someone writing an automated test using a hard-coded key was beyond their control). Incidents like this are generally why rate limits exist, which they don't currently have [0], but perhaps they'll consider a burst limiter in place to dissuade automated tests but not organic human load spikes. Unfortunately there doesn't seem to be an easy way to fix the per-user ID write bottleneck, short of adding a rate limit to the API – which would push backpressure from Dynamo to the Segment API consumer. Round-robin partitioning of values would fix the write bottleneck, but has heavy read costs because you have to query all partitions. They undoubtedly performed such analysis and found that it didn't fit their desired tradeoffs :) Great post, very informative. Thanks for sharing! Also, love the slight irony of loading AWS log data into an AWS product (Redshift) to find cost centers. [0]: https://segment.com/docs/sources/server/http/#rate-limits https://segment.com/docs/sources/server/http/#rate-limits
- user5994461 10y ago
  > the per-user ID write bottleneck The basics is to have less machines but more powerful. It helps to handle the targeted bursts. The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic.
  - kornish 10y ago
    Yep, you're absolutely right. In multitenant SaaS apps with extremely uneven distribution of traffic, it's pretty common for large customers to get their own dedicated DB servers. > The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic. And batch loading – don't forget batching!
    - user5994461 10y ago
      I didn't mean to have special databases, that is another level of optimization. I mean to have bigger servers for everything. For instance, a farm of 4x 10cores servers is likely to process data more consistently than 10x 4cores servers.
qaq 10y ago
Here's amazing way to optimize your AWS bill don't use it. Compared to dedicated you overpaid millions than you spent non-trivial developer time/money to get the number down but you are still overpaying. At 5K/month AWS might make sense (although debatable) at your level of spend it's a really bad idea. At this level an ops team of 2 1 on-site 1 remote (diff time zone) would give you way more flexibility and a ver low bill.
joshua_wold 10y ago
Can you lower our bill now? :)
- QuinnyPig 10y ago
  My one step solution to bill lowering involves "blackmailing your TAM." Have you tried that? ;-)
gravypod 10y ago
I've always been set aback by how much AWS servers cost. Maybe I'm just too cheap but you can go to very reputable hosting companies and get things at fractions of the price. For example, if I need 16GB of memory and 4 cores here are my options: * AWS (t2.xlarge) $137.62 * OVH (EG-16) $79 * So You Start/OVH (SYS-IP-1S) $42 * Kimsufi/OVH (KS-3B) $29 * Hetzner (EX41) €46.41 [Lowest cost model is 32GB of ram] * Joe's Data Center (No model name) $45 [Duel socketed L5420 which is 8 cores] That's a crazy price difference and, unlike frmo what I understand about AWS, you don't need to pay for bandwith for most of these companies or they have some crazy high limit. IIRC it's common for 20TB to be the standard (FREE) bandwith limit. You're also on real hardware, not a VM. Unfortunately not all of these are perfect systems. There are issues with them. Some have slower CPUs, some have slower disks, some are in other countries but you can just pick the ones that suit your need for whatever you have to run. You can also use VMs that are far cheaper then these dedicated systems. If your workload is under 50% duty cycle on a CPU and under 4GB of memory you don't need a dedicated server. Buy a VM from one of these companies: * RamNode (8GB SVZS) $28 [Fully SSD storage, very fast network] * OVH (VPS CLOUD RAM 2) $22.39 [12GB(!) of ram and 2 vCores] * Linode (Linode 4GB) $20 These are all VPSs but they will get the job done and are cheap enough that you can build something great on a budget. This is just from a few minutes of remembering when I had to buy my own hardware. I'm sure this isn't an exhaustive list. You can usually find information on the web hosting talk forums and on a site called low end box. I don't have links on hand but they're worth a read.
- bluedino 10y ago
  AWS is so much more than servers. If you just want a server to run some stuff on, you're right, it's faster and cheaper to buy a dedicated server from a reseller. With Amazon you have so many options when it comes to storage, compute power, load balancing... They have a huge portfolio of products and you can get everything in one location from one vendor.
  - zzzcpan 10y ago
    "With Amazon you have so many options when it comes to storage, compute power, load balancing..." It used to be different, but today their most important offerings have open source equivalents. They are essentially off the shelf, because for your own projects there is no need to care about abuse or noisy neighbors. Most importantly, though, it's going to be cheaper and you still will be able to use cloud where it shines.
WithHighProb 10y ago
For us the ECS autoscaling feature is way too naive. We need to do our own controller on some more responsive metrics. The customization support is pretty bad.
- TheHydroImpulse 10y ago
  CPU + memory autoscaling has worked pretty well for most services at Segment. We did have to build a custom queue-based autoscaler that feeds into ECS for our integration stack but the limitations of the ECS autoscaling rules makes it a bit clunky.
- dastbe 10y ago
  Would you mind saying more about what you had to build, and what you would liked to see? One thing I'd like to see in the future is scaling actions that supported multipliers, as well as metric math support (scale service to request_count/acceptable_requests_per_service).
  - TheHydroImpulse 10y ago
    One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that. We settled on constant increases/decreases using queue depth thresholds but ideally ECS would support feeding multiple metrics and doing some basic math to figure out how much we're currently process, how long it would take to flush the queue, and how many more containers we need to drain in a timely fashion.
    - cobookman 10y ago
      Have you looked into kubernetes?
      - TheHydroImpulse 10y ago
        A bunch of people at Segment have looked into it. We manage our infrastructure with Terraform and operationally IMO Kubernetes introduces too much complexity compared with ECS and Terraform. ECS is pretty dead simple (just run an agent on the host) and while it doesn't offer nearly the same feature set, it's really good operationally. Some people here are tinkering with it for some non-core services.
divbit 10y ago
Idk how common the setup in the article is, but seems they could easily test for heatmap singularities such as the one that happened and alert for that.
- divbit 10y ago
  (just alert any keys with heat > mean +- x *standard deviation or something stupid like that)
  - divbit 10y ago
    Actually if you have 3d features, could use edge detection algorithm like Kirsch after fuzzing the colors a bit.
burntwater 10y ago
Had an interesting experience when I went to the pricing page for Segment - Had a little sales guy pop up in the lower right corner asking "Would (my enterprise employer name) like to know if you qualify for a volume discount?" I only recently started working (and web browsing) from an enterprise network, so maybe this has been done before, but it's my first time seeing it and I thought it was really interesting. Edit: To be clear, I'm not intrigued at the little guy himself, I'm intrigued at how personalized it was.
- ropiku 10y ago
  I don't get it right now but it's possible that is Intercom (chat) with data from Clearbit (https://clearbit.com/reveal https://clearbit.com/reveal).
  - davidlumley 10y ago
    Close! It's Drift (drift.com), not Intercom.
    - dcancel 10y ago
      Thanks David. You're right it's Drift. I'm the CEO. Love working with Segment, Mixpanel, Clearbit and other great companies to deliver a personalized experience on their sites like in this example.
      - dcancel 10y ago
        We just published an article on how Segment doubled their sales using Drift. Here's all the details on how it works: https://blog.drift.com/segment-case-study/ https://blog.drift.com/segment-case-study/
- kornish 10y ago
  Did the box have a "Powered by x" statement anywhere? Would be curious to know if it's Intercom or something else.
Dannymetconan 10y ago
A great write up. Really wish that they had given some indication about how much the original bill was. Sounds very effective but 1m of 100m isn't that much!
CobrastanJorji 10y ago
> After a three months of focused work, we managed to cut our AWS bill by over one million dollars annually. That's great, but now I'm wondering how many engineers were paid for three months of full time work to save $1 million/year on AWS, and how much it costs the company to pay those engineers for 3 months. It's also interesting to wonder about how much growth your startup could have managed if it had had three additional months of work from those engineers.
- grkvlt 10y ago
  If you assume USD 250k per engineer, they would need to have had more than 16 engineers working on this for the 3 months to spend more than a million on salary. So, I'm guessing they probably came out ahead on that...?
bearjaws 10y ago
I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, clean up, and remove any data we don't need. We will not get the weeks of development effort back, we will continue to spend more money on hardware, and we will continue to suffer to delay projects because of poor choices. If we had AWS, we could have written an entire sprints worth of work (also ~$14k worth of IT time) and moved on with our lives. Clearly, some companies let the budget keep growing, but for most companies AWS flexibility is worth its weight in gold.
- QuinnyPig 10y ago
  Exactly. Neither path is easy to walk; it always comes down to trade-offs. What do you value?
- ryeguy 10y ago
  Why is it always colo vs cloud in these arguments? What about renting everything monthly from ovh, rackspace, softlayer etc? Many of them can provision dedicated servers within minutes, too.
  - flukus 10y ago
    Also some combination of cloud and private servers. If I was starting up a Netflix competitor for instance, the site could probably run fine from owned hardware, but the content itself would be better off being cloud based.
    - ryeguy 10y ago
      Ironically netflix does the exact opposite. Also, you definitely don't want to have the content on the cloud - the data transfer pricing is where they make significant profit. Both AWS and GCE are ~$100/TB outgoing.
deleted 10y ago
[deleted]
dredmorbius 10y ago
Clickbait title.
williamscales 10y ago
Is it just me or is segment.com down?
nemothekid 10y ago
>The ALB allows ECS to set ports dynamically for individual containers, and then pack as many containers as can fit onto a given instance. Maybe I have a different view, but I've been avoiding running the same service on the same instance. If I'm running multiple services, I'm doing it for HA, and if an instance goes down, you could lose all your services. Why would you run the same service on the same instance?
pascalxus 10y ago
Umm... i take issue with the comment "distribution is essentially free". nonononnononooooo. Other than the rare case of virality, I'd like to know in what software industry where distribution is essentially free - i'm happy to be wrong about this, let me know. the rest of the article was fine.
ryanmarsh 10y ago
> blocking any new discovered badly behaved keys. Eventually we reduced capacity by 4x. I wonder how it felt when they discovered how much money they'd burned on such a dumb little bug. I'd feel like throwing up.
abalone 10y ago
> In some cases, we’re currently packing 100-200 containers per instance. I have a very basic "Docker 101" question here.. Can someone explain how you might get to >100 containers on a given instance? That seems to imply that there are multiple containers of the same microservice running on one machine. Why would you pack duplicate containers like that instead of using one larger container?
- eastbayjake 10y ago
  > Why would you pack duplicate containers like that instead of using one larger container? Microservices that are scaling horizontally to handle concurrent requests, etc. Depending on how seriously your architects take the "micro" in microservice, you may easily get to 100+ unique microservices/processes each running in their own Docker container.
  - abalone 10y ago
    That's not what's going on here (100-200 unique microservices). Aside from the fact that that's insane, they specifically state they want to "pack" multiple containers of the same service on one machine: "You cannot run two containers of the same service on a single host because they will collide and attempt to listen on the same port. (no bin packing)" So, what's the sense in doing that instead of one bigger container?
    - biot 10y ago
      They're reimplementing the actor model using Docker containers. Let it crash.
- bpicolo 10y ago
  The goal is just having a compute pool where you don't have to care at all where things are actually running. It definitely depends on the use case, but I could imagine more, smaller units allowing for more granular (and therefore a bit more cost-effective) scaling.
  - 10y ago
cowardlydragon 10y ago
It is my impression that AWS makes its bank on I/O, while everyone makes buying decisions on CPU/RAM
londons_explore 10y ago
If only they'd spent a couple more dollars on hosting for https://segment.com https://segment.com so it didn't return HTTP 503 errors...
ig1 10y ago
However they miss the easiest fix: Calling their account rep at AWS and cutting a deal. AWS loves startups that could end up being huge customers so they're willing to slash bills upfront to help you get to growth stage; not only will they assign you an account rep but they'll have a rep whose job it is to build a good relationship with your VC. Speak to your account rep. Have your VC speak to their Amazon rep. Push them to cut your prices. I've known startups which have cut their monthly bill by six figures going this route. AWS doesn't want your startup to fail and on the margin these resources cost them close to zero so they're willing to deal, make use of it.
- brianwawok 10y ago
  Why not both? Get the best deal and turn off unused servers.
- rbranson 10y ago
  That was definitely part of the effort; it just doesn't make for terribly compelling technical content. We have a healthy, active relationship with our AWS team. They worked with us to put in place minimum commits for several products, and examined both EDP as well as bulk RI purchase discounts.
  - ig1 10y ago
    I'd say especially in your case you could be pushing for additional discounts beyond what's available to everyone else. You're the ideal case for this, not only will you be a large customer in your own right if you succeed but you have a multiplier effect in that your own clients become heavier users of AWS because of your product encouraging users to move large amounts of data from Salesforce, etc. into AWS. I don't what level the rep you're dealing with is, but if you're not doing so already I'd strongly encourage you to build higher level relationships with AWS.
    - rbranson 10y ago
      Some of the minimum commits aren't exactly standard. I'd talk more about them but they're covered by non-disclosure. I see what you mean by the multiplier effect increasing our bargaining position — totally makes sense.
dangoldin 10y ago
Really interesting read. We've been in a similar mindset and it really comes down to understanding what's driving your costs and being creative around ways to minimize it. As others on here mentioned bandwidth and data transfer is a huge cost so if there's a way to make your caching more aggressive it's a big and easy win. Beyond that the low hanging fruit is in using reserved instances - there are a few options there but are significantly cheaper than paying the on demand price. A shameless plug is I built a small visualization for the AWS detail log to help understand the cost drivers. All it is loading the data and visualizing it across a few dimensions but might come in handy for others - https://github.com/dangoldin/aws-billing-details-analysis https://github.com/dangoldin/aws-billing-details-analysis
qwerty2020 10y ago
how does a bar chart with no y-axis label show any trend? scale could be $100-101/mo
rumcajz 10y ago
Given that Elite game [1] used to run on Sinclair with Z80 processor and 48kB of memory, any complaints about the costs nowadays sound ridiculous. If our software infrastructure was coded by Elite programmers, we would be able to run in at 0.01% of the cost. [1] https://en.wikipedia.org/wiki/Elite_(video_game) https://en.wikipedia.org/wiki/Elite_(video_game)
- jaggederest 10y ago
  > It was written in machine code using assembly language, giving much care to maximum compactness of code. From the link. It seems unlikely that people would/could build applications in the modern world from scratch in assembler. Start with reimplementing the entire HTTP parsing engine and go from there? No thanks.
gregjw 10y ago
Dollar dollar bills y'all
marknadal 10y ago
Great article. I wish/hope/expect everybody to take this type of work/attention as seriously as they have. I am as much of a penny pincher as you can get, so our team built a database adapter on AWS that can handle 100M+ records for $10/day, demo here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . I'm surprised not more people are price/performance sensitive. My goal is to get a system up to 10K table inserts/second on a free Heroku instance, we're already doing on low end hardware 1750 table inserts/second across a federated system. But price tuning is one of the most important things when engineering for high load, as well. Glad to see articles like this. :)
stymaar 10y ago
I don't know if LBO funds are doing this already but I think it's most likely going to become a trend in the future : - buy a SaaS company which hosts all its infra on the cloud - move everything to bare metal - double the company's margin - profit
julienba 10y ago
DISCLAIMER: It can be viewed as an advertisement for my project Few months back I have start a company for that exact purpose : helping you decreasing you AWS bill. https://wizardly.eu/ https://wizardly.eu/ The focus is around reservation, unused resources and tracking cost evolution on a day to day basics. I don't get that much traction ( I'm not a marketer and it's my first SaaS project, still a lot to learn in that area). I'm looking for more users feedback, if you willing to invested few minutes in my project that would really helpful. It's also completely free now. As the article say, their is no silver bullet. Specially if you need infrastructure changed. But it's possible to have an interesting cut by using all the AWS pricing options. Making teams aware and responsable of their costs is also making a big differences. Often, people creating and managing infrastructure have no view of what it is costing for their company.
wazoox 10y ago
A friend of mine developed something to do just that: understand AWS/Google/Azure costs and compare prices (and save big). Shameless plug: https://trackit.io/ https://trackit.io/
- kierenj 10y ago
  Probably not the ideal response, sorry, but I only got as far as the homepage - really slow, jerky scrolling, scroll-to-anchors miss the mark, and the social media links don't work. Then, mouse wheel/trackpad scrolling stopped working. :/
  - wazoox 10y ago
    THe website is probably hosted on AWS :) As for the social media links, they all work for me...

570 points by gwintrob 10y ago

officelineback 10y ago

Is this the same thing as segment.io?

kornish 10y ago
Yes; they changed their name to Segment at the Series A.
deleted 10y ago
[deleted]

nik736 10y ago

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

asptimothy 10y ago
I was curious about that myself. What's the best way to model the costs associated with maintenance overhead that comes with bare-metal, vs the savings from managing bare-metal servers using a service (like packet), co-locating your own hardware or running your own datacenter. My gut feeling is that you have to get to an extraordinary size to realize any meaningful savings, but that's primarily based on Dropbox's migration off AWS (https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...).
- GABeech 10y ago
  That's a big giant "it depends." It depends on what your service looks like: CPU intensive, Memory Intensive, Storage intensive? (In reality some unique mix). You probably won't see a huge savings year one, as you'll be spinning up a lot of new things and have a fairly large CapEx expenditure. Now if your growth pattern is steady/predictable then you should be able to plan out your hardware buys or do a hybrid solution to handle traffic bursts. One of the nice things about running your own hardware is that there are some costs that are easier to control. Don't need new hardware? Don't need to spend on new hardware for example. You also have much more control over your environment so you are able to really optimize your code, and infrastructure so that you don't need to scale as large system wise. But, back to the question on how to model it? You just gotta dig in, and make some educated guesses about performance,test and repeat.
- user5994461 10y ago
  It's rather simple, you count the days you are wasting on handling the low level hardware. The entry cost is £500 per day or $1000 for an engineer. 5 hours on the phone to find the hardware and agree on the order with DELL + an afternoon with customer support because they shipped the servers without hard drives + your project is delayed by an entire week because you don't have the resources => 1 day + 1 day + half a week. These things would have been 5 minutes on a cloud.
- QuinnyPig 10y ago
  It depends upon your constraints. As a for-instance, I was asked this exact question a year or so ago about an on-prem object store instead of S3. The break-even price to go multi-region was ~15PB on paper; that included DC space, hardware, software (build vs buy was another factored discussion), and the staff to run it. That assessment was delivered with the caveat that their uptime was not going to approach S3's by any stretch of the imagination-- and their infrastructure outages weren't going to sync with "most of the internet's." It's a complex topic, and there are many hidden costs...
rbranson 10y ago
It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows down the decision-making process and seriously complicates experimentation. The number of people who know how to build out datacenters at the scale of thousands of machines is vanishingly small. We'd also need to replace IaaS services like ECS, ELB, ElastiCache, RDS, and DynamoDB. There are certainly off-the-shelf replacements, but we'd need to build-out the expertise within our teams to operate these systems. We're talking roughly a dozen or so engineers working full time for many, many months to get these systems in place from scratch, on top of the even larger effort to design and build out datacenters. I'd much rather plow those cycles into efforts like expanding to multiple regions and improving reliability of internal services. That's a much better return on investment for our customers. Right now we're in the sweet spot for the cloud. We're way too big to run on trivial amounts of hardware, growing at a rate that makes it difficult to stay ahead of demand in a datacenter-centric world, and too small to justify investing in a scalable hardware build-out.
- user5994461 10y ago
  Try all the same on Google Cloud, you'll save half the money compared to AWS, probably more. Disclaimer: I am NOT affiliated to Google in any way.
- QuinnyPig 10y ago
  > It could be if our workload was relatively stable +1. If your entire environment is steady-state, it may be worth considering a migration to bare metal. That said, you can expect: * Worse uptime * A fair bit of retraining / hiring different skillsets * A loss of engineering focus as they work on the migration instead of feature expansion It can make sense economically, but that discussion goes well past "the monthly AWS bill."
- lafay 10y ago
  This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure: http://firstround.com/review/the-three-infrastructure-mistakes-your-company-must-not-make/ http://firstround.com/review/the-three-infrastructure-mistak...

teh 10y ago

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

andreapaiola 10y ago
If I have to give that to you it's not "savings"... :D
- xj9 10y ago
  but you do save money every month after that!
cheez 10y ago
I don't think that's a bad proposition at all. If I were a business person running on AWS, I'd do it.
officelineback 10y ago
I'm pretty much that in my current job only with a salary. Here's why: I can make all the recommendations I want but change has to be driven by the will of the higher ups, often as high as C-level folks (CIO/CTO). So you pay me to make recommendations, not to actually save you the money, because the second part is largely out of my control. Having said that, all the cost savings initiatives I've spearheaded are on my resume and LinkedIn profile and I take great satisfaction in optimizing those environments to save the client money.
sly010 10y ago
The simplest one might be to convince a company to reserve 3 years worth of AWS resources and paying upfront. I am in this situation right now, but and it's a tough pill to swallow. I decided that all of my personal projects will be GCE. It is much more cost efficient already and Google will soon allow me to commit to future usage and pay my commitment as I go (Right now AWS forces you to pay upfront to get the same discount (~50%))

fourstar 10y ago

I wish I could invest in Segment. Lot of smart people over there.

eastbayjake 10y ago
These last two engineering blog posts[1] were great reads and they're doing good open source work. Not looking for a job but would love to meet the team if they host an event! [1] https://segment.com/blog/ui-testing-with-nightmare/ https://segment.com/blog/ui-testing-with-nightmare/

dastbe 10y ago

Reading through this, I would change "Dynamo is Amazon’s hosted version of Cassandra" to "DynamoDB is Amazon's hosted key-value store, similar to Cassandra". The former (to me) sounds like you're saying they vend a managed Cassandra.

rbranson 10y ago
You are totally right. This was an oversight that we didn't catch during our review process. Thanks so much for the tip, we've updated the post.

Negitivefrags 10y ago

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service is profitable, and will stay up. It runs way faster on the single server. It also has required less maintenance after the move too. This kind of shit is everywhere. At this point simply not using AWS is a competitive advantage.

throwawaysed 10y ago
A lot of people are in denial that there isn't using some magic efficiency to cloud services that other datacenters don't have. Primary cost savings on cloud VM's is from overprovisioning. The more abstracted away the service is from the hardware, the more they can overprovision without customers noticing. The 50%+ profit margins have to be coming from somewhere. AWS is not made of magic, it's made from largely the same PC parts you buy on newegg
- user5994461 10y ago
  > The 50%+ profit margins Charge more than what it costs you. That's how to make money.
  - deleted 10y ago
    [deleted]
- Shivetya 10y ago
  well to be honest its because it sounds cool, its in the news, and so how can such proposals be wrong. Seriously the number of times I heard "the cloud" bandied about by people who have continuously failed or were "thwarted" by people who knew better is getting silly. I am sure it happens in other places but too many think it makes them look smart to suggest it. if anything its a diversion from fixing what is broke or even admitting to it within some organizations

deleted 10y ago

[deleted]

user5994461 10y ago

Addendum missing from the article: Segment charges per request, these users who run performance testing with analytics enabled should be very profitable. Business rule => You gotta optimize for your cash cows. ;)

tejasmanohar 10y ago
Actually, our pricing is no longer based on API calls-- https://segment.com/pricing https://segment.com/pricing.
- user5994461 10y ago
  $10 / 1000 MTUs "Monthly Tracked User, or the sum of logged in plus anonymous users that access your product in a given month. " P.S. I understand that you have volume discounts.

jamiesonbecker 10y ago

Great writeup. The "user_id" one really hit home for me. @ Userify (ssh key management for on-prem and cloud) we currently have hotspots where some companies integrate the userify shim and put 'change_me' (or similar) in their API ID fields. Apparently, sometimes they don't always update it before putting into production... so we get lots and lots of "Change Me" attempted logins! It's not just one company, but dozens. Fortunately, we cache everything (including failures) with Redis, so the actual cost is tiny at most, but if you are not caching failures as well as successes, this can result in unexpected and really hard to track down cost spikes. (disclaimer: AWS cert SA, AWS partner) Segment's trick to detect and record when throttling, and using that as a template for "bad keys" (which presumably are manually validated as well) seems like a great idea as well, but I'd suggest first caching even failure calls on logins if possible, as that probably would have mitigated the need to ever hit dynamo. PS the name 'project benjamin' for the cost cutting efforts.. pure genius.

kornish 10y ago

The Dynamo incident highlights an important lesson when using consistently hashed distributed data stores: make sure the actual distribution of hash keys mirrors the expected distribution. (though to their credit, someone writing an automated test using a hard-coded key was beyond their control). Incidents like this are generally why rate limits exist, which they don't currently have [0], but perhaps they'll consider a burst limiter in place to dissuade automated tests but not organic human load spikes. Unfortunately there doesn't seem to be an easy way to fix the per-user ID write bottleneck, short of adding a rate limit to the API – which would push backpressure from Dynamo to the Segment API consumer. Round-robin partitioning of values would fix the write bottleneck, but has heavy read costs because you have to query all partitions. They undoubtedly performed such analysis and found that it didn't fit their desired tradeoffs :) Great post, very informative. Thanks for sharing! Also, love the slight irony of loading AWS log data into an AWS product (Redshift) to find cost centers. [0]: https://segment.com/docs/sources/server/http/#rate-limits https://segment.com/docs/sources/server/http/#rate-limits

user5994461 10y ago
> the per-user ID write bottleneck The basics is to have less machines but more powerful. It helps to handle the targeted bursts. The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic.
- kornish 10y ago
  Yep, you're absolutely right. In multitenant SaaS apps with extremely uneven distribution of traffic, it's pretty common for large customers to get their own dedicated DB servers. > The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic. And batch loading – don't forget batching!
  - user5994461 10y ago
    I didn't mean to have special databases, that is another level of optimization. I mean to have bigger servers for everything. For instance, a farm of 4x 10cores servers is likely to process data more consistently than 10x 4cores servers.

qaq 10y ago

Here's amazing way to optimize your AWS bill don't use it. Compared to dedicated you overpaid millions than you spent non-trivial developer time/money to get the number down but you are still overpaying. At 5K/month AWS might make sense (although debatable) at your level of spend it's a really bad idea. At this level an ops team of 2 1 on-site 1 remote (diff time zone) would give you way more flexibility and a ver low bill.

joshua_wold 10y ago

Can you lower our bill now? :)

QuinnyPig 10y ago
My one step solution to bill lowering involves "blackmailing your TAM." Have you tried that? ;-)

gravypod 10y ago

I've always been set aback by how much AWS servers cost. Maybe I'm just too cheap but you can go to very reputable hosting companies and get things at fractions of the price. For example, if I need 16GB of memory and 4 cores here are my options: * AWS (t2.xlarge) $137.62 * OVH (EG-16) $79 * So You Start/OVH (SYS-IP-1S) $42 * Kimsufi/OVH (KS-3B) $29 * Hetzner (EX41) €46.41 [Lowest cost model is 32GB of ram] * Joe's Data Center (No model name) $45 [Duel socketed L5420 which is 8 cores] That's a crazy price difference and, unlike frmo what I understand about AWS, you don't need to pay for bandwith for most of these companies or they have some crazy high limit. IIRC it's common for 20TB to be the standard (FREE) bandwith limit. You're also on real hardware, not a VM. Unfortunately not all of these are perfect systems. There are issues with them. Some have slower CPUs, some have slower disks, some are in other countries but you can just pick the ones that suit your need for whatever you have to run. You can also use VMs that are far cheaper then these dedicated systems. If your workload is under 50% duty cycle on a CPU and under 4GB of memory you don't need a dedicated server. Buy a VM from one of these companies: * RamNode (8GB SVZS) $28 [Fully SSD storage, very fast network] * OVH (VPS CLOUD RAM 2) $22.39 [12GB(!) of ram and 2 vCores] * Linode (Linode 4GB) $20 These are all VPSs but they will get the job done and are cheap enough that you can build something great on a budget. This is just from a few minutes of remembering when I had to buy my own hardware. I'm sure this isn't an exhaustive list. You can usually find information on the web hosting talk forums and on a site called low end box. I don't have links on hand but they're worth a read.

bluedino 10y ago
AWS is so much more than servers. If you just want a server to run some stuff on, you're right, it's faster and cheaper to buy a dedicated server from a reseller. With Amazon you have so many options when it comes to storage, compute power, load balancing... They have a huge portfolio of products and you can get everything in one location from one vendor.
- zzzcpan 10y ago
  "With Amazon you have so many options when it comes to storage, compute power, load balancing..." It used to be different, but today their most important offerings have open source equivalents. They are essentially off the shelf, because for your own projects there is no need to care about abuse or noisy neighbors. Most importantly, though, it's going to be cheaper and you still will be able to use cloud where it shines.

WithHighProb 10y ago

For us the ECS autoscaling feature is way too naive. We need to do our own controller on some more responsive metrics. The customization support is pretty bad.

TheHydroImpulse 10y ago
CPU + memory autoscaling has worked pretty well for most services at Segment. We did have to build a custom queue-based autoscaler that feeds into ECS for our integration stack but the limitations of the ECS autoscaling rules makes it a bit clunky.
dastbe 10y ago
Would you mind saying more about what you had to build, and what you would liked to see? One thing I'd like to see in the future is scaling actions that supported multipliers, as well as metric math support (scale service to request_count/acceptable_requests_per_service).
- TheHydroImpulse 10y ago
  One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that. We settled on constant increases/decreases using queue depth thresholds but ideally ECS would support feeding multiple metrics and doing some basic math to figure out how much we're currently process, how long it would take to flush the queue, and how many more containers we need to drain in a timely fashion.
  - cobookman 10y ago
    Have you looked into kubernetes?
    - TheHydroImpulse 10y ago
      A bunch of people at Segment have looked into it. We manage our infrastructure with Terraform and operationally IMO Kubernetes introduces too much complexity compared with ECS and Terraform. ECS is pretty dead simple (just run an agent on the host) and while it doesn't offer nearly the same feature set, it's really good operationally. Some people here are tinkering with it for some non-core services.

divbit 10y ago

Idk how common the setup in the article is, but seems they could easily test for heatmap singularities such as the one that happened and alert for that.

divbit 10y ago
(just alert any keys with heat > mean +- x *standard deviation or something stupid like that)
- divbit 10y ago
  Actually if you have 3d features, could use edge detection algorithm like Kirsch after fuzzing the colors a bit.

burntwater 10y ago

Had an interesting experience when I went to the pricing page for Segment - Had a little sales guy pop up in the lower right corner asking "Would (my enterprise employer name) like to know if you qualify for a volume discount?" I only recently started working (and web browsing) from an enterprise network, so maybe this has been done before, but it's my first time seeing it and I thought it was really interesting. Edit: To be clear, I'm not intrigued at the little guy himself, I'm intrigued at how personalized it was.

ropiku 10y ago
I don't get it right now but it's possible that is Intercom (chat) with data from Clearbit (https://clearbit.com/reveal https://clearbit.com/reveal).
- davidlumley 10y ago
  Close! It's Drift (drift.com), not Intercom.
  - dcancel 10y ago
    Thanks David. You're right it's Drift. I'm the CEO. Love working with Segment, Mixpanel, Clearbit and other great companies to deliver a personalized experience on their sites like in this example.
    - dcancel 10y ago
      We just published an article on how Segment doubled their sales using Drift. Here's all the details on how it works: https://blog.drift.com/segment-case-study/ https://blog.drift.com/segment-case-study/
kornish 10y ago
Did the box have a "Powered by x" statement anywhere? Would be curious to know if it's Intercom or something else.

Dannymetconan 10y ago

A great write up. Really wish that they had given some indication about how much the original bill was. Sounds very effective but 1m of 100m isn't that much!

CobrastanJorji 10y ago

> After a three months of focused work, we managed to cut our AWS bill by over one million dollars annually. That's great, but now I'm wondering how many engineers were paid for three months of full time work to save $1 million/year on AWS, and how much it costs the company to pay those engineers for 3 months. It's also interesting to wonder about how much growth your startup could have managed if it had had three additional months of work from those engineers.

grkvlt 10y ago
If you assume USD 250k per engineer, they would need to have had more than 16 engineers working on this for the 3 months to spend more than a million on salary. So, I'm guessing they probably came out ahead on that...?

bearjaws 10y ago

I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, clean up, and remove any data we don't need. We will not get the weeks of development effort back, we will continue to spend more money on hardware, and we will continue to suffer to delay projects because of poor choices. If we had AWS, we could have written an entire sprints worth of work (also ~$14k worth of IT time) and moved on with our lives. Clearly, some companies let the budget keep growing, but for most companies AWS flexibility is worth its weight in gold.

QuinnyPig 10y ago
Exactly. Neither path is easy to walk; it always comes down to trade-offs. What do you value?
ryeguy 10y ago
Why is it always colo vs cloud in these arguments? What about renting everything monthly from ovh, rackspace, softlayer etc? Many of them can provision dedicated servers within minutes, too.
- flukus 10y ago
  Also some combination of cloud and private servers. If I was starting up a Netflix competitor for instance, the site could probably run fine from owned hardware, but the content itself would be better off being cloud based.
  - ryeguy 10y ago
    Ironically netflix does the exact opposite. Also, you definitely don't want to have the content on the cloud - the data transfer pricing is where they make significant profit. Both AWS and GCE are ~$100/TB outgoing.

deleted 10y ago

[deleted]

dredmorbius 10y ago

Clickbait title.

williamscales 10y ago

Is it just me or is segment.com down?

nemothekid 10y ago

>The ALB allows ECS to set ports dynamically for individual containers, and then pack as many containers as can fit onto a given instance. Maybe I have a different view, but I've been avoiding running the same service on the same instance. If I'm running multiple services, I'm doing it for HA, and if an instance goes down, you could lose all your services. Why would you run the same service on the same instance?

pascalxus 10y ago

Umm... i take issue with the comment "distribution is essentially free". nonononnononooooo. Other than the rare case of virality, I'd like to know in what software industry where distribution is essentially free - i'm happy to be wrong about this, let me know. the rest of the article was fine.

ryanmarsh 10y ago

> blocking any new discovered badly behaved keys. Eventually we reduced capacity by 4x. I wonder how it felt when they discovered how much money they'd burned on such a dumb little bug. I'd feel like throwing up.

abalone 10y ago

> In some cases, we’re currently packing 100-200 containers per instance. I have a very basic "Docker 101" question here.. Can someone explain how you might get to >100 containers on a given instance? That seems to imply that there are multiple containers of the same microservice running on one machine. Why would you pack duplicate containers like that instead of using one larger container?

eastbayjake 10y ago
> Why would you pack duplicate containers like that instead of using one larger container? Microservices that are scaling horizontally to handle concurrent requests, etc. Depending on how seriously your architects take the "micro" in microservice, you may easily get to 100+ unique microservices/processes each running in their own Docker container.
- abalone 10y ago
  That's not what's going on here (100-200 unique microservices). Aside from the fact that that's insane, they specifically state they want to "pack" multiple containers of the same service on one machine: "You cannot run two containers of the same service on a single host because they will collide and attempt to listen on the same port. (no bin packing)" So, what's the sense in doing that instead of one bigger container?
  - biot 10y ago
    They're reimplementing the actor model using Docker containers. Let it crash.
bpicolo 10y ago
The goal is just having a compute pool where you don't have to care at all where things are actually running. It definitely depends on the use case, but I could imagine more, smaller units allowing for more granular (and therefore a bit more cost-effective) scaling.
- 10y ago

cowardlydragon 10y ago

It is my impression that AWS makes its bank on I/O, while everyone makes buying decisions on CPU/RAM

londons_explore 10y ago

If only they'd spent a couple more dollars on hosting for https://segment.com https://segment.com so it didn't return HTTP 503 errors...

ig1 10y ago

However they miss the easiest fix: Calling their account rep at AWS and cutting a deal. AWS loves startups that could end up being huge customers so they're willing to slash bills upfront to help you get to growth stage; not only will they assign you an account rep but they'll have a rep whose job it is to build a good relationship with your VC. Speak to your account rep. Have your VC speak to their Amazon rep. Push them to cut your prices. I've known startups which have cut their monthly bill by six figures going this route. AWS doesn't want your startup to fail and on the margin these resources cost them close to zero so they're willing to deal, make use of it.

brianwawok 10y ago
Why not both? Get the best deal and turn off unused servers.
rbranson 10y ago
That was definitely part of the effort; it just doesn't make for terribly compelling technical content. We have a healthy, active relationship with our AWS team. They worked with us to put in place minimum commits for several products, and examined both EDP as well as bulk RI purchase discounts.
- ig1 10y ago
  I'd say especially in your case you could be pushing for additional discounts beyond what's available to everyone else. You're the ideal case for this, not only will you be a large customer in your own right if you succeed but you have a multiplier effect in that your own clients become heavier users of AWS because of your product encouraging users to move large amounts of data from Salesforce, etc. into AWS. I don't what level the rep you're dealing with is, but if you're not doing so already I'd strongly encourage you to build higher level relationships with AWS.
  - rbranson 10y ago
    Some of the minimum commits aren't exactly standard. I'd talk more about them but they're covered by non-disclosure. I see what you mean by the multiplier effect increasing our bargaining position — totally makes sense.

dangoldin 10y ago

Really interesting read. We've been in a similar mindset and it really comes down to understanding what's driving your costs and being creative around ways to minimize it. As others on here mentioned bandwidth and data transfer is a huge cost so if there's a way to make your caching more aggressive it's a big and easy win. Beyond that the low hanging fruit is in using reserved instances - there are a few options there but are significantly cheaper than paying the on demand price. A shameless plug is I built a small visualization for the AWS detail log to help understand the cost drivers. All it is loading the data and visualizing it across a few dimensions but might come in handy for others - https://github.com/dangoldin/aws-billing-details-analysis https://github.com/dangoldin/aws-billing-details-analysis

qwerty2020 10y ago

how does a bar chart with no y-axis label show any trend? scale could be $100-101/mo

rumcajz 10y ago

Given that Elite game [1] used to run on Sinclair with Z80 processor and 48kB of memory, any complaints about the costs nowadays sound ridiculous. If our software infrastructure was coded by Elite programmers, we would be able to run in at 0.01% of the cost. [1] https://en.wikipedia.org/wiki/Elite_(video_game) https://en.wikipedia.org/wiki/Elite_(video_game)

jaggederest 10y ago
> It was written in machine code using assembly language, giving much care to maximum compactness of code. From the link. It seems unlikely that people would/could build applications in the modern world from scratch in assembler. Start with reimplementing the entire HTTP parsing engine and go from there? No thanks.

gregjw 10y ago

Dollar dollar bills y'all

marknadal 10y ago

Great article. I wish/hope/expect everybody to take this type of work/attention as seriously as they have. I am as much of a penny pincher as you can get, so our team built a database adapter on AWS that can handle 100M+ records for $10/day, demo here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . I'm surprised not more people are price/performance sensitive. My goal is to get a system up to 10K table inserts/second on a free Heroku instance, we're already doing on low end hardware 1750 table inserts/second across a federated system. But price tuning is one of the most important things when engineering for high load, as well. Glad to see articles like this. :)

stymaar 10y ago

I don't know if LBO funds are doing this already but I think it's most likely going to become a trend in the future : - buy a SaaS company which hosts all its infra on the cloud - move everything to bare metal - double the company's margin - profit

julienba 10y ago

DISCLAIMER: It can be viewed as an advertisement for my project Few months back I have start a company for that exact purpose : helping you decreasing you AWS bill. https://wizardly.eu/ https://wizardly.eu/ The focus is around reservation, unused resources and tracking cost evolution on a day to day basics. I don't get that much traction ( I'm not a marketer and it's my first SaaS project, still a lot to learn in that area). I'm looking for more users feedback, if you willing to invested few minutes in my project that would really helpful. It's also completely free now. As the article say, their is no silver bullet. Specially if you need infrastructure changed. But it's possible to have an interesting cut by using all the AWS pricing options. Making teams aware and responsable of their costs is also making a big differences. Often, people creating and managing infrastructure have no view of what it is costing for their company.

wazoox 10y ago

A friend of mine developed something to do just that: understand AWS/Google/Azure costs and compare prices (and save big). Shameless plug: https://trackit.io/ https://trackit.io/

kierenj 10y ago
Probably not the ideal response, sorry, but I only got as far as the homepage - really slow, jerky scrolling, scroll-to-anchors miss the mark, and the social media links don't work. Then, mouse wheel/trackpad scrolling stopped working. :/
- wazoox 10y ago
  THe website is probably hosted on AWS :) As for the social media links, they all work for me...