9 ms·
The optimizations are no doubt impressive, but looking at it from a costs perspective – they run their company on two DB instances, each with two CPU cores and
by paxys 3y ago
The optimizations are no doubt impressive, but looking at it from a costs perspective – they run their company on two DB instances, each with two CPU cores and 4 GB of ram. The annual cost of running such a machine on AWS (with SSD included) is ~$4000. Even if their DB infra was only half as efficient as it is today, their costs would increase by a whopping $8000/yr.
How much do they pay an average software developer? How many hours have developers spent on query monitoring, optimization, removing joins, removing aggregations? What is the ongoing cost of learning and applying these techniques to all new code written at the company? Even with the most conservative estimates I'm confident that it costs them several orders of magnitude more in developer productivity than they are saving on hardware.
Optimization is good, but every company needs to recognize where they are throwing away dollars to save cents. There is an optimal level of inefficiency that companies want to have.
- mcqueenjordan 3y agoHow did you arrive at $4000? I used https://calculator.aws/#/estimate https://calculator.aws/#/estimate and my number is over an order of magnitude less than $8000 for 2 instances with SSDs, 4GB RAM, and 2 cores.
- deleted 3y ago[deleted]
- wheaties 3y agoI/O, backups, and such. Just having the instances cost money. Using them costs even more!
- sgt 3y agoAnd egress at AWS.
- mcqueenjordan 3y agoDo you expect a database to do much egress?
- 0x6c6f6c 3y agoDepends on the queries and the application. Even just the case of sitemap access alone can very well read half a database's tables and many of each table's columns in the act. That being done dozens of times a week between various search vendors and yeah your egress from just SEO can exceed the size of your database alone. Just depends on the nature of your data. Then consider paging views, infinite scroll. Then toss in something like Algolia or other third-party tools ingesting your data. But it really boils down to your application. You might have NONE of these and egress is just crumbs.
- mcqueenjordan 3y agoEgress is defined as outbound traffic to the public internet. Unless your database is publicly accessible (which it really should not be), its egress should be zero. What your /application/ returns to the public internet, is a different conversation.
- SteveNuts 3y agoI've always thought there should be a meta "AWS Calculator" Calculator tool, which takes the output from the official AWS calculator and applies an algorithm to make it more realistic to factor in the things people forget about (mostly data transfer)
- fatherzine 3y agoHow much would they pay an average SRE to deal with random fires operating a poorly designed, poorly instrumented and poorly understood system?
- jezovuk 3y agoThis. Optimized system means the devs took the time to understand the system, know the (most frequent) usage patterns and will not be nagged by 'something is slow' problems every so often. Of course, as with everything, there is a danger of going overboard. (And industry is firmly headed in the 'just throw more people/hardware/AI at the problem' direction anyway).
- YetAnotherNick 3y agoReserved m6a.xlarge(4vcpu, 16 GB RAM) costs <$1000. Assuming most companies use reserved for things like DB. So yeah their savings is like less than $1000, at which point it feels more like a developer satisfaction project rather than having any ROI on money terms.
- _a_a_a_ 3y agoLink? ta
- rob742 3y agoThe balancing act between one-time developer hours and the recurring costs associated with scaling up database resources can be challenging. If sufficient funds are available, scaling up may seem like an easy decision. However, investing in the development of expertise in efficient database operations is usually more cost-effective in the long run. Additionally, acquiring the skills to debug queries and optimize database configurations can ultimately save on developer hours.
- mamcx 3y ago> How much do they pay an average software developer? Let's say that you pay a lot. Like you expend $20.000 USD doing this. But we forget the cost power of software. Your monthly savings extend to the infinity. Your dev is now freer to do other things that are precluded in a more complex scenario ("you save pennies here, but expend pounds elsewhere"), because complexity DEMANDS MORE ATTENTION. This is something I have observed all my life. I have never regretted to optime]ize for operational costs even if this means I "waste" a lot of time for this. I work in a more cutthroat environment than most startups (ERPs), where my time to deploy to production something is measured in hours, and is an environment that is filled by the graveyard of my competitors that have drink all the kool-aids. Complex operational environments ALWAYS have turned worse/more complex than do the simplest things. P.D: I think is similar to spending time in refactoring, cleanups, testing, security audits, training, and even resting. All of that sounds like "costs". But you are doing this stuff alongside, just need to budget a few hours a week on this and the profits come and come. Never regret using time on this stuff. NEED to be smart on this? Sure. But is pure muscle and after a while is great.
- nerdponx 3y agoCarefully designing your application around an underpowered database is almost surely more complicated than just paying for a more powerful database. We're not talking about K8s or sharding. We're talking about adding more cores and RAM to the single database node that they're already using. Imagine how much easier the backend dev's life would be if they could just write SELECT FROM JOIN WHERE like everyone else, and not have to pore over query plans and design/implement/test a bunch of custom joining routines. This path sounds like a high-effort artisanal approach to something where it doesn't seem warranted. It's penny-wise and pound-foolish in exactly the way you (very nicely) described.
- hobs 3y agoIt's true, but its great experience for future things when the costs actually matter - I cut my teeth at a biz that refused to pay more for database hardware and now my performance skills pay the bills at much nicer companies.
- renegade-otter 3y agoAnd now your system can scale better, saving you enough money to justify the optimization work long-term. The StackOverflow folks famously take their DB design and performance very seriously, and now they do not need an army of overpriced "architects" to plug the performance holes with more insane and expensive complexity. https://stackexchange.com/performance https://stackexchange.com/performance Coincidentally I just wrote up a post about the importance of minding your database skills: https://renegadeotter.com/2023/11/12/your-database-skills-are-not-good-to-have.html https://renegadeotter.com/2023/11/12/your-database-skills-ar...
- paxys 3y agoStack Overflow is the poster child of over-optimizing DB engineering while neglecting every other product feature that end users actually want, so not sure if that should be your example.
- renegade-otter 3y agoThere is a case to be made they took it to an extreme, but there is too much going in the other direction as well. It's not like most teams are working on the meat of the product anyway - they probably spend 40% of their time running a small project on Kubernetes and getting rid of daily security warnings from NPM.
- adrianN 3y agoSomehow they managed to become one of the most important websites for software developers despite neglecting every product feature you miss.
- paxys 3y agoThey have never made a dollar of profit in their 15 years of operation and just sold themselves to a private equity firm for pennies. "Most important website for software developers" doesn't really mean anything unless one can also be a sustainable company in the process.
- sharadov 3y agoAs someone who does DB optimization for a living - there is a tradeoff, I've seen far too many places where db instances run with default settings out of the box, queries missing basic indexes, no one even cared to run an explain analyze. I've seen far too experienced developers with no basic understanding of how relational databases operate. And it's gotten worse over the years - as they can keep throwing hardware at the problem. Performance optimization has become a lost art. This is as much a problem of negligence as much as it is of education. I won't even touch on proper design, which requires someone with significant database expertise from the get-go ( which most startups may not have).
- dkjaudyeqooe 3y ago> Performance optimization has become a lost art. It's really not very hard, basically add indexes wherever lookups are done but don't go overboard, especially with tables that see a lot of inserts. The DB will tell you what it's doing if you ask, it's all easily investigated. If your joins resist optimisation your data structure is probably off. > I won't even touch on proper design, which requires someone with significant database expertise from the get-go In more extreme cases this is true, but a few basic principles and an understanding of databases will get you 95% of the way there and is quite intuitive. Something as basic as "don't copy data around, each individual entity exists in one place, and use its key to refer to it otherwise" will help enormously.
- alex7734 3y ago> It's really not very hard, basically add indexes It's really not but you'd be surprised how many people don't know the basics of how indexes work. I've had people tell me that their queries couldn't be optimized further because "they had already added an index for every column of the table". It doesn't help that SQL, by design, hides the actual algorithm doing the data access from the users while simultaneously relying on them to add indexes to achieve performance, which is in my humble opinion the worst mistake of SQL.
- corytheboyd 3y ago
- qingcharles 3y agoYou can run this on Hetzner ARM hosts for about $30/year o_O
- jerf 3y agoThe last line from the article: "We must not forget that 20 years ago only a fraction of the performance of current hardware resources was available, and despite these restrictions, performance database-based applications were developed." I think you have a mental model in which these people are spending weeks and months optimizing a system they should just upgrade while the real tasks languish, but in my experience, probably all they are doing is just paying a bit of attention every so often and simply doing things that don't require 32 cores and terabytes of RAM. A lot of things don't need that power. 20 years ago you had a lot more real variance in power. If your 100MHz server couldn't handle it, maybe your 400MHz server could, and the gap between those two systems is probably even bigger than the numbers imply because that 400MHz was probably equipped with better everything else (RAM, disk, cache, better performance/cycle for the CPU) to go with it. Nowadays, with CPU speeds having stalled, it's easy to go wide and service lots of diverse queries at a time, but if you have a particular query that your lowest-end DB node is struggling with, there's a pretty decent chance that just pushing the button to go to a bigger node will have a distinctly sub-linear result on performance improvement. I've certainly seen this from a number of teams in a number of places. You can push a button to get more CPUs but you can't push a button to get much better CPUs. Some better, yes, but not like it used to be. If you just try a little, modern commodity systems can do a lot. Even with only 4 cores and 4GB. Just try a little. Yes, there are absolutely tasks that that doesn't work with; if you've got 10,000 hours of video to re-encode that system isn't going to cut it. But a lot of developers are really bad at understanding how long things should take and don't realize that the thing that takes two minutes during CI really should take something more like 50ms because https://accidentallyquadratic.tumblr.com/ https://accidentallyquadratic.tumblr.com/ and somebody really ought to look into that rather than just bump the CI node size again hoping it solves the problem. (And for Pete's sake, when bumping the instance size does happen to not solve the problem... turn it back down again! Despite what I just wrote I do understand there's a time and a place to spray money at the problem, but when spraying money at the problem doesn't even help, STOP.)
- jjav 3y ago> The optimizations are no doubt impressive The sad thing is that the things they discuss like monitoring and optimizing slow queries have always, until recently, been baseline competence. It's only recently that the universal answer to something being too slow is to simply spin up another dozen, or heck make it two dozen, cloud instances to compensate. > Even if their DB infra was only half as efficient as it is today, their costs would increase by a whopping $8000/yr. I think the interesting comparison is not so much the same team doing things slightly less efficiently, but comparing to the average status quo in the industry. So many companies are spending $100K+/month (month not year) on AWS costs to run very tiny minimal loads due to the attitude that efficiency never matters, just spin up more instances. Pretty soon you're talking real money. And by the time the CFO starts screaming, it's very difficult to go back to a more sane infrastructure.
- deleted 3y ago[deleted]
- sgarland 3y ago> How many hours have developers spent on query monitoring Install Percona Monitoring & Management. Clickhouse + Grafana all in one, dashboards ready to go, and free. Not hard. > optimization, removing joins, removing aggregations? How is this not part of the job already? Too many people seem to have it in their head that the database is an inconvenience that gets in the way of their application. It is the heart and soul of every application, because all applications boil down to CRUD. You should learn SQL, full stop. Relation algebra is not that difficult. The time people spending learning the newest stupid JS framework could instead be spent learning a language that hasn't changed in decades. It will serve you well.
- dzolob 3y agoThese optimizations should be done regardless of db power. You don’t need an extra eng to do them, you just do them. They are part of the job.
- SergeAx 3y ago> The annual cost of running such a machine on AWS (with SSD included) is ~$4000 Are you sure about this estimation? Because Digital Ocean offers 2vCPU + 8GB machines for $25 / month, which is more than 10 times cheaper.