3 ms·
Ah yes. The horizontal scalability endgame. Amazon designed a system that is perfectly horizontally scalable. The only issue is that they now need 33,000 machi
by chmod775 6y ago
Ah yes. The horizontal scalability endgame.
Amazon designed a system that is perfectly horizontally scalable. The only issue is that they now need 33,000 machines to achieve the throughput of TimescaleDB running on a single node.
I would call the results this new school of software development produces horrifying, but I already get lynched every time I tell a startup that their horrifyingly complex horizontally 'scalable' AWS setup that costs them $10k a month could be replaced by a handful of ducktape running on a rPI.
The best part is how the scaling never really works and just adding a load balancer and a second rPI would've worked better.
"We didn't think we'd get that many users. I mean 10k! Whew! It just wasn't designed for this!"
"Oh. That sucks. My rPI is serving 2 million monthly visitors right now and I'm still waiting for CPU usage to dip into the double digits."
I'm not saying people should design their stuff to handle millions of users on underpowered hardware, but I am suggesting they not lose their sense of perspective while building their whatever.
Because if you do, you may end up thinking you did well when your DB maxes out at 500 inserts/second - Another DB running on a Nintendo DS might just outperform it.
- valyala 6y agoHorizontally scalable systems are usually slower than single-node solutions because of overhead on passing data between nodes. We at VictoriaMetrics clearly see this - cluster version of VictoriaMetrics needs up to 2x more CPU and RAM resources comparing to a single-node version for achieving the same performance. Single-node VictoriaMetrics provides almost perfect vertical scalability though. Its' performance scales almost linearly from rPI to a monster with hundreds of CPU cores and terabytes of RAM. See https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae https://valyala.medium.com/measuring-vertical-scalability-fo... and https://valyala.medium.com/billy-how-victoriametrics-deals-with-more-than-500-billion-rows-e82ff8f725da https://valyala.medium.com/billy-how-victoriametrics-deals-w... .