6 ms·
Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you thi
by justinlloyd 3y ago
Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you think about a problem, suddenly you've got a lot left to get.
I spent two or so months optimizing the crap out of a majestic monolith and went from under 2K RPS when the PM thought, and the team repeatedly reported, that everything had been squeezed as much as it could, then changing the hardware, which got us to less than 3200 RPS, then to 4K RPS after just a few days of tinkering, to 10K RPS with a bit more effort, to 40K RPS a week or so later. "Oh that's, enough, we don't need to go further." I then changed "quite a bit of stuff" which then jumped us to 2M+ RPS, and then a month later, a consistent 40M+ RPS with low latency on a single box and there is still some juice left in the box should we want to go a little harder.
Right now we're not even touching 5% of the capacity of what we can pull from, it was that much of a change, simply by changing how we think about the problems. Moving from the old server to the new server let us jump from around 1800 RPS to a hair over 3000 RPS. Adding more hardware didn't fix our underlying problems. Adding more complexity was just punting the problem down the road. But changing how to think about the problem? _That_ changed the problem. And changed our answer to the problem.
- smarkov 3y agoVery curious to learn more about what the monolith was doing so incredibly poorly that you managed to squeeze that much performance out of it. Poorly written queries? Too many queries? Lack of any caching? Doing things synchronously when they could've been done concurrently?
- porridgeandrice 3y ago+1 I'd like to know as well
- justinlloyd 3y agohttps://justinlloyd.li/blog/how-much-cache-you-got-on-you/ https://justinlloyd.li/blog/how-much-cache-you-got-on-you/
- justinlloyd 3y agoSome of that, some other bad practices. Lots of low-hanging fruit, then more esoteric changes. https://justinlloyd.li/blog/how-much-cache-you-got-on-you/ https://justinlloyd.li/blog/how-much-cache-you-got-on-you/
- zepolen 3y agoHe cached everything and delayed writes too. It's easy to make a system fast when it's not realtime.
- justinlloyd 3y agoIncredibly dismissive of somebody's work, aren't ya? I regret breaking my own self-imposed rule of never answering follow-up questions on HN because there's always somebody willing to hand-wave away six months of my life and 40 years of real-world experience with a flippant comment of "oh, but that's easy if you don't have too..."
- zepolen 3y agoFragile ego too.
- justinlloyd 3y agoAnd you sir, are fucking toxic, just looking at your comment history I can tell that. So you calling me "fragile" carries about as much weight as the next idiot with an opinion.
- zepolen 3y agoToxicity is in the eye of the beholder.
- deleted 3y ago[deleted]
- high_priest 3y agoQuick google yields very good examples in huge improvements on a single algorithm level: https://youtu.be/c33AZBnRHks https://youtu.be/c33AZBnRHks Easy to imagine how even smaller improvements across 10-100 steps of data processing, can become even better.
- therealdrag0 3y agoYou gotta share more lol. These are insane gains.
- justinlloyd 3y agoI cannot go too deep. It was some client work. https://justinlloyd.li/blog/how-much-cache-you-got-on-you/ https://justinlloyd.li/blog/how-much-cache-you-got-on-you/
- therealdrag0 3y agoThank you! Even that was insightful.
- dist1ll 3y agoCould you share what's the content/protocol of these requests? And what's the average payload size (for req and response each)?