5 ms·
I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit s
by latch 5y ago
I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget.
_How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing.
Off the top of my head, the "how" will often involve the following (just to list a few):
1 - Baremetal
2 - Cache
3 - Denormalize
4 - Append-only
5 - Shard
6 - Performance focused clients/api
7 - Async / background everything
These strategies work _really_ well for catalog-type systems: amazon.com, wiki, shopify, spotify, stackoverflow. The list is virtually endless.
But it doesn't take much more complexity for it to become more difficult/expensive.
Twitter's a good example. Forget twitter-scale, just imagine you've outgrown what 1 single DB server can do, how do you scale? You can't shard on the `author_id` because the hot path isn't "get all my tweets", the hot path is "get all the tweets of the people I follow". If you shard on `author_id`, you now need to visit N shards. To optimize the hot path, you need to duplicate tweets into each "recipient" shard so that you can do: "select tweet from tweets where recipient_id = $1 order by created desc limit 50". But this duplication is never going to be cheap (to compute or store).
(At twitter's scale, even though it's a simple graph, you have the case of people with millions of followers which probably need special handling. I assume this involves a server-side merge of "tweets from normal people" & RAM["tweets from the popular people"].)
- 29athrowaway 5y agoTry to convert as much content as you can into static content, and serve it via CDN. Then, use your servers only for dynamic stuff. Also, put the browser to work for you, caching via Cache-Control, ETag, etc. Only then, optimize your server...
- ignoramous 5y ago> Twitter's a good example. Mike Cvet's talk about Twitter's fan-in/fan-out problem and its solution makes for a fascinating watch: https://www.youtube-nocookie.com/embed/WEgCjwyXvwc https://www.youtube-nocookie.com/embed/WEgCjwyXvwc
- Cipater 5y agoI appreciate the no-cookie embed. Learned something new today.
- trampi 5y agoReads like a small excerpt out of "Designing Data-Intensive Applications" :)
- chairmanwow1 5y agoThis is an amazing book that improved my effectiveness as an engineer by an undefinable amount. Instead of just randomly picking components for a cloud application, I learned that I could pick the right tools for the job. This book does a really good job communicating the trade-offs between different designs and tools.
- ignoramous 5y agoI have always wondered "what next" after having read data-intensive. Some suggested looking at research publications by Google, Facebook, and Microsoft. What do others interested in the field read?
- kalev 5y agoThe 1-7 list you mention definitely deserves it’s own blogpost and how to implement these. I’m currently not using any of these except 1, and probably don’t need the rest for a while but I do want to know what I should do when I need it. For example: what and how should things be cached? When and how to denormalize, why is it needed? Why append-only and how? Never ‘sharded’ before, no idea how that works. Heard some things of everything async/in the background, but how would that work practically?
- BrentOzar 5y ago> what and how should things be cached? If something is read much more frequently than it changes, store it client-side, or store it temporarily in an in-memory-only, not-persisted-to-disk "persistence" layer like Redis. For example, if you're running an online store, your product list doesn't change all that often, but it's queried constantly. The single source of truth lives in a relational database, but when your app needs to fetch the list of products, it should first check the caching layer to see if it's available there. If not, fetch it from the database, but then write it into the cache so that it's available more quickly the next time you need it. > When and how to denormalize, why is it needed? When you need to join several tables together in order to retrieve a result set, and especially when you need to do grouping to get the result set, and the retrieval & grouping is presenting a performance problem, then pre-bake that data on a regular basis, flattening it out into a table optimized for read performance. Again with the online store example, let's say you want to show the 10 most popular products, with the average review score for each product. As your store grows and you have millions of reviews, you don't really want to calculate that data every time the web page renders. You would build a simpler table that just has the top 10 products, names, IDs, average rating, etc. Rendering the page becomes much more simple because you can just fetch that list from the table. If the average review counts are slightly out of date by a day or two, it doesn't really matter. > Why append-only and how? If you have a lot of users fighting over the same row, trying to update it, you can run into blocking problems. Consider just storing new versions of rows. But now we're starting to get into the much more challenging things that require big application code changes - that's why the grandparent post listed 'em in this order. If you do the first two things I cover above there, you can go a long, long, long way.
- 5y ago
- winrid 5y agoI've heard in a few talks how at Twitter engineers have accidentally ran into OOM problems by loading up too big of a follower graph in memory in application code. I think it's a nice reminder that at scale even big companies make the easy mistakes and you have to architect for them.
- Ginden 5y agoI would like to notice that many of these techniques can incur significant cost of developer or sysadmin time.