4 ms·

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and a

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization and all of a sudden none of that matters any more. Or you accept that there's a hard ceiling on what your system will ever be able to do. This is like building a video game that pushes a specific generation of console hardware to its limit - fantastic! You got it to do realtime shadows and 100 simultaneous NPCs on screen! But when the level designer asks if they can have water in one level you have to say 'no', there's no room to add screenspace reflections, the console can't handle that as well. And that's just a compromise you have to make, and ship the game with the best set of features you can cram into that specific hardware. You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model.

eismcc 4y ago
You’d end up synchronizing feature releases to Moores law. Which while it sounds untenable there are large corporations that continue to use monolithic approach and vertical scaling.
nyanpasu64 4y ago
Video games have had 3D water for decades before screen-space reflections, and many look serviceable decades later (Super Mario Sunshine looks great at 480p though dated at higher resolutions).
- jameshart 4y ago
  Curses. That undermines my argument completely. It wasn’t merely a random example of how you have to compromise on features to fit in the available hardware budget, but actually the premise upon which my entire argument rested.
joshspankit 4y ago
Weird criticism but o.k. Edit: Unless I missed something, the author never argued that Twitter should be hosted on one machine and therefore criticizing the “fun stunt” like this makes no sense to me
- jameshart 4y ago
  There are certainly people reading this and nodding and thinking ‘yeah, this makes sense! Why don't we build services like this?’ And adding it to their mental list of arguments against microservices or whatever - and I wanted to make sure people hear the reasons why this kind of performance maximization tends not to be the norm.
p-e-w 4y ago
> You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model. That "fundamental" thing is the cultural expectation that SaaS offerings constantly grow in features, rather than in reliability or performance. As your example from the world of video games demonstrates, there is no industry-wide belief that things must be able to do ever-more, forever. It's really mostly SaaS and desktop software that has this weird and unreasonable culture around it. That's why your word processor can now send emails, and your email provider now does translations as well.
- nkellenicki 4y ago
  You're not taking into account data, you're only talking about features. What about when the data no longer fits on the one machine? Or processing the data exceeds the capacity of the machine? Data growth through user growth or just normal day-to-day usage is expected.
  - p-e-w 4y ago
    If Twitter's data can fit on one machine, then the data of 99.99% of companies can. Not every product needs a billion users with Gigabytes of storage each. The assumption that if your startup's tech isn't scalable enough to become the next Google then it's the wrong tech is hilarious nonsense driven largely by ego fantasies.
    - mapme 4y ago
      Twitters data cannot fit on one machine. In 2015 their Hadoop cluster was 30 PB per earlier comments/their blog. How do you fit that on one machine?
    - nkellenicki 4y ago
      It does not fit on one machine. Tweets alone generate petabytes of data a year, and other events are petabytes per day. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2021/processing-billions-of-events-in-real-time-at-twitter- https://blog.twitter.com/engineering/en_us/topics/infrastruc... https://ankush-chavan.medium.com/twitter-data-storage-and-processing-dd13fd0fdb30 https://ankush-chavan.medium.com/twitter-data-storage-and-pr...
seanhunter 4y ago
It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux became dominant primarily because horizontal scaling won that argument and it won it pretty much exactly because of: 1)what you said - you hit a hard cap on how big you can make your main server at which point you are really screwed. Scalability pain points become a hard wall. 2) when I say "server" I mean "servers" of course because you'd need an H/A failover, at which point you've eaten the cost of replication, handling failover etc and you may as well distribute 3) cost. Because hardware cost vs capability is exponential, as your requirements become bigger you pretty rapidly hit a point where lots of commodity hardware becomes cheaper for a given performance point than few big servers So there's a reason that distributed systems on commodity hardware became the dominant architectural paradigm. It's not the only way to do it, but it's a reasonable default for many use cases. For a very high-throughput system like twitter it seems a very obvious choice. Clearly there are costs to distribution, so if you can get away with a simpler architecture then as always Occam's razor applies. Also if you can easily distribute later then it probably makes sense to leave that option open and explore it when you need it rather than overcomplicate too early.
- Joeri 4y ago
  The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost. I’m always reminded of how stackoverflow essentially runs off a single database server. If they can do it, most web properties can do it.
  - saagarjha 4y ago
    Stack Overflow happens to go down fair maintenance a fair bit. Now, it’s not really a service that might need reliability like Twitter does*, but it’s important to keep in mind. *Some engineers may disagree
Winsaucerer 4y ago
How about smaller projects that might be hugely successful, but will never be remotely close to a Twitter level of success? It’s interesting to see how much can be done with a single machine, because most projects will never be this big. Though there will still be other concerns like redundancy to deal with.
- Ingaz 4y ago
  Can you explain me what exactly meant by "success of Twitter"? It's not sarcasm, I have twitter account but I never understood hype about twitter. I see nothing in twitter from technical POW, closed twitter protocol looks very strange, they banned Trump, they were profitable in 2021, Elon Musk bought them for ~44 bln. Maybe sellout of company with problems for such price is success.
deleted 4y ago
[deleted]
vidarh 4y ago
Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore. Of course they could fit a much larger dataset on one machine today. (But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)
- gnuvince 4y ago
  There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy and avoid the latest non-proven fads, we can surely write software that is 10x faster and requires much less scaling. [1] https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s
  - kaba0 4y ago
    Twitter used to (not sure how much did that idiot fck it up) be quite great from this perspective from what I gathered. They had technical blog posts and are even featured in the famous data intensive book as an example for many scaling problems. Sure, they didn’t write it in assembly but instead use Java/Scala with Graal, but architecturally they had a sound system (plus what people routinely forget, the system should handle the general load, but it worths nothing if it fails at peaks, and twitter can easily hit a billion daily users during big global events)z
  - vidarh 4y ago
    Oh, absolutely. I think in terms of making people think about putting things in-process in RAM and being a bit more imaginative at looking at machines sizes as worth exploring is a good thing, and the article is interesting as a look at what you can do. Many things can plausibly fit on a single machine irrespective of uptake (e.g. I worked on a system not long ago where even if we cornered the entire global market and the market expanded several magnitudes in size, our entire working set would still fit comfortable in memory). But even when you need to scale, it absolutely can scale better if you're willing to not automatically resort to a standard database stack for everything for example.
Ingaz 4y ago
I think that the main point of OP is that it's possible to serve production load using just one server. I did not looked into source code yet but I suppose that OP if not implemented already than there should ideas for implementation. In addition: from my POV implementation of scaling for such service should be trivial: - sharding of data between instances by a criteria (e.g. regional) or by hash - configure network routing I think it should work
vidarh 4y ago
> And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization and all of a sudden none of that matters any more. Indeed, with the hyperoptimized version here, the moment you tip over into two machines each machine will need two copies of every tweet from anyone who has followers sharded to both machines, so the capacity of two machines is going to be far less than twice the capacity of one as a large proportion of tweets will cause writes on both shards. This inefficiency will now always be with you - the average number of writes per user per tweet will go up until your number of shards approaches average follower counts. This is why it's common to model this with fan-out on write, because the moment you accept that there is a risk you'll tip over into a sharded model you need to account for that. If asked the question of such a design, it's worth pointing out that if you can guarantee it fits on one machine, and this is true for many more problems than people expect, then you can save a lot, but then I'd set out the more complex model and contrast it to the single-machine model. You don't need to fan-out to every account even in such a distributed system, certainly. You can fan-out to every shard/instance, and keeping that cached in RAM would still allow you to be far more efficient than e.g. Mastodon (which does fan-out to every instance for the actual post data, but relies on a Postgres database)

eismcc 4y ago

You’d end up synchronizing feature releases to Moores law. Which while it sounds untenable there are large corporations that continue to use monolithic approach and vertical scaling.

nyanpasu64 4y ago

Video games have had 3D water for decades before screen-space reflections, and many look serviceable decades later (Super Mario Sunshine looks great at 480p though dated at higher resolutions).

jameshart 4y ago
Curses. That undermines my argument completely. It wasn’t merely a random example of how you have to compromise on features to fit in the available hardware budget, but actually the premise upon which my entire argument rested.

joshspankit 4y ago

Weird criticism but o.k. Edit: Unless I missed something, the author never argued that Twitter should be hosted on one machine and therefore criticizing the “fun stunt” like this makes no sense to me

jameshart 4y ago
There are certainly people reading this and nodding and thinking ‘yeah, this makes sense! Why don't we build services like this?’ And adding it to their mental list of arguments against microservices or whatever - and I wanted to make sure people hear the reasons why this kind of performance maximization tends not to be the norm.

p-e-w 4y ago

> You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model. That "fundamental" thing is the cultural expectation that SaaS offerings constantly grow in features, rather than in reliability or performance. As your example from the world of video games demonstrates, there is no industry-wide belief that things must be able to do ever-more, forever. It's really mostly SaaS and desktop software that has this weird and unreasonable culture around it. That's why your word processor can now send emails, and your email provider now does translations as well.

nkellenicki 4y ago
You're not taking into account data, you're only talking about features. What about when the data no longer fits on the one machine? Or processing the data exceeds the capacity of the machine? Data growth through user growth or just normal day-to-day usage is expected.
- p-e-w 4y ago
  If Twitter's data can fit on one machine, then the data of 99.99% of companies can. Not every product needs a billion users with Gigabytes of storage each. The assumption that if your startup's tech isn't scalable enough to become the next Google then it's the wrong tech is hilarious nonsense driven largely by ego fantasies.
  - mapme 4y ago
    Twitters data cannot fit on one machine. In 2015 their Hadoop cluster was 30 PB per earlier comments/their blog. How do you fit that on one machine?
  - nkellenicki 4y ago
    It does not fit on one machine. Tweets alone generate petabytes of data a year, and other events are petabytes per day. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2021/processing-billions-of-events-in-real-time-at-twitter- https://blog.twitter.com/engineering/en_us/topics/infrastruc... https://ankush-chavan.medium.com/twitter-data-storage-and-processing-dd13fd0fdb30 https://ankush-chavan.medium.com/twitter-data-storage-and-pr...

seanhunter 4y ago

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux became dominant primarily because horizontal scaling won that argument and it won it pretty much exactly because of: 1)what you said - you hit a hard cap on how big you can make your main server at which point you are really screwed. Scalability pain points become a hard wall. 2) when I say "server" I mean "servers" of course because you'd need an H/A failover, at which point you've eaten the cost of replication, handling failover etc and you may as well distribute 3) cost. Because hardware cost vs capability is exponential, as your requirements become bigger you pretty rapidly hit a point where lots of commodity hardware becomes cheaper for a given performance point than few big servers So there's a reason that distributed systems on commodity hardware became the dominant architectural paradigm. It's not the only way to do it, but it's a reasonable default for many use cases. For a very high-throughput system like twitter it seems a very obvious choice. Clearly there are costs to distribution, so if you can get away with a simpler architecture then as always Occam's razor applies. Also if you can easily distribute later then it probably makes sense to leave that option open and explore it when you need it rather than overcomplicate too early.

Joeri 4y ago
The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost. I’m always reminded of how stackoverflow essentially runs off a single database server. If they can do it, most web properties can do it.
- saagarjha 4y ago
  Stack Overflow happens to go down fair maintenance a fair bit. Now, it’s not really a service that might need reliability like Twitter does*, but it’s important to keep in mind. *Some engineers may disagree

Winsaucerer 4y ago

How about smaller projects that might be hugely successful, but will never be remotely close to a Twitter level of success? It’s interesting to see how much can be done with a single machine, because most projects will never be this big. Though there will still be other concerns like redundancy to deal with.

Ingaz 4y ago
Can you explain me what exactly meant by "success of Twitter"? It's not sarcasm, I have twitter account but I never understood hype about twitter. I see nothing in twitter from technical POW, closed twitter protocol looks very strange, they banned Trump, they were profitable in 2021, Elon Musk bought them for ~44 bln. Maybe sellout of company with problems for such price is success.

deleted 4y ago

[deleted]

vidarh 4y ago

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore. Of course they could fit a much larger dataset on one machine today. (But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

gnuvince 4y ago
There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy and avoid the latest non-proven fads, we can surely write software that is 10x faster and requires much less scaling. [1] https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s
- kaba0 4y ago
  Twitter used to (not sure how much did that idiot fck it up) be quite great from this perspective from what I gathered. They had technical blog posts and are even featured in the famous data intensive book as an example for many scaling problems. Sure, they didn’t write it in assembly but instead use Java/Scala with Graal, but architecturally they had a sound system (plus what people routinely forget, the system should handle the general load, but it worths nothing if it fails at peaks, and twitter can easily hit a billion daily users during big global events)z
- vidarh 4y ago
  Oh, absolutely. I think in terms of making people think about putting things in-process in RAM and being a bit more imaginative at looking at machines sizes as worth exploring is a good thing, and the article is interesting as a look at what you can do. Many things can plausibly fit on a single machine irrespective of uptake (e.g. I worked on a system not long ago where even if we cornered the entire global market and the market expanded several magnitudes in size, our entire working set would still fit comfortable in memory). But even when you need to scale, it absolutely can scale better if you're willing to not automatically resort to a standard database stack for everything for example.

Ingaz 4y ago

I think that the main point of OP is that it's possible to serve production load using just one server. I did not looked into source code yet but I suppose that OP if not implemented already than there should ideas for implementation. In addition: from my POV implementation of scaling for such service should be trivial: - sharding of data between instances by a criteria (e.g. regional) or by hash - configure network routing I think it should work

vidarh 4y ago

> And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization and all of a sudden none of that matters any more. Indeed, with the hyperoptimized version here, the moment you tip over into two machines each machine will need two copies of every tweet from anyone who has followers sharded to both machines, so the capacity of two machines is going to be far less than twice the capacity of one as a large proportion of tweets will cause writes on both shards. This inefficiency will now always be with you - the average number of writes per user per tweet will go up until your number of shards approaches average follower counts. This is why it's common to model this with fan-out on write, because the moment you accept that there is a risk you'll tip over into a sharded model you need to account for that. If asked the question of such a design, it's worth pointing out that if you can guarantee it fits on one machine, and this is true for many more problems than people expect, then you can save a lot, but then I'd set out the more complex model and contrast it to the single-machine model. You don't need to fan-out to every account even in such a distributed system, certainly. You can fan-out to every shard/instance, and keeping that cached in RAM would still allow you to be far more efficient than e.g. Mastodon (which does fan-out to every instance for the actual post data, but relies on a Postgres database)