3 ms·

Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in para

by hank777 18y ago

Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.

nostrademons 18y ago
So, it's scalable, but the total number of writes/sec necessary is O(messages sent/sec * average number of recipients per message). I could see this working as long as the average number of recipients per message remains constant as the service grows, though it's still a lot of hardware. How much funding does Twitter have again? If the average number of recipients grows along with the number of users, then there's a problem, because average number of messages should be growing linearly with userbase, so the total hardware required is quadratic. No way that would work out economically...
- hank777 18y ago
  I think your math is off here. Basically you end up with this equation cost = ((NumOfUsers * avgMsgsRevceivedPerUser)/maxWritesPerCPU) * costOfCPU There is nothing quadratic here. Your are able to scale linearly with the number of messages received in the system. There is a fixed cost per message received which is exactly what you want. By the way, this is exactly how the internet mail system works and it is why it is so scalable.
  - nostrademons 18y ago
    It's NumOfMessages, not NumOfUsers, and Messages = Messages/User * Users. Then the question is whether Messages/User and Recipients/Message remain constant as NumOfUsers goes up. If both do, then your total cost is linear with the number of users, which is what you want. But if either of them is linear in the number of users, then the total number of messages becomes quadratic in the number of users: a linear factor from growth in userbase, and a linear factor factor from increased usage by each user. Most likely there's a logarithmic correlation (Recipients/Message goes up as more people are on the service, but not by as much as total users), which is still generally fine for scalability concerns. E-mail's actually a really great example. It worked great as long as it was used for person-to-person communication, which held both the number of messages/user and the number of recipients/message constant. Then spammers found out that the more people who had e-mail, the more they could send to at once, and sent recipients/message through the roof. In 1997/98, there were actually a number of articles about how the e-mail architecture was in serious trouble, and unless some way could be found to limit spam, it would collapse within a couple years. Of course, they dealt with it through spam filters and blacklists on the server, so now e-mail works fine. Twitter's problem is that both Messages/User and Recipients/Message average significantly higher than e-mail, and yet the resources they have available are significantly lower than the full resources of the Internet. So even if they don't have an architectural scalability problem, they could very well have an economics problem.
    - hank777 18y ago
      I don't understand your correction, but my lack of understanding is really immaterial. I actually agree that twitter has a problem with economics if they cant sell anything. Whether it is worth solving the scalability problem is a good question. Certainly if they keep growing with no revenue, at some point it is definitely an economic problem. Indeed I posted a while ago about the community taking over twitter as a distributed public service (like email). I do think that there is an upper limit to the amount of messages the average user will send.

by hank777 18y ago

nostrademons 18y ago

So, it's scalable, but the total number of writes/sec necessary is O(messages sent/sec * average number of recipients per message). I could see this working as long as the average number of recipients per message remains constant as the service grows, though it's still a lot of hardware. How much funding does Twitter have again? If the average number of recipients grows along with the number of users, then there's a problem, because average number of messages should be growing linearly with userbase, so the total hardware required is quadratic. No way that would work out economically...

hank777 18y ago
I think your math is off here. Basically you end up with this equation cost = ((NumOfUsers * avgMsgsRevceivedPerUser)/maxWritesPerCPU) * costOfCPU There is nothing quadratic here. Your are able to scale linearly with the number of messages received in the system. There is a fixed cost per message received which is exactly what you want. By the way, this is exactly how the internet mail system works and it is why it is so scalable.
- nostrademons 18y ago
  It's NumOfMessages, not NumOfUsers, and Messages = Messages/User * Users. Then the question is whether Messages/User and Recipients/Message remain constant as NumOfUsers goes up. If both do, then your total cost is linear with the number of users, which is what you want. But if either of them is linear in the number of users, then the total number of messages becomes quadratic in the number of users: a linear factor from growth in userbase, and a linear factor factor from increased usage by each user. Most likely there's a logarithmic correlation (Recipients/Message goes up as more people are on the service, but not by as much as total users), which is still generally fine for scalability concerns. E-mail's actually a really great example. It worked great as long as it was used for person-to-person communication, which held both the number of messages/user and the number of recipients/message constant. Then spammers found out that the more people who had e-mail, the more they could send to at once, and sent recipients/message through the roof. In 1997/98, there were actually a number of articles about how the e-mail architecture was in serious trouble, and unless some way could be found to limit spam, it would collapse within a couple years. Of course, they dealt with it through spam filters and blacklists on the server, so now e-mail works fine. Twitter's problem is that both Messages/User and Recipients/Message average significantly higher than e-mail, and yet the resources they have available are significantly lower than the full resources of the Internet. So even if they don't have an architectural scalability problem, they could very well have an economics problem.
  - hank777 18y ago
    I don't understand your correction, but my lack of understanding is really immaterial. I actually agree that twitter has a problem with economics if they cant sell anything. Whether it is worth solving the scalability problem is a good question. Certainly if they keep growing with no revenue, at some point it is definitely an economic problem. Indeed I posted a while ago about the community taking over twitter as a distributed public service (like email). I do think that there is an upper limit to the amount of messages the average user will send.