10 ms·

The Infrastructure Behind Twitter: Scale

348 points by kungfudoi 10y ago

rollulus 10y ago
It strikes me that so much of the components they use (e.g. under "Storage") are in-house built (several dbs, blob store, caches, etc). Is that because at that time equivalent solutions didn't exist? Is that because Twitter suffers from NIH?
- daveFNbuck 10y ago
  Probably a bit of both. I know the team I was on when I worked there was fairly proud of duplicating effort implementing things that other teams had already done. But Twitter does tend to build really good stuff.
- fizx 10y ago
  I built custom storage for Twitter back in the 2010-12 period. There wasn't much off the shelf in those days that worked out-of-the-box at scale besides Cassandra, and Twitter had a well-documented attempt at using Cassandra for primary workloads that failed due to the amount of variance in IO and latency for high-volume workloads. Most of the time, we were building custom distribution layers on top of open source storage (e.g. memcached/mysql/redis/etc). I think blobstore was the first thing twitter put in production that was mostly custom, followed a year or two later by manhattan. I'm not sure if there's even now good open-source competitors for those projects, largely because any reasonable smaller company uses s3 or dynamo. There's plenty of open-source things twitter created, or nurtured out of the existing ecosystem, from mesos to memcached to some of the hadoop/scalding/parquet stuff.
  - seanp2k2 10y ago
    The flip side of that imo is that if you don't have Twitter-scale needs for the specific things they've optimized their infrastructure for, you probably don't need their solutions :)
  - TheAceOfHearts 10y ago
    I haven't used it, but one tool I've read good things about Minio [0]. I don't know if it's able to match twitter-scale, though. https://minio.io/ https://minio.io/
- throwawasiudy 10y ago
  Sounds like it to me. Storing a crap ton of 300 byte messages is pretty common. Thousands of companies have been doing it for a decade. Anyone that does analytics or email probably stores far more of these than twitter. Blob store is perhaps the only forgivable custom solution. Besides the eventually expensive S3 you pretty much have to roll your own large binary storage at that scale. For regular DB sized workloads 1-16k bytes they had hundreds of options to choose from. Same with caching. They're both relatively solved problems.
- thinkingfish 10y ago
  In general, Twitter infrastructure didn't start by building solutions in house. Like most early stage companies, it started by using whatever it can find in the OSS repertoire. But a lot of problems that are non-existing at a smaller scale or at least tolerable become more acute when you scale up. This is when the developers face a decision: to improve the existing solution, or develop a new one? There are many factors that impact the final decision. Maturity of the technology? Community support/inclusion? How big is the necessary change? What scale can existing solution handle by design? How does in-house talent compare to those maintaining the original? What's the focus/need of the in-house use cases compared to the broader objective of the OSS solution? It should not be a surprise that the answer differs from project to project. But if the decision is to use the existing solutions, it probably won't raise much eyebrow/question. It is very common for larger-scale operations to write up their own solutions. Because a different scale can change the nature of the problem fundamentally. Google, Facebook and whoever has production fleet above the 10K range tend to build a lot of in-house solutions, often after similar considerations as mentioned above.
atcole 10y ago
This is a fairly technical analysis, and the terminology used in many cases is above what I know about networking. But the one quote that will stick is this. "There is no such a thing as a temporary change or workaround: In most cases, workarounds are tech debt."
- jlgaddis 10y ago
  I've been telling cow-orkers this for a long time. They're finally starting to realize that it's true.
- kristianc 10y ago
  Tech debt is a choice. Sometimes you'll want to embrace some level of technical debt in order to bring something into production quickly, with the understanding that you will fix later. It's part of a triad with Speed and Quality. That may not be the choice Twitter has made in this instance, but it's a viable choice nonetheless. Defaulting to tech debt = evil wrong imo.
  - cel1ne 10y ago
    And sometime a workaround is the best solution you'll get. Because solving the problem properly might introduce new issues, which might require new workarounds.
    - paulddraper 10y ago
      Exactly. The operative part of "workaround" is "work".
  - dingaling 10y ago
    The problem with technical debt versus financial debt is that the latter has a monthly mandatory cost ( interest payments ) that can't be ignored and which is visible all the way up to the C-level, whereas middle-management can keep obscuring the presence of technical debt and pushing its repayment out to the right. Essentially it's a 'free' internal debt, regardless of how often architects and developers complain of its cost. Thus in a contest between doing something right, but expensively, versus good-enough-for-now but technically-constrained the latter will usually win.
jzl 10y ago
Lots of CIO-type buzzwords and acronyms in this, including many coined by Twitter themselves. More worthy of a skim than a rigorous reading. But interesting for a bird's eye view of how complex Twitter is behind the scenes.
mnutt 10y ago
They mention that they used to base their geo routing on the location of the client's DNS server but have moved to BGP Anycast. I've heard that there can potentially be routing issues for long-running connections using anycast to end users, is anybody else doing something like this and do these issues happen in practice?
- smueller1234 10y ago
  As far as I know, using anycast for geo routing is pretty standard. Eg. I'm not aware that my employer (large ecommerce) has had any routing issues. This being said, we mostly avoid long running connections today. With more messenger like products becoming ubiquitous, I expect that to change though.
- pyvpx 10y ago
  TCP Anycast - Don’t believe the FUD Matt Levine (CacheNetworks), Barrett Lyon (BitGravity), Todd Underwood (Renesys) Operational experience with TCP and Anycast https://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf https://www.nanog.org/meetings/nanog37/presentations/matt.le...
seanmccann 10y ago
> Fast forward a few years and we were running a network with POPs on five continents and data centers with hundreds of thousands of servers. That seems high given Twitter's size and the hardware distribution pie chart they showed. Does anybody have an idea how this compares?
- smueller1234 10y ago
  I'm working for a very successful Internet company with 9 figure user count. We have 7 or 8 POPs on 4 continents and run a low 5 digit number of servers. This being said: we do not have to deal with anything like the network effect and lifestyle type usage that twitter has. Once you pass a certain size and complexity of infrastructure it happens really easily that economy of scale doesn't help and the complexity of everything having to be distributed works against your efficiency big time. So I'm not shocked by their numbers.
- easytiger 10y ago
  Web companies think 12k transactions per node is fine. It isn't fine. They just have an idiotic funding ans scaling model
tabeth 10y ago
I know nothing about storage, so I'm a bit confused about why Twitter needed: 1. Hadoop 2. Graph 3. Redis/Memcache 4. Blobstore 5. SQL variants (and a few others). I do see that the post has a short snippet briefly describing what they're storing, but I'd be curious to know why (speed, cost, latency, space tradeoffs/constraints). Also, if any more experienced folks want to chime in: Elixir/Erlang is "built for concurrency" as they say. I'd love to hear people's opinions on their one sentence simplifications of what kind of situation Hadoop/SQL/Redis/etc should be used for (similar to how Erlang is best used for situations where concurrency and fault tolerance is desired). In particular, is there a "Code Complete" type book for storage?
- redler 10y ago
  You might try "Seven Databases in Seven Weeks". It's a few years old now but still solid.
- timClicks 10y ago
  Erlang was behind the NoSQL database that started the NoSQL hype train (CouchDB), a document database. The Erlang distribution contains an amazing distributed database called Mnesia. For books, read the AOSA Chapter on Riak (http://www.aosabook.org/en/riak.html http://www.aosabook.org/en/riak.html) and then read through the tutorials on Riak Core. Oh, and Erlang in Anger https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf
  - rozap 10y ago
    Mnesia is a joy to use, but keep mind it was written before the CAP theorem existed, so it doesn't behave that well in the face of network partitions.
- marcinzm 10y ago
  That seems a pretty standard stack, I don't know about twitter but I can say why I'd use them in my experience: 5. SQL variants: This is where you keep your source of truth data needed to serve request. It's flexible, reliable but moderately fast. The backbone of your infrastructure and you can consider everything else as enhancements to it to overcome specific limitations. 4. Blobstore: Your SQL database is not going to be good at storing large binary blobs such as files or images. 3. Redis/Memcache: Your SQL database may be too slow for some requests and/or it's data model may require too much work in the API layer to align with the final presentation model. So you want a blazing fast store that you can use for that final presentation view when requests don't differ from each other. 2. Graph: SQL databases aren't the best for graph queries or data (I mean you can hack it but it's going to be slower and uglier code wise) so if you need to look at data as a graph something dedicated makes sense. 1. Hadoop: You're getting lot's of log data from users, what they do, when they do it and so on. You want to analyze this data, build models (who to recommend for people to follow for example) and use it to probably serve ads (gotta pay the bills somehow). You could put this in SQL but it's a lot of data and you'll be doing a lot of read requests which will end up being slow and messy in SQL. Hadoop, on the other hand, is well suited for this type of data and workload. Large slow read heavy queries on data that is appended to but not modified in place.
therealmarv 10y ago
No Ansible? In an older (2014) Ansible video they claim Twitter is using Ansible but I only see Puppet mentioned.
- rco8786 10y ago
  Only ever used puppet during my time there, fwiw
marknadal 10y ago
I've talked/pitched (full disclosure: got a "no") some of the bigger names behind the product at Twitter. I was a little disappointed because they seem to be proud of their at least $15M+/month server costs which is partly driving their company into the ground (the user facing product hasn't improved despite "re-engineering" of their backend, at non-substantial price differences, and lack of innovation for consumers have made them lose everyone to Snapchat or anti-censorship sites). Of tweets alone (about 200bytes per tweet), over the last decade, they probably have about 3 petabytes. Unknown to them (because of the aforementioned pride) they have 1.5 petabytes a month of free storage/caching they aren't even touching. If they switched to a P2P model like IPFS or (full disclosure: I work here) http://gunDB.io/ http://gunDB.io/ , but Twitter seems determined to stay as a centralized monolith. Which is too bad, because that has now become their own death - the regime change is happening and decentralized services will win instead. Edit: Compare against 100M+ messages (1000bytes each) for $10 a day. 2 min screencast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . Even if you multiplied the feature set by 10K times, you would still be saving $12M+ a month. At this rate the Discord guys (pushing 120M+ messages a day) are doing way better - there post is on top of HN right now too, see my comment there as well. And they only have 4 backend engineers.
- jhoechtl 10y ago
  Why the downvote? The post contains valuable architectural links worth a discussion
  - rapsey 10y ago
    1. Because it's a .js database. 2. Because the website does a poor job explaining what problem it solves.
    - deleted 10y ago
      [deleted]
    - eduren 10y ago
      Agreed, their website definitely turned me off. Quick tip for them: Having a section for showing that a bunch of people are using and "Trusting" the product is good, but your's seems to mostly (4/7 from what I can see) be the investors behind your product. Edit: Looked at the names more, and it seems 7/7 are individual investors or VCs https://angel.co/gun https://angel.co/gun Which is all well and good, but maybe the wording should be "Investors and Backers" rather than "Trusted By".
burgreblast 10y ago
100,000's of servers for 100,000,000 of messages/day ? I understand that half the servers aren't even doing messages, but, isn't WhatsApp doing 2 orders of magnitude more messages with 3 orders of magnitude (?) fewer servers? Is that right? I'm curious how one would justify 10,000X worse? So for each message, 10,000X more equipment is needed?
- kalleboo 10y ago
  In WhatsApp, a typical message goes to 1 other person. On Twitter, it can go to millions of people. When Twitter initially got their failwhaling under control, I recall reading they solved it by changing from a relational "join in and merge the timelines of everyone you're following on each refresh" model to a messagebox model. If that's true, maybe that naive model is now showing its limitations (I doubt they stopped there though, it seems like they have things under control)
- nrjdhsbsid 10y ago
  That's only 1000 messages per second on average. A single database +app server could handle that load. Assuming a bunch of other stuff is happening 500 servers sounds generous. Wtf are they doing that each server can only handles one tweet every two minutes?
  - burgreblast 10y ago
    Actually, it's 1000-9000 messages per server per day. Or about 1 message every 10-100 seconds. Of course, that's just the new messages inbound. They may need to distribute that single message to 100M people (who likely won't even see it, but still.) Problems that are trivally solvable with one database don't simply scale by adding more DBs or machines. Scaling isn't easy or they would have done it. I'm in no way disparaging their team, because I don't know what kind of constraints they had getting to this point. Still, I'd bet it could be optimized by 2+ of orders of magnitude if people sat down and re-evaluated the whole structure again at this point in time. Regardless, is that really a priority? They may have bigger issues on their plate now (growing revenue, growing users, making users happy). Assuming their business can generate the cash flow to overcome the inefficiencies, they may be better served to focus on growth.
niftich 10y ago
The day before, Discord did high-quality a write-up on why they chose Cassandra [1], and now this post hits explaining how one of the world's most popular and trafficked service has engineered their infrastructure; it's like a dream. I'll echo the praise I wrote earlier, that insights like this aren't only some of the best content to hit HN, but become some of the most valuable resources for designers who have yet to face a scaling issue, but know they will soon. Since you have developed custom layers on top of open-source software to fit your particular usecase and load profile, and host all this in-house, have you considered monetizing your infrastructure for outsiders who may have similar needs? Today, one has limited, unpleasant choices: either pay out the nose for something like AWS or Google Cloud to get elastic scaling and the captive storage systems that can be made to handle these kinds of workloads and still have to write a fair bit of custom glue to get all pieces to play nice, or you can build out the servers yourself, but have to employ dedicated talent with the requisite expertise. Either way, the barriers are fairly steep; you could tap into an under-served market should you choose to sell IaaS (edit: or, more accurately, PaaS). Has this conversation come up in the past? [1] https://news.ycombinator.com/item?id=13439725 https://news.ycombinator.com/item?id=13439725
turbohz 10y ago
If Twitter can manage to operate at profit, then I might be interested in Twitter's infrastructure.
ashayh 10y ago
| We have over 100 committers per month, over 500 modules, and over 1,000 roles. | we were able to reduce our average Puppet runtimes on our Mesos clusters from well over 30 minutes to under 5 minutes. This isn't just tech debt .. it's poorly designed, poorly thought, poorly architect-ed and poorly managed in the first place. Is it because Twitter cannot find good talent because of its falling stock?
hueving 10y ago
They switched to a BGP anycast model for twitter.com, which implies TCP. I'm curious how they deal with situations where route preferences change in intermediary ISPs mid-TCP stream. Does the new server reject the TCP connection or are they synchronizing TCP sessions across clusters?

348 points by kungfudoi 10y ago

It strikes me that so much of the components they use (e.g. under "Storage") are in-house built (several dbs, blob store, caches, etc). Is that because at that time equivalent solutions didn't exist? Is that because Twitter suffers from NIH?

daveFNbuck 10y ago
Probably a bit of both. I know the team I was on when I worked there was fairly proud of duplicating effort implementing things that other teams had already done. But Twitter does tend to build really good stuff.
fizx 10y ago
I built custom storage for Twitter back in the 2010-12 period. There wasn't much off the shelf in those days that worked out-of-the-box at scale besides Cassandra, and Twitter had a well-documented attempt at using Cassandra for primary workloads that failed due to the amount of variance in IO and latency for high-volume workloads. Most of the time, we were building custom distribution layers on top of open source storage (e.g. memcached/mysql/redis/etc). I think blobstore was the first thing twitter put in production that was mostly custom, followed a year or two later by manhattan. I'm not sure if there's even now good open-source competitors for those projects, largely because any reasonable smaller company uses s3 or dynamo. There's plenty of open-source things twitter created, or nurtured out of the existing ecosystem, from mesos to memcached to some of the hadoop/scalding/parquet stuff.
- seanp2k2 10y ago
  The flip side of that imo is that if you don't have Twitter-scale needs for the specific things they've optimized their infrastructure for, you probably don't need their solutions :)
- TheAceOfHearts 10y ago
  I haven't used it, but one tool I've read good things about Minio [0]. I don't know if it's able to match twitter-scale, though. https://minio.io/ https://minio.io/
throwawasiudy 10y ago
Sounds like it to me. Storing a crap ton of 300 byte messages is pretty common. Thousands of companies have been doing it for a decade. Anyone that does analytics or email probably stores far more of these than twitter. Blob store is perhaps the only forgivable custom solution. Besides the eventually expensive S3 you pretty much have to roll your own large binary storage at that scale. For regular DB sized workloads 1-16k bytes they had hundreds of options to choose from. Same with caching. They're both relatively solved problems.
thinkingfish 10y ago
In general, Twitter infrastructure didn't start by building solutions in house. Like most early stage companies, it started by using whatever it can find in the OSS repertoire. But a lot of problems that are non-existing at a smaller scale or at least tolerable become more acute when you scale up. This is when the developers face a decision: to improve the existing solution, or develop a new one? There are many factors that impact the final decision. Maturity of the technology? Community support/inclusion? How big is the necessary change? What scale can existing solution handle by design? How does in-house talent compare to those maintaining the original? What's the focus/need of the in-house use cases compared to the broader objective of the OSS solution? It should not be a surprise that the answer differs from project to project. But if the decision is to use the existing solutions, it probably won't raise much eyebrow/question. It is very common for larger-scale operations to write up their own solutions. Because a different scale can change the nature of the problem fundamentally. Google, Facebook and whoever has production fleet above the 10K range tend to build a lot of in-house solutions, often after similar considerations as mentioned above.

atcole 10y ago

This is a fairly technical analysis, and the terminology used in many cases is above what I know about networking. But the one quote that will stick is this. "There is no such a thing as a temporary change or workaround: In most cases, workarounds are tech debt."

jlgaddis 10y ago
I've been telling cow-orkers this for a long time. They're finally starting to realize that it's true.
kristianc 10y ago
Tech debt is a choice. Sometimes you'll want to embrace some level of technical debt in order to bring something into production quickly, with the understanding that you will fix later. It's part of a triad with Speed and Quality. That may not be the choice Twitter has made in this instance, but it's a viable choice nonetheless. Defaulting to tech debt = evil wrong imo.
- cel1ne 10y ago
  And sometime a workaround is the best solution you'll get. Because solving the problem properly might introduce new issues, which might require new workarounds.
  - paulddraper 10y ago
    Exactly. The operative part of "workaround" is "work".
- dingaling 10y ago
  The problem with technical debt versus financial debt is that the latter has a monthly mandatory cost ( interest payments ) that can't be ignored and which is visible all the way up to the C-level, whereas middle-management can keep obscuring the presence of technical debt and pushing its repayment out to the right. Essentially it's a 'free' internal debt, regardless of how often architects and developers complain of its cost. Thus in a contest between doing something right, but expensively, versus good-enough-for-now but technically-constrained the latter will usually win.

jzl 10y ago

Lots of CIO-type buzzwords and acronyms in this, including many coined by Twitter themselves. More worthy of a skim than a rigorous reading. But interesting for a bird's eye view of how complex Twitter is behind the scenes.

mnutt 10y ago

They mention that they used to base their geo routing on the location of the client's DNS server but have moved to BGP Anycast. I've heard that there can potentially be routing issues for long-running connections using anycast to end users, is anybody else doing something like this and do these issues happen in practice?

smueller1234 10y ago
As far as I know, using anycast for geo routing is pretty standard. Eg. I'm not aware that my employer (large ecommerce) has had any routing issues. This being said, we mostly avoid long running connections today. With more messenger like products becoming ubiquitous, I expect that to change though.
pyvpx 10y ago
TCP Anycast - Don’t believe the FUD Matt Levine (CacheNetworks), Barrett Lyon (BitGravity), Todd Underwood (Renesys) Operational experience with TCP and Anycast https://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf https://www.nanog.org/meetings/nanog37/presentations/matt.le...

seanmccann 10y ago

> Fast forward a few years and we were running a network with POPs on five continents and data centers with hundreds of thousands of servers. That seems high given Twitter's size and the hardware distribution pie chart they showed. Does anybody have an idea how this compares?

smueller1234 10y ago
I'm working for a very successful Internet company with 9 figure user count. We have 7 or 8 POPs on 4 continents and run a low 5 digit number of servers. This being said: we do not have to deal with anything like the network effect and lifestyle type usage that twitter has. Once you pass a certain size and complexity of infrastructure it happens really easily that economy of scale doesn't help and the complexity of everything having to be distributed works against your efficiency big time. So I'm not shocked by their numbers.
easytiger 10y ago
Web companies think 12k transactions per node is fine. It isn't fine. They just have an idiotic funding ans scaling model

tabeth 10y ago

I know nothing about storage, so I'm a bit confused about why Twitter needed: 1. Hadoop 2. Graph 3. Redis/Memcache 4. Blobstore 5. SQL variants (and a few others). I do see that the post has a short snippet briefly describing what they're storing, but I'd be curious to know why (speed, cost, latency, space tradeoffs/constraints). Also, if any more experienced folks want to chime in: Elixir/Erlang is "built for concurrency" as they say. I'd love to hear people's opinions on their one sentence simplifications of what kind of situation Hadoop/SQL/Redis/etc should be used for (similar to how Erlang is best used for situations where concurrency and fault tolerance is desired). In particular, is there a "Code Complete" type book for storage?

redler 10y ago
You might try "Seven Databases in Seven Weeks". It's a few years old now but still solid.
timClicks 10y ago
Erlang was behind the NoSQL database that started the NoSQL hype train (CouchDB), a document database. The Erlang distribution contains an amazing distributed database called Mnesia. For books, read the AOSA Chapter on Riak (http://www.aosabook.org/en/riak.html http://www.aosabook.org/en/riak.html) and then read through the tutorials on Riak Core. Oh, and Erlang in Anger https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf
- rozap 10y ago
  Mnesia is a joy to use, but keep mind it was written before the CAP theorem existed, so it doesn't behave that well in the face of network partitions.
marcinzm 10y ago
That seems a pretty standard stack, I don't know about twitter but I can say why I'd use them in my experience: 5. SQL variants: This is where you keep your source of truth data needed to serve request. It's flexible, reliable but moderately fast. The backbone of your infrastructure and you can consider everything else as enhancements to it to overcome specific limitations. 4. Blobstore: Your SQL database is not going to be good at storing large binary blobs such as files or images. 3. Redis/Memcache: Your SQL database may be too slow for some requests and/or it's data model may require too much work in the API layer to align with the final presentation model. So you want a blazing fast store that you can use for that final presentation view when requests don't differ from each other. 2. Graph: SQL databases aren't the best for graph queries or data (I mean you can hack it but it's going to be slower and uglier code wise) so if you need to look at data as a graph something dedicated makes sense. 1. Hadoop: You're getting lot's of log data from users, what they do, when they do it and so on. You want to analyze this data, build models (who to recommend for people to follow for example) and use it to probably serve ads (gotta pay the bills somehow). You could put this in SQL but it's a lot of data and you'll be doing a lot of read requests which will end up being slow and messy in SQL. Hadoop, on the other hand, is well suited for this type of data and workload. Large slow read heavy queries on data that is appended to but not modified in place.

therealmarv 10y ago

No Ansible? In an older (2014) Ansible video they claim Twitter is using Ansible but I only see Puppet mentioned.

rco8786 10y ago
Only ever used puppet during my time there, fwiw

marknadal 10y ago

I've talked/pitched (full disclosure: got a "no") some of the bigger names behind the product at Twitter. I was a little disappointed because they seem to be proud of their at least $15M+/month server costs which is partly driving their company into the ground (the user facing product hasn't improved despite "re-engineering" of their backend, at non-substantial price differences, and lack of innovation for consumers have made them lose everyone to Snapchat or anti-censorship sites). Of tweets alone (about 200bytes per tweet), over the last decade, they probably have about 3 petabytes. Unknown to them (because of the aforementioned pride) they have 1.5 petabytes a month of free storage/caching they aren't even touching. If they switched to a P2P model like IPFS or (full disclosure: I work here) http://gunDB.io/ http://gunDB.io/ , but Twitter seems determined to stay as a centralized monolith. Which is too bad, because that has now become their own death - the regime change is happening and decentralized services will win instead. Edit: Compare against 100M+ messages (1000bytes each) for $10 a day. 2 min screencast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . Even if you multiplied the feature set by 10K times, you would still be saving $12M+ a month. At this rate the Discord guys (pushing 120M+ messages a day) are doing way better - there post is on top of HN right now too, see my comment there as well. And they only have 4 backend engineers.

jhoechtl 10y ago
Why the downvote? The post contains valuable architectural links worth a discussion
- rapsey 10y ago
  1. Because it's a .js database. 2. Because the website does a poor job explaining what problem it solves.
  - deleted 10y ago
    [deleted]
  - eduren 10y ago
    Agreed, their website definitely turned me off. Quick tip for them: Having a section for showing that a bunch of people are using and "Trusting" the product is good, but your's seems to mostly (4/7 from what I can see) be the investors behind your product. Edit: Looked at the names more, and it seems 7/7 are individual investors or VCs https://angel.co/gun https://angel.co/gun Which is all well and good, but maybe the wording should be "Investors and Backers" rather than "Trusted By".

burgreblast 10y ago

100,000's of servers for 100,000,000 of messages/day ? I understand that half the servers aren't even doing messages, but, isn't WhatsApp doing 2 orders of magnitude more messages with 3 orders of magnitude (?) fewer servers? Is that right? I'm curious how one would justify 10,000X worse? So for each message, 10,000X more equipment is needed?

kalleboo 10y ago
In WhatsApp, a typical message goes to 1 other person. On Twitter, it can go to millions of people. When Twitter initially got their failwhaling under control, I recall reading they solved it by changing from a relational "join in and merge the timelines of everyone you're following on each refresh" model to a messagebox model. If that's true, maybe that naive model is now showing its limitations (I doubt they stopped there though, it seems like they have things under control)
nrjdhsbsid 10y ago
That's only 1000 messages per second on average. A single database +app server could handle that load. Assuming a bunch of other stuff is happening 500 servers sounds generous. Wtf are they doing that each server can only handles one tweet every two minutes?
- burgreblast 10y ago
  Actually, it's 1000-9000 messages per server per day. Or about 1 message every 10-100 seconds. Of course, that's just the new messages inbound. They may need to distribute that single message to 100M people (who likely won't even see it, but still.) Problems that are trivally solvable with one database don't simply scale by adding more DBs or machines. Scaling isn't easy or they would have done it. I'm in no way disparaging their team, because I don't know what kind of constraints they had getting to this point. Still, I'd bet it could be optimized by 2+ of orders of magnitude if people sat down and re-evaluated the whole structure again at this point in time. Regardless, is that really a priority? They may have bigger issues on their plate now (growing revenue, growing users, making users happy). Assuming their business can generate the cash flow to overcome the inefficiencies, they may be better served to focus on growth.

niftich 10y ago

The day before, Discord did high-quality a write-up on why they chose Cassandra [1], and now this post hits explaining how one of the world's most popular and trafficked service has engineered their infrastructure; it's like a dream. I'll echo the praise I wrote earlier, that insights like this aren't only some of the best content to hit HN, but become some of the most valuable resources for designers who have yet to face a scaling issue, but know they will soon. Since you have developed custom layers on top of open-source software to fit your particular usecase and load profile, and host all this in-house, have you considered monetizing your infrastructure for outsiders who may have similar needs? Today, one has limited, unpleasant choices: either pay out the nose for something like AWS or Google Cloud to get elastic scaling and the captive storage systems that can be made to handle these kinds of workloads and still have to write a fair bit of custom glue to get all pieces to play nice, or you can build out the servers yourself, but have to employ dedicated talent with the requisite expertise. Either way, the barriers are fairly steep; you could tap into an under-served market should you choose to sell IaaS (edit: or, more accurately, PaaS). Has this conversation come up in the past? [1] https://news.ycombinator.com/item?id=13439725 https://news.ycombinator.com/item?id=13439725

turbohz 10y ago

If Twitter can manage to operate at profit, then I might be interested in Twitter's infrastructure.

ashayh 10y ago

| We have over 100 committers per month, over 500 modules, and over 1,000 roles. | we were able to reduce our average Puppet runtimes on our Mesos clusters from well over 30 minutes to under 5 minutes. This isn't just tech debt .. it's poorly designed, poorly thought, poorly architect-ed and poorly managed in the first place. Is it because Twitter cannot find good talent because of its falling stock?

hueving 10y ago

They switched to a BGP anycast model for twitter.com, which implies TCP. I'm curious how they deal with situations where route preferences change in intermediary ISPs mid-TCP stream. Does the new server reject the TCP connection or are they synchronizing TCP sessions across clusters?