6 ms·

> I am convinced you can run the entire tech stack with a team of a 100 people Please tell us in detail about the Twitter stack. Because I always find it fasc

> I am convinced you can run the entire tech stack with a team of a 100 people Please tell us in detail about the Twitter stack. Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.

WJW 4y ago
I assume the reason for statements like that goes something like: 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple. 2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere. 3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances. 4. Twitter is just a big mastodon instance. 5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done. Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.
- brobdingnagians 4y ago
  That is a straw man. There are other perfectly valid interpretations and distributions of a statement that 100 could maintain it, like this: 15 database admins 10 linux sys admins 5 kubernetes specialists 10 windows tech support 25 front end developers 15 back end developers w/ Scala 10 machine learning experts Whether that makeup could or couldn't do it is a different question, or whether it would be a different mix; all of that is up for debate, but the 1/99 ratio is just one very specific, extreme, and laughable mix for anyone who has supported a system of any real size.
  - lightbendover 4y ago
    How is that valid for creating and maintaining Twitter? 15 backend engineers are not even sufficient for any single facet of Twitter. Plus, there are other initiaves that Twitter takes on that are probably outside the scope of what you consider Twitter. Take a look at just their open source initiatives: https://opensource.twitter.dev/projects/ https://opensource.twitter.dev/projects/. Go to, e.g., Finagle's Github page and look at the number of contributors and commits. While you may disagree that all of this is essential for creating and maintaining Twitter, there are problems that creep in when your platform supports billions of users that are not solved either correctly or at all by existing libraries and services; plus, open source projects serve as important recruiting collateral for engineers, especially for companies that do not have a set of widely used tools that entrench engineers (e.g. AWS, GC, Azure). Creating a husk of an app that looks vaguely like Twitter and supports hundreds of users is a weekend project for anyone with a modicum of talent; building a platform that supports billions of users and is monetized well enough to support itself is an entirely different beast.
- threeseed 4y ago
  - withinboredom 4y ago
    Flat files and replication would probably get you unreasonably far.
    - mhuffman 4y ago
      I don't know why you are being down-voted. Premateralized flat files pushed out to heavy-duty CDNs would get you very, very far. I would think that dealing with CSAM, death-threats, and other things that would get your platform into trouble would be the more difficult problem to manage. I could very easily see the technical parts of Twitter being run with a few hundred people. I don't know what else would be involved for legal, finance, marketing, sales, etc. But I doubt it is 8,000+ people.
  - WJW 4y ago
    Did you read the last line where I mentioned it was faulty logic?
  - SadTrombone 4y ago
    Did you actually read the whole comment you replied to?
  - chasd00 4y ago
    also, iirc a database based architecture was why the failwhale image was made. iirc the threw out databases and went with something more analogous to email with tweets represented as files on file systems. That was was a lonnnng time ago though, they may have reinvented the architecture a dozen times since then.
- pmoriarty 4y ago
  There are some major pieces missing from your analysis: legacy code and infrastructure, and lack of good documentation. These can make for a massive hairball of complexity that can swell the number of people needed to support it. This reminds me of a talk I once saw by a Netflix SRE, who showed a crazy convoluted mess of a diagram with thousands of crisscrossing lines going everywhere, and him screaming "No one understands Netflix!!!"
  - lightbendover 4y ago
    If you think Netflix is bad, you should see architecture diagrams for one of the major ad platforms; 30+ service sub-systems get encapsulated as a tiny block of the massive diagram. Anyone who thinks complex systems can be built and maintained by a skeleton crew hasn't worked on complex systems or hasn't been exposed to their full scope.
- KoftaBob 4y ago
  > 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple. Yes, using open source code built and maintained by 732 contributors: https://github.com/mastodon/mastodon/graphs/contributors https://github.com/mastodon/mastodon/graphs/contributors
criddell 4y ago
I couldn’t give you details, but I do know Instagram had 13 employees when Facebook bought them. Is Twitter really two-orders of magnitude more difficult to run?
- jsjohnst 4y ago
  Well, let’s see. IG was acquired by FB in April 2012 and had somewhere above 10M users at the time. Around the same time, Twitter had around 140M accounts in US alone, nearly 500M worldwide. Do we want to continue the apples vs oranges comparison further. Happy to keep citing scale differences. :) Source: Wikipedia pages for both
  - mdorazio 4y ago
    Do you also have a source showing that the number of engineers required scales linearly with number of users?
    - lightbendover 4y ago
      It in fact scales exponentially. Internal sources only, but topics like regionalization, localization, bespoke caching implementations, hardware-level optimizations, content moderation, policy adherence (e.g. GDPR, CCPA), long-term monetization (especially if supporting advertisers as a direct customer), 3P support, public APIs, documentation, SRE (a fledgling product doesn't need 5 9s), analytics (internal and for advertisers and 3P partners) and for the most part, security, are non-exhaustive examples of things you can mostly ignore when your product supports 10M users that are unavoidable when your product supports half of the US, a good chunk of the rest of the world, and other large businesses that consume you at large business scale.
      - naasking 4y ago
        You listed a bunch of requirements without an argument that fulfilling those requirements actually entails an exponential growth in employees.
  - 4y ago
P5fRxh5kUvp2th 4y ago
When people say this it's because they assume there's not stupid ass politics or moat building. THEY could probably do it with 100 people, YOU cannot. 100 people is most likely within the ballpark for a group of people whose sole purpose is to write and maintain twitter's tech stack. Unfortunately, that is not NEARLY the sole purpose of most people in businesses and that adds all kinds of productivity hits. What happens is that people like yourself become convinced that's the only way to operate.
- TheCoelacanth 4y ago
  The question is not how many people to maintain an app with the functionality of Twitter, it's how many people to maintain Twitter. Twitter has to maintain Twitter's actual app, not an app that you built to be maintainable by 100 people. You can't say whether that's possible without knowing how Twitter actually works.
  - P5fRxh5kUvp2th 4y ago
    I could certainly build a version of twitter that didn't require so many people.
    - ric2b 4y ago
      You could have saved Musk a lot of money then, too late now.

WJW 4y ago

I assume the reason for statements like that goes something like: 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple. 2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere. 3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances. 4. Twitter is just a big mastodon instance. 5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done. Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.

brobdingnagians 4y ago
That is a straw man. There are other perfectly valid interpretations and distributions of a statement that 100 could maintain it, like this: 15 database admins 10 linux sys admins 5 kubernetes specialists 10 windows tech support 25 front end developers 15 back end developers w/ Scala 10 machine learning experts Whether that makeup could or couldn't do it is a different question, or whether it would be a different mix; all of that is up for debate, but the 1/99 ratio is just one very specific, extreme, and laughable mix for anyone who has supported a system of any real size.
- lightbendover 4y ago
  How is that valid for creating and maintaining Twitter? 15 backend engineers are not even sufficient for any single facet of Twitter. Plus, there are other initiaves that Twitter takes on that are probably outside the scope of what you consider Twitter. Take a look at just their open source initiatives: https://opensource.twitter.dev/projects/ https://opensource.twitter.dev/projects/. Go to, e.g., Finagle's Github page and look at the number of contributors and commits. While you may disagree that all of this is essential for creating and maintaining Twitter, there are problems that creep in when your platform supports billions of users that are not solved either correctly or at all by existing libraries and services; plus, open source projects serve as important recruiting collateral for engineers, especially for companies that do not have a set of widely used tools that entrench engineers (e.g. AWS, GC, Azure). Creating a husk of an app that looks vaguely like Twitter and supports hundreds of users is a weekend project for anyone with a modicum of talent; building a platform that supports billions of users and is monetized well enough to support itself is an entirely different beast.
threeseed 4y ago
- withinboredom 4y ago
  Flat files and replication would probably get you unreasonably far.
  - mhuffman 4y ago
    I don't know why you are being down-voted. Premateralized flat files pushed out to heavy-duty CDNs would get you very, very far. I would think that dealing with CSAM, death-threats, and other things that would get your platform into trouble would be the more difficult problem to manage. I could very easily see the technical parts of Twitter being run with a few hundred people. I don't know what else would be involved for legal, finance, marketing, sales, etc. But I doubt it is 8,000+ people.
- WJW 4y ago
  Did you read the last line where I mentioned it was faulty logic?
- SadTrombone 4y ago
  Did you actually read the whole comment you replied to?
- chasd00 4y ago
  also, iirc a database based architecture was why the failwhale image was made. iirc the threw out databases and went with something more analogous to email with tweets represented as files on file systems. That was was a lonnnng time ago though, they may have reinvented the architecture a dozen times since then.
pmoriarty 4y ago
There are some major pieces missing from your analysis: legacy code and infrastructure, and lack of good documentation. These can make for a massive hairball of complexity that can swell the number of people needed to support it. This reminds me of a talk I once saw by a Netflix SRE, who showed a crazy convoluted mess of a diagram with thousands of crisscrossing lines going everywhere, and him screaming "No one understands Netflix!!!"
- lightbendover 4y ago
  If you think Netflix is bad, you should see architecture diagrams for one of the major ad platforms; 30+ service sub-systems get encapsulated as a tiny block of the massive diagram. Anyone who thinks complex systems can be built and maintained by a skeleton crew hasn't worked on complex systems or hasn't been exposed to their full scope.
KoftaBob 4y ago
> 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple. Yes, using open source code built and maintained by 732 contributors: https://github.com/mastodon/mastodon/graphs/contributors https://github.com/mastodon/mastodon/graphs/contributors

criddell 4y ago

I couldn’t give you details, but I do know Instagram had 13 employees when Facebook bought them. Is Twitter really two-orders of magnitude more difficult to run?

jsjohnst 4y ago
Well, let’s see. IG was acquired by FB in April 2012 and had somewhere above 10M users at the time. Around the same time, Twitter had around 140M accounts in US alone, nearly 500M worldwide. Do we want to continue the apples vs oranges comparison further. Happy to keep citing scale differences. :) Source: Wikipedia pages for both
- mdorazio 4y ago
  Do you also have a source showing that the number of engineers required scales linearly with number of users?
  - lightbendover 4y ago
    It in fact scales exponentially. Internal sources only, but topics like regionalization, localization, bespoke caching implementations, hardware-level optimizations, content moderation, policy adherence (e.g. GDPR, CCPA), long-term monetization (especially if supporting advertisers as a direct customer), 3P support, public APIs, documentation, SRE (a fledgling product doesn't need 5 9s), analytics (internal and for advertisers and 3P partners) and for the most part, security, are non-exhaustive examples of things you can mostly ignore when your product supports 10M users that are unavoidable when your product supports half of the US, a good chunk of the rest of the world, and other large businesses that consume you at large business scale.
    - naasking 4y ago
      You listed a bunch of requirements without an argument that fulfilling those requirements actually entails an exponential growth in employees.
- 4y ago

P5fRxh5kUvp2th 4y ago

When people say this it's because they assume there's not stupid ass politics or moat building. THEY could probably do it with 100 people, YOU cannot. 100 people is most likely within the ballpark for a group of people whose sole purpose is to write and maintain twitter's tech stack. Unfortunately, that is not NEARLY the sole purpose of most people in businesses and that adds all kinds of productivity hits. What happens is that people like yourself become convinced that's the only way to operate.

TheCoelacanth 4y ago
The question is not how many people to maintain an app with the functionality of Twitter, it's how many people to maintain Twitter. Twitter has to maintain Twitter's actual app, not an app that you built to be maintainable by 100 people. You can't say whether that's possible without knowing how Twitter actually works.
- P5fRxh5kUvp2th 4y ago
  I could certainly build a version of twitter that didn't require so many people.
  - ric2b 4y ago
    You could have saved Musk a lot of money then, too late now.