3 ms·

You need to: * Apply security patches - there's thousands of dependencies. * Manage hardware/cloud resources according to volume of data you need to handle.

You need to: * Apply security patches - there's thousands of dependencies. * Manage hardware/cloud resources according to volume of data you need to handle. * React to the changes in operating systems/browsers you run on. * Fix bugs - they are there, because simply there's not enough time, money, and need to write "perfect" software. Competition will not do that and will beat you to the market. There's no internet connected software, that can "just run", because world around is changing and you must catch up. Fells a bit weird to explain that on HN :-|

site-packages1 4y ago
This is all needed to continue to iterate on software. It’s not as needed for the ossified state Twitter has been in for many years, as the parent pointed out. Acknowledge there is _some_ work needed, maybe refreshing certs (though software systems I have built have always been set up to do that automatically), apply security patches, keep an eye on dashboards. But this is a job for a skeleton team, not 7500 engineers or whatever. It’s honestly kind of sad it took that many people to basically “keep Twitter up” all these years.
- dzikimarian 4y ago
  I believe that 7500 is total headcount, not just engineers, but yeah software development can be terribly inefficient, when all problems are solved by throwing money at them, like VC funded startups like to do. We'll see what happens now, when there seems to be a drought.
jrumbut 4y ago
There are also integrations with partners, new regulatory requirements, changes to support ownership's new priorities (and to undo the old priorities). I don't work at Twitter but these things happen at every business.
mihaic 4y ago
What your saying is true for almost every web app. I don't see any reason to justify Twitter's complexity that requires 1000 engineers. Scale is such a weak argument since their entire product paralelizez very easily.
- Xorlev 4y ago
  I don't think you understand the fanout/fanin problems their product has. Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.
  - mihaic 4y ago
    I'm not saying that it's easy, just that most work could be done by much fewer people. I'm not talking out of my ass, I have been working on scalable systems for more than a decade. I'm reminded of when, out of sheer curiosity, I once skimmed the proof for Fermat's last theorem by Andrew Willes. It was a monstrous work of which I could understand very little, but that clearly couldn't have been made by a team of a hundred mathematicians working together.
    - Xorlev 4y ago
      It's not that it isn't possible for a single person to do a monstrous amount of work, but that it takes an organization to do a breadth of things and maintain it over time. I also don't disagree that some problems are self-inflicted. Large organizations breed large organization problems. But large organizations usually grow from seemingly simple requirements and compliance efforts. My team certainly got a lot more done before we required tons of process, safety checks, data storage requirements, etc., where we now spend a non-trivial amount of HC as overhead and avoid building other things entirely. > I'm not talking out of my ass, I have been working on scalable systems for more than a decade. Ditto, friend. I currently work in an organization that builds a billion user product with far fewer than Twitter does. However, we also depend on a battalion of internal services, and other shared resources (HR, recruiting, PR, finance, ...). Twitter maintains their own datacenters, are among the few users of Aurora and Mesos at this point, and purpose-builds a lot of their infra for requirements we probably don't have a great grasp on. Twitter, like many products, is like an iceberg: the majority of the complexity is underneath.
rdiddly 4y ago
If you're gonna condescend like in that last line, your spelling ought to be impeccable. Security patches... so 10 full-time engineers ought to take care of that? "Thousands of dependencies" was a mistake. The ideal is minimal complexity. Any browser or OS decides to commit a breaking change (rare), you bring in contractors. Scaling up and down (and out and in) is handled for you automatically by cloud services now without anyone lifting a finger. "Fells" weird to be explaining that on HN. Bottom line, you don't need thousands of engineers to keep Twitter running for "the next few days to week or two (at most)." When has the world NOT been changing? The leadership of Twitter changed also.
- MAGZine 4y ago
  twitter is mostly run in their own datacenters, and does not have autoscaling. so capacity overall is more or less fixed, though how much capacity is given to each service seems to be human adjusted. the lack of autoscaling at twitter, imo, is worrisome as it sort of hints that the system might not be as mature in the self-healing department as compared to other large tech companies. time will tell. you are right though, you don't need thousand of engineers to keep twitter running a few weeks. you also don't need a driver for a car driving a straight highway (until it bends). how many engineers are needed for maintenance? who knows. I don't think we have answers, we just have what elon decided. disclaimer: facts based on twitter threads/online reading, not a twitter employee or anyone with first-hand inside knowledge.
  - acdha 4y ago
    By what I’ve heard from people who worked there they do have autoscaling but it’s services running on things like Mesos clusters instead of someone else’s data centers. That can lead to some real cost savings but requires people to run, and most of them just left the building.
monkeydust 4y ago
As as PM my headspace is very much split on this. On one hand I totally get your arguments, it's what I would expect my engineering team to say and they are often right. On the other hand I am watching this space utterly intrigued if somehow he manages to pull it off. If he does it might say more about the engineering team left than him though.
- dzikimarian 4y ago
  To be clear - I don't argue that it's impossible to run Twitter with smaller crew. Hard to say without a look at the internals, but if I had to bet - yeah, probably there are inefficiences. But OP was coming from "you can leave it and it will just run by itself" angle, which is very much different thing.

site-packages1 4y ago

This is all needed to continue to iterate on software. It’s not as needed for the ossified state Twitter has been in for many years, as the parent pointed out. Acknowledge there is _some_ work needed, maybe refreshing certs (though software systems I have built have always been set up to do that automatically), apply security patches, keep an eye on dashboards. But this is a job for a skeleton team, not 7500 engineers or whatever. It’s honestly kind of sad it took that many people to basically “keep Twitter up” all these years.

dzikimarian 4y ago
I believe that 7500 is total headcount, not just engineers, but yeah software development can be terribly inefficient, when all problems are solved by throwing money at them, like VC funded startups like to do. We'll see what happens now, when there seems to be a drought.

jrumbut 4y ago

There are also integrations with partners, new regulatory requirements, changes to support ownership's new priorities (and to undo the old priorities). I don't work at Twitter but these things happen at every business.

mihaic 4y ago

What your saying is true for almost every web app. I don't see any reason to justify Twitter's complexity that requires 1000 engineers. Scale is such a weak argument since their entire product paralelizez very easily.

Xorlev 4y ago
I don't think you understand the fanout/fanin problems their product has. Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.
- mihaic 4y ago
  I'm not saying that it's easy, just that most work could be done by much fewer people. I'm not talking out of my ass, I have been working on scalable systems for more than a decade. I'm reminded of when, out of sheer curiosity, I once skimmed the proof for Fermat's last theorem by Andrew Willes. It was a monstrous work of which I could understand very little, but that clearly couldn't have been made by a team of a hundred mathematicians working together.
  - Xorlev 4y ago
    It's not that it isn't possible for a single person to do a monstrous amount of work, but that it takes an organization to do a breadth of things and maintain it over time. I also don't disagree that some problems are self-inflicted. Large organizations breed large organization problems. But large organizations usually grow from seemingly simple requirements and compliance efforts. My team certainly got a lot more done before we required tons of process, safety checks, data storage requirements, etc., where we now spend a non-trivial amount of HC as overhead and avoid building other things entirely. > I'm not talking out of my ass, I have been working on scalable systems for more than a decade. Ditto, friend. I currently work in an organization that builds a billion user product with far fewer than Twitter does. However, we also depend on a battalion of internal services, and other shared resources (HR, recruiting, PR, finance, ...). Twitter maintains their own datacenters, are among the few users of Aurora and Mesos at this point, and purpose-builds a lot of their infra for requirements we probably don't have a great grasp on. Twitter, like many products, is like an iceberg: the majority of the complexity is underneath.

rdiddly 4y ago

If you're gonna condescend like in that last line, your spelling ought to be impeccable. Security patches... so 10 full-time engineers ought to take care of that? "Thousands of dependencies" was a mistake. The ideal is minimal complexity. Any browser or OS decides to commit a breaking change (rare), you bring in contractors. Scaling up and down (and out and in) is handled for you automatically by cloud services now without anyone lifting a finger. "Fells" weird to be explaining that on HN. Bottom line, you don't need thousands of engineers to keep Twitter running for "the next few days to week or two (at most)." When has the world NOT been changing? The leadership of Twitter changed also.

MAGZine 4y ago
twitter is mostly run in their own datacenters, and does not have autoscaling. so capacity overall is more or less fixed, though how much capacity is given to each service seems to be human adjusted. the lack of autoscaling at twitter, imo, is worrisome as it sort of hints that the system might not be as mature in the self-healing department as compared to other large tech companies. time will tell. you are right though, you don't need thousand of engineers to keep twitter running a few weeks. you also don't need a driver for a car driving a straight highway (until it bends). how many engineers are needed for maintenance? who knows. I don't think we have answers, we just have what elon decided. disclaimer: facts based on twitter threads/online reading, not a twitter employee or anyone with first-hand inside knowledge.
- acdha 4y ago
  By what I’ve heard from people who worked there they do have autoscaling but it’s services running on things like Mesos clusters instead of someone else’s data centers. That can lead to some real cost savings but requires people to run, and most of them just left the building.

monkeydust 4y ago

As as PM my headspace is very much split on this. On one hand I totally get your arguments, it's what I would expect my engineering team to say and they are often right. On the other hand I am watching this space utterly intrigued if somehow he manages to pull it off. If he does it might say more about the engineering team left than him though.

dzikimarian 4y ago
To be clear - I don't argue that it's impossible to run Twitter with smaller crew. Hard to say without a look at the internals, but if I had to bet - yeah, probably there are inefficiences. But OP was coming from "you can leave it and it will just run by itself" angle, which is very much different thing.