4 ms·

This is also largely the answer to why your weekend garage project doesn't need 1000+ developers that an actual in production project needs. At scale every edge

This is also largely the answer to why your weekend garage project doesn't need 1000+ developers that an actual in production project needs. At scale every edge case is a bug that someone has to deal with. Not just the hard crash software/hardware edge cases either. Regulatory edge cases, abuse vector edge cases, localization and accessibility edge cases will all need to be dealt with. I remember people exclaiming 'musk is right, Twitter doesn't need 1000s of developers' only for Twitter to start failing a lot of regulatory and abuse vector edge cases shortly after their layoffs. It's no good judging business needs based on a basic garage implementation of Twitter. The real world and a few billion users makes for an entirely different set of problems.

compiler-guy 2y ago
Such scale also creates opportunities. If you have 10 servers, a developer can't spend a couple of months to get a 0.1% performance boost--the benefit will never cover the cost. If you have 100,000 servers, it just might. If you have 1,000,000 servers, it almost certainly is worth an entire team looking for that much performance, year in and year out. This is true of performance, avoiding downtime, or whatever else. You could never justify such an expense at a startup, but with sufficient scale they pay for themselves many times over.
- closeparen 2y ago
  A large company's million servers are going to be partitioned across thousands of separate applications. The biggest ones do benefit from performance work, but a large portion of the server count is going to be a long tail of services that aren't individually worth that much effort.
  - vlovich123 2y ago
    They would rarely be partitioned because of all the waste - workloads are typically placed cotenant into 1 machine (VMs, containers, normal multi-processing, etc). But I believe you’re right that cumulatively there would be a lot of long tail services that consume resources and the cumulative inefficiencies aren’t being optimized because it’s work that doesn’t scale (i.e. you’d have to fix too many performance issues on services no one cares to make a visible overall dent). Not sure why you’re downvoted though - it’s an astute observation.
brabel 2y ago
Do you have more information about the Twitter abuse vector edge cases? And how would developers mitigate that? Isn't that the kind of thing you need human content reviewers to handle (assuming any automated tool Twitter had was still there after the layoffs)?
- AnotherGoodName 2y ago
  The development work comes in stopping repeated abuse. Eg. A block of up addresses known to be bad should not be able to repeatedly sign up for new accounts after a ban. You may need to add catchas for users which have certain criteria met. Call pumping needs to be constantly re-scripted to stop bad phone companies tricking your systems into sending SMS to expensive numbers (Twitter turned off all SMS two factor auth to stop this which is telling of the gap they left here).
- closeparen 2y ago
  Rules engines and ML models to escalate interesting Tweets, accounts, and networks of related accounts for human review. Data pipelines to feed them. Change management and observability so that analysts can safely keep them up to speed with threats. Case management systems to put the findings into. Reviews, approvals, deviations, escalations, M of N reviewer consensus. Each one of these things is at least as complex as Twitter's core product.
pessimizer 2y ago
> I remember people exclaiming 'musk is right, Twitter doesn't need 1000s of developers' only for Twitter to start failing a lot of regulatory and abuse vector edge cases shortly after their layoffs. It's no good judging business needs based on a basic garage implementation of Twitter. The real world and a few billion users makes for an entirely different set of problems. It's strange this is the lesson you took from Twitter. Twitter fired 90% of their developers, and almost nothing changed. Twitter is a site that exists in the real world, at scale. People don't even talk about the fact that Twitter fired 90% of their devs anymore, they just began firing devs themselves.
- MrDarcy 2y ago
  Elon Musk fired Twitter employees. Have you not noticed how the site has deteriorated technically? I personally know people who have suffered abuse on Twitter, reported the abuse, and had no action taken because Elon fired the trust and safety org.
  - 4hg4ufxhy 2y ago
    Do you also know personally people who reported it and had action taken prior?
    - MrDarcy 2y ago
      Yes
  - thfuran 2y ago
    Even if it did, that's not really evidence that it'd take nearly as many people to run a Twitter clone at Twitter scale. It may well, but suddenly losing 90% of personnel is some serious bus factor woes.
  - canoebuilder 2y ago
    If reading something is hurting your feelings, you can stop reading it. Twitter even provides mute, block, and whatnot functionality to prevent specified things from even showing up in your line of sight to begin with. And if the app is really bothering you, you can always set it down and go outside, take a walk, meet somebody new, do something that will put a smile on your face on your deathbed. Lumping in mean comments online, with actual abuse is approaching risible. Words have meanings, we shouldn’t dilute or distort them. By Twitter “not taking action,” sounds like your friend is upset that he or she can no longer co-opt the proprietors of the site into enacting punitive measures on people who draw his or her ire. Maybe some mean things were said or whatever, but at the end of the day it’s just text on a screen isn’t it? And there’s a lot more to life than text on a screen, isn’t there? It’s also weird how you mention the technical functioning of the site, then bring up the “Trust & Safety Org” when the legacy of “Trust & Safety” is a small cabal with extremist views arbitrarily deciding what information to censor and suppress based on their own viewpoints, whims, and influence from government agencies. That has nothing to do with the technical functioning of the site which is a matter of reproducible, specifiable, determinate functions implemented in computer code to produce a useful product. The kind of thing that really turns the mind of an autist on. P.S. Not to be too blasé about your friend, mean words can be an issue, especially an ongoing pattern, but anonymous strangers online seems like less of an issue than irl, and was this really an issue where block or mute wasn’t sufficient? How so?
nsguy 2y ago
Your garage project also doesn't need to support 100's of random features some product manager thought are critical but nobody uses. Quality issues are exponential to the complexity of the software. Out of those 1000's of engineers there will also lot of variability in terms of productivity and quality. Some engineers are likely doing most of the heavy lifting. Some spend all their time in meetings. Some engineers might be working on their novel or startup while sitting in the office. Extremely unlikely they're all operating at the same level. With 1000's of engineers there's a lot of effort synchronizing those engineers, communication overhead, people stepping over each other's work etc. The mythical tower of babel: https://en.wikipedia.org/wiki/Tower_of_Babel https://en.wikipedia.org/wiki/Tower_of_Babel Lots of reliable/production software was built by teams much smaller than 1000's. Things like operating systems (e.g. the Linux kernel), compilers, tools or libraries, come to mind. Some even by single people. That's not to say that all weekend garage projects are better than what 1000+ developer teams produce but don't knock the ability of small teams to do amazing things. Twitter isn't exactly the pinnacle of engineering accomplishment either. I'm pretty sure Twitter suffered from bloat similar to many other large tech companies. Elon taking a sledgehammer to that is probably not the best approach though. That said some people were saying the whole thing will fall apart in days, and it didn't. [EDITed for typo]

compiler-guy 2y ago

Such scale also creates opportunities. If you have 10 servers, a developer can't spend a couple of months to get a 0.1% performance boost--the benefit will never cover the cost. If you have 100,000 servers, it just might. If you have 1,000,000 servers, it almost certainly is worth an entire team looking for that much performance, year in and year out. This is true of performance, avoiding downtime, or whatever else. You could never justify such an expense at a startup, but with sufficient scale they pay for themselves many times over.

closeparen 2y ago
A large company's million servers are going to be partitioned across thousands of separate applications. The biggest ones do benefit from performance work, but a large portion of the server count is going to be a long tail of services that aren't individually worth that much effort.
- vlovich123 2y ago
  They would rarely be partitioned because of all the waste - workloads are typically placed cotenant into 1 machine (VMs, containers, normal multi-processing, etc). But I believe you’re right that cumulatively there would be a lot of long tail services that consume resources and the cumulative inefficiencies aren’t being optimized because it’s work that doesn’t scale (i.e. you’d have to fix too many performance issues on services no one cares to make a visible overall dent). Not sure why you’re downvoted though - it’s an astute observation.

brabel 2y ago

Do you have more information about the Twitter abuse vector edge cases? And how would developers mitigate that? Isn't that the kind of thing you need human content reviewers to handle (assuming any automated tool Twitter had was still there after the layoffs)?

AnotherGoodName 2y ago
The development work comes in stopping repeated abuse. Eg. A block of up addresses known to be bad should not be able to repeatedly sign up for new accounts after a ban. You may need to add catchas for users which have certain criteria met. Call pumping needs to be constantly re-scripted to stop bad phone companies tricking your systems into sending SMS to expensive numbers (Twitter turned off all SMS two factor auth to stop this which is telling of the gap they left here).
closeparen 2y ago
Rules engines and ML models to escalate interesting Tweets, accounts, and networks of related accounts for human review. Data pipelines to feed them. Change management and observability so that analysts can safely keep them up to speed with threats. Case management systems to put the findings into. Reviews, approvals, deviations, escalations, M of N reviewer consensus. Each one of these things is at least as complex as Twitter's core product.

pessimizer 2y ago

> I remember people exclaiming 'musk is right, Twitter doesn't need 1000s of developers' only for Twitter to start failing a lot of regulatory and abuse vector edge cases shortly after their layoffs. It's no good judging business needs based on a basic garage implementation of Twitter. The real world and a few billion users makes for an entirely different set of problems. It's strange this is the lesson you took from Twitter. Twitter fired 90% of their developers, and almost nothing changed. Twitter is a site that exists in the real world, at scale. People don't even talk about the fact that Twitter fired 90% of their devs anymore, they just began firing devs themselves.

MrDarcy 2y ago
Elon Musk fired Twitter employees. Have you not noticed how the site has deteriorated technically? I personally know people who have suffered abuse on Twitter, reported the abuse, and had no action taken because Elon fired the trust and safety org.
- 4hg4ufxhy 2y ago
  Do you also know personally people who reported it and had action taken prior?
  - MrDarcy 2y ago
    Yes
- thfuran 2y ago
  Even if it did, that's not really evidence that it'd take nearly as many people to run a Twitter clone at Twitter scale. It may well, but suddenly losing 90% of personnel is some serious bus factor woes.
- canoebuilder 2y ago
  If reading something is hurting your feelings, you can stop reading it. Twitter even provides mute, block, and whatnot functionality to prevent specified things from even showing up in your line of sight to begin with. And if the app is really bothering you, you can always set it down and go outside, take a walk, meet somebody new, do something that will put a smile on your face on your deathbed. Lumping in mean comments online, with actual abuse is approaching risible. Words have meanings, we shouldn’t dilute or distort them. By Twitter “not taking action,” sounds like your friend is upset that he or she can no longer co-opt the proprietors of the site into enacting punitive measures on people who draw his or her ire. Maybe some mean things were said or whatever, but at the end of the day it’s just text on a screen isn’t it? And there’s a lot more to life than text on a screen, isn’t there? It’s also weird how you mention the technical functioning of the site, then bring up the “Trust & Safety Org” when the legacy of “Trust & Safety” is a small cabal with extremist views arbitrarily deciding what information to censor and suppress based on their own viewpoints, whims, and influence from government agencies. That has nothing to do with the technical functioning of the site which is a matter of reproducible, specifiable, determinate functions implemented in computer code to produce a useful product. The kind of thing that really turns the mind of an autist on. P.S. Not to be too blasé about your friend, mean words can be an issue, especially an ongoing pattern, but anonymous strangers online seems like less of an issue than irl, and was this really an issue where block or mute wasn’t sufficient? How so?

nsguy 2y ago

Your garage project also doesn't need to support 100's of random features some product manager thought are critical but nobody uses. Quality issues are exponential to the complexity of the software. Out of those 1000's of engineers there will also lot of variability in terms of productivity and quality. Some engineers are likely doing most of the heavy lifting. Some spend all their time in meetings. Some engineers might be working on their novel or startup while sitting in the office. Extremely unlikely they're all operating at the same level. With 1000's of engineers there's a lot of effort synchronizing those engineers, communication overhead, people stepping over each other's work etc. The mythical tower of babel: https://en.wikipedia.org/wiki/Tower_of_Babel https://en.wikipedia.org/wiki/Tower_of_Babel Lots of reliable/production software was built by teams much smaller than 1000's. Things like operating systems (e.g. the Linux kernel), compilers, tools or libraries, come to mind. Some even by single people. That's not to say that all weekend garage projects are better than what 1000+ developer teams produce but don't knock the ability of small teams to do amazing things. Twitter isn't exactly the pinnacle of engineering accomplishment either. I'm pretty sure Twitter suffered from bloat similar to many other large tech companies. Elon taking a sledgehammer to that is probably not the best approach though. That said some people were saying the whole thing will fall apart in days, and it didn't. [EDITed for typo]