11 ms·

2 Years at Twitter

113 points by stanislavb 4y ago

metadat 4y ago
An unfortunate title, completely devoid of any hint about the Very Interesting content in this post: This is an insider story chronicling the evolution of Twitter 1.0's approach to build infrastructure and developer efficiency strategies at scale on teams of hundreds to thousands of SWEs.
- MuffinFlavored 4y ago
  Please excuse my ignorance. Why are 1,000+ SWEs needed (let alone custom tooling to support them)? Twitter features I know about: 1. Timeline recommendations (with personalized/relevant ads sprinkled in) 2. Notifications (so you get pushed an iOS/Android alert when somebody you subscribe to tweets) 3. Being able to post/view media (image/video/polls) 4. Direct private messaging 5. Concept of following/followers with recommendations 6. Threads/retweets/replies 7. Advertising dashboard/analytics for advertisers to create + track campaigns 8. "Explore" / "What's happening" / automatically serve "trending" content / topics 9. Lists / bookmarks 10. Anti-spam / anti-fraud 11. User profiles / settings If I missed noteworthy/sizable/heavily used by masses features, please let me know. Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Genuinely asking. Would love to learn what I'm missing.
  - dvt 4y ago
    You're not missing anything. WhatsApp had 14 million active users with 32 engineers. Instagram had 30 million active users with 13 employees. It's just delusional to think you need four-figure engineers to run Twitter—a product that basically hasn't changed for over 15 years.
    - woodruffw 4y ago
      I can't find it now, but someone in another post on Twitter pointed out that these comparisons aren't very meaningful: Twitter's social topology is any-any between users. A major (and unique) aspect of Twitter is that everybody on Twitter can talk to just about anybody else, and is encouraged to. This is also a critical value proposition for advertising on Twitter, and probably explains the size of their ads team. This is very different from WhatsApp's topology, which is a relatively traditional chat system (small groups of shared state). Instagram's is closer, but still favors smaller social groups over a global feed. Beyond that, Twitter is well known for running its own DCs, etc. It doesn't seem inconceivable that all of this, plus internationalization, mature platforms teams, etc. all adds up to four figures.
      - VectorLock 4y ago
        It boggles my mind how much people think "active users" is like some kind of fungible unit for engineering work required.
        KptMarchewa 4y ago
        Sometimes it is, until you start comparing between wildly different companies and business models.
        dvt 4y ago
        It's absolutely an OKR which implies higher workloads, are you kidding me‽ I'm sure you can build some architectural rat's nest where you need 100 engineers to maintain systems that barely serve a dozen requests a second (kind of like the blockchain) but that doesn't mean you couldn't do a much better job.
        jsnell 4y ago
        Everything else being equal, sure, more traffic is harder. But everything else is generally not equal. And despite your sleight of hand in this comment, your original claim about what's "delusional" was based on asserting an equivalence between the WhatsApp and Twitter workloads.
        woodruffw 4y ago
        I don't think there's a inviolable ratio between active users and engineers, but you do absolutely need more engineers as your user base increases: you need more people on call, more people maintaining the servers you're running (especially if, like Twitter, you run your own DCs), more people doing QA and reliability engineering because of increased load, etc.
    - vvillena 4y ago
      While the product looks like it hasn't changed much from the outside, it's been completely rewritten once or twice, at least. When Twitter launched it was a Ruby on Rails app. They scaled it to the absolute limit, then started breaking services away using Scala, then they kept at it adding several other languages to the mix. While all this happened, there was almost zero product guidance. Instead, they watched for usage patterns, and started to incorporate those into the product. The original Twitter didn't have mentions, replies, or hashtags. Adding support for all that on a product not designed for it while managing exponential growth is an extremely difficult challenge. On top of that, Twitter added divisions that are known to be engineer-heavy, such as ads, search, and infrastructure. --- The comparison with Whatsapp and Instagram is interesting. Whatsapp's case is special: they famously opted to use Erlang and it turned out to be a genius decision, as that allowed them to scale beyond expectations. Whatsapp never changed much from its initial design, which means no need for deep rewrites. Instagram was a Django app running on AWS servers, and that setup was enough until they got bought by Facebook and eventually moved from AWS to FB's infrastructure.
      - ilyt 4y ago
        That does sound like a task for few dozen very skilled people per bigger system, not thousand+
        npn 4y ago
        Well, since you only have like 2 hours of productive work per week, thousands might still not enough.
  - deleted 4y ago
    [deleted]
  - atsjie 4y ago
    At my company there are like 20-30 engineers involved in search (using ElasticSearch). The code has grown massively and they are all bogged down fixing small stuff, improvements, doing A/B-tests and the like. They also have a whole ETL setup to gather and transform data from a lot of different sources. In my spare-time I build a small search function with ElasticSearch for a big dataset. The use case was almost identical; and I feel my small setup works better, faster and has more options than where I work. I don't know, things just grow over time, old developers leave and new ones join who will never be as productive as the original devs were (it's basically rewriting someone else's book without breaking it). But I'm puzzled about it too sometimes.
  - Macha 4y ago
    > with ads sprinkled in They aren't just splatting a tag from a third party ad platform into the tweets, they're building on your own data and handling the entire auction/bidding process internally along with all the tools for advertiser to manage that, click fraud prevention, etc. There are many 100-500 engineer companies built on doing this, along with lots of 50-100 engineer companies doing just one part of this.
  - lmarcos 4y ago
    > Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Because the bigger the company the more valuation it gets (although, with the latest layoffs and all, I think this won't be the case much more). Why would Twitter (or any tech company) would remain static in size (let's say at 500 engineers) when it can just grow and grow? As long as money comes in (funding) they company has to do something with it (hiring and invent new areas to explore that requires more hiring). It's not a question of technical requirements, it's just all about money.
  - VectorLock 4y ago
    Each bullet point can be a massive undertaking, especially at scale. "Being able to post/view media (image/video/polls)" think about how much media that is. How much infrastructure you need to support that. How much it takes to make tools to manage that infrastructure.
  - KptMarchewa 4y ago
    Google has thousands of engineers just on point 1 of your list - search results.
    - lightbendover 4y ago
      And way more on the “ads sprinkled in” bit that wasn’t even its own line.
  - deleted 4y ago
    [deleted]
  - slig 4y ago
    See how long it took them to develop their own shorted links and image uploads.
  - lightbendover 4y ago
    “ads sprinkled in” could be 1,000 engineers without surprising me.
    - ilyt 4y ago
      Anything could be 1k engineers if you mismanage it enough
  - twblalock 4y ago
    Almost all of the things in your list are harder to do at scale, especially anti-spam/anti-fraud. Plus, all of this needs to be done in compliance with the legal regimes of many countries where Twitter operates -- especially regarding advertising and privacy. Having seen how some other large web services work from the inside, I'm not surprised Twitter needs a lot of people. It would be possible to pull this off with far fewer engineers, but not sustainably. People aren't robots, they can't work 80-hour weeks forever. People also need to be able to take vacations, and sometimes they will leave the company -- so it's not a good idea to run super lean. Plus, when companies run lean, the security and privacy aspects of the engineering tend not to be done very well (if at all), and for a social network that is a serious issue. Twitter was definitely bloated, but I've seen and heard of worse bloat at other tech companies.
  - guelo 4y ago
    You could probably build it in a weekend.
  - mtejo 4y ago
    twitter runs its own datacenters and has teams to support that. Since its not in a cloud the company runs all their own internal platform stuff you would get on something like aws. that is a large engineering effort. theres also the andorid and ios app, more teams. elon tweeted out a picture for tweet or timeline reads i think. each square is basically a team with an oncall rotation. this is off the top of my head, there's plenty more not visible stuff happening
PeterStuer 4y ago
2000 people on a build team. ...
- brian-armstrong 4y ago
  I think that comment is saying there were 2000 ICs merging code at Twitter, at least according to publicly released figures
  - PeterStuer 4y ago
    thx, I misread obviously
that_guy_iain 4y ago
For me, the interesting part was that there were 10+ people the build team and it seems like there are still people left since he didn't say all the people left. How large was the team? I personally find teams reaching 6+ members too large. and > For example, I think it was Adam who pointed out that the Mac laptops were unable to fully utilize the remote cache because action cache was platform specific. This remains to be an open challenge of Bazel to date. It's always macs that are suffering from problems in my experience.
- ladberg 4y ago
  The article says about 12 people plus a few extras and that 10+ left, so definitely a huge chunk of the team. Reading that line it's evident that the issue isn't Mac-specific, just that you can't reuse build artifacts across different platforms (which is expected for pretty much any build system).
- KptMarchewa 4y ago
  >How large was the team? I personally find teams reaching 6+ members too large. That does not mean that all of those people had a single manager and were all responsible for the same chunks of code. Most likely there were 3-4 sub-teams responsible for different parts of the system.
ralph84 4y ago
Another company that copied the engineering practices of Google without the profitability to support a huge internal tools team.
- rnk 4y ago
  Google had great engineering practices and infrastructure, the best of my career across several FAANGS. If only they didn't make launching products (and killing the old one) the way for advancement.
  - dekhn 4y ago
    Some of Google's great build tools came about simply because they built absurdly large binaries (GWS and GFE). IE, they own-goaled themselves repeatedly and had to invent entire new technologies to get themselves out of it. THis was a consistent pattern; the much-vaunted GFS was actually a bear to run in production. Some things I really liked about working there include: Monorepo for the entire front end and back end of Google services and products (excluding the Android OS code, and some other projects) made it really easy to get started doing interesting things without having to do a lot of work. LexicographicRangeSharder, which shards data streams using reservoir sampling, is a great example that I wouldn't have been able to build myself, but could trivially link into my code. Blaze was an excellent build tool most of the time, and the team that build it/supported it was excellent. The build/test platform (Forge and TAP) were both absolutely genius, and made large-scale testing easy. I was the owner of numpy and scipy and frequently had to do megabuilds to ensure that upgrades didn't break important things (IE, we forward-fixed all the dependencies that would break when the upgrade was committed). A highly scalable version control system with serial commit numbers, making all sorts of projects involving bisection and problem hunting far easier than git. A wide collection of very smart people who could answer nearly any technical question that came up After returning to more conventional company, as well as a startup I really miss a lot of that. but blaze and TAP were partly necessary because of the design decisions made by early (pre-google3) engineers.
deleted 4y ago
[deleted]
msoad 4y ago
20m lines of code is not large enough to have 6 people in build tooling. A lot of companies at Twitter size end up over engineering and resisting buying vs. building in house big shot projects that often times can't compete with some 3rd party solution they could just buy. For example my company is reinventing Next.js in house because the politics in the company won't allow us ditch this half-baked Next.js clone and just use the real thing. Our deployment pipeline will be much simpler if we simply bought Vercel. I'm sure we can negotiate prices with them that is better than hiring 12 engineers
- skilled 4y ago
  Sooo…. Your company can afford to buy Vercel?
  - valarauko 4y ago
    Presumably they mean Vercel licenses.
- woodruffw 4y ago
  I think the more relevant number is the number of ICs (around 2000, according to TFA). 12 build engineers (I don't know where you got 6 from) for 2000 ICs doesn't seem large at all to me; it's actually less than I would have expected.
  - marcinzm 4y ago
    Yeah, as long as the team makes the rest of the engineers 1% more efficient it's a win in terms of investment.
  - LudwigNagasena 4y ago
    What is TFA?
    - woodruffw 4y ago
      "The Fucking Article[1]." Alternatively, "the fine article." [1]: https://www.urbandictionary.com/define.php?term=TFA https://www.urbandictionary.com/define.php?term=TFA
black_13 4y ago
softwaredoug 4y ago
> Looking back, I feel like this has been among the most productive 2 years of my career. This was facilitated precisely because of the work/life balance afforded by the “peak-Kumbaya” Twitter culture, and the psychological safety I feel like the most productive I’ve been is 1. Psychological safety and trust in engineering to lead 2. Coworkers that intrinsically care about their work 3. Coworkers that are fantastic people 4. Meaningful work The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear. If the goal is engineering excellence, it just doesn’t work. A lot of people shut down and won’t take any risks or innovate in that situation. Or they leave. There may be other goals of course, but there’s a valuable reason tech companies have been selective in hiring and generous in trust.
- mfer 4y ago
  Twitter was losing millions of dollars each day. It wasn’t profitable or even break even. An engineering culture that’s awesome at a company like that will only last until the money runs out. I’m not supporting what Elon is doing. I’m just saying that what Twitter was doing before wasn’t sustainable or on a path to being sustainable. The culture was part of that
  - ketzo 4y ago
    I don’t understand why people keep saying this. Twitter was a public company. You can see from their financial statements that they recorded positive operating income for all of 2021. Not very positive! They weren’t a great business! Then they had to pay a huge fine, and lost a bunch of money because of it. But they weren’t operating at a loss. Certainly not “millions a day.” Where do people get that figure? 2020, they had a $1B net loss for the year. 2019, they had a $1B net gain. I dunno, it’s clear they were a weird and not great business. But like, they weren’t JUST lighting cash on fire.
    - foundajob 4y ago
    - 4y ago
gravypod 4y ago
There are many people who are commenting that this is so much work for 20m lines of code, they're reinventing the wheel, they should have just bought something off the shelf, etc. Two things to consider: 1. It is sometimes hard to buy things that works well: https://danluu.com/nothing-works/ https://danluu.com/nothing-works/ 2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo. It is very difficult to understand how big these two things play into these decisions. Having ~20 people manage build tooling, build caching, source control, CI (build+test), and building release artifacts (which it sounds like Pants did) is not a bad deal. Consider: 1. GitLab will charge you $20/month/user. 2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website. 3. You'll still need to hire a person or two to negotiate the purchasing of the GitLab. Now remember that GitLab might not be prefect for your use case. GitLab doesn't increase the performance of builds on your laptop by sharing a cache with all engineers. There's also a limit to the mount of data you can store for source + build artifacts (50GB). The CI runners they host for you are very expensive for how slow the machines are ($10/1000 minutes). Sure, you can do things like run your own GitLab runners but now you're back to running your own infrastructure which is an engineer's job. Sure, you could save money by using Gitea or something that if free but again hosting your own infra. As soon as you have someone managing that infrastructure they're bound to say "how else can I save time and money" and at a certain scale these things become reasonable or even required.
- xbar 4y ago
  Well said. The outcome of the Bazel/Pants changes along with the avoidance of the "just buy it" from the outset will pay dividends in efficiency and savings forever (current-events-doomsaying aside).
- atsjie 4y ago
  > If you have 2000 SWEs you'll pay 40k/month just to have access to the website. That doesn't sound like a bad deal if you have to hire such a big build team; let's say you could fire 10 people, 10 * 150.000 (rough estimate) / 12 = 125k/month. That's a 85k/month saving. I highly doubt GitLab fits the need of such massive monorepos; my only point is that the financial argument is not very strong IF GitLab were a viable alternative to such a big build team.
mirror_maze 4y ago
Does this mean Musk was right?
- xbar 4y ago
  I find it orthogonal.
xster 4y ago
Does anyone know overall how migrated to bezel Twitter is?
deleted 4y ago
[deleted]
andrew_ 4y ago
my takeaway is that I'm impressed they're running such a large scale on scala/JVM. I was at a retail startup in the mid 2010s that was doing their own 3PL, $500m / year and the JVM was a constant pain while scaling up.
- KptMarchewa 4y ago
  JVM is one of the most brilliant pieces of engineering in history. Also, there are more large companies running on way more "shoddy" technologies like Rails, which Twitter also used up to some point.
  - Fiahil 4y ago
    > JVM is one of the most brilliant pieces of engineering in history. Yet, such a pain in the ass…
    - KptMarchewa 4y ago
      Java might be. JVM? No.
- lightbendover 4y ago
  I assure you that some of the highest-throughput consumer-facing systems in the world run on the JVM.
w0mbat 4y ago
All those thousands of engineers, millions of lines of code, and they couldn’t add an edit button. Burn that bloated mess to the ground and do version 2, a fraction of the code size, a fraction of the engineers required to write and maintain it. Musk didn’t arrive soon enough.

113 points by stanislavb 4y ago

An unfortunate title, completely devoid of any hint about the Very Interesting content in this post: This is an insider story chronicling the evolution of Twitter 1.0's approach to build infrastructure and developer efficiency strategies at scale on teams of hundreds to thousands of SWEs.

MuffinFlavored 4y ago
Please excuse my ignorance. Why are 1,000+ SWEs needed (let alone custom tooling to support them)? Twitter features I know about: 1. Timeline recommendations (with personalized/relevant ads sprinkled in) 2. Notifications (so you get pushed an iOS/Android alert when somebody you subscribe to tweets) 3. Being able to post/view media (image/video/polls) 4. Direct private messaging 5. Concept of following/followers with recommendations 6. Threads/retweets/replies 7. Advertising dashboard/analytics for advertisers to create + track campaigns 8. "Explore" / "What's happening" / automatically serve "trending" content / topics 9. Lists / bookmarks 10. Anti-spam / anti-fraud 11. User profiles / settings If I missed noteworthy/sizable/heavily used by masses features, please let me know. Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Genuinely asking. Would love to learn what I'm missing.
- dvt 4y ago
  You're not missing anything. WhatsApp had 14 million active users with 32 engineers. Instagram had 30 million active users with 13 employees. It's just delusional to think you need four-figure engineers to run Twitter—a product that basically hasn't changed for over 15 years.
  - woodruffw 4y ago
    I can't find it now, but someone in another post on Twitter pointed out that these comparisons aren't very meaningful: Twitter's social topology is any-any between users. A major (and unique) aspect of Twitter is that everybody on Twitter can talk to just about anybody else, and is encouraged to. This is also a critical value proposition for advertising on Twitter, and probably explains the size of their ads team. This is very different from WhatsApp's topology, which is a relatively traditional chat system (small groups of shared state). Instagram's is closer, but still favors smaller social groups over a global feed. Beyond that, Twitter is well known for running its own DCs, etc. It doesn't seem inconceivable that all of this, plus internationalization, mature platforms teams, etc. all adds up to four figures.
    - VectorLock 4y ago
      It boggles my mind how much people think "active users" is like some kind of fungible unit for engineering work required.
      - KptMarchewa 4y ago
        Sometimes it is, until you start comparing between wildly different companies and business models.
      - dvt 4y ago
        It's absolutely an OKR which implies higher workloads, are you kidding me‽ I'm sure you can build some architectural rat's nest where you need 100 engineers to maintain systems that barely serve a dozen requests a second (kind of like the blockchain) but that doesn't mean you couldn't do a much better job.
        jsnell 4y ago
        Everything else being equal, sure, more traffic is harder. But everything else is generally not equal. And despite your sleight of hand in this comment, your original claim about what's "delusional" was based on asserting an equivalence between the WhatsApp and Twitter workloads.
      - woodruffw 4y ago
        I don't think there's a inviolable ratio between active users and engineers, but you do absolutely need more engineers as your user base increases: you need more people on call, more people maintaining the servers you're running (especially if, like Twitter, you run your own DCs), more people doing QA and reliability engineering because of increased load, etc.
  - vvillena 4y ago
    While the product looks like it hasn't changed much from the outside, it's been completely rewritten once or twice, at least. When Twitter launched it was a Ruby on Rails app. They scaled it to the absolute limit, then started breaking services away using Scala, then they kept at it adding several other languages to the mix. While all this happened, there was almost zero product guidance. Instead, they watched for usage patterns, and started to incorporate those into the product. The original Twitter didn't have mentions, replies, or hashtags. Adding support for all that on a product not designed for it while managing exponential growth is an extremely difficult challenge. On top of that, Twitter added divisions that are known to be engineer-heavy, such as ads, search, and infrastructure. --- The comparison with Whatsapp and Instagram is interesting. Whatsapp's case is special: they famously opted to use Erlang and it turned out to be a genius decision, as that allowed them to scale beyond expectations. Whatsapp never changed much from its initial design, which means no need for deep rewrites. Instagram was a Django app running on AWS servers, and that setup was enough until they got bought by Facebook and eventually moved from AWS to FB's infrastructure.
    - ilyt 4y ago
      That does sound like a task for few dozen very skilled people per bigger system, not thousand+
      - npn 4y ago
        Well, since you only have like 2 hours of productive work per week, thousands might still not enough.
- deleted 4y ago
  [deleted]
- atsjie 4y ago
  At my company there are like 20-30 engineers involved in search (using ElasticSearch). The code has grown massively and they are all bogged down fixing small stuff, improvements, doing A/B-tests and the like. They also have a whole ETL setup to gather and transform data from a lot of different sources. In my spare-time I build a small search function with ElasticSearch for a big dataset. The use case was almost identical; and I feel my small setup works better, faster and has more options than where I work. I don't know, things just grow over time, old developers leave and new ones join who will never be as productive as the original devs were (it's basically rewriting someone else's book without breaking it). But I'm puzzled about it too sometimes.
- Macha 4y ago
  > with ads sprinkled in They aren't just splatting a tag from a third party ad platform into the tweets, they're building on your own data and handling the entire auction/bidding process internally along with all the tools for advertiser to manage that, click fraud prevention, etc. There are many 100-500 engineer companies built on doing this, along with lots of 50-100 engineer companies doing just one part of this.
- lmarcos 4y ago
  > Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Because the bigger the company the more valuation it gets (although, with the latest layoffs and all, I think this won't be the case much more). Why would Twitter (or any tech company) would remain static in size (let's say at 500 engineers) when it can just grow and grow? As long as money comes in (funding) they company has to do something with it (hiring and invent new areas to explore that requires more hiring). It's not a question of technical requirements, it's just all about money.
- VectorLock 4y ago
  Each bullet point can be a massive undertaking, especially at scale. "Being able to post/view media (image/video/polls)" think about how much media that is. How much infrastructure you need to support that. How much it takes to make tools to manage that infrastructure.
- KptMarchewa 4y ago
  Google has thousands of engineers just on point 1 of your list - search results.
  - lightbendover 4y ago
    And way more on the “ads sprinkled in” bit that wasn’t even its own line.
- deleted 4y ago
  [deleted]
- slig 4y ago
  See how long it took them to develop their own shorted links and image uploads.
- lightbendover 4y ago
  “ads sprinkled in” could be 1,000 engineers without surprising me.
  - ilyt 4y ago
    Anything could be 1k engineers if you mismanage it enough
- twblalock 4y ago
  Almost all of the things in your list are harder to do at scale, especially anti-spam/anti-fraud. Plus, all of this needs to be done in compliance with the legal regimes of many countries where Twitter operates -- especially regarding advertising and privacy. Having seen how some other large web services work from the inside, I'm not surprised Twitter needs a lot of people. It would be possible to pull this off with far fewer engineers, but not sustainably. People aren't robots, they can't work 80-hour weeks forever. People also need to be able to take vacations, and sometimes they will leave the company -- so it's not a good idea to run super lean. Plus, when companies run lean, the security and privacy aspects of the engineering tend not to be done very well (if at all), and for a social network that is a serious issue. Twitter was definitely bloated, but I've seen and heard of worse bloat at other tech companies.
- guelo 4y ago
  You could probably build it in a weekend.
- mtejo 4y ago
  twitter runs its own datacenters and has teams to support that. Since its not in a cloud the company runs all their own internal platform stuff you would get on something like aws. that is a large engineering effort. theres also the andorid and ios app, more teams. elon tweeted out a picture for tweet or timeline reads i think. each square is basically a team with an oncall rotation. this is off the top of my head, there's plenty more not visible stuff happening

PeterStuer 4y ago

2000 people on a build team. ...

brian-armstrong 4y ago
I think that comment is saying there were 2000 ICs merging code at Twitter, at least according to publicly released figures
- PeterStuer 4y ago
  thx, I misread obviously

that_guy_iain 4y ago

For me, the interesting part was that there were 10+ people the build team and it seems like there are still people left since he didn't say all the people left. How large was the team? I personally find teams reaching 6+ members too large. and > For example, I think it was Adam who pointed out that the Mac laptops were unable to fully utilize the remote cache because action cache was platform specific. This remains to be an open challenge of Bazel to date. It's always macs that are suffering from problems in my experience.

ladberg 4y ago
The article says about 12 people plus a few extras and that 10+ left, so definitely a huge chunk of the team. Reading that line it's evident that the issue isn't Mac-specific, just that you can't reuse build artifacts across different platforms (which is expected for pretty much any build system).
KptMarchewa 4y ago
>How large was the team? I personally find teams reaching 6+ members too large. That does not mean that all of those people had a single manager and were all responsible for the same chunks of code. Most likely there were 3-4 sub-teams responsible for different parts of the system.

ralph84 4y ago

Another company that copied the engineering practices of Google without the profitability to support a huge internal tools team.

rnk 4y ago
Google had great engineering practices and infrastructure, the best of my career across several FAANGS. If only they didn't make launching products (and killing the old one) the way for advancement.
- dekhn 4y ago
  Some of Google's great build tools came about simply because they built absurdly large binaries (GWS and GFE). IE, they own-goaled themselves repeatedly and had to invent entire new technologies to get themselves out of it. THis was a consistent pattern; the much-vaunted GFS was actually a bear to run in production. Some things I really liked about working there include: Monorepo for the entire front end and back end of Google services and products (excluding the Android OS code, and some other projects) made it really easy to get started doing interesting things without having to do a lot of work. LexicographicRangeSharder, which shards data streams using reservoir sampling, is a great example that I wouldn't have been able to build myself, but could trivially link into my code. Blaze was an excellent build tool most of the time, and the team that build it/supported it was excellent. The build/test platform (Forge and TAP) were both absolutely genius, and made large-scale testing easy. I was the owner of numpy and scipy and frequently had to do megabuilds to ensure that upgrades didn't break important things (IE, we forward-fixed all the dependencies that would break when the upgrade was committed). A highly scalable version control system with serial commit numbers, making all sorts of projects involving bisection and problem hunting far easier than git. A wide collection of very smart people who could answer nearly any technical question that came up After returning to more conventional company, as well as a startup I really miss a lot of that. but blaze and TAP were partly necessary because of the design decisions made by early (pre-google3) engineers.

deleted 4y ago

[deleted]

msoad 4y ago

20m lines of code is not large enough to have 6 people in build tooling. A lot of companies at Twitter size end up over engineering and resisting buying vs. building in house big shot projects that often times can't compete with some 3rd party solution they could just buy. For example my company is reinventing Next.js in house because the politics in the company won't allow us ditch this half-baked Next.js clone and just use the real thing. Our deployment pipeline will be much simpler if we simply bought Vercel. I'm sure we can negotiate prices with them that is better than hiring 12 engineers

skilled 4y ago
Sooo…. Your company can afford to buy Vercel?
- valarauko 4y ago
  Presumably they mean Vercel licenses.
woodruffw 4y ago
I think the more relevant number is the number of ICs (around 2000, according to TFA). 12 build engineers (I don't know where you got 6 from) for 2000 ICs doesn't seem large at all to me; it's actually less than I would have expected.
- marcinzm 4y ago
  Yeah, as long as the team makes the rest of the engineers 1% more efficient it's a win in terms of investment.
- LudwigNagasena 4y ago
  What is TFA?
  - woodruffw 4y ago
    "The Fucking Article[1]." Alternatively, "the fine article." [1]: https://www.urbandictionary.com/define.php?term=TFA https://www.urbandictionary.com/define.php?term=TFA

black_13 4y ago

softwaredoug 4y ago

> Looking back, I feel like this has been among the most productive 2 years of my career. This was facilitated precisely because of the work/life balance afforded by the “peak-Kumbaya” Twitter culture, and the psychological safety I feel like the most productive I’ve been is 1. Psychological safety and trust in engineering to lead 2. Coworkers that intrinsically care about their work 3. Coworkers that are fantastic people 4. Meaningful work The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear. If the goal is engineering excellence, it just doesn’t work. A lot of people shut down and won’t take any risks or innovate in that situation. Or they leave. There may be other goals of course, but there’s a valuable reason tech companies have been selective in hiring and generous in trust.

mfer 4y ago
Twitter was losing millions of dollars each day. It wasn’t profitable or even break even. An engineering culture that’s awesome at a company like that will only last until the money runs out. I’m not supporting what Elon is doing. I’m just saying that what Twitter was doing before wasn’t sustainable or on a path to being sustainable. The culture was part of that
- ketzo 4y ago
  I don’t understand why people keep saying this. Twitter was a public company. You can see from their financial statements that they recorded positive operating income for all of 2021. Not very positive! They weren’t a great business! Then they had to pay a huge fine, and lost a bunch of money because of it. But they weren’t operating at a loss. Certainly not “millions a day.” Where do people get that figure? 2020, they had a $1B net loss for the year. 2019, they had a $1B net gain. I dunno, it’s clear they were a weird and not great business. But like, they weren’t JUST lighting cash on fire.
  - foundajob 4y ago
  - 4y ago

gravypod 4y ago

There are many people who are commenting that this is so much work for 20m lines of code, they're reinventing the wheel, they should have just bought something off the shelf, etc. Two things to consider: 1. It is sometimes hard to buy things that works well: https://danluu.com/nothing-works/ https://danluu.com/nothing-works/ 2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo. It is very difficult to understand how big these two things play into these decisions. Having ~20 people manage build tooling, build caching, source control, CI (build+test), and building release artifacts (which it sounds like Pants did) is not a bad deal. Consider: 1. GitLab will charge you $20/month/user. 2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website. 3. You'll still need to hire a person or two to negotiate the purchasing of the GitLab. Now remember that GitLab might not be prefect for your use case. GitLab doesn't increase the performance of builds on your laptop by sharing a cache with all engineers. There's also a limit to the mount of data you can store for source + build artifacts (50GB). The CI runners they host for you are very expensive for how slow the machines are ($10/1000 minutes). Sure, you can do things like run your own GitLab runners but now you're back to running your own infrastructure which is an engineer's job. Sure, you could save money by using Gitea or something that if free but again hosting your own infra. As soon as you have someone managing that infrastructure they're bound to say "how else can I save time and money" and at a certain scale these things become reasonable or even required.

xbar 4y ago
Well said. The outcome of the Bazel/Pants changes along with the avoidance of the "just buy it" from the outset will pay dividends in efficiency and savings forever (current-events-doomsaying aside).
atsjie 4y ago
> If you have 2000 SWEs you'll pay 40k/month just to have access to the website. That doesn't sound like a bad deal if you have to hire such a big build team; let's say you could fire 10 people, 10 * 150.000 (rough estimate) / 12 = 125k/month. That's a 85k/month saving. I highly doubt GitLab fits the need of such massive monorepos; my only point is that the financial argument is not very strong IF GitLab were a viable alternative to such a big build team.

mirror_maze 4y ago

Does this mean Musk was right?

xbar 4y ago
I find it orthogonal.

xster 4y ago

Does anyone know overall how migrated to bezel Twitter is?

deleted 4y ago

[deleted]

andrew_ 4y ago

my takeaway is that I'm impressed they're running such a large scale on scala/JVM. I was at a retail startup in the mid 2010s that was doing their own 3PL, $500m / year and the JVM was a constant pain while scaling up.

KptMarchewa 4y ago
JVM is one of the most brilliant pieces of engineering in history. Also, there are more large companies running on way more "shoddy" technologies like Rails, which Twitter also used up to some point.
- Fiahil 4y ago
  > JVM is one of the most brilliant pieces of engineering in history. Yet, such a pain in the ass…
  - KptMarchewa 4y ago
    Java might be. JVM? No.
lightbendover 4y ago
I assure you that some of the highest-throughput consumer-facing systems in the world run on the JVM.

w0mbat 4y ago

All those thousands of engineers, millions of lines of code, and they couldn’t add an edit button. Burn that bloated mess to the ground and do version 2, a fraction of the code size, a fraction of the engineers required to write and maintain it. Musk didn’t arrive soon enough.