5 ms·

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their righ

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compare Mastodon to Twitter 0.1 and don't do false advertising > 100M bots posting 3,500 times per second... to demonstrate its scale I'm wondering why 100M bots post only 3500 times per second? Is it 3500 per second for each bot? Seems like it's not, since https termination will consume the most of resources in this case. So I'm afraid it's just not enough. When I worked in Statuspage, we had support of 50-100k requests per second, because this is how it works - you have spikes, and traffic which is not evenly distributed. TBH, if it's only 3500 per second total, then I have to admit it is not enough.

nathanmarz 3y ago
We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a scalable Twitter with modern tools, which is why we also included the comparison against Threads. That's a very recent data point showing how ridiculously expensive it is to build applications like this, and they didn't even start from scratch as Instagram/Meta already had infrastructure powering similar products.
- ehutch79 3y ago
  Are you sure about that. With things like twitter, the ui is not the hard part. Things like moderation are the secret sauce. All the corner cases and support for devopsy stuff likely account for a lot. Routing to specific instances for celebrities and such.
  - justrealist 3y ago
    Nathan worked at Twitter so while he might be wrong, I don't think it's reasonable to assume he's just naive http://nathanmarz.com/blog/leaving-twitter.html http://nathanmarz.com/blog/leaving-twitter.html.
    - ehutch79 3y ago
      I misread the parent. I originally read it as it's easy to clone twitter. My response is it's easy enough to build a micro-blogging platform/service. it's all the other shit like moderation, regulatory/legal compliance, making a profit, keeping advertisers happy, etc that's hard.
- doctorpangloss 3y ago
  I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation. In practice, this means ECS is used to make first person shooters, which have decades of well specified traditions and behavior, and V2 of simulators, like for Cities Skylines 2 and Two Point Campus. So this is not meant to diminish the achievements of what you have built at all, it is more intellectually honest to say that "any high performance framework is most suitable for projects that are exact clones of pre-existing, mature things with battle-hardened specifications and end user behavior." While this might cover some greenfield projects, including the best capitalized ones that may matter to you, it does diminish the appeal of a framework for the vast majority of success stories from small & poorly capitalized teams. Those small & poor teams are very innovation and serendipity driven and hence rarely copying a pre-existing thing. And even if they try to become well-capitalized, they are almost always doing so by having worked on the thing they are copying already (i.e., already shipping version 1.0 for years).
  - syntheweave 3y ago
    Yes, this is along the lines of what I'm suspicious of too, as an also-gamedev that has done some ECS. It's too easy to study an existing system and, given the resources, create a perfect demo for how to dramatically improve it in certain ways. You can go on to ship the demo, but the demo wasn't made by the same kind of organization, with the same kinds of goals, as the original system builders. The extreme example of this is, of course, the demoscene and its hardware-bending tricks that achieve the impossible through a significant modification to the design of a "production" equivalent. So it's better by performance metrics, better by codesize, but unknown on other metrics. Like, "do I know how to start building new things with this?"
    - moate 3y ago
      Asking “what are you optimizing for/building towards” and getting different answers for different products isn’t proof that either is inferior, just that they may be better at different things. Your comment makes it seem like their post is misleading, when it isn’t it just might be that what they do best isn’t useful to you (or possibly anyone).
- deltree7 3y ago
  [dead]
- reissbaker 3y ago
  I worked on a previous iteration of Threads at Instagram, and I don't think the 25 person-years number you're estimating spent on the project was spent primarily on backend engineering. They concurrently launched native iOS and Android apps, and I think you're also factoring in the IC count from the linked article to mean full-time ICs dedicated exclusively to Threads, which is probably not how it worked. (Although TBH I'm not sure how you arrived at 25-person-years from the article, which said the project started with "dozens" of engineers and peaked at 50, and was 7 months long in total — a simple average of 24-50 engineers for 7 months, even if you assume they're all full time, would be about 21 person-years). When I was at IG, my team was 20-something ICs full-time on our project, bursted to maybe double that as necessary in part-time help from ICs in the wider org. We had a total of three backend engineers, of which I was one, as well as a backend intern for a few months, although we bursted the backend team for a couple of months to four for some ML help. Your project sounds pretty cool! But I don't think the comparison to Threads productivity is quite right. The majority of IG engineering is focused on building polished native clients, not generally on backend infrastructure. It's true that Meta already has a lot of the backend infra built — if Red Planet Labs can come close to what you get at Meta, that's pretty amazing. But I don't think the numbers you're quoting are apples-to-apples, or mean quite what you think they do. I don't know if this version of Threads operated exactly like the one I worked on (a Snapchat competitor), but I'd be pretty surprised if there was a product team at IG that was majority backend. (Edit: I also think it's worth keeping in mind the experience levels of who works on these projects — at IG it's usually a smaller number of E6/E5s guiding a larger number of E4/E3s. Person-years are not all equivalent! If you spent ten years building the Red Planet Labs infrastructure based on your time at Twitter, nine person-months of your team's time building a product on your infra might not be the same as nine person-months of someone else's.) Anyway — I don't mean to downplay your product, and really, if it's anything like the backend productivity I experienced at Meta, that would be pretty groundbreaking. Curious to see what you launch :)
- dpweb 3y ago
  Built some log databases and back end frameworks myself with some of the same concepts. I applaud the creativity in rethinking how back ends should work. Please now do frontends next! :) "But it's not a fully functioning 2023 Twitter!!!" I think some people miss the point. This is not about hey we built a Twitter clone. This is about a POC for a novel app architecture. We need to be constantly examining and re-examining our thoughts about the best way to deal with distributed systems, scale, developer workflow. Even inventing new ones.
  - philipwhiuk 3y ago
    > This is not about hey we built a Twitter clone. Then they shouldn’t have titled it “we built a Twitter clone”
hosh 3y ago
How much of Twitter’s code base is dedicated to things like security, compliance, and moderation? Granted, a decentralized platform would eliminate some of those, just by being decentralized
- asu_thomas 3y ago
  And advertising. Twitter is an advertising platform.
- adventured 3y ago
  None of those things get eliminated by decentralization, they get distributed to whatever the point of control / ownership is. Mastodon still requires security, compliance and moderation. And those requirements are going to keep getting more challenging by the year. It'll end up being another reason nobody will want to host content in a decentralized manner, the burden will become obnoxious.
  - hosh 3y ago
    An organization trying to maintain an ISO certification will have drastically different policies and controls than a small shop, or even a hobbyist group. Everyone (theoretically) would be complying to statues in its broadest sense, but jurisdiction, regulations, industry best practices, reporting requirements, and appetite for risk is going to be different from organization to organization. It’s not one-size-fits-all. So some of these things are eliminated because the people hosting those are not put under the same kind of scrutiny as say, a Twitter.
    - jeremyjh 3y ago
      The scrutiny from investors, legal and compliance exists because the risks are real, and they don't go away just because there isn't a Serious Business involved. Once someone operating one of these is publicly damaged, the risk will be better understood and marked up accordingly.
      - 3y ago
MikePlacid 3y ago
So >> 100M bots posting 3,500 times per second... and > We used the OpenAI API to generate 50,000 statuses for the bots to choose from at random. I wonder: 100M OpenAI bots talking to each other continuously and with much vigor - how is this affecting OpenAI’s uhm… intellect?
- sdwr 3y ago
  They generated 50,000 statuses once, put them in a text file, and pick between them randomly. So not at all.
littlestymaar 3y ago
> Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. But Twitter isn't, and was never, about live video support: this is pure feature creep and that's how you get headcount inflation and a company that can be run for 17 years without making profit (AKA terrible business). > When I worked in Statuspage, we had support of 50-100k requests per second Having served 150kqps in the past as part of a very small team (3 back-end eng.), this isn't necessarily as big of a deal as you make it sound: it mostly depends on your workload and whether or not you need consistency (or even persistence at all) in your data. In practice, building scalable system is hard mostly because it's hard to get the management forgot their vanity ideas that go against your (their, actually) system's scalability.
- RomanPushkin 3y ago
  > Having served 150kqps in the past as part of a very small team (3 back-end eng.), this isn't necessarily as big of a deal I would love to see a blog post about your experience!
WheelsAtLarge 3y ago
We see this type of post regularly. Something like, "How I built a better <pick your app> clone by myself in a month." Well, no, usually it's just a bare skeleton with the least amount of functionality. Not only that, the software is the least of the functionality. The organizational structure around the app is what matters most to keep it going. It's an attention seeking ploy and the whole thing usually disappears real quick.
- gexla 3y ago
  If you want background, you can go to the blog in his bio and read the first post, which mentioned he worked at Twitter starting in 2011. He mentioned elsewhere that his comparison is early Twitter, as opposed to current Twitter. The background makes this a bit more interesting, because you can imagine how those early days impacted the arc of his work.
otikik 3y ago
Indeed. Add a single JavaScript dependency… you will get the banana, the gorilla holding the banana, the tree holding the gorilla, and the whole jungle.
codedokode 3y ago
Twitter has many features, but not all of them are necessary. For example, there is no need to implement popups blocking the page and demanding a user to register, or detailed telemetry collecting user data from 20 different providers.
codedokode 3y ago
> When I worked in Statuspage, we had support of 50-100k requests per second Serving one cached HTML with Nginx and serving dynamically generated content, updating a database is a different thing.
anothernewdude 3y ago
It takes more self-control and effort to reduce the number of features to the ones that matter. Twitter having more features is a liability, not a benefit. > Add large-scale distributed live video support at the top of that, Why? For the love of all that is good and efficient, why? Why not have a separate platform for that? Or link to a different federated video service? Why does every platform need to do all the things?
KingOfCoders 3y ago
The 100x less code always reminds me of a hypothetical Half Life 2 game engine. Then your code would be: StartHalfLife2LikeGame() and you'd replace millions LOC with one line. If there is a perfect match between the framework and your app, there is no code. The more your app diverges from the ideal app the framework was written for, the more code you have.
- lbalazscs 3y ago
  They claim that "Reddit, Slack, Gmail, Uber, etc" could also be done with a similar level of effort.
  - KingOfCoders 3y ago
    Yes, all applications that are aligned with the framework (message spreading from one user to many others) - not that I want to put down their effort in any way. But this is the core [0] - a simple event location plattform my wife founded hat easily >200k LOC while the core (edit/search/detail page event location) had <10k LOC. [0] If building Twitter I perhaps would try their setup, although I have been bitten with "magic" JVM frameworks in the past, e.G. we used one commercially and the license cost went up from 80k to 800k YoY. On top of scaling problems we could do nothing about because the framework was a black box.
smcl 3y ago
Not to mention the phrase "x times less than" doesn't really make sense the way it's often used. For it to make sense you have to reinterpret it to mean something that it doesn't based on being the opposite of "x times more than" (which is also often misused).

nathanmarz 3y ago

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a scalable Twitter with modern tools, which is why we also included the comparison against Threads. That's a very recent data point showing how ridiculously expensive it is to build applications like this, and they didn't even start from scratch as Instagram/Meta already had infrastructure powering similar products.

ehutch79 3y ago
Are you sure about that. With things like twitter, the ui is not the hard part. Things like moderation are the secret sauce. All the corner cases and support for devopsy stuff likely account for a lot. Routing to specific instances for celebrities and such.
- justrealist 3y ago
  Nathan worked at Twitter so while he might be wrong, I don't think it's reasonable to assume he's just naive http://nathanmarz.com/blog/leaving-twitter.html http://nathanmarz.com/blog/leaving-twitter.html.
  - ehutch79 3y ago
    I misread the parent. I originally read it as it's easy to clone twitter. My response is it's easy enough to build a micro-blogging platform/service. it's all the other shit like moderation, regulatory/legal compliance, making a profit, keeping advertisers happy, etc that's hard.
doctorpangloss 3y ago
I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation. In practice, this means ECS is used to make first person shooters, which have decades of well specified traditions and behavior, and V2 of simulators, like for Cities Skylines 2 and Two Point Campus. So this is not meant to diminish the achievements of what you have built at all, it is more intellectually honest to say that "any high performance framework is most suitable for projects that are exact clones of pre-existing, mature things with battle-hardened specifications and end user behavior." While this might cover some greenfield projects, including the best capitalized ones that may matter to you, it does diminish the appeal of a framework for the vast majority of success stories from small & poorly capitalized teams. Those small & poor teams are very innovation and serendipity driven and hence rarely copying a pre-existing thing. And even if they try to become well-capitalized, they are almost always doing so by having worked on the thing they are copying already (i.e., already shipping version 1.0 for years).
- syntheweave 3y ago
  Yes, this is along the lines of what I'm suspicious of too, as an also-gamedev that has done some ECS. It's too easy to study an existing system and, given the resources, create a perfect demo for how to dramatically improve it in certain ways. You can go on to ship the demo, but the demo wasn't made by the same kind of organization, with the same kinds of goals, as the original system builders. The extreme example of this is, of course, the demoscene and its hardware-bending tricks that achieve the impossible through a significant modification to the design of a "production" equivalent. So it's better by performance metrics, better by codesize, but unknown on other metrics. Like, "do I know how to start building new things with this?"
  - moate 3y ago
    Asking “what are you optimizing for/building towards” and getting different answers for different products isn’t proof that either is inferior, just that they may be better at different things. Your comment makes it seem like their post is misleading, when it isn’t it just might be that what they do best isn’t useful to you (or possibly anyone).
deltree7 3y ago
[dead]
reissbaker 3y ago
I worked on a previous iteration of Threads at Instagram, and I don't think the 25 person-years number you're estimating spent on the project was spent primarily on backend engineering. They concurrently launched native iOS and Android apps, and I think you're also factoring in the IC count from the linked article to mean full-time ICs dedicated exclusively to Threads, which is probably not how it worked. (Although TBH I'm not sure how you arrived at 25-person-years from the article, which said the project started with "dozens" of engineers and peaked at 50, and was 7 months long in total — a simple average of 24-50 engineers for 7 months, even if you assume they're all full time, would be about 21 person-years). When I was at IG, my team was 20-something ICs full-time on our project, bursted to maybe double that as necessary in part-time help from ICs in the wider org. We had a total of three backend engineers, of which I was one, as well as a backend intern for a few months, although we bursted the backend team for a couple of months to four for some ML help. Your project sounds pretty cool! But I don't think the comparison to Threads productivity is quite right. The majority of IG engineering is focused on building polished native clients, not generally on backend infrastructure. It's true that Meta already has a lot of the backend infra built — if Red Planet Labs can come close to what you get at Meta, that's pretty amazing. But I don't think the numbers you're quoting are apples-to-apples, or mean quite what you think they do. I don't know if this version of Threads operated exactly like the one I worked on (a Snapchat competitor), but I'd be pretty surprised if there was a product team at IG that was majority backend. (Edit: I also think it's worth keeping in mind the experience levels of who works on these projects — at IG it's usually a smaller number of E6/E5s guiding a larger number of E4/E3s. Person-years are not all equivalent! If you spent ten years building the Red Planet Labs infrastructure based on your time at Twitter, nine person-months of your team's time building a product on your infra might not be the same as nine person-months of someone else's.) Anyway — I don't mean to downplay your product, and really, if it's anything like the backend productivity I experienced at Meta, that would be pretty groundbreaking. Curious to see what you launch :)
dpweb 3y ago
Built some log databases and back end frameworks myself with some of the same concepts. I applaud the creativity in rethinking how back ends should work. Please now do frontends next! :) "But it's not a fully functioning 2023 Twitter!!!" I think some people miss the point. This is not about hey we built a Twitter clone. This is about a POC for a novel app architecture. We need to be constantly examining and re-examining our thoughts about the best way to deal with distributed systems, scale, developer workflow. Even inventing new ones.
- philipwhiuk 3y ago
  > This is not about hey we built a Twitter clone. Then they shouldn’t have titled it “we built a Twitter clone”

hosh 3y ago

How much of Twitter’s code base is dedicated to things like security, compliance, and moderation? Granted, a decentralized platform would eliminate some of those, just by being decentralized

asu_thomas 3y ago
And advertising. Twitter is an advertising platform.
adventured 3y ago
None of those things get eliminated by decentralization, they get distributed to whatever the point of control / ownership is. Mastodon still requires security, compliance and moderation. And those requirements are going to keep getting more challenging by the year. It'll end up being another reason nobody will want to host content in a decentralized manner, the burden will become obnoxious.
- hosh 3y ago
  An organization trying to maintain an ISO certification will have drastically different policies and controls than a small shop, or even a hobbyist group. Everyone (theoretically) would be complying to statues in its broadest sense, but jurisdiction, regulations, industry best practices, reporting requirements, and appetite for risk is going to be different from organization to organization. It’s not one-size-fits-all. So some of these things are eliminated because the people hosting those are not put under the same kind of scrutiny as say, a Twitter.
  - jeremyjh 3y ago
    The scrutiny from investors, legal and compliance exists because the risks are real, and they don't go away just because there isn't a Serious Business involved. Once someone operating one of these is publicly damaged, the risk will be better understood and marked up accordingly.
    - 3y ago

MikePlacid 3y ago

So >> 100M bots posting 3,500 times per second... and > We used the OpenAI API to generate 50,000 statuses for the bots to choose from at random. I wonder: 100M OpenAI bots talking to each other continuously and with much vigor - how is this affecting OpenAI’s uhm… intellect?

sdwr 3y ago
They generated 50,000 statuses once, put them in a text file, and pick between them randomly. So not at all.

littlestymaar 3y ago

> Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. But Twitter isn't, and was never, about live video support: this is pure feature creep and that's how you get headcount inflation and a company that can be run for 17 years without making profit (AKA terrible business). > When I worked in Statuspage, we had support of 50-100k requests per second Having served 150kqps in the past as part of a very small team (3 back-end eng.), this isn't necessarily as big of a deal as you make it sound: it mostly depends on your workload and whether or not you need consistency (or even persistence at all) in your data. In practice, building scalable system is hard mostly because it's hard to get the management forgot their vanity ideas that go against your (their, actually) system's scalability.

RomanPushkin 3y ago
> Having served 150kqps in the past as part of a very small team (3 back-end eng.), this isn't necessarily as big of a deal I would love to see a blog post about your experience!

WheelsAtLarge 3y ago

We see this type of post regularly. Something like, "How I built a better <pick your app> clone by myself in a month." Well, no, usually it's just a bare skeleton with the least amount of functionality. Not only that, the software is the least of the functionality. The organizational structure around the app is what matters most to keep it going. It's an attention seeking ploy and the whole thing usually disappears real quick.

gexla 3y ago
If you want background, you can go to the blog in his bio and read the first post, which mentioned he worked at Twitter starting in 2011. He mentioned elsewhere that his comparison is early Twitter, as opposed to current Twitter. The background makes this a bit more interesting, because you can imagine how those early days impacted the arc of his work.

otikik 3y ago

Indeed. Add a single JavaScript dependency… you will get the banana, the gorilla holding the banana, the tree holding the gorilla, and the whole jungle.

codedokode 3y ago

Twitter has many features, but not all of them are necessary. For example, there is no need to implement popups blocking the page and demanding a user to register, or detailed telemetry collecting user data from 20 different providers.

codedokode 3y ago

> When I worked in Statuspage, we had support of 50-100k requests per second Serving one cached HTML with Nginx and serving dynamically generated content, updating a database is a different thing.

anothernewdude 3y ago

It takes more self-control and effort to reduce the number of features to the ones that matter. Twitter having more features is a liability, not a benefit. > Add large-scale distributed live video support at the top of that, Why? For the love of all that is good and efficient, why? Why not have a separate platform for that? Or link to a different federated video service? Why does every platform need to do all the things?

KingOfCoders 3y ago

The 100x less code always reminds me of a hypothetical Half Life 2 game engine. Then your code would be: StartHalfLife2LikeGame() and you'd replace millions LOC with one line. If there is a perfect match between the framework and your app, there is no code. The more your app diverges from the ideal app the framework was written for, the more code you have.

lbalazscs 3y ago
They claim that "Reddit, Slack, Gmail, Uber, etc" could also be done with a similar level of effort.
- KingOfCoders 3y ago
  Yes, all applications that are aligned with the framework (message spreading from one user to many others) - not that I want to put down their effort in any way. But this is the core [0] - a simple event location plattform my wife founded hat easily >200k LOC while the core (edit/search/detail page event location) had <10k LOC. [0] If building Twitter I perhaps would try their setup, although I have been bitten with "magic" JVM frameworks in the past, e.G. we used one commercially and the license cost went up from 80k to 800k YoY. On top of scaling problems we could do nothing about because the framework was a black box.

smcl 3y ago

Not to mention the phrase "x times less than" doesn't really make sense the way it's often used. For it to make sense you have to reinterpret it to mean something that it doesn't based on being the opposite of "x times more than" (which is also often misused).