4 ms·

> I often wonder this too, but... I think making "another Google" is not the best example. 100% agreed. They really should have used Twitter as an example. A

> I often wonder this too, but... I think making "another Google" is not the best example. 100% agreed. They really should have used Twitter as an example. Any competent full-stack developer could probably have a working clone of Twitter running in a weekend. The challenge I think is scaling it to millions of daily active users. But, even then, Twitter has been around for over 10 years. The scaling problem seems to have been solved. I would think that these days, Twitter engineering would just be a whack-a-mole of odd obscure bugs. I can't think of any useful new features it's gotten in the last two years. Why do they still need so many engineers?

liquidpele 3y ago
Not just scaling it… scaling it in a way that doesn’t immediately bankrupt you. Bit of a chicken and egg problem with advertisers.
- MrLeap 3y ago
  This is one of those _things_ that I suspect causes many greybeards to grimace. What's a tweet? We can haggle this up and down so cut me some slack. 200 bytes? 50 bytes? gzipped -90%, 5 bytes. Holy shit what absurd amount of traffic could a 1gb fiber connection handle? oh huh 200 million tweets a second. That's silly. How wrong could I be? -- then you run down the numbers and you realize that no matter how wrong you are, a twitter-clone-for-most-people oughta be cheap. The network effects and fuzzy human reasons are their moat, nothing else. One of those fuzzy human reasons is how amazon EC2 eats your ass. Just gobbles it from stem to stern.
  - tehjoker 3y ago
    They have pictures and video now. You also have to sort them in real time, check for blocked, mutes, etc, and each tweet is amplified to followers so one tweet requires e.g. 700 copies or 1M copies depending on follower count. The text part is basically solved at this point, but not quite that easy. I hear dealing with blocks is one of the slow parts.
    - MrLeap 3y ago
      HN doesn't host pictures or video. I come here well before I do twitter, reddit, facebook, youtube or anywhere else on the internet commons. I hazard there's a verdant, halcyon prairie between the two zones. The common zone of maximum extraction, and the hypothetical 200$-a-month-operating-costs text only twitter that can service 5% of the entire planet.
      - tehjoker 3y ago
        Perhaps, but you'd be missing a lot of good memes
  - nemothekid 3y ago
    When I look at Twitter, I think first about the business model, and then I think about the app. I think, today, it's a lot easier to build Twitter the user facing app, even at scale. Where I see your engineering headcount blow up is in moderation, legal and advertising; which I think everyone skips over when doing their mental math. Campaign management, targeting, reporting, and optimization on such a large corpus of data is where I struggle to see even a team of 5 building that in a couple of weekends.
  - sgarland 3y ago
    > 200 million/sec OK, but you aren’t just pushing raw ASCII across the wire, you’re getting those results out of a database. For a weekend clone, someone will probably just use an RDBMS (a very reasonable choice). I’ll be generous and say they can retrieve a given tweet in sub-msec time, say 10 usec. Suddenly, you can only send 100,000 tweets/sec, ignoring all other overhead (which of course you can’t). I’m sure there is unoptimized code and some unnecessary bloat, but fundamentally there are limits on data retrieval that are very, very hard to get close to.
cloudquelle 3y ago
It’s not about features, it’s about their algorithms. Think about TikTok: It’s essentially just an app that lets you scroll through videos, yet it became breathtakingly popular just due to the fact that they seem to know exactly which videos you may want to see.
pjerem 3y ago
Also, for Twitter, it may be easy to clone the "public facing" product which is basically a list of filtered tweets from the DB with probably some tweaks for the scale. But the point is that this product is not what makes the company money. If you only consider this, sure it’s a easy to clone product, backoffice for the advertisers, algorithms to maximize engagement, external integrations, marketing, PRs, partnerships are the thing that brings money home. Of course if there was no Twitter, I imagine you could earn some money/be profitable with just the public product and some basic backend for ads publishers. But you’ll probably encounter a revenue bottleneck and start researching fresh income which you can do by … becoming Twitter, the big company.
danjac 3y ago
That appears to be an iceberg problem. While the public UI of Twitter is simple and is often used as the basis of a starter project in training courses, there's also likely to be a lot of backend admin tools for things like moderation, advertising, sales, etc (not to mention as others have the scaling: at this level it's not so much a "solved" problem but one that requires constant monitoring, tweaking, and preventing newly-minted CEOs from literally pulling the plug on running servers).
FragrantRiver 3y ago
The weekend is coming up. I bet you a $100 that you can't do that.
- Sohcahtoa82 3y ago
  I never claimed to be a competent full-stack web developer. I'm an AppSec engineer. I don't write a lot of code, and when I do, it's entirely scripts and occasionally modifying existing code (both front- and back-end, but more often back-end) to implement security fixes. That said, I could probably hobble it together. It'd be ugly as sin with hand-written Web 1.0 style HTML and likely server-side rendering since I hate writing front-end code, but it really wouldn't be hard as long as you're keeping it simple - Just text posts (No video/pictures, though maybe that wouldn't be too hard to add?), no algorithm for engagement (just chronological ordered tweets), no analytics to determine what's trending. Obviously no mobile app.
nonameiguess 3y ago
The NSA actually created clones of Twitter, Facebook, and ironically enough, tumblr back in maybe 2014 or so? In order to be able to deploy classified versions on the JWICS network that people could use for cross-agency discussion that can't happen on the Internet. I don't think it took them much more than a weekend to do all three and they're maintained by a only a few people working in their spare time. The obvious differences being the scale, not really having to worry about the complexities of running your own data centers and overlay networks or whatever, and the fact there is no advertising, no sales, so none of the clutter that comes from trying to optimize that as well as defeat your user's attempts to subvert being tracked, nor any tracking of what you're doing off-site. The signup process is really nice, though. You don't need individual accounts on each site. They pull your identity automatically from your IC PKI client cert and create the account for you when you first use it. I work from home now, so I'm not on there any more, but they called the Twitter clone "eChirp" and the Facebook one "Tapioca." They also clone Wikipedia and Stack Overflow, but those are just read-only mirrors that get updated once a day.
philjohn 3y ago
Cool. Now, how are you going to report CSAM to NCMEC, or human trafficking to law enforcement as per Sesta-Fosta? And then how are you going to ensure you don't provide content rabbit holes that can be harmful to mental health. Then detect credible cases of people threatening suicide to get a welfare check run. That's just one of the things you don't see on the surface, there are hundreds more functions like that, that you'll very quickly realise you need to do so you don't get fined out of existence and/or end up with execs going to jail.

liquidpele 3y ago

Not just scaling it… scaling it in a way that doesn’t immediately bankrupt you. Bit of a chicken and egg problem with advertisers.

MrLeap 3y ago
This is one of those _things_ that I suspect causes many greybeards to grimace. What's a tweet? We can haggle this up and down so cut me some slack. 200 bytes? 50 bytes? gzipped -90%, 5 bytes. Holy shit what absurd amount of traffic could a 1gb fiber connection handle? oh huh 200 million tweets a second. That's silly. How wrong could I be? -- then you run down the numbers and you realize that no matter how wrong you are, a twitter-clone-for-most-people oughta be cheap. The network effects and fuzzy human reasons are their moat, nothing else. One of those fuzzy human reasons is how amazon EC2 eats your ass. Just gobbles it from stem to stern.
- tehjoker 3y ago
  They have pictures and video now. You also have to sort them in real time, check for blocked, mutes, etc, and each tweet is amplified to followers so one tweet requires e.g. 700 copies or 1M copies depending on follower count. The text part is basically solved at this point, but not quite that easy. I hear dealing with blocks is one of the slow parts.
  - MrLeap 3y ago
    HN doesn't host pictures or video. I come here well before I do twitter, reddit, facebook, youtube or anywhere else on the internet commons. I hazard there's a verdant, halcyon prairie between the two zones. The common zone of maximum extraction, and the hypothetical 200$-a-month-operating-costs text only twitter that can service 5% of the entire planet.
    - tehjoker 3y ago
      Perhaps, but you'd be missing a lot of good memes
- nemothekid 3y ago
  When I look at Twitter, I think first about the business model, and then I think about the app. I think, today, it's a lot easier to build Twitter the user facing app, even at scale. Where I see your engineering headcount blow up is in moderation, legal and advertising; which I think everyone skips over when doing their mental math. Campaign management, targeting, reporting, and optimization on such a large corpus of data is where I struggle to see even a team of 5 building that in a couple of weekends.
- sgarland 3y ago
  > 200 million/sec OK, but you aren’t just pushing raw ASCII across the wire, you’re getting those results out of a database. For a weekend clone, someone will probably just use an RDBMS (a very reasonable choice). I’ll be generous and say they can retrieve a given tweet in sub-msec time, say 10 usec. Suddenly, you can only send 100,000 tweets/sec, ignoring all other overhead (which of course you can’t). I’m sure there is unoptimized code and some unnecessary bloat, but fundamentally there are limits on data retrieval that are very, very hard to get close to.

cloudquelle 3y ago

It’s not about features, it’s about their algorithms. Think about TikTok: It’s essentially just an app that lets you scroll through videos, yet it became breathtakingly popular just due to the fact that they seem to know exactly which videos you may want to see.

pjerem 3y ago

Also, for Twitter, it may be easy to clone the "public facing" product which is basically a list of filtered tweets from the DB with probably some tweaks for the scale. But the point is that this product is not what makes the company money. If you only consider this, sure it’s a easy to clone product, backoffice for the advertisers, algorithms to maximize engagement, external integrations, marketing, PRs, partnerships are the thing that brings money home. Of course if there was no Twitter, I imagine you could earn some money/be profitable with just the public product and some basic backend for ads publishers. But you’ll probably encounter a revenue bottleneck and start researching fresh income which you can do by … becoming Twitter, the big company.

danjac 3y ago

That appears to be an iceberg problem. While the public UI of Twitter is simple and is often used as the basis of a starter project in training courses, there's also likely to be a lot of backend admin tools for things like moderation, advertising, sales, etc (not to mention as others have the scaling: at this level it's not so much a "solved" problem but one that requires constant monitoring, tweaking, and preventing newly-minted CEOs from literally pulling the plug on running servers).

FragrantRiver 3y ago

The weekend is coming up. I bet you a $100 that you can't do that.

Sohcahtoa82 3y ago
I never claimed to be a competent full-stack web developer. I'm an AppSec engineer. I don't write a lot of code, and when I do, it's entirely scripts and occasionally modifying existing code (both front- and back-end, but more often back-end) to implement security fixes. That said, I could probably hobble it together. It'd be ugly as sin with hand-written Web 1.0 style HTML and likely server-side rendering since I hate writing front-end code, but it really wouldn't be hard as long as you're keeping it simple - Just text posts (No video/pictures, though maybe that wouldn't be too hard to add?), no algorithm for engagement (just chronological ordered tweets), no analytics to determine what's trending. Obviously no mobile app.

nonameiguess 3y ago

The NSA actually created clones of Twitter, Facebook, and ironically enough, tumblr back in maybe 2014 or so? In order to be able to deploy classified versions on the JWICS network that people could use for cross-agency discussion that can't happen on the Internet. I don't think it took them much more than a weekend to do all three and they're maintained by a only a few people working in their spare time. The obvious differences being the scale, not really having to worry about the complexities of running your own data centers and overlay networks or whatever, and the fact there is no advertising, no sales, so none of the clutter that comes from trying to optimize that as well as defeat your user's attempts to subvert being tracked, nor any tracking of what you're doing off-site. The signup process is really nice, though. You don't need individual accounts on each site. They pull your identity automatically from your IC PKI client cert and create the account for you when you first use it. I work from home now, so I'm not on there any more, but they called the Twitter clone "eChirp" and the Facebook one "Tapioca." They also clone Wikipedia and Stack Overflow, but those are just read-only mirrors that get updated once a day.

philjohn 3y ago

Cool. Now, how are you going to report CSAM to NCMEC, or human trafficking to law enforcement as per Sesta-Fosta? And then how are you going to ensure you don't provide content rabbit holes that can be harmful to mental health. Then detect credible cases of people threatening suicide to get a welfare check run. That's just one of the things you don't see on the surface, there are hundreds more functions like that, that you'll very quickly realise you need to do so you don't get fined out of existence and/or end up with execs going to jail.