10 ms·

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think th

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or other in place of the text of the quoted tweet itself but regardless a tweet takes up more than 280 unicode characters. Also, nowhere in the article are hashtags mentioned. For a system like this to work you need some indexing of hashtags so you aren't doing a full scan of the entire tweet text of every tweet anytime someone decides to search for #YOLO. The system as proposed is missing a highly critical feature of the platform it purports to emulate. I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform, apart from the tweets themselves.

chippiewill 4y ago
> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform, apart from the tweets themselves. I'd probably go as far to say that the indexes _generally_ at twitter could be larger than the tweets
trishume 4y ago
Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashtag-only search might fit, mainly because my impression is people doing hashtag searches are a small fraction of traffic nowadays so the main cost is disk, not sure though. I did run the post by 5 ex-Twitter engineers and none of them said any of my estimates were super wrong, mainly just brought up additional features and things I didn't discuss (which I edited into the post before publishing). Still possible that they just didn't divulge or didn't know some number they knew that I estimated very wrong.
- mr90210 4y ago
  [flagged]
  - Aeolun 4y ago
    I’m not sure why you are asking him to do the thing you’ve literally quoted him doing?
    - literallyroy 4y ago
      I believed he was saying the author should run his idea of indexes on disk taking up a lot of space by the engineers.
  - 867-5309 4y ago
    >Absence of evidence is not evidence of absence. >possible that they just didn't divulge
  - lelandfe 4y ago
    If an inspector reviews your house and finds no issues, that is indeed evidence of absence.
    - icedchai 4y ago
      Not necessarily. The inspector could be corrupt, bad at their job, whatever.
    - quickthrower2 4y ago
      But evidence is not necessarily proof.
      - lelandfe 4y ago
        Sure, but if someone accuses your house of having issues, and you retort that you've had it inspected by professionals, a reply of "Hah! That's evidence, not proof!" is just a bit smarmy.
Xeoncross 4y ago
> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. While true, and not to take away from the parent comment, I've noticed that the size of things is often partially the result of scaling out horizontally. Most companies I've worked at end up with a lot of duplicate records as each subsystem might want a copy or to cache a copy.
- vidarh 4y ago
  This is indeed a problem, and one of the reasons to be careful about knowing you can fit things in one machine before taking that approach, because the moment you're forced to move to a distributed model it's rarely one machine to two but one machine to a dozen coupled with a major rearchitecturing effort at just the wrong moment. It's often fine to start without a fully decoupled system (net present value of the time and money needed to scale out might be far too high), but you need to know whether or not it's likely to come and what to look for so you can start preparing in time.
mrtksn 4y ago
Twitter's "linked" tweets seems to be strangely unattached from context. What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database. So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously. I also heard that a method for spamming Twitter trending topics is to send tweets and delete them quickly. My impression is that Twitter is big on real time processing. They definitely don't search the entire database for #YOLO tweets, instead they seem to be searching the almost-live stuff and some archived stuff(probably ranked and saved as noteworthy tweets or something).
- madeofpalk 4y ago
  Old stuff on twitter is weird. Tweets seem to eventually forget that You specifically liked a tweet, and will allow you to like a tweet again, and the like count will reflect you liking it twice.
  - kaba0 4y ago
    Like count is almost surely just a locally incremented number, your like request will eventually get processed and will be dropped if it was already liked. It only has to be an eventually consistent value.
    - madeofpalk 4y ago
      This is beyond optimistic updates (which twitter also has plenty of issues with - you'll reply to your tweet and it'll jump immediately to two replies, but on refresh just show your one).
walrus01 4y ago
> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform I really wonder how much of a challenge this is and how much it occupies, not even talking about disk, but continuing the theoretical exercise in the linked URL, you can get 1U size servers with 2TB of RAM these days.
api 4y ago
I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of the water, but... maybe not by as much as people think...? I would not be surprised if a very performance-engineered version of Twitter could run on a few dozen racks full of beefy machines like this with HPC-grade super-fast fabric interconnects. I have a strong sense that most large scale systems are way less efficient than what's possible. They trade development ease, modularity, and velocity for performance by using a lot of microservices, flabby but easy and flexible protocols (e.g. JSON over HTTP), slow dynamic languages, and layers of abstraction that could be integrated a lot more tightly. Of course that may be a rational trade-off if velocity, flexibility, and labor costs and logistics matter more than hardware, power, or data center floor space costs.
- BeefWellington 4y ago
  I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. I agree mostly. Where I differ in that I would argue that hashtags were THE thing that Twitter is most known for but that could be a perspective from having been on the platform for forever and a day and I recognize not everyone may make that same association anymore.
  - saagarjha 4y ago
    FWIW my opinion of hashtags (relatively new to Twitter) is that they’re only used by brands and mostly cringe people
    - jwmoz 4y ago
      Originally they were the main feature I'd say. Now they've been deprecated by the algorithm that surfaces and recommends stuff.
vidarh 4y ago
Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision. While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet views.
- wtallis 4y ago
  A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.
  - poslathian 4y ago
    Best comment. And one you don’t hear nearly enough from typical product and eng managers.
    - tinyspacewizard 4y ago
      Ehhh a large company would much prefer: $$$$$$ income on $$$ costs to $$$ income on $ costs
      - gonzo 4y ago
        Perhaps. But VCs prefer the first.
        yuhong 4y ago
        [dead]
  - serverholic 4y ago
    I think about this often. Specifically, how much bloat exists in the world because individuals in our society are forced to justify their existence on a daily basis. Everyone has to be employed so it's better to keep adding more crap to products and make those products disposable in order to give people a job.
dspillett 4y ago
The hashtag index is not going to be any bigger than the tweets storage though, and may be significantly smaller, so this part is not if by an order of magnitude (even a binary one). Assuming something like a common SQL database is used for storage there would be a tags table (one row per unique tag, tag string plus a numeric identifier, indexed by both which bloats the size a bit but it'll still be small) and a link table (one row per tag in a message, containing the tag and message ids). Even if using 64-bit IDs because you don't want to fall over at 2 thousand million messages (or 4, if your DB supports unsigned types or you start at MININT instead of zero or one) then that structure is going to consume about 32 bytes per tag per message (plus some static overheads and a little more for non-leaf index pages of course). In theory this could be the same size as the messages table or even larger (if most messages contain many small tags), but in practise it is going to be significantly smaller. Yes, this would be big enough to need specifically factoring into a real implementation design. But it would not be big enough to invalidate the proposed idea so I understand leaving it off, at least initially, to simplify the thought experiment. Similarly to support a message responding to, or quoting, a single other you only need one or two NULLable ID references, 16 bytes per message, which will likely be dwarfed by the message text in the average case. Given it likely makes sense to use something like SQL Server's compression options for data like this the average load imposed will be much smaller than 16 bytes/message. We are fiddling, fairly insignificantly, measurable but to massively, with constants a & b in O(a+bN) here, so the storage problem is still essentially of order O(N) [where N is total length of the stored messages].
gorgoiler 4y ago
Isn’t a hashtag just another kind of user account — an account from which anyone can post? The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.
- bithaze 4y ago
  Hashtags aren't like user accounts, no - they're strings that are part of a tweet. In theory, a separate data structure shouldn't be needed since you can just search the full text of tweets, but in practice, I don't know how that scales for the number of all-time tweets.
  - HyperSane 4y ago
    I think he was talking about how hashtags are implemented.
- deleted 4y ago
  [deleted]
redbell 4y ago
> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. I want to add another concept that may impact, considerably, the storage, which is "threads". I'm not sure what is the percentage of threads/tweets but what I consider an important factor is that threads do not have a maximum number of characters.
knorker 4y ago
I would be surprised if indexes were not larger than the raw tweets. Text search, hashtag index, some structured data for popular tweets, etc... In order to deliver search results I would not be surprised if tweets are duplicated/denormalized, for quick search/lookup.
zxcvbn4038 4y ago
This is absolutely the sort of thing I wish more developers did - and I think the good ones already do. Most of what you find in blogs will work just fine at 1 request per second (OMG! 1M Hits!!) or 10 requests per second (and I think someone did post their “how I scaled to 10 million hits per month” blog to Hacker News once which sounds impressive until you do the math) but when you get into thousands of requests per second you really do need to understand the network stack, the different storage tiers, your choice of algorithms, how to interact with CDNs, etc. a lot more then any blog will tell you. When interviewing developers I always ask them what is the largest public web site they ever worked on and then probe about performance issues they encountered and how they resolved them in order gague how far along they are in their skill development. I would never plan to run a production service on a single server just because coordinating changes in the active dataset among two or more production servers often changes your design significantly, and you want to plan for that because the consumer grade hardware we all use has a nasty habit of not working after power cycles (which still tends to be the most strain a system goes through, even in a world of SSD storage).

chippiewill 4y ago

> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform, apart from the tweets themselves. I'd probably go as far to say that the indexes _generally_ at twitter could be larger than the tweets

trishume 4y ago

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashtag-only search might fit, mainly because my impression is people doing hashtag searches are a small fraction of traffic nowadays so the main cost is disk, not sure though. I did run the post by 5 ex-Twitter engineers and none of them said any of my estimates were super wrong, mainly just brought up additional features and things I didn't discuss (which I edited into the post before publishing). Still possible that they just didn't divulge or didn't know some number they knew that I estimated very wrong.

mr90210 4y ago
[flagged]
- Aeolun 4y ago
  I’m not sure why you are asking him to do the thing you’ve literally quoted him doing?
  - literallyroy 4y ago
    I believed he was saying the author should run his idea of indexes on disk taking up a lot of space by the engineers.
- 867-5309 4y ago
  >Absence of evidence is not evidence of absence. >possible that they just didn't divulge
- lelandfe 4y ago
  If an inspector reviews your house and finds no issues, that is indeed evidence of absence.
  - icedchai 4y ago
    Not necessarily. The inspector could be corrupt, bad at their job, whatever.
  - quickthrower2 4y ago
    But evidence is not necessarily proof.
    - lelandfe 4y ago
      Sure, but if someone accuses your house of having issues, and you retort that you've had it inspected by professionals, a reply of "Hah! That's evidence, not proof!" is just a bit smarmy.

Xeoncross 4y ago

> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. While true, and not to take away from the parent comment, I've noticed that the size of things is often partially the result of scaling out horizontally. Most companies I've worked at end up with a lot of duplicate records as each subsystem might want a copy or to cache a copy.

vidarh 4y ago
This is indeed a problem, and one of the reasons to be careful about knowing you can fit things in one machine before taking that approach, because the moment you're forced to move to a distributed model it's rarely one machine to two but one machine to a dozen coupled with a major rearchitecturing effort at just the wrong moment. It's often fine to start without a fully decoupled system (net present value of the time and money needed to scale out might be far too high), but you need to know whether or not it's likely to come and what to look for so you can start preparing in time.

mrtksn 4y ago

Twitter's "linked" tweets seems to be strangely unattached from context. What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database. So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously. I also heard that a method for spamming Twitter trending topics is to send tweets and delete them quickly. My impression is that Twitter is big on real time processing. They definitely don't search the entire database for #YOLO tweets, instead they seem to be searching the almost-live stuff and some archived stuff(probably ranked and saved as noteworthy tweets or something).

madeofpalk 4y ago
Old stuff on twitter is weird. Tweets seem to eventually forget that You specifically liked a tweet, and will allow you to like a tweet again, and the like count will reflect you liking it twice.
- kaba0 4y ago
  Like count is almost surely just a locally incremented number, your like request will eventually get processed and will be dropped if it was already liked. It only has to be an eventually consistent value.
  - madeofpalk 4y ago
    This is beyond optimistic updates (which twitter also has plenty of issues with - you'll reply to your tweet and it'll jump immediately to two replies, but on refresh just show your one).

walrus01 4y ago

> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform I really wonder how much of a challenge this is and how much it occupies, not even talking about disk, but continuing the theoretical exercise in the linked URL, you can get 1U size servers with 2TB of RAM these days.

api 4y ago

I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of the water, but... maybe not by as much as people think...? I would not be surprised if a very performance-engineered version of Twitter could run on a few dozen racks full of beefy machines like this with HPC-grade super-fast fabric interconnects. I have a strong sense that most large scale systems are way less efficient than what's possible. They trade development ease, modularity, and velocity for performance by using a lot of microservices, flabby but easy and flexible protocols (e.g. JSON over HTTP), slow dynamic languages, and layers of abstraction that could be integrated a lot more tightly. Of course that may be a rational trade-off if velocity, flexibility, and labor costs and logistics matter more than hardware, power, or data center floor space costs.

BeefWellington 4y ago
I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. I agree mostly. Where I differ in that I would argue that hashtags were THE thing that Twitter is most known for but that could be a perspective from having been on the platform for forever and a day and I recognize not everyone may make that same association anymore.
- saagarjha 4y ago
  FWIW my opinion of hashtags (relatively new to Twitter) is that they’re only used by brands and mostly cringe people
  - jwmoz 4y ago
    Originally they were the main feature I'd say. Now they've been deprecated by the algorithm that surfaces and recommends stuff.

vidarh 4y ago

Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision. While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet views.

wtallis 4y ago
A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.
- poslathian 4y ago
  Best comment. And one you don’t hear nearly enough from typical product and eng managers.
  - tinyspacewizard 4y ago
    Ehhh a large company would much prefer: $$$$$$ income on $$$ costs to $$$ income on $ costs
    - gonzo 4y ago
      Perhaps. But VCs prefer the first.
      - yuhong 4y ago
        [dead]
- serverholic 4y ago
  I think about this often. Specifically, how much bloat exists in the world because individuals in our society are forced to justify their existence on a daily basis. Everyone has to be employed so it's better to keep adding more crap to products and make those products disposable in order to give people a job.

dspillett 4y ago

The hashtag index is not going to be any bigger than the tweets storage though, and may be significantly smaller, so this part is not if by an order of magnitude (even a binary one). Assuming something like a common SQL database is used for storage there would be a tags table (one row per unique tag, tag string plus a numeric identifier, indexed by both which bloats the size a bit but it'll still be small) and a link table (one row per tag in a message, containing the tag and message ids). Even if using 64-bit IDs because you don't want to fall over at 2 thousand million messages (or 4, if your DB supports unsigned types or you start at MININT instead of zero or one) then that structure is going to consume about 32 bytes per tag per message (plus some static overheads and a little more for non-leaf index pages of course). In theory this could be the same size as the messages table or even larger (if most messages contain many small tags), but in practise it is going to be significantly smaller. Yes, this would be big enough to need specifically factoring into a real implementation design. But it would not be big enough to invalidate the proposed idea so I understand leaving it off, at least initially, to simplify the thought experiment. Similarly to support a message responding to, or quoting, a single other you only need one or two NULLable ID references, 16 bytes per message, which will likely be dwarfed by the message text in the average case. Given it likely makes sense to use something like SQL Server's compression options for data like this the average load imposed will be much smaller than 16 bytes/message. We are fiddling, fairly insignificantly, measurable but to massively, with constants a & b in O(a+bN) here, so the storage problem is still essentially of order O(N) [where N is total length of the stored messages].

gorgoiler 4y ago

Isn’t a hashtag just another kind of user account — an account from which anyone can post? The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.

bithaze 4y ago
Hashtags aren't like user accounts, no - they're strings that are part of a tweet. In theory, a separate data structure shouldn't be needed since you can just search the full text of tweets, but in practice, I don't know how that scales for the number of all-time tweets.
- HyperSane 4y ago
  I think he was talking about how hashtags are implemented.
deleted 4y ago
[deleted]

redbell 4y ago

> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. I want to add another concept that may impact, considerably, the storage, which is "threads". I'm not sure what is the percentage of threads/tweets but what I consider an important factor is that threads do not have a maximum number of characters.

knorker 4y ago

I would be surprised if indexes were not larger than the raw tweets. Text search, hashtag index, some structured data for popular tweets, etc... In order to deliver search results I would not be surprised if tweets are duplicated/denormalized, for quick search/lookup.

zxcvbn4038 4y ago

This is absolutely the sort of thing I wish more developers did - and I think the good ones already do. Most of what you find in blogs will work just fine at 1 request per second (OMG! 1M Hits!!) or 10 requests per second (and I think someone did post their “how I scaled to 10 million hits per month” blog to Hacker News once which sounds impressive until you do the math) but when you get into thousands of requests per second you really do need to understand the network stack, the different storage tiers, your choice of algorithms, how to interact with CDNs, etc. a lot more then any blog will tell you. When interviewing developers I always ask them what is the largest public web site they ever worked on and then probe about performance issues they encountered and how they resolved them in order gague how far along they are in their skill development. I would never plan to run a production service on a single server just because coordinating changes in the active dataset among two or more production servers often changes your design significantly, and you want to plan for that because the consumer grade hardware we all use has a nasty habit of not working after power cycles (which still tends to be the most strain a system goes through, even in a world of SSD storage).