4 ms·

I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers an

by pornel 2y ago

I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served). When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.

ericvolp12 2y ago
This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.
- curious_cat_163 2y ago
  That's insightful. Keep up the good work!
- Xunjin 2y ago
  Just to be clear, you are a Bluesky engineer, right? off-topic: how has been dealing with the influx of new users after X political/legals problems aftermath? Did you see an increase in toxicity around the network? And how has you (Bluesky moderation) dealing with it.
  - ToucanLoucan 2y ago
    [flagged]
    - deleted 2y ago
      [deleted]
    - Sloowms 2y ago
      You're not less partisan if you prefer a slimmer range of political leanings.
    - breakyerself 2y ago
      There's nothing wrong with being partisan if you're partisan against fascists who want to destroy democracy and the rule of law.
      - Imustaskforhelp 2y ago
        Funny how you call trump administration fascist. (theoretically its anti fascist but its still bad , Taking from the description of the video since this was what immediately ringed when you said trump===fascism The liberal theory of the rise of Trumpism and its supposed fascistic features is inadequate in both effectively analysing and offering solutions to the present situation. Liberals often personalise or individualise people like Donald Trump and Elon Musk, casting them as deviations, as opposed to manifestations of class society. Class analysis suggests that fascism was a unique response to growing anti-capitalist organisations, socialist and/or anarchist, gaining prominence and posing threats to the economic base. The owning class required a mass movement which enveloped otherwise disillusioned people into a political project which had the collectivist, anti-free market appeal that socialist and anarchist organisations had, but nonetheless committed to solidifying and strengthening the economic base and profit motive. In modern America, no such anti-capitalist threat exists. Neoliberalism has created significant disillusionment with mainstream social and political institutions and systems, but this disillusionment hasn’t been captured by anti-capitalist forces, but rather by the populist right. As such, the populist right doesn’t need to give up the economic game, i.e. free markets, deregulation, privatisation, austerity, etc (with the exception of tariffs), but can purely rely on minorities as scapegoats in a constructed culture war, such as immigrants, ‘wokeness’, transgender people, etc. Therefore, capital doesn’t need to be subordinated to the nation-state, like pursued by contemporary fascist governments. Rather, in this ‘inverted’ fascism, capital takes over and exploits the state in a rather oligarchic manner. https://www.youtube.com/watch?v=pqdLwkyfLdM https://www.youtube.com/watch?v=pqdLwkyfLdM This video is really great , I spent 10 minutes looking for this. I am not a trump supporter , The title might be a little clickbaity (basically the opposite of what it really is) You might find it really great. It is one of the best videos I have ever watched on politics.
        meowface 2y ago
- petra 2y ago
  Maybe this would be helpful:http://daslab.seas.harvard.edu/datacalculator/ http://daslab.seas.harvard.edu/datacalculator/
locusofself 2y ago
Why do they "insert" even non-celebrity posts into each follower's timeline? That is not intuitive to me.
- giovannibonetti 2y ago
  To serve a user timeline in single-digit milliseconds, it is not practical for a data store to load each item in a different place. Even with an index, the index itself can be contiguous in disk, but the payload is scattered all over the place if you keep it in a single large table. Instead, you can drastically speed up performance if you are able to store data for each timeline somewhat contiguously on disk.
- wlonkly 2y ago
  Think of it as pre-rendering. Of pre-rendering and JIT collecting, pre-rendering means more work but it's async, and it means the timeline is ready whenever a user requests it, to give a fast user experience. (Although I don't understand the "non-celebrity" part of your comment -- the timeline contains (pointers to) posts from whoever someone follows, and doesn't care who those people are.)
  - locusofself 2y ago
    Perhaps I misunderstanding, I thought the actual content of each tweet was being duplicated to every single timeline who followed the author, which sounded extremely wasteful, especially in the case of someone who has 200 million followers.
    - TimK65 2y ago
      From the linked article: "Additionally, a reference to your post is 'fanned out' to your followers so they can see it in their Timelines." So not the content, just a sort of link to it.
rubslopes 2y ago
This problem is discussed in the beginning of the Designing Data-Intensive Applications book. It's worth a read!
- Brystephor 2y ago
  Do you know the name of the problem or strategy used for solving the problem? I'd be interested in looking it up! I own DDIA but after a few chapters of how database work behind the scenes, I begin to fall asleep. I have trouble understanding how to apply the knowledge to my work but this seems like a useful thing with a more clear application.
  - bitbckt 2y ago
    Yes, we used the Yahoo! “Feeding Frenzy” paper as the basis for the design of Haplocheirus (the timeline service).
VWWHFSfQ 2y ago
At some point they'll end up just doing the Bieber rack [1]. It's when a shard becomes so hot that it just has to be its own thing entirely. [1] - https://www.themarysue.com/twitter-justin-bieber-servers/ https://www.themarysue.com/twitter-justin-bieber-servers/ @bluesky devs, don't feel ashamed for doing this. It's exactly how to scale these kinds of extreme cases.
- stavros 2y ago
  Given that BlueSky is funded by Twitter, I'm assuming they know a lot more than us on how Twitter architects systems.
- genewitch 2y ago
  I've stood up machines for this before I did not know they had a name, and I worked at the mouse company and my parking spot was two over from a J. Beibe'rs spot. So now we have Slashdot effect, HN hug, and its not Clarkson its... Stephen Fry effect? Maybe can be Cross-Discipline - there's a term for when lots of UK turns their kettles on at the same time. I should make a blog post to record all the ones I can remember.
  - k1t 2y ago
    TV Pickup aka the Half Time Kettle Effect. https://en.wikipedia.org/wiki/TV_pickup https://en.wikipedia.org/wiki/TV_pickup
- bitbckt 2y ago
  We never actually had a literal “Bieber Box”, but the joke took off. Hot shards were definitely an issue, though.
- Imustaskforhelp 2y ago
  Its so crazy. Thanks a lot for sharing this link.
rsynnott 2y ago
> and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline I think then you still have the 'weird user who follows hundreds of thousands of people' problem, just at read time instead of write time. It's unclear that this is _better_, though, yeah, caching might help. But if you follow every celeb on Bluesky (and I guarantee you this user exists) you'd be looking at fetching and merging _thousands_ of timelines (again, I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users). Given the nature of the service, making read predictably cheap and writes potentially expensive (which seems to be the way they've gone) seems like a defensible practice.
- christkv 2y ago
  You might mix the approaches based on some cut off point
- fc417fc802 2y ago
  > I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users Random sampling? It's not as though the user needs thousands of posts returned for a single fetch. Scrolling down and seeing some stuff that's not in chronological order seems like an acceptable tradeoff.

by pornel 2y ago

ericvolp12 2y ago

This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.

curious_cat_163 2y ago
That's insightful. Keep up the good work!
Xunjin 2y ago
Just to be clear, you are a Bluesky engineer, right? off-topic: how has been dealing with the influx of new users after X political/legals problems aftermath? Did you see an increase in toxicity around the network? And how has you (Bluesky moderation) dealing with it.
- ToucanLoucan 2y ago
  [flagged]
  - deleted 2y ago
    [deleted]
  - Sloowms 2y ago
    You're not less partisan if you prefer a slimmer range of political leanings.
  - breakyerself 2y ago
    There's nothing wrong with being partisan if you're partisan against fascists who want to destroy democracy and the rule of law.
    - Imustaskforhelp 2y ago
      Funny how you call trump administration fascist. (theoretically its anti fascist but its still bad , Taking from the description of the video since this was what immediately ringed when you said trump===fascism The liberal theory of the rise of Trumpism and its supposed fascistic features is inadequate in both effectively analysing and offering solutions to the present situation. Liberals often personalise or individualise people like Donald Trump and Elon Musk, casting them as deviations, as opposed to manifestations of class society. Class analysis suggests that fascism was a unique response to growing anti-capitalist organisations, socialist and/or anarchist, gaining prominence and posing threats to the economic base. The owning class required a mass movement which enveloped otherwise disillusioned people into a political project which had the collectivist, anti-free market appeal that socialist and anarchist organisations had, but nonetheless committed to solidifying and strengthening the economic base and profit motive. In modern America, no such anti-capitalist threat exists. Neoliberalism has created significant disillusionment with mainstream social and political institutions and systems, but this disillusionment hasn’t been captured by anti-capitalist forces, but rather by the populist right. As such, the populist right doesn’t need to give up the economic game, i.e. free markets, deregulation, privatisation, austerity, etc (with the exception of tariffs), but can purely rely on minorities as scapegoats in a constructed culture war, such as immigrants, ‘wokeness’, transgender people, etc. Therefore, capital doesn’t need to be subordinated to the nation-state, like pursued by contemporary fascist governments. Rather, in this ‘inverted’ fascism, capital takes over and exploits the state in a rather oligarchic manner. https://www.youtube.com/watch?v=pqdLwkyfLdM https://www.youtube.com/watch?v=pqdLwkyfLdM This video is really great , I spent 10 minutes looking for this. I am not a trump supporter , The title might be a little clickbaity (basically the opposite of what it really is) You might find it really great. It is one of the best videos I have ever watched on politics.
      - meowface 2y ago
petra 2y ago
Maybe this would be helpful:http://daslab.seas.harvard.edu/datacalculator/ http://daslab.seas.harvard.edu/datacalculator/

locusofself 2y ago

Why do they "insert" even non-celebrity posts into each follower's timeline? That is not intuitive to me.

giovannibonetti 2y ago
To serve a user timeline in single-digit milliseconds, it is not practical for a data store to load each item in a different place. Even with an index, the index itself can be contiguous in disk, but the payload is scattered all over the place if you keep it in a single large table. Instead, you can drastically speed up performance if you are able to store data for each timeline somewhat contiguously on disk.
wlonkly 2y ago
Think of it as pre-rendering. Of pre-rendering and JIT collecting, pre-rendering means more work but it's async, and it means the timeline is ready whenever a user requests it, to give a fast user experience. (Although I don't understand the "non-celebrity" part of your comment -- the timeline contains (pointers to) posts from whoever someone follows, and doesn't care who those people are.)
- locusofself 2y ago
  Perhaps I misunderstanding, I thought the actual content of each tweet was being duplicated to every single timeline who followed the author, which sounded extremely wasteful, especially in the case of someone who has 200 million followers.
  - TimK65 2y ago
    From the linked article: "Additionally, a reference to your post is 'fanned out' to your followers so they can see it in their Timelines." So not the content, just a sort of link to it.

rubslopes 2y ago

This problem is discussed in the beginning of the Designing Data-Intensive Applications book. It's worth a read!

Brystephor 2y ago
Do you know the name of the problem or strategy used for solving the problem? I'd be interested in looking it up! I own DDIA but after a few chapters of how database work behind the scenes, I begin to fall asleep. I have trouble understanding how to apply the knowledge to my work but this seems like a useful thing with a more clear application.
- bitbckt 2y ago
  Yes, we used the Yahoo! “Feeding Frenzy” paper as the basis for the design of Haplocheirus (the timeline service).

VWWHFSfQ 2y ago

At some point they'll end up just doing the Bieber rack [1]. It's when a shard becomes so hot that it just has to be its own thing entirely. [1] - https://www.themarysue.com/twitter-justin-bieber-servers/ https://www.themarysue.com/twitter-justin-bieber-servers/ @bluesky devs, don't feel ashamed for doing this. It's exactly how to scale these kinds of extreme cases.

stavros 2y ago
Given that BlueSky is funded by Twitter, I'm assuming they know a lot more than us on how Twitter architects systems.
genewitch 2y ago
I've stood up machines for this before I did not know they had a name, and I worked at the mouse company and my parking spot was two over from a J. Beibe'rs spot. So now we have Slashdot effect, HN hug, and its not Clarkson its... Stephen Fry effect? Maybe can be Cross-Discipline - there's a term for when lots of UK turns their kettles on at the same time. I should make a blog post to record all the ones I can remember.
- k1t 2y ago
  TV Pickup aka the Half Time Kettle Effect. https://en.wikipedia.org/wiki/TV_pickup https://en.wikipedia.org/wiki/TV_pickup
bitbckt 2y ago
We never actually had a literal “Bieber Box”, but the joke took off. Hot shards were definitely an issue, though.
Imustaskforhelp 2y ago
Its so crazy. Thanks a lot for sharing this link.

rsynnott 2y ago

> and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline I think then you still have the 'weird user who follows hundreds of thousands of people' problem, just at read time instead of write time. It's unclear that this is _better_, though, yeah, caching might help. But if you follow every celeb on Bluesky (and I guarantee you this user exists) you'd be looking at fetching and merging _thousands_ of timelines (again, I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users). Given the nature of the service, making read predictably cheap and writes potentially expensive (which seems to be the way they've gone) seems like a defensible practice.

christkv 2y ago
You might mix the approaches based on some cut off point
fc417fc802 2y ago
> I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users Random sampling? It's not as though the user needs thousands of posts returned for a single fetch. Scrolling down and seeing some stuff that's not in chronological order seems like an acceptable tradeoff.