15 ms·

When imperfect systems are good: Bluesky's lossy timelines

• nightpool 2y ago
  Note that all of this reflects design decisions on Bluesky's closed-source "AppView" server—any federated servers interacting with Bluesky would need to construct their own timelines, and do not get the benefit of the work described here.
  • xrisk 2y ago
    What reason does Bluesky give for not opening up their AppView code? Another notable component that is closed source is the discovery feed generator, where at least there is some reason.
    • muscomposter 2y ago
      what else? profit by means of doing work that benefits first and foremost the private proprietors of the closed source if they gave it away (which used to be unfeasible until the digital era) they feel they’re loosing their valuable effort which they’re wont on concentrating, not diluting.
    • verdverm 2y ago
      The App View frontend is open source: https://github.com/bluesky-social/social-app https://github.com/bluesky-social/social-app Much of the backend is open source as well: https://github.com/bluesky-social/atproto/tree/main/packages https://github.com/bluesky-social/atproto/tree/main/packages What is not are the extra services they run to provide a better and faster UX. Even if it was open source, it likely costs 10s of thousands to run per month (they have moved largely to "onprem" hardware instead of the cloud aiui)
      • nightpool 2y ago
        That's the frontend code, it doesn't include the backend API services, which are closed source.
        • verdverm 2y ago
          Which is what I said in the second sentence
          • nightpool 2y ago
            AppView is a specific term of art within the Bluesky federation architecture: https://atproto.com/guides/glossary#app-view https://atproto.com/guides/glossary#app-view, you were incorrect in identifying the public frontend repo as the AppView.
            • verdverm 2y ago
              A frontend is (can be) part of an App View. It is quite literally the app you view the network through. There can also be headless app views and app views which have no backend
              • half-kh-hacker 2y ago
                this is not correct
            • verdverm 2y ago
              The glossary specifically calls out the UI as part of an app view. Can you explain why it is not according to you?
        • half-kh-hacker 2y ago
          the backend (the AppView) can be found here: https://github.com/bluesky-social/atproto/tree/main/packages/bsky https://github.com/bluesky-social/atproto/tree/main/packages... there are various supporting services written in Go as well https://github.com/bluesky-social/indigo https://github.com/bluesky-social/indigo
      • half-kh-hacker 2y ago
        that's not the appview, that's the client
        • verdverm 2y ago
          App View is a bit fuzzy of a term. To me it seems like a combination of frontend, backend, custom lexicon, and supporting services. There isn't really another place in the spec or design where clients or browsers fit in, which do in fact provide a view of the network via an app.
          • verdverm 2y ago
            "UI" is part of the definition they give in the glossary https://atproto.com/guides/glossary#app-view https://atproto.com/guides/glossary#app-view
    • dingnuts 2y ago
      when I read the spec it seemed like the operator of an AppView & Relay would be most in need of compensation for their hosting costs due to the amount of demand on those components so I believe the spec allows an operator to implement their own AppView & monetize it as that operator sees fit, so that they can afford to operate the service and maybe even make money off of it so that they can make it their full time jobs.
      • verdverm 2y ago
        It seems this way to me as well. ATProto fundamentally changes how monetization works in social media by removing lockin. It's going to be interesting to see what emerges from this design decision. Another interesting way to view ATProto is that it could be a collection of headless features and network browsers that leverage those feature providers.
    • iameli 2y ago
      I asked this and got > We did a backend rewrite from postgres to scylla and it has a bunch of deployment specific stuff, but is functionally identical to the open source postgres version. Its not really a "v2" in terms of new features, we just made it make use of our hardware really well[1] [1]: https://bsky.app/profile/iame.li/post/3l7e3jfqit22s https://bsky.app/profile/iame.li/post/3l7e3jfqit22s
      • nightpool 2y ago
        Thanks, so are both the Postgres and Scylla versions maintained in terms of new features? I wasn't aware that AppView v1 was open source, and the most recent info I'm aware of on the topic is https://alice.bsky.sh/post/3laega7icmi2q https://alice.bsky.sh/post/3laega7icmi2q, https://github.com/bluesky-social/atproto/discussions/2961 https://github.com/bluesky-social/atproto/discussions/2961 and https://docs.bsky.app/docs/advanced-guides/federation-architecture https://docs.bsky.app/docs/advanced-guides/federation-archit..., and everything I've heard about Bluesky was that open source appview is "still coming".
        • psionides 2y ago
          It's not coming, it never went away… As I understand it, the "business layer" with all the logic is above the data later, shared by the Postgres and Scylla versions, and the data layer just makes queries to the database. I think they are using the Postgres version locally for development.
  • haileyok 2y ago
    This is not true. Third party PDSes are fully supported by our app view, and our app view generates timelines for all the users on those PDSes.
    • nightpool 2y ago
      What does this have to do with third party app views?
      • psionides 2y ago
        You didn't specify what kind of federated servers you were thinking about
      • madeofpalk 2y ago
        The statement "any federated servers interacting with Bluesky" is ambiguous, because Bluesky's federated model means there's many different types of servers, and one user's view of what a "federated server" could be vastly different from another. Federated PDS-s (which is probably the closest to what people mean when they say they want to federate on bluesky) would not need to reconstruct timelines if their users use the bsky.app appview.
        • nightpool 2y ago
          Thanks, that's a fair point that I was overlooking. When I say a "federated server", I don't just mean a self-hosted PDS, I mean a third party app that potentially has its own lexicon and design decisions. Creating a robust third-party app that can meaningfully interact with the Bluesky network is still a very difficult engineering challenge, which I think this article does a good job demonstrating—that was the tension I was trying to underscore in my comment. Bluesky may be solving those engineering challenges for those clients who are satisfied with Bluesky's frontend and AppView, but every single other app built on top of ATProto will have to resolve those same challenges. This is directly downstream from Bluesky's "global firehose" topology and various design decisions that stem from that.
  • pfraze 2y ago
    As others have noted, the appview is open source. The dataplane has two implementations, one in postgres and another in scylla. The scylla dataplane is closed, the postgres one is open. The interesting next stage for the postgres implementation is to create a sync engine for partial syncs of the network, so that an appview can run affordably. We ran some benches on the current state of the postgres implementation and found we could index 300k users on a $100/mo vps. I think with a couple of weeks of optimization that could reach 1mm users.
    • nightpool 2y ago
      This is great to hear—my current understanding of the most recent state of the art on the topic is https://alice.bsky.sh/post/3laega7icmi2q https://alice.bsky.sh/post/3laega7icmi2q which mentions that the self-hosted appview is not yet open source. So I'm glad to hear the situation has changed in the past 3 months.
      • psionides 2y ago
        It was open source (except the Scylla database layer) from the beginning, AFAIK - that blog post just says that they haven't set it up yet, because that's the hardest part to run
  • evbogue 2y ago
    My thinking has evolved on this topic significantly as of late. My current thinking is we should create a secure gossip network on top of the Bluesky API, and forgot about all the DAG-CBOR stuff that gets stripped from the Jetstream. Hash the posts on the gossip layer and if posts change then diff them. This is all prep for when X billionaire buys out Bluesky then we just pop some signing key crypto on top of this gossip layer and wow! It's distributed!
    • pfraze 2y ago
      isnt that ssb?
      • evbogue 2y ago
        reverse-ssb
• dang 2y ago
  [stub for offtopicness]
  • amazingamazing 2y ago
    I don’t understand the infatuation with blue sky. The minute they need money it’ll go the way of the Reddit and twitter.
    • xrisk 2y ago
      People want the old Twitter, and Bluesky is close to that. It also cosplays being decentralized to people who don’t look too closely.
      • dom96 2y ago
        What makes it not decentralised?
        • xrisk 2y ago
          The fact that you have to be on “the” relay to meaningfully participate on the network. If you instead claim that users can always choose to use other 3P relays, then you immediately lose all the nice things that Bluesky is able to do well today (search, discoverability, a “discover” algorithm). Indeed, you fall back to the same old problems that every other decentralized social network has. Bluesky is just a shittier version of Nostr, except that the people over at Nostr don’t pretend.
        • immibis 2y ago
          The approximately a million dollars a year that it costs to run another copy.
        • BizarroLand 2y ago
          https://dustycloud.org/blog/how-decentralized-is-bluesky/ https://dustycloud.org/blog/how-decentralized-is-bluesky/
    • 2y ago
• nasso_dev 2y ago
  Interesting! I wonder what value they chose for the `reasonable_limit`.
  • Retr0id 2y ago
    ought to be possible to reverse-engineer it by following a large number of active accounts and seeing what percentage of their posts actually hit your feed
  • nasso_dev 2y ago
    It's 4k: https://bsky.app/profile/jaz.bsky.social/post/3likncbqutk2y https://bsky.app/profile/jaz.bsky.social/post/3likncbqutk2y
• bitmasher9 2y ago
  It’s really impressive how well Bluesky is performing. It really feels like a throwback to older social media platforms with its simplicity and lack of dark-patterns. I’m concerned that all the great work on the platform, protocol, etc won’t shine in the long term as they eventually need to find a revenue source.
  • autobodie 2y ago
    Absolutely. The profit motive is the root of most evil. It is a shame that so many are trained to believe it is the only motive available.
    • gkoberger 2y ago
      I completely agree with this... but without profit, people can't get paid, and they'll stop building. I do hate this incredibly need for growth, of course, but financial growth is necessary to pay people and give them raises and allow them to have upward mobility at the company. I hope Bluesky is able to find a model that works for them AND for consumers. (I do know it's an open protocol, so it'll live on without Bluesky itself! However, as this post shows, it's a lot of work to build on the prototype... so if not them, who? And if someone else, how will they become sustainable?)
      • jandrese 2y ago
        At the same time I feel like a lot of companies grow much larger than they need to be simply because of bigger is better mentality. How many of Uber's 30,000ish employees are involved with making sure the app and backend database are working properly? Are they really doing 600 times more work than Craigslist at connecting sellers with buyers?
        • bitmasher9 2y ago
          You cannot compare uber to Craigslist. Uber takes on so much more responsibility of the transaction. Setting price, handling disputes, real time coordination, etc.
• knallfrosch 2y ago
  Anyone following hundreds of thousands of users is obviously a bot account scraping content. I'd ban them and call it a day. However, I do love reading about the technical challenge. I think Twitter has a special architecture for celebrities with millions of followers. Given Bluesky is a quasi-clone, I wonder why they did not follow in these footsteps.
  • deleted 2y ago
    [deleted]
  • psionides 2y ago
    You don't need to follow anyone (or even have an account) to scrape content… Someone following a huge amount of accounts usually wants to get a lot of followers quickly this way through follow-backs.
    • mikemitchelldev 2y ago
      Yes, and Starter Packs make this possible.
  • ruined 2y ago
    if you want to scrape all the content, that's what the firehose is for, and it's allowed. the only reason to mass-follow is for spam purposes.
    • Retr0id 2y ago
      This does assume that scrapers are smart, and often they're really not. They have infrastructure for scraping HTML from webpages at scale and that is the hammer they use for all nails. (e.g. Wikipedia has to fight off scraper traffic despite full archives being available as torrents, etc.) In this case I agree though, they're all spammers and/or "clout farmers", or trying to make an account seem more authentic for future scams. They want to generate follow notifications in the hope that some will follow them back (and if they don't, they unfollow again after some interval).
• deleted 2y ago
  [deleted]
• ChuckMcM 2y ago
  As a systems enthusiast I enjoy articles like this. It is really easy to get into the mindset of "this must be perfect". In the Blekko search engine back end we built an index that was 'eventually consistent' which allowed updates to the index to be propagated to the user facing index more quickly, at the expense that two users doing the exact same query would get slightly different results. If they kept doing those same queries they would eventually get the exact same results. Systems like this bring in a lot of control systems theory because they have the potential to oscillate if there is positive feedback (and in search engines that positive feedback comes from the ranker which is looking at which link you clicked and giving it a higher weight) and it is important that they not go crazy. Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly. Reading this description of how user's timelines are sharded and the same sorts of feedback loops (in this case 'likes' or 'reposts') sounds like a pretty interesting problem space to explore.
  • culi 2y ago
    What became of Blekko?
    • an_ko 2y ago
      > It was acquired by IBM in March 2015, and the service was discontinued. — https://en.wikipedia.org/wiki/Blekko https://en.wikipedia.org/wiki/Blekko Perhaps GP has a more interesting answer though.
      • ChuckMcM 2y ago
        That's the correct answer, IBM wanted the crawler mostly to feed Watson. Building a full search engine (crawler, indexer, ranker, API, web application) for the English language was a hell of an accomplishment but by the time Blekko was acquired Google was paying out tens of billions of dollars to people to send them and only them their search queries. For a service that nominally has to live on advertising revenue getting humans to use it was the only way to be net profitable, and you can't spend billions buying traffic and hope to make it back on advertising as the #3 search engine in the English speaking markets. There are other ways to monetize search (look at Kagi for example) than advertising. Blekko missed that window though. (too early, Google needed to get a crappy as it is today to make the value of a spam free search engine desirable)
• sphars 2y ago
  When I go directly to a user's profile and see all their posts, sometimes one of their posts isn't in my timeline where it should be. I follow less than 100 users on Bluesky, but I guess this explains why I occasionally don't see a user's post in my timeline. Lossy indeed.
  • Eric_WVGG 2y ago
    Are you using an app, website, or combination? Various clients (I’m writing one) interpret the timeline differently, as a feed that shows literally everything includes could things that most people would find undesirable or irrelevant. (replies to strangers, replies to replies to replies, etc)
    • sphars 2y ago
      I'm using the official android app. There has been an update or two so I'll have to confirm it's still happening
  • Retr0id 2y ago
    If another user you follow reposted or replied to a post, it can affect its order in your following feed. You shouldn't be seeing any loss as described in the article from following only 100 users.
    • sphars 2y ago
      I've experienced it with "first-party" posts, not replies. A post wouldn't show in my timeline but would on the user's profile. This is the official android app, but there has been an update or two so I'll have to double check again
• trhway 2y ago
  So the system design puts the burden on what seems to be synchronous, not queued, writes to get easy reads. I usually prefer simpler cheaper writes at the cost of more complicated reads as the reads scale and parallelize better.
  • pfraze 2y ago
    you're underestimating the read load, by a lot
• skybrian 2y ago
  This design makes sense if you didn’t previously have any limit on the number of people an account could follow. But why not have a limit?
  • whyrusleeping 2y ago
    people get so up in arms when you suggest there might be a limit on how many people they can follow.
• timewizard 2y ago
  > This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post. Seriously? Isn't this the nut of your problem right here?
  • jsnell 2y ago
    What alternative design did you have in mind, given that a Twitter-like data model of individual follows is likely a strict product requirement? There are obviously other ways of doing it (doing the timeline propagation in a batch job, fanning out the reads rather than the writes), but they've got their own problems. Probably worse ones.
    • pphysch 2y ago
      Wouldn't a hybrid approach makes sense? Periodically classify users as hot/cold based on their activity, build hot-follower timelines on write, and build cold-follower timelines on read.
      • jsnell 2y ago
        You'd still have exactly the same hot write path, it'd just have maybe 50% of the load. That could be a legit optimization, but not having it hardly warrants an incredulous "seriously?" like the OP's. (And the same for the inverse hybrid strategy of quarantining the writes of highly followed users and handling their fan-out at read time. A neat optimization, and maybe even absolutely once you have accounts with 100M followers. But the vast majority of posts would still be handled via the original strategy.)
• PaulHoule 2y ago
  An airline reservation system has to be perfect (no slack in today's skies), a hotel reservation can be 98% perfect so long as there is some slack and you don't mind putting somebody up in a better room than they paid for from time to time. A social media system doesn't need to be perfect at all. It was clear to me from the beginning that Bluesky's feeds aren't very fast, not like they are crazy slow, but if it saves money or effort it's no problem if notifications are delayed 30s.
  • singleshot_ 2y ago
    Does the fact that an airline booking system must be perfect explain why so many flights are overbooked or cancelled?
    • rconti 2y ago
      No, overbooking is a business decision justified by the fact that, statistically, not all passengers will actually show up for their flight, and lower load factors cost money.
      • josefresco 2y ago
        What is the "no show" rate?
        • nightpool 2y ago
          A 2019 study of 5 European airports in 2019 had no-show rates of 14.4%: https://www.ozion-airport.com/product/comparative-analysis-no-show-rates-2020-vs-2019/#:~:text=The%202019%20no%2Dshow%20rates%20among%20the%20airports%20sampled%20demonstrate%20stability%2C%20averaging%2014.40%25%20throughout%20the%20year.%20January%20demonstrated%20the%20lowest%20no%2Dshow%20rate%20at%2014.35%25%20whilst%20August%20witnessed%20a%20peak%20at%2018.30%25%20(%2B3.95%25). https://www.ozion-airport.com/product/comparative-analysis-n... However, my understanding is that airlines have much more sophisticated per-flight and per-passenger models that calculate the predicted no-show factor based on the historical rates for that particular route (e.g. you're more likely to get more no-shows in business class flying from NYC to SF compared to holiday travelers with a reservation on the Florida Keys)
• rakoo 2y ago
  Ok I'm curious: since this strategy sacrifices consistency, has anyone thoughts about something that is not full fan-out on reads or on writes ? Let's imagine something like this: instead of writing to every user's timeline, it is written once for each shard containing at least one follower. This caps the fan-out at write time to hundreds of shards. At read time, getting the content for a given users reads that hot slice and filters actual followers. It definitely has more load but - the read is still colocated inside the shard, so latency remains low - for mega-followers the page will not see older entries anyway There are of course other considerations, but I'm curious about what the load for something like that would look like (and I don't have the data nor infrastructure to test it)
• rconti 2y ago
  > Additionally, beyond this point, it is reasonable for us to not necessarily have a perfect chronology of everything posted by the many thousands of users they follow, but provide enough content that the Timeline always has something new. While I'm fine with the solution, the wording of this sentence led me to believe that the solution was going to be imperfect chronology, not dropped posts in your feed.
• deleted 2y ago
  [deleted]
• artee_49 2y ago
  I am a bit perplexed though as to why they have implemented fan-out in a way that each "page" is blocking fetching further pages, they would not have been affected by the high tail latencies if they had not done this, "In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page." Basically means that they block on each page, process all the items on the page, and then move on to the next page. Why wouldn't you rather decouple page fetcher and the processing of the pages? A page fetching activity should be able to continuously keep fetching further set of followers one after another and should not wait for each of the items in the page to be updated to continue. Something that comes to mind would be to have a fetcher component that fetches pages, stores each page in S3 and publishes the metadata (content) and the S3 location to a queue (SQS) that can be consumed by timeline publishers which can scale independently based on load. You can control the concurrency in this system much better, and you could also partition based on the shards with another system like Kafka by utilizing the shards as keys in the queue to even "slow down" the work without having to effectively drop tweets from timelines (timelines are eventually consistent regardless). I feel like I'm missing something and there's a valid reason to do it this way.
  • abound 2y ago
    I interpreted this as a batch write, e.g. "write these 10k entries and then come back". The benefit of that is way less overhead versus 10k concurrent background routines each writing individual rows to the DB. The downside is, as you've noted, that you can't "stream" new writes in as older ones finish. There's a tradeoff here between batch size and concurrency, but perhaps they've already benchmarked it and "single-threaded" batches of 10k writes performed best.
• mifydev 2y ago
  "Hot Shards in Your Area" - 10/10 heading
• NoGravitas 2y ago
  The funny thing is that all of the centralization in Bluesky is defended as being necessary to provide things like global search and all replies in a thread, things that Mastodon simply punts on in the name of decentralization. But then ultimately, Bluesky has to relax those goals after all.
  • ramblejam 2y ago
    True. In context though Bluesky can tweak the volume knob as and when they see fit, whereas for Mastodon it's stuck where it is.
• arcastroe 2y ago
  I found it odd to base the loss-factor on the number of people you follow, rather than a truer indication of timeline-update-frequency. What if I follow 4k accounts, but each of those accounts only posts once a decade? My timeline would be become unnecessarily lossy.
• crabbone 2y ago
  Anecdotally, I ran into a similar solution "by chance". Long ago, I worked for a dating site. Our CTO at the time was a "guest of honor" who was brought in by a family friend who was working in the marketing at the time. The CTO was a university professor who took on a job as a courtesy (he didn't need the money nor fame, he had enough of both, and actually liked teaching). But he instituted a lot of experimental practices in the company. S.a. switching roles every now and then (anyone in the company could apply for a different role except administration and try themselves wearing a different hat), or having company-wide discussions of problems where employees would have to prepare a presentation on their current work (that was very unusual at the time, but the practice became more institutional in larger companies afterwards). Once he announced a contest for the problem he was trying to solve. Since we were building a dating site, the obvious problem was matching. The problem was that the more properties there were to match on, the longer it would take (beside other problems that is). So, the program was punishing site users who took time to fill out the questionnaires as well as they could and favored the "slackers". I didn't have any bright ideas on how to optimize the matching / search for matches. So, ironically, I asked "what if we just threw away properties beyond certain threshold randomly?" I was surprised that my idea received any traction at all. And the answer was along the lines of "that would definitely work, but I wouldn't know how to explain this behavior to the users". Which, at the time, I took to be yet another eccentricity of the old man... but hey, the idea stuck with me for a long time!
  • detuur 2y ago
    The answer to that reply is you don't need to explain it to your users. People are used to fuzzy/best-effort sort of matching, especially when it's specifically presented as a "matching algorithm" instead of a "filter".
• cavisne 2y ago
  AWS has a cool general approach to this problem (one badly behaving user effecting others on their shard) https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/ https://aws.amazon.com/builders-library/workload-isolation-u... The basic idea is to assign each user to multiple shards, decreasing the changes of another user sharing all their shards with the badly behaving user. Fixing this issue as described in the article makes sense, but if they did shuffle sharding in the first place it would cover any new issues without effecting many other users.
  • artee_49 2y ago
    I think shuffle sharding is beneficial for read-only replica cases, not for writing scenarios like this. You'll have to write to the primary and not to a "virtual node". Right? Or am I understand it incorrectly? I just read that article now.
• pornel 2y ago
  I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served). When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.
  • ericvolp12 2y ago
    This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.
    • curious_cat_163 2y ago
      That's insightful. Keep up the good work!
    • Xunjin 2y ago
      Just to be clear, you are a Bluesky engineer, right? off-topic: how has been dealing with the influx of new users after X political/legals problems aftermath? Did you see an increase in toxicity around the network? And how has you (Bluesky moderation) dealing with it.
      • ToucanLoucan 2y ago
        [flagged]
        • deleted 2y ago
          [deleted]
        • 2y ago
• JadeNB 2y ago
  I understand that it's a different point, but how can someone write a whole essay called "When imperfect systems are good" without once mentioning Gabriel or https://en.wikipedia.org/wiki/Worse_is_better https://en.wikipedia.org/wiki/Worse_is_better?
• Artoooooor 2y ago
  Are users informed that they follow too many creators and now they will not see every post on their timelines?
• fastest963 2y ago
  To help avoid the hot shard problem, I wonder how capping followers per "timeline" would perform. Especially each user would have a separate timeline per 1000 followers and the client would merge them. You could still do the lossy part, if necessary, by only loading a percent of the actual timelines. That wouldn't help the celebrity problem but it was already acknowledged earlier that the solution to that is to not fan out celebrity accounts.
• cush 2y ago
  "Hot Shards in Your Area"... I died
• inportb 2y ago
  An interesting solution to a challenging problem. Thank you for sharing it. I must admit, I had some trouble following the author's transition from "celebrity" with many followers to "bot" with many follows. While I assume the work done for a celebrity to scatter a bunch of posts would be symmetric to the work done for a commensurate bot to gather a bunch of posts, I had the impression that the author was introducing an entirely different concept in "Lossy Timelines."
• jadbox 2y ago
  So, let's say I follow 4k people in the example and have a 50% drop rate. It seems a bit weird that if all (4k - 1) accounts I follow end up posting nothing in a day, that I STILL have a 50% chance that I won't see the 1 account that posts in a day. It seems to me that the algorithm should consider my feed's age (or the post freshness of my followers). Am I overthinking?
  • imrehg 2y ago
    This feels like an edge case. The "reasonable limit" is likely set based on experimentation, and thus on how much people post on average and the load it generates (so the real number is unlikely to be exactly "2000", IMHO). If you follow a lot of people, how likely it is that their posting pattern is so different from the average? The more people you follow, the less likely that is. So while you can end up in such situation in theory, it would need to be a very unusual (and rare) case.
  • kevincox 2y ago
    Yeah, this seems concerning to me. Maybe now as the platform is new this isn't much of an issue. But as accounts go inactive people will naturally collect "dead" accounts that they are still following. On Facebook it isn't uncommon of to have old accounts of sociable people naturally collect thousands of friends. It seems that what they are trying to measure is "busy timelines" and it seems bag they could probably measure that more directly. For example what is the number of posts in the timeline over theast 24h? It seems that it should be fairly easy to use this as the metric for calculating drop rate.
  • brianolson 2y ago
    I think the 'law of large numbers' says that it's very unlikely for you to follow 4k and have _none_ of them posting. You could artificially construct a counter-example by finding 4k open but silent accounts, but that's silly. The other workaround is: follow everyone. Write some code to get what you want out of the jetstream event feed. https://docs.bsky.app/blog/jetstream https://docs.bsky.app/blog/jetstream
• spoaceman7777 2y ago
  Hmm. Twitter/X appears to do this at quite a low number, as the "Following" tab is incredibly lossy (some users are permanently missing) at only 1,200 followed people. It's insanely frustrating. Hopefully you're adjusting the lossy-ness weighting and cut-off by whether a user is active at any particular time? Because, otherwise, applying this rule, if the cap is set too low, is a very bad UX in my experience x_x
  • VWWHFSfQ 2y ago
    > It's _insanely_ frustrating. > at only 1,200 followed people. I follow like, 50 people on bluesky. Who is following 1,200 people? What kind of value do you even get out of your feed?
    • peoplepostphew 2y ago
      1200 people is really nothing, specially if you have a job tangentially related to social media (for example journalists). It's really simple, you are not the same type of user. You have 50 "acquaintances", they have 1200 "sources". The article is talking about people who have following/follower counts in the millions. Those are dozens of writes per second in one feed and a fannout of potentially millions. Someone with 1200 followers, if everyone actually posts once a day (most people do not) gets... a rate of 0.138 writes per second. They should be background noise, irrelevant to the discussion. That level of work is within reasonable expectation. What they're pointing out is that Twitter is aggressively anti-perfectionist for no good technical reason - so there must be a business reason for it.
      • VWWHFSfQ 2y ago
        Why are you following 1,200 people? What is the point of your home feed? What are you trying to see?
        • rjeli 2y ago
          there are a lot of interesting people on twitter
• yibg 2y ago
  I think something like this was a FB engineering interview (several years ago), just for instagram feeds.
• alexnewman 2y ago
  I don’t see much call for blusky anymore….
  • rsynnott 2y ago
    I mean: https://bskycharts.edavis.dev/static/dynazoom.html?cgiurl_graph=/munin-cgi/munin-cgi-graph&plugin_name=edavis.dev/bskycharts.edavis.dev/bsky&size_x=800&size_y=400&start_epoch=1705502705&stop_epoch=1740062705 https://bskycharts.edavis.dev/static/dynazoom.html?cgiurl_gr... Posts/sec are just off record levels.
    • Claudus 2y ago
      Record seems to be ~115, while current is ~63, or 55%.
      • rsynnott 2y ago
        Yeah, there was a short spike directly after the US elections, but, ignoring that, post rates are stable to somewhat rising.
    • Vaslo 2y ago
      So it’s bluesky this week? What about mastodon? What about threads? I thought X was going to die. I’ll just stay with X, that’s where everybody is, contrary to the bluesky cheerleaders here.
  • alexnewman 2y ago
    Wow this got super downvoted for nothing??? It was honest are people seeing as much references in the media
• andsoitis 2y ago
  Principle: Progress over perfection.
• udioron 2y ago
  > some of them will do abnormal things like… well… following hundreds of thousands of other users. Sounds like Bluesky Pro.
• Nemo_bis 2y ago
  "Lossy timelines" have already been implemented in ActivityPub and Mastodon by design. Will Bluesky ever catch up? It remains to be seen.
• dsauerbrun 2y ago
  I'm a bit confused. The lossy timeline solution basically means you skip updating the feed for some people who are above the number of reasonable followers. I get that Seeing them get 96% improvements is insane, does that mean they have a ton of users following an unreasonable number of people or do they just have a very low number for reasonable followers. I doubt it's the latter since that would mean a lot of people would be missing updates. How is it possible to get such massive improvements when you're only skipping a presumably small % of people per new post? EDIT: nvm, I rethought about it, the issue is that a single user with millions of follows will constantly be written to which will slow down the fanout service when a celebrity makes a post since you're going through many db pages.
  • friendzis 2y ago
    When a system gets "overloaded", typically it enters exponential degradation of performance state, i.e. performs self ddos. > Seeing them get 96% improvements is insane TFA is talking about P99 tail latencies. It does not sound too insane to reduce tail latencies by extraordinary margins. Remember, it's just reshaping of latency distribution. In this case pathological cases get dropped.
  • aloha2436 2y ago
    > does that mean they have a ton of users following an unreasonable number of people They do, there are groups of users on bluesky who follow inordinate numbers of other accounts to try and get follows back.
  • Beretta_Vexee 2y ago
    > does that mean they have a ton of users following an unreasonable number of people Look at the accounts of OnlyFans models, crypto influencers, etc. They follow thousands or even tens of thousands of accounts in the hope that we will follow them in return.
    • mapt 2y ago
      I don't see that accommodating this behavior is prosocial or technically desirable. Can you think of a use case? All sorts of bots want this sort of access, but whether there are legitimate reasons to grant it to them on a non-sharded basis is another question since a lot of these queries do not scale resources with O(n) even on a centralized server architecture.
• thmrtz 2y ago
  That’s quite interesting and a challenge I have not thought of. I understand the need for a solution and I believe this works reasonably well, but I am wondering what is happening to users that follow a lot of accounts with below-average activity. This may naturally happen on new social media platforms with people trying out the service and possibly abandoning it. The „reasonable limit“ is likely set to account for such an effect, but I am wondering if a per-user limit based on the activity of the accounts one follows will be an improvement on this approach.
• flaburgan 2y ago
  The solution to this problem is known and implemented already: the social web should be distributed between thousands of pods which should contain at the maximum a few thousands users. Diaspora is already working like this for 15 years. It is technically harder to build initially but it then divide all the problems (maintenance, moderation, load, censorship, trust of the owner...) Which makes the network much more resilient. Bluesky knows that and they are allowing other people to host their software but they are really not pushing for it and it highly doubt that the experience of a user on a small external pod is the same than on bluesky.com.
  • grishka 2y ago
    This particular problem will still exist for a fediverse server. You follow 10k people? Nice, now you're getting ddos'd by their activities. Though, most fediverse servers being monolithic applications definitely helps.
• dtonon 2y ago
  The typical problem of a centralized infrastructure. Indeed: > This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability
• buxidao 2y ago
  In the fanout design, why not dynamically move on to the next 10,000-user page as soon as all tasks for the current page are either queued or processing? Would that approach improve throughput, or could it introduce issues like resource contention?
• ramblejam 2y ago
  Nice problem to have, though. Over on Nostr they're finding it a real struggle to get to the point where you're confident you won't miss replies to your own notes, let alone replies from other people in threads you haven't interacted with. The current solution is for everyone to use the same few relays, which is basically a polite nod to Bluesky's architecture. The long-term solution is—well it involves a lot of relay hint dropping and a reliance on Japanese levels of acuity when it comes to picking up on hints (among clinets). But (a) it's proving extreme slow going and (b) it only aims to mitigate the "global as relates to me" problem.
• ultra-boss 2y ago
  Love reading these sorts of "technical problem + solution" pieces. The world does not need more content, in general, but it does need more of this kind of quality information sharing.
• robbale 2y ago
  the use of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served a good implementations in hot reload scenarios
• KolmogorovComp 2y ago
  A simpler option is to put a limit on the number of accounts one’s can follow. Who needs to follow more than 4k followers if not bots?
• mpweiher 2y ago
  On a related note, I am pretty confident that one of the main reasons the WWW succeeded where previous attempts failed was that it very specifically allowed 404s.