9 ms·
Jetstream: Shrinking the AT Protocol Firehose by >99%
- londons_explore 2y ago> Before this new surge in activity, the firehose would produce around 24 GB/day of traffic. The firehose is all public data going into the network right? Isn't that pretty tiny for a worldwide social network? And the fact one country can cause a 10x surge in traffic also suggests its worldwide footprint must be tiny...
- londons_explore 2y ago24GB/day is very much a "we could host this on one server with a bit of read caching" scale. Every year or so, you plop in one more 10TB SSD.
- ericvolp12 2y agoYes the actual record content on the network isn't huge at the moment but the firehose doesn't include blobs (images and videos) which take up significantly more space. Either way, yeah it's pretty lightweight. Total number of records on the network is around 2.5Bn in the ~1.5 years Bluesky has been around. I aggregated some stats when we hit 10M users here - https://bsky.app/profile/did:plc:q6gjnaw2blty4crticxkmujt/post/3l47puwmjnl2x https://bsky.app/profile/did:plc:q6gjnaw2blty4crticxkmujt/po...
- ericvolp12 2y agoThe 10x surge in traffic was us gaining 3.5M new users over the course of a week (growing the entire userbase by >33%) and all these users have been incredibly active on a daily basis. Lots of these numbers are public and the impact of the surge can be seen here: https://bskycharts.edavis.dev/static/dynazoom.html?plugin_name=edavis.dev%2Fbskycharts.edavis.dev%2Fbsky_users_total&start_iso8601=2024-04-22T11%3A15%3A23-0700&stop_iso8601=2024-09-24T11%3A15%3A23-0700&start_epoch=1713809723&stop_epoch=1727201723&lower_limit=&upper_limit=&size_x=800&size_y=400&cgiurl_graph=%2Fmunin-cgi%2Fmunin-cgi-graph https://bskycharts.edavis.dev/static/dynazoom.html?plugin_na... Note the graphs in that link only show users that take a publicly visible action (i.e. post, like, follow, etc.) and won't show lurkers at all.
- str4d 2y ago> The firehose is all public data going into the network right? It's the "main subset" of the public data, being "events on the network": for the Bluesky app that's posts, reposts, replies, likes, etc. Most of those records are just metadata (e.g. a repost record references the post being reposted, rather than embedding it). Post / reply records include the post text (limited to 300 graphemes). In particular, the firehose traffic does _not_ include images or videos; it only includes references to "blob"s. Clients separately fetch those blobs from the PDSs (or from a CDN caching the data).
- deleted 2y ago[deleted]
- bcrl 2y agoI find it baffling that the difference in cost of serving 41GB/day vs 232GB/day is worth spending any dev time on. We're talking about a whopping 21.4Mbps on average, which costs me roughly CAD$3.76/month in transit (and my transit costs are about to be cut in half for 2 x 10Gbps links thanks to contracts being up and the market being very competitive). 1 hour of dev time is upwards of 2 years of bandwidth usage at that rate.
- ericvolp12 2y agoThe current relay firehose has more than 250 subscribers. It's served more than 8.5Gbps in real-world peak traffic sustained for ~12 hours a day. That being said, Jetstream is a lot more friendly for devs to get started with consuming than the full protocol firehose, and helps grow the ecosystem of cool projects people build on the open network. Also, this was a fun thing I built mostly in my free time :)
- tecleandor 2y agoAlso it's just not those concrete 190GBs a day. It's the 6x traffic you can fit in the same pipe :D
- ericvolp12 2y agoYeah exactly! The longer we can make it without having to shard the firehose, the better. It's a lot less complex to consume as a single stream.
- tecleandor 2y agoAlso, being some sort of streaming firehose with the same data for everyone (if I understood what this is in a quick read) I guess if it makes any sense to do any kind of P2P/distributed transfer of it to ease the load...
- wkat4242 2y agoIt's a benefit on the receiving side too. And it has ecological benefits. Nothing to sneeze at.
- ChicagoDave 2y agoI'm just popping in here to say this (and BlueSky and atproto) are two of the coolest technical feats in today's tech world.
- BobbyTables2 2y ago[flagged]
- squidgedcricket 2y agoFor real. It's 2024, which in my millennial mind is squarely 'the future', and I'm writing bespoke UART drivers and command parsers for a protocol that's older than I am.
- deleted 2y ago[deleted]
- out_of_protocol 2y agoWhy providing non-compressed version at all? This is new protocol, no need for backwards compatibility. Dictionary may be baked into protocol itself, being fixed for specific version. E.g. protocol v1 uses that fixed v1 dictionary. Useful for replaying stored events on both sides
- CuriousCosmic 2y agoA non-compressed version is almost certainly cheaper for anything local (ex self-hosting your own services that consume the firehose on the same machine or for testing). There's not really a good reason to do compression if the stream is just going to be consumed locally. Instead you can skip that step and broadcast over memory to the other local services.
- out_of_protocol 2y agoIt could be a flag, normally disabled. Also, i'm not sure about "cheaper" side, since disk ops are not free, maybe uncompressing zstd IS cheaper than writing, reading huge blobs from disk, exchanging info between apps
- cowsandmilk 2y agoThis isn’t compression, they are throwing features of the original stream out.
- deleted 2y ago[deleted]
- out_of_protocol 2y agoWe're discussing here compiled output, plain json and the same json but zstd-compressed
- ericvolp12 2y agoJetstream isn't an official change to the Protocol, it's an optimization I made for my own services that I realized a lot of other devs would appreciate. The major driving force behind it was both the bandwidth savings but also making the Firehose a lot easier to use for devs that aren't familiar with AT Proto and MSTs. Jetstream is a much more approachable way for people to dip their toe into my favorite part of AT Proto: the public event stream.
- wrigby 2y agoI thought this was going to be a strange read about the Hayes command set[1] at first glance. 1: https://en.m.wikipedia.org/wiki/Hayes_AT_command_set https://en.m.wikipedia.org/wiki/Hayes_AT_command_set
- gooseus 2y agoI thought this was going to be about NATS Jetstream, but it is not. https://docs.nats.io/nats-concepts/jetstream https://docs.nats.io/nats-concepts/jetstream
- deleted 2y ago[deleted]
- fakwandi_priv 2y agoWhy is this being downvoted? Seems like a valid concern to raise if you find two pieces of software somewhat having the same functionality.
- gooseus 2y agoWas I being downvoted? It wasn't even a criticism, just an observation for anyone else who was thinking the same or was interested in another popular project with a similar name (and seemingly similar functions? didn't look too hard). Naming things is hard and we all kinda share one global tech namespace, so this is gonna inevitably happen.
- kylecazar 2y agoIt has the same name, not the same functionality. I am not a downvoter... but it's probably because reading a few sentences of this blog post would reveal what it is.
- JoshMandel 2y agoServer-Sent Events (SSE) with standard gzip compression could be a simpler solution -- or maybe I'm missing something about the websocket + zstd approach. SSE Benefits: Standard HTTP protocol, Built-in gzip compression, Simpler client implementation
- jeroenhd 2y agoWell-configured zstd can save a lot of bandwidth over gzip at this scale without major performance impact, especially with the custom dictionary. Initialising zstd with a custom dictionary also isn't very difficult for the client side. As for application development, I think web socket APIs are generally exposed much better and used much easier than SSEs. I agree that SSEs are a more appropriate technology to use here, but they're used so little that I don't think the tooling is good. Just about every language has a dedicated websocket client library, but SSEs are usually implemented as a weird side effect of a HTTP connection you need to keep alive manually. The stored ZSTD objects make sense, as you only need to compress once rather than compress for every stream (as the author details). It also helps store the data collected more efficiently on the server side if that's what you want to do.
- qixxiq 2y agoI don't have an understanding of SSE in depth, but one of the points the post is arguing for is compress once (using zstd dictionary) and send that to every client. The dictionary allows for better compression without needing a large amount of data, and sending every client the same compressed binary data saves a lot of CPU time in compression. Streams, usually, require running the compression for each client.
- scirob 2y agoWas expecting Nats Jetstream but this is also cool
- vundercind 2y agoWas expecting the Hayes modem command language.
- deleted 2y ago[deleted]
- pohl 2y agoThe "bring it all home" screenshot shows a CPU Utilization graph, and the units of measurements on the vertical axis appears to be milliseconds. Could someone help me understand what that measurement might be?
- Ericson2314 2y agoI gotta say, I am not very excited about "let's throw away all the security properties for performance!" (and also "CBOR is too hard!") If everyone is on one server (remains to be seen), and all the bots blindly trust it because they are cheap and lazy, what the hell is the point?
- steveklabnik 2y ago> If everyone is on one server (remains to be seen) Even internally, BlueSky runs a number of "servers." It's already internally federated. And you can transfer your account to your own server if you want to, though that is very much still beta quality, to be fair. You're not really "on a server" in the same sense as other things. It's closer to "I create content addressable storage" than "The database on this instance knows my username/password."
- apitman 2y agoAnd if you're curious what your Bluesky server is: DID=$(curl -s https://<username>.bsky.social/.well-known/atproto-did) curl https://plc.directory/$DID | jq '.service[0].serviceEndpoint' Or if you're using a custom domain (a la example.com), you can get your DID from the DNS: dig +short _atproto.example.com TXT
- steveklabnik 2y agoThanks for this! I'd never checked. Turns out I'm on https://morel.us-east.host.bsky.network https://morel.us-east.host.bsky.network. I do want to host my own PDS someday, but then I'd be responsible for keeping it up...
- apitman 2y agoIndie hosted PDS is on my list as well. I'm also currently trying to understand the tradeoffs for did:plc more. It's unclear to me just how centralized it is. Will it always require a single central directory, or is it more like Certificate Transparency? Based on what I've heard about the recover process, I believe it's the latter, but I still need to dig into it more.
- wowtraffic 2y ago[flagged]
- ebiester 2y ago...if you're doing this on a 5 dollar OVH VPS as a solo developer where you don't control all pieces of the puzzle?
- deleted 2y ago[deleted]
- oliver_jack 2y ago[flagged]
- xbar 2y agoI'm never not going to look for Hayes command set topics when people talk about BlueSky.
- gs17 2y agoYou're not the only one. I don't get why they couldn't have named it something that wasn't very similar to something already around for several decades, or at least insist on the shortened ATproto name (one word, lower case p). Sure, in practice, no one will actually confuse them, but that could be said for Java and JavaScript.
- deleted 2y ago[deleted]
- madduci 2y agoNice feat! I wonder if a rewrite of this in C++ would even bump further the performance and optimise the overall system.
- szundi 2y agoOr in rust haha
- S0y 2y ago>Before this new surge in activity, the firehose would produce around 24 GB/day of traffic. After the surge, this volume jumped to over 232 GB/day! >Jetstream is a streaming service that consumes an AT Proto com.atproto.sync.subscribeRepos stream and converts it into lightweight, friendly JSON. So let me get this straight. if you did want to run Jetstream yourself you'd still need to be able to handle the 232 GB/day of bandwidth? This always has been my issue with Bluesky/AT Protocol, For all the talk about their protocol being federated, It really doesn't seem realistic for anyone to run any of the infrastructure themselves. You're always going to be reliant on a big player that has the capital to keep everything running smoothly. At this point I don't really see how it's any different then being on any of the old centralized social media.
- rudyfraser 2y ago> It really doesn't seem realistic for anyone to run any of the infrastructure themselves. You're always going to be reliant on a big player that has the capital to keep everything running smoothly I run a custom rust-based firehose consumer on the main firehose using a cheap digitalocean droplet and don't cross 15% CPU usage even during the peak 33 Mb/s bandwidth described in this article. The core team seem to put a lot of intention into being able to host different parts of the network. The most resource intensive of which is the relay which produces the event stream that Jetstream and others consume from. One of their devs did a breakdown of that and showed you could run it at $150 per month which is pricey but not unattainable with grants or crowdfunding https://whtwnd.com/bnewbold.net/entries/Notes%20on%20Running%20a%20Full-Network%20atproto%20Relay%20(July%202024) https://whtwnd.com/bnewbold.net/entries/Notes%20on%20Running...
- purlane 2y agoI can attest secondhand that it's possible to run a relay for ~€75/month, which is well within the range of many hobbyists.
- AlienRobot 2y agoSometimes I think all of this could have been avoided if people knew how to use RSS.
- hinkley 2y agoGiven that “AT Protocol” already has a definition in IT that’s as old as OP’s grandma, what is this AT Protocol they are talking about here? Introduce your jargon before expositing, please.
- purlane 2y agoThe Wikipedia article does a pretty good job at giving an overview for it. https://en.wikipedia.org/wiki/AT_Protocol https://en.wikipedia.org/wiki/AT_Protocol
- deleted 2y ago[deleted]
- dboreham 2y agoHad same confusion. Wondered why it would need compression...
- ChicagoDave 2y agoBlueSky is mentioned. The ATProto thing has been discussed often in relation to an opensource social media protocol and all of this started from Jack Dorsey and Twitter. If it helps, it's thoroughly documented on Wikipedia. https://en.wikipedia.org/wiki/Bluesky https://en.wikipedia.org/wiki/Bluesky And the protocol is documented here: https://atproto.com/ https://atproto.com/ Hope this helps with the confusion.
- gs17 2y agoThe article kind of assumes you know what it is in order to be interested in it, but it's the protocol used by Bluesky instead of ActivityPub.
- bschmidt1 2y ago[flagged]
- kemotep 2y agoWhere does Gaddafi come into this? That seems like a complete non-sequitur with the previous sentence talking about people involved in Start-ups in the early 2010s.
- bschmidt1 2y agoWere you born yesterday? https://en.wikipedia.org/wiki/Social_media%27s_role_in_the_Arab_Spring https://en.wikipedia.org/wiki/Social_media%27s_role_in_the_A...
- kemotep 2y agoGaddafi was overthrown as part of the Arab Spring. He wasn’t a guy involved in tech startups in the early 2010s hence my confusion as to how he was related to Aaron Swartz.
- bschmidt1 2y agoAh you misread, I'm saying anyone reading the comment who was active in startups during that time would know about those events.
- arghandugh 2y agoThis is top-to-bottom 100% nonsense composed out of imagined facts. Please do not contribute this.
- bschmidt1 2y agoState what's not factual instead of just saying it's not factual. Otherwise I can easily say your statement is 100% top-down, in-and-out, all-around, unsubstantiated.
- 2y ago
- deleted 2y ago[deleted]