3 ms·

So is the protocol sort of like a big public kafka topic? If you want to create a new web app, say, you don't actually store the data yourself. You basically s

So is the protocol sort of like a big public kafka topic? If you want to create a new web app, say, you don't actually store the data yourself. You basically set up a listener to "the big topic in the sky" and ask your users to save all their data in their own respective places, which the protocol will propagate out into the world such that your app's listener will hear about and cache it. Do I have that right? It's a very interesting concept for how to build a web app while the users still own their data. I don't have a great understanding of the process, though. Are their "offsets" or something to make sure you don't miss updates? Partitions for scalability? Maybe I'm pattern matching too hard to Kafka here.

verdverm 1y ago
> So is the protocol sort of like a big public kafka topic? yes, at least the firehose is that, and anyone can subscribe or run their own https://atproto.com/articles/atproto-for-distsys-engineers https://atproto.com/articles/atproto-for-distsys-engineers > to make sure you don't miss updates Yes, the firehose and jetstreams have a cursor, though with limited time window depending on the instance. (1-3h -> 72h) If you want the entire history you do what's called a backfill
- maelito 1y ago
  Hi, thanks for these info. I have a web app that produces atproto records, app.cartes.comment. My server listens to Jetstream and builds its DB based on this. But what about if my server goes down for say 4 days ? How can I rebuild my DB to catch up those days ? It looks like Jetstream cannot work for this. My need is simple : give me all the records of the collection on the whole atmosphere. How to do this ?