7 ms·

ATProto for distributed system engineers

200 points by danabramov 2y ago

omnicarinha 2y ago
One thing I still didn't quite grasp with BlueSky yet is if it's a decentralized platform or not... ATProto seems technically capable of supporting decentralized platforms.
- wmf 2y ago
  It's pretty decentralized. You can run your own PDS, relay, and appview (some of these are more expensive than others). I'm not sure if you can configure the official clients to use an alternate server.
  - nunobrito 2y ago
    Decentralization on this social network context means to have users accessing data from other users even when other third-parties don't want you to. That platform is (today) a centralized walled garden. As others detailed, it is difficult for anyone to add new servers and even more difficult to convince the official client to support them. It is a complete contrast to NOSTR that has zero official servers and zero official clients to access the data. It has hundreds of relays from different people, along with several clients from different developers that compete for your preference.
    - GaryNumanVevo 2y ago
      There are already alternative clients that support custom AppViews and relays
      - nunobrito 2y ago
        Described as "3rd party client" and which aims as goal to connect with NOSTR: https://docs.bsky.app/blog/feature-skyfeed https://docs.bsky.app/blog/feature-skyfeed Twitter also had 3rd party clients until one day they turned off the switch. Around NOSTR there are no labels as "3rd party clients" because they can't lock down your data with the push of a switch nor block your use of the platform.
        GaryNumanVevo 2y ago
        What's not to understand here? The PDS isn't a relay, it's a repository. The data layer is decoupled from the message passing layer, that's it.
        nunobrito 2y ago
        That's decoupling, not decentralization.
    - pfraze 2y ago
      What are you talking about? It’s not difficult to add servers or get the official client to use them.
- jazzyjackson 2y ago
  Bsky wants to be one entry point into a decentralized network but there's little incentive to spin up your own Personal Data Server (since you're still subject to the moderation of the one front end everyone uses (AppView in ATProto parlance)) and still less incentive to host your own front end since you'd just be burdening yourself with all the same moderation problems bsky is trying to stay on top of IMO the devs have been so overburdened with trying to nail moderation that they're actually disincentivized from onboarding new populations, since multiple entry points to the network just lands in their lap as more difficult moderation problems - that is, they're still figuring out how to moderate people on their own servers and haven't yet decided how they're going to moderate with a federation of servers with different cultures than their own I don't think they're avoiding the big problems, but it does seem like they're taking the slow careful route, maybe this is for the best.
- shafyy 2y ago
  In theory it is decentralized. But if you compare it to Mastodon for example, it's pretty centralized in practice. I haven't come across any people running their larger own servers, like I do on Mastodon.
  - danabramov 2y ago
    Note the shape of decentralization is very different from Mastodon — there's no concept of "running a Bluesky instance". What you can run is a personal server to host your data (which would work for any atproto apps, not just Bluesky). The Bluesky web app (which is ran by Bluesky) would aggregate data from your server (and all other servers on the network). Unlike Mastodon, you don't have people running copies of the Bluesky app because it is simply unnecessary — each copy would "see" the same network. If you wanted to fork the Bluesky product (e.g. different branding, different moderation decisions, different product decisions) then yes, you'd run your own product on your own backend and it would be able to ingest Bluesky app data (and vice versa, the Bluesky app would be able to ingest the data from your product).
    - shafyy 2y ago
      Sure, but I still haven't seen people really doing that.
      - danabramov 2y ago
        Which part are you referring to by “that”? There’s definitely people self-hosting their data (not a lot cause the process is pretty technical and manual atm). Note that you can always move hosts (without asking permission from the previous host) so you can start using the Bluesky hosting and then switch it.
        shafyy 2y ago
        I mean it does not seem like there's a bunch of personal data servers or web app instances people choose from, like there's in Mastodon (I know it's not the same technically). Therefore, it looks like Bluesky is much more centralized than Mastodon. I wonder if that's because it's technically harder to run your own software for Bluesky or what?
- brianolson 2y ago
  atproto PDSes are like blog servers with RSS (but better) and bsky.app is the prevailing RSS reader. It's an open protocol because anyone can host a source and anyone can run a different reader.
- GaryNumanVevo 2y ago
  BlueSky is just a reference implementation using AT Proto. They namespace anything bsky related in the lexicon as such.
apitman 2y ago
Has anyone played around with ATProto yet? ActivityPub is pretty easy to get started with, especially if you just ignore JSON-LD and parse what you see. I'm curious how ATProto compares.
- __loam 2y ago
  There's a lot fewer resources for AT than ActivityPub. Last time I checked which was a few months ago, the official documentation for AT was pretty sparse if you're interested in building to a spec. You'll find a lot more in the ActivityPub specs, plus a lot of open implementations and helpful guides.
  - m_eiman 2y ago
    > You'll find a lot more in the ActivityPub specs, plus a lot of open implementations and helpful guides. I've read that there's a problem with interacting with Mastodon if you only rely on the protocol specs, that they do things their own way and have different requirements than the official specs. Is this still a problem? If it is, are Mastodon moving to be more closely aligned with the spec, or to doing more of their own thing?
    - zimpenfish 2y ago
      From what I've seen, Mastodon sticks to the spec but a lot of clients and servers then stick to Mastodon's interpretation of the spec rather than the spec. e.g. for status IDs, the spec says "String (cast from an integer but not guaranteed to be a number)", Mastodon uses numerical IDs, some clients[1] see this as "Ah, IDs are numbers!" and break horribly when they're not numerically parseable (Akkoma, Pleroma, GotoSocial...) (IIRC there was another thing where `created_at` is described as "The date when this status was created" but the type is given as "String (ISO 8601 Datetime)" which led some code to crash when Mastodon started outputting just dates instead of datetimes.) [1] Including some from people who Really Should Know Better.
      - vidarh 2y ago
deleted 2y ago
[deleted]
openrisk 2y ago
What might be useful for the re-decentralized web community is a detailed comparison of the ATProto, ActivityStreams/Pub and maybe Solid specifications, protocols, standards, vocabularies (or whatever exactly these blueprints actually are). As the blog post illustrates quite nicely (literally), ATProto is a fairly complete, bottom-up type specification that makes concrete various server/database aspects that in the ActivityPub spec are somewhere in the remote background, "left to the implementation". One could almost think of implementing AP over ATProto, and sure enough somebody wrote about this [1]. One can also not miss the (at least) linguistic affinity of a Personal Data Server with a Personal Data Store (Solid) and sure enough somebody did and asked [2]. [1] https://berjon.com/ap-at/ https://berjon.com/ap-at/ [2] https://www.reddit.com/r/BlueskySocial/comments/ywrw3f/whats_the_difference_between_the_at_protocol_and/ https://www.reddit.com/r/BlueskySocial/comments/ywrw3f/whats...
swyx 2y ago
always enjoy your writing, Dan. at:// seems like its close enough to DNS to warrant just using DNS. why not? (im sure theres a good reason so just asking)
- danabramov 2y ago
  Oh this is not mine actually — Paul wrote this one :) atproto does use DNS under the hood for domain verification but atproto itself is a bit higher-level. It builds on top of DNS, HTTP, JSON, web sockets, and a few other specs.
  - danabramov 2y ago
    If you’re specifically asking why the identity system is not “rooted” in DNS (i.e. why at://danabra.mov resolves to another host than my website) — it’s because we want users to be able to change their hosting over time without breaking links between records. The actual identity system is “rooted” in a stable identifier (which is a hash of the first version of your identity record). That’s your global immutable ID in the entire network. The identity record for your ID specifies your current public key, your current domain name (which acts as a human-readable handle), and your current host (which actually contains your data). This extra level of indirection ensures you’re always able to change your user-readable handle (eg if you get a new domain or your domain expires etc), and that you’re always able to change your host (eg if your host goes down or you don’t like its services or you want to host data yourself). The key piece allowing this is the identity registry of course. Think of it similar to npm registry. We run a centralized one, but all records are signed so you can always recursively verify that we haven’t tampered with any of the records. This layer is already very thin but in longer term we’d like to move this layer outside the company to be governed independently, similar to ICANN.
    - str4d 2y ago
      Additionally, a user _can_ root their identity in DNS if they want, by using did:web instead of did:plc [0]. The main Bluesky client doesn't expose this (presumably because did:web cannot provide a mechanism for automatic migration between PDSs (due to the PDS having no control over the DID document) or recovering from loss of control of the domain name, so it requires more technical expertise), but there are users successfully using this method for their identities. [0] https://atproto.com/specs/did#blessed-did-methods https://atproto.com/specs/did#blessed-did-methods
badgersnake 2y ago
I was expecting something about modems. ATDT (555)-COOL-BBS (Totally decentralised btw)
- kragen 2y ago
  the nanpa is not decentralized at all, though it does delegate phone number assignment to local telecom companies, and nowadays even to sip providers but yeah it seems pretty suboptimal that they decided to reuse the name of the protocol you use to talk to most cellular modems
  - lifthrasiir 2y ago
    In fact, there had been some complaints [1] about the `at` URI scheme itself as well, even though its registration itself is valid as per RFC 7595 (First Come First Served for provisional entries). [1] https://www.iana.org/assignments/uri-schemes/expert-notes/at-expert-notes https://www.iana.org/assignments/uri-schemes/expert-notes/at...
  - badgersnake 2y ago
    > the nanpa is not decentralized at all Sure but BBSs tend to be. And then you’ve got systems like fidonet to connect them up.
    - kragen 2y ago
      i don't know if you've ever been a regional coordinator but fidonet is not that decentralized either, though the pstn (and nowadays the internet) do put limits on how much power such offices can wield each bbs is usually very centralized
sebstefan 2y ago
So the user's repo is decentralized but the event-log services and view services are centrally managed by bluesky?
- danabramov 2y ago
  There's a few layers to the system. - Identity layer: This is where your identity information (public key, current domain handle, current user repo host) is stored, essentially as a piece of JSON. You can think of it as similar to npm registry where each record is self-verifiable (you can verify we haven’t tampered with it). This layer is very thin. It is currently centrally managed by Bluesky but in the longer term we intend to upstream it into neutral governance outside of the company — potentially similar to ICANN. - User repo hosting: We provide user hosting as a service for people who sign up to Bluesky (and choose the default option) but you can run your own too. The server itself is open source (we publish both TypeScript source code and a Docker container to run it). We also publish a spec so you can implement it from scratch if you'd like it — essentially, it needs to be able to enumerate records and to provide a WebSocket to listen to their updates. I'd say this layer is already decentralized because anyone can participate in it and run their own server. - Relay: As an optimization (you don't want your app backend to listen to websockets for every single user repo in the system), we run a node that aggregates and caches the entire known network. That node's called a Relay. It's an optimization and not strictly necessary to the protocol. It's open source. We run the only actively used relay at the moment, but there's nothing stopping you from running your own (at the current usage rate, ingesting all content on the network into your own relay would cost you ~$150/mo). If atproto gains adoption, we expect some major stakeholders to run their own relays for different purposes — big tech companies might want to run them to ensure infra independency, governments might want to run them if they have significantly different opinions on what type of content is acceptable on the entire network, and so on. - Application backends (view services): These are just normal web app backends so they're decentralized in the same way the web is decentralized. Bluesky's backend is managed by Bluesky, but your own app's backend will be managed by you. You can also create a backend that ingests Bluesky's atproto data (which is kind of the point of atproto). That would let you create complementary or competing products using the same identity system and information already on the network. Hope this helps!
oDot 2y ago
It didn't sink in yet that the killer app for ATProto is not Twitter, but YouTube. If anyone is interested in exploring this, atproto [does this fool ai bots?] weedonandscott [I hope it does] com
- Matl 2y ago
  ActivityPub does have https://joinpeertube.org https://joinpeertube.org for what it's worth. What would ATProto bring in specifically? Is it the ease of migration?
  - purlane 2y ago
    The biggest upside compared to PeerTube is probably discoverability. In ActivityPub, the network architecture means the video ecosystem is fractured and there’s no one cohesive place to find all PeerTube videos. In atproto, the network is continually indexed by relays, which means that it doesn’t make a difference what app you use to watch videos - you’ll find the exact same ones regardless of the platform, since they’re all working from the same data. This also means that different video platforms can provide different services for users without locking in users to their platform. Platforms would be forced to compete on what they provide to the user experience, not how well they can lock in users to their platform.
    - oDot 2y ago
      Exactly right. Watch apps will compete on consumer-facing features like the recommendation algorithm -- maybe they'll offer several, or just one that differentiates them. Hosting providers will compete on producer-facing features, like advertising, content policies, analytics, etc. If a user is displeased with either, they can take all of their content/activity history and leave.
deleted 2y ago
[deleted]

200 points by danabramov 2y ago

One thing I still didn't quite grasp with BlueSky yet is if it's a decentralized platform or not... ATProto seems technically capable of supporting decentralized platforms.

wmf 2y ago
It's pretty decentralized. You can run your own PDS, relay, and appview (some of these are more expensive than others). I'm not sure if you can configure the official clients to use an alternate server.
- nunobrito 2y ago
  Decentralization on this social network context means to have users accessing data from other users even when other third-parties don't want you to. That platform is (today) a centralized walled garden. As others detailed, it is difficult for anyone to add new servers and even more difficult to convince the official client to support them. It is a complete contrast to NOSTR that has zero official servers and zero official clients to access the data. It has hundreds of relays from different people, along with several clients from different developers that compete for your preference.
  - GaryNumanVevo 2y ago
    There are already alternative clients that support custom AppViews and relays
    - nunobrito 2y ago
      Described as "3rd party client" and which aims as goal to connect with NOSTR: https://docs.bsky.app/blog/feature-skyfeed https://docs.bsky.app/blog/feature-skyfeed Twitter also had 3rd party clients until one day they turned off the switch. Around NOSTR there are no labels as "3rd party clients" because they can't lock down your data with the push of a switch nor block your use of the platform.
      - GaryNumanVevo 2y ago
        What's not to understand here? The PDS isn't a relay, it's a repository. The data layer is decoupled from the message passing layer, that's it.
        nunobrito 2y ago
        That's decoupling, not decentralization.
  - pfraze 2y ago
    What are you talking about? It’s not difficult to add servers or get the official client to use them.
jazzyjackson 2y ago
Bsky wants to be one entry point into a decentralized network but there's little incentive to spin up your own Personal Data Server (since you're still subject to the moderation of the one front end everyone uses (AppView in ATProto parlance)) and still less incentive to host your own front end since you'd just be burdening yourself with all the same moderation problems bsky is trying to stay on top of IMO the devs have been so overburdened with trying to nail moderation that they're actually disincentivized from onboarding new populations, since multiple entry points to the network just lands in their lap as more difficult moderation problems - that is, they're still figuring out how to moderate people on their own servers and haven't yet decided how they're going to moderate with a federation of servers with different cultures than their own I don't think they're avoiding the big problems, but it does seem like they're taking the slow careful route, maybe this is for the best.
shafyy 2y ago
In theory it is decentralized. But if you compare it to Mastodon for example, it's pretty centralized in practice. I haven't come across any people running their larger own servers, like I do on Mastodon.
- danabramov 2y ago
  Note the shape of decentralization is very different from Mastodon — there's no concept of "running a Bluesky instance". What you can run is a personal server to host your data (which would work for any atproto apps, not just Bluesky). The Bluesky web app (which is ran by Bluesky) would aggregate data from your server (and all other servers on the network). Unlike Mastodon, you don't have people running copies of the Bluesky app because it is simply unnecessary — each copy would "see" the same network. If you wanted to fork the Bluesky product (e.g. different branding, different moderation decisions, different product decisions) then yes, you'd run your own product on your own backend and it would be able to ingest Bluesky app data (and vice versa, the Bluesky app would be able to ingest the data from your product).
  - shafyy 2y ago
    Sure, but I still haven't seen people really doing that.
    - danabramov 2y ago
      Which part are you referring to by “that”? There’s definitely people self-hosting their data (not a lot cause the process is pretty technical and manual atm). Note that you can always move hosts (without asking permission from the previous host) so you can start using the Bluesky hosting and then switch it.
      - shafyy 2y ago
        I mean it does not seem like there's a bunch of personal data servers or web app instances people choose from, like there's in Mastodon (I know it's not the same technically). Therefore, it looks like Bluesky is much more centralized than Mastodon. I wonder if that's because it's technically harder to run your own software for Bluesky or what?
brianolson 2y ago
atproto PDSes are like blog servers with RSS (but better) and bsky.app is the prevailing RSS reader. It's an open protocol because anyone can host a source and anyone can run a different reader.
GaryNumanVevo 2y ago
BlueSky is just a reference implementation using AT Proto. They namespace anything bsky related in the lexicon as such.

apitman 2y ago

Has anyone played around with ATProto yet? ActivityPub is pretty easy to get started with, especially if you just ignore JSON-LD and parse what you see. I'm curious how ATProto compares.

__loam 2y ago
There's a lot fewer resources for AT than ActivityPub. Last time I checked which was a few months ago, the official documentation for AT was pretty sparse if you're interested in building to a spec. You'll find a lot more in the ActivityPub specs, plus a lot of open implementations and helpful guides.
- m_eiman 2y ago
  > You'll find a lot more in the ActivityPub specs, plus a lot of open implementations and helpful guides. I've read that there's a problem with interacting with Mastodon if you only rely on the protocol specs, that they do things their own way and have different requirements than the official specs. Is this still a problem? If it is, are Mastodon moving to be more closely aligned with the spec, or to doing more of their own thing?
  - zimpenfish 2y ago
    From what I've seen, Mastodon sticks to the spec but a lot of clients and servers then stick to Mastodon's interpretation of the spec rather than the spec. e.g. for status IDs, the spec says "String (cast from an integer but not guaranteed to be a number)", Mastodon uses numerical IDs, some clients[1] see this as "Ah, IDs are numbers!" and break horribly when they're not numerically parseable (Akkoma, Pleroma, GotoSocial...) (IIRC there was another thing where `created_at` is described as "The date when this status was created" but the type is given as "String (ISO 8601 Datetime)" which led some code to crash when Mastodon started outputting just dates instead of datetimes.) [1] Including some from people who Really Should Know Better.
    - vidarh 2y ago

deleted 2y ago

[deleted]

openrisk 2y ago

What might be useful for the re-decentralized web community is a detailed comparison of the ATProto, ActivityStreams/Pub and maybe Solid specifications, protocols, standards, vocabularies (or whatever exactly these blueprints actually are). As the blog post illustrates quite nicely (literally), ATProto is a fairly complete, bottom-up type specification that makes concrete various server/database aspects that in the ActivityPub spec are somewhere in the remote background, "left to the implementation". One could almost think of implementing AP over ATProto, and sure enough somebody wrote about this [1]. One can also not miss the (at least) linguistic affinity of a Personal Data Server with a Personal Data Store (Solid) and sure enough somebody did and asked [2]. [1] https://berjon.com/ap-at/ https://berjon.com/ap-at/ [2] https://www.reddit.com/r/BlueskySocial/comments/ywrw3f/whats_the_difference_between_the_at_protocol_and/ https://www.reddit.com/r/BlueskySocial/comments/ywrw3f/whats...

swyx 2y ago

always enjoy your writing, Dan. at:// seems like its close enough to DNS to warrant just using DNS. why not? (im sure theres a good reason so just asking)

danabramov 2y ago
Oh this is not mine actually — Paul wrote this one :) atproto does use DNS under the hood for domain verification but atproto itself is a bit higher-level. It builds on top of DNS, HTTP, JSON, web sockets, and a few other specs.
- danabramov 2y ago
  If you’re specifically asking why the identity system is not “rooted” in DNS (i.e. why at://danabra.mov resolves to another host than my website) — it’s because we want users to be able to change their hosting over time without breaking links between records. The actual identity system is “rooted” in a stable identifier (which is a hash of the first version of your identity record). That’s your global immutable ID in the entire network. The identity record for your ID specifies your current public key, your current domain name (which acts as a human-readable handle), and your current host (which actually contains your data). This extra level of indirection ensures you’re always able to change your user-readable handle (eg if you get a new domain or your domain expires etc), and that you’re always able to change your host (eg if your host goes down or you don’t like its services or you want to host data yourself). The key piece allowing this is the identity registry of course. Think of it similar to npm registry. We run a centralized one, but all records are signed so you can always recursively verify that we haven’t tampered with any of the records. This layer is already very thin but in longer term we’d like to move this layer outside the company to be governed independently, similar to ICANN.
  - str4d 2y ago
    Additionally, a user _can_ root their identity in DNS if they want, by using did:web instead of did:plc [0]. The main Bluesky client doesn't expose this (presumably because did:web cannot provide a mechanism for automatic migration between PDSs (due to the PDS having no control over the DID document) or recovering from loss of control of the domain name, so it requires more technical expertise), but there are users successfully using this method for their identities. [0] https://atproto.com/specs/did#blessed-did-methods https://atproto.com/specs/did#blessed-did-methods

badgersnake 2y ago

I was expecting something about modems. ATDT (555)-COOL-BBS (Totally decentralised btw)

kragen 2y ago
the nanpa is not decentralized at all, though it does delegate phone number assignment to local telecom companies, and nowadays even to sip providers but yeah it seems pretty suboptimal that they decided to reuse the name of the protocol you use to talk to most cellular modems
- lifthrasiir 2y ago
  In fact, there had been some complaints [1] about the `at` URI scheme itself as well, even though its registration itself is valid as per RFC 7595 (First Come First Served for provisional entries). [1] https://www.iana.org/assignments/uri-schemes/expert-notes/at-expert-notes https://www.iana.org/assignments/uri-schemes/expert-notes/at...
- badgersnake 2y ago
  > the nanpa is not decentralized at all Sure but BBSs tend to be. And then you’ve got systems like fidonet to connect them up.
  - kragen 2y ago
    i don't know if you've ever been a regional coordinator but fidonet is not that decentralized either, though the pstn (and nowadays the internet) do put limits on how much power such offices can wield each bbs is usually very centralized

sebstefan 2y ago

So the user's repo is decentralized but the event-log services and view services are centrally managed by bluesky?

danabramov 2y ago
There's a few layers to the system. - Identity layer: This is where your identity information (public key, current domain handle, current user repo host) is stored, essentially as a piece of JSON. You can think of it as similar to npm registry where each record is self-verifiable (you can verify we haven’t tampered with it). This layer is very thin. It is currently centrally managed by Bluesky but in the longer term we intend to upstream it into neutral governance outside of the company — potentially similar to ICANN. - User repo hosting: We provide user hosting as a service for people who sign up to Bluesky (and choose the default option) but you can run your own too. The server itself is open source (we publish both TypeScript source code and a Docker container to run it). We also publish a spec so you can implement it from scratch if you'd like it — essentially, it needs to be able to enumerate records and to provide a WebSocket to listen to their updates. I'd say this layer is already decentralized because anyone can participate in it and run their own server. - Relay: As an optimization (you don't want your app backend to listen to websockets for every single user repo in the system), we run a node that aggregates and caches the entire known network. That node's called a Relay. It's an optimization and not strictly necessary to the protocol. It's open source. We run the only actively used relay at the moment, but there's nothing stopping you from running your own (at the current usage rate, ingesting all content on the network into your own relay would cost you ~$150/mo). If atproto gains adoption, we expect some major stakeholders to run their own relays for different purposes — big tech companies might want to run them to ensure infra independency, governments might want to run them if they have significantly different opinions on what type of content is acceptable on the entire network, and so on. - Application backends (view services): These are just normal web app backends so they're decentralized in the same way the web is decentralized. Bluesky's backend is managed by Bluesky, but your own app's backend will be managed by you. You can also create a backend that ingests Bluesky's atproto data (which is kind of the point of atproto). That would let you create complementary or competing products using the same identity system and information already on the network. Hope this helps!

oDot 2y ago

It didn't sink in yet that the killer app for ATProto is not Twitter, but YouTube. If anyone is interested in exploring this, atproto [does this fool ai bots?] weedonandscott [I hope it does] com

Matl 2y ago
ActivityPub does have https://joinpeertube.org https://joinpeertube.org for what it's worth. What would ATProto bring in specifically? Is it the ease of migration?
- purlane 2y ago
  The biggest upside compared to PeerTube is probably discoverability. In ActivityPub, the network architecture means the video ecosystem is fractured and there’s no one cohesive place to find all PeerTube videos. In atproto, the network is continually indexed by relays, which means that it doesn’t make a difference what app you use to watch videos - you’ll find the exact same ones regardless of the platform, since they’re all working from the same data. This also means that different video platforms can provide different services for users without locking in users to their platform. Platforms would be forced to compete on what they provide to the user experience, not how well they can lock in users to their platform.
  - oDot 2y ago
    Exactly right. Watch apps will compete on consumer-facing features like the recommendation algorithm -- maybe they'll offer several, or just one that differentiates them. Hosting providers will compete on producer-facing features, like advertising, content policies, analytics, etc. If a user is displeased with either, they can take all of their content/activity history and leave.

deleted 2y ago

[deleted]