20 ms·
A Social Filesystem
- elbci 9mo agoagree! Social-media contributions as files on your system: owned by you, served to the app. Like .svg specifications allows editing in inkscape or illustrator a post on my computer would be portable on mastodon or bluesky or a fully distributed p2p network.
- catapart 9mo agoyeah yeah yeah, everyone get on the AT protocol, so that the bluesky org can quickly get all of these filthy users off of their own servers (which costs money) while still maintaining the original, largest, and currently only portal to actually publish the content (which makes money[0]). let them profit from a technical "innovation" that is 6 levels of indirection to mimic activity pub. if they were decent people, that would be one thing. but if they're going to be poisoned with the same faux-libertarian horseshit that strangled twitter, I don't see any value in supporting their protocol. there's always another protocol. but assuming I was willing to play ball and support this protocol, they STILL haven't solved the actual problem that no one else is solving either: your data exists somewhere else. until there's a server that I can bring home and plug in with setup I can do using my TV's remote, you're not going to be able to move most people to "private" data storage. you're just going to change which massive organization is exploiting them. I know, I know: hardware is a bitch and the type of device I'm even pitching seems like a costly boondoggle. but that's the business, and if you're not addressing it, you're not fomenting real change; you're patting yourself on the back for pretending we can algorithm ourselves out of late-stage capitalism. [0] *potentially/eventually
- lou1306 9mo ago> until there's a server that I can bring home and plug in with setup I can do using my TV's remote, you're not going to be able to move most people to "private" data storage Quite some BSky users are publishing on their own PDS (Personal Data Server) right now. They have been for a while. There are already projects that automate moving or backign up your PDS data from BSky, like https://pdsmoover.com/ https://pdsmoover.com/
- andai 9mo agoWhat's a PDS?
- treetalker 9mo agoPersonal Data Server. See https://atproto.com/guides/glossary#pds-personal-data-server https://atproto.com/guides/glossary#pds-personal-data-server
- Zircom 9mo agohttps://docs.bsky.app/docs/advanced-guides/atproto https://docs.bsky.app/docs/advanced-guides/atproto
- avsm 9mo agoMicroblogging is also the least interesting part of the ATProto ecosystem. I've switched all my git hosting over to https://tangled.org https://tangled.org and am loving it, not least of which is that my git server (a 'knot' in Tangled parlance) is under my control as a PDS and has no storage limits!
- skybrian 9mo agoIs it as easy for other people to read as a Github repo? Want to share?
- icy 9mo agoTangled founder here; it's just as easy! For example, here's the entire Tangled codebase monorepo: https://tangled.org/tangled.org/core https://tangled.org/tangled.org/core — you can clone this directly as you would a git repo anywhere else.
- skybrian 9mo agoLooks nice! New user sign up is a bit wonky. It asked for an email, login and password, then it's asking for a bsky sign-in too? This seems a little weird. (Minor nit: for some reason, Google didn't auto-suggest a strong password for the password field.) Then I got to the screen where it asks for full read-write access to my PDS and stopped there. It's kind of a lot to ask! I believe this is Bluesky's fault, but I don't think I can really use third-party bluesky apps until they implement finer-grained permissions.
- danabramov 9mo ago>that the bluesky org can quickly get all of these filthy users off of their own servers (which costs money) That's not correct, actually hosting user data is cheap. Most users' repos are tiny. Bluesky doesn't save anything by having someone move to their own PDS. What's expensive is stuff like video processing and large scale aggregation. Which has to be done regardless of where the user is hosting their data.
- catapart 9mo agocome on, man, let's be real. you're talking modern, practical application; I'm talking reasonable user buy in at big boy social media levels. the video hosting IS what I'm talking about being expensive. you think bsky is going to be successfully while ignoring the instagram crowd forever? what are we doing here? bsky saves the video processing and bandwidth by not hosting that content on bsky. it's a smaller problem, but in a large enough pool, images become heavy, too. and, either way, the egress of that content is expensive if you're doing it for the entire world, instead of letting each individual's computer (their pds) do it. I'm happy to admit that text is cheap and bsky isn't looking to offload their data as it stands now. but let's be honest about the long term, which is what my original comment takes aim at.
- danabramov 9mo agoI still don't think this is correct. The Bluesky app always processes video, whether you're self-hosting or not. The personal data serves stores the original blob, but the Bluesky's video service will have to pick it up and transcode it (to serve it from CDN) either way. Also, this: >let them profit from a technical "innovation" that is 6 levels of indirection to mimic activity pub. is also wrong because AT solves completely different problems. None of the stuff I wrote about in the post can be solved or or is being solved by ActivityPub. Like, AP is just message passing. It doesn't help with aggregation, identity, signing, linking cross-application data, any of that.
- catapart 9mo agoright on, man, that must be the only way transcoding can be done and completely future proof so it will never change to let the user transcode their own damn content. I get it; you're frustrated so you're nitpicking. funny how dang doesn't swoop in to tut tut you for not steelmanning instead of strawmanning. in any case, you're completely right about the activity pub comment. that was absolutely mockery and not actually a complaint. artistic license and all that. god forbid we express ourselves. but sure, I can recognize that AT proto is useful in that it provides mechanisms we didn't really have before. that said, it's not novel (just new) and it's not irreplaceable. LIKE I SAID: there's always another protocol. any time you want to actually address the point of the comments, I'd be happy to get your take on why it's fine, actually, for the CEO to imply she doesn't have to care what users think as long as they aren't paying her. but if you're not ready to have the real conversation, I'll let you be satisfied with whatever other potshots you want to take at my reasonable indignation.
- ninkendo 9mo ago> When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. https://www.joelonsoftware.com/2001/04/21/dont-let-architecture-astronauts-scare-you/ https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
- dang 9mo ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- danabramov 9mo agoAuthor here! I grew up reading Joel's blog and am familiar with this post. Do you have a more pointed criticism? I agree something like "hyperlinked JSON" maybe sounds too abstract, but so does "hyperlinked HTML". But I doubt you see web as being vague? This is basically web for data.
- ninkendo 9mo ago> Do you have a more pointed criticism? Sure. After taking the time to re-read the article since I initial posted my (admittedly shallow) dismissal, I realized this article is really a primer/explainer for the AT protocol, which I don't really have enough background in to criticize. My criticism is more about the usefulness of saying "what if we treated social networking as a filesystem": which is that this doesn't actually solve any problems or add any value. The idea of modeling a useful thing (social media)[0] as a filesystem is generalizing the not-useful parts of it (ie. the minutia of how you actually read/write to it) and not actually addressing any of the interesting or difficult parts of it (how you come up with relevant things to look at, whether a "feed" should be a list of people you follow or suggestions from an algorithm, how you deal with bad actors, sock puppets, the list goes on forever.) This is relevant to Joel's blog because of the point he makes about Napster: It was never about the "peer to peer" or "sharing", that was the least interesting part. The useful thing about Napster was that you could type in a song and download it. It would have been popular if it wasn't peer to peer, so long as you could still get any music you wanted for free. Modeling social media as a filesystem, or constructing a data model about how to link things together, and hypergeneralizing all the way to "here's how to model any graph of data on the filesystem!" is basically a "huh, that's neat" little tech demo but doesn't actually solve anything. Yes, you can take any graph-like structured data and treat it as files and folders. I can write a FUSE filesystem to browse HN. I can spend the 20 minutes noodling on how the schema should work, what a "symlink" should represent, etc... but at the end of the day, you've just taken data and changed how it's presented. There's no reason for the filesystem to be the "blessed" metaphor here. Why not a SQL database? You can `SELECT * FROM posts WHERE like_count > 100`, how neat! Or how about a git repo? You can represent posts as commits, and each person's timeline as a branch, and ooh then you could cherry-pick to retweet! These kind of exercises basically just turn into nerd-sniping: You think of a clever "what if we treated X as Y" abstraction, then before you really stop to think "what problem does that actually solve", you get sucked into thinking about various implementation details and how it to model things. The AT protocol may be well-designed, it may not be, but my point is more that it's not protocols that we're lacking. It's a lack of trust, lack of protection from bad actors, financial incentives that actively harm the experience for users, and the negative effects on what social media does to people. Nobody's really solved any of this: Not ActivityPub, not Mastadon, not BlueSky, not anyone. Creating a protocol that generalizes all of social media so that you can now treat it all homogeneously is "neat", but it doesn't solve anything that you couldn't solve via a simple (for example) web browser extension that aggregated the data in the same way for you. Or bespoke data transformations between social media sites to allow for federation/replication. You can just write some code to read from site A and represent it in site B (assuming sites A and B are willing.) Creating a protocol for this? Meh, it's not a terrible idea but it's also not interesting. - [0] You could argue whether social media is "useful", let's just stipulate that it is.
- metabagel 9mo agoHow does this relate to the SOLID project? https://solidproject.org/ https://solidproject.org/
- danabramov 9mo agoI'd say some of the worldview is shared but the architecture and ethos is very different. Some major differences: - AT tries to solve aggregation of public data first. I.e. it has to be able to express modern social media. Bluesky is a proof that it would work in production. AFAIK, Solid doesn't try to solve aggregation, and is focused on private data first. (AT plans private data support but not now.) - AT embraces "apps describe on their own formats" (Lexicons). Solid uses RDF which is a very different model. My impression is RDF may be more powerful but is a lot more abstract. Lexicon is more or less like *.d.ts for JSON.
- Jonovono 9mo agoI can’t remember how many times I’ve read an article and enjoyed it so much and then looked and saw it was written by Dan ;) always a pleasure !
- danabramov 9mo agoThank you!
- geokon 9mo agoThis was a nice intro to AT (though I feel it could have been a bit shorter) The whole things seems a bit over engineered with poor separation of concerns. It feels like it'd be smarter to flatten the design and embed everything in the Records. And then other layers can be built on top of that Making every record includes the author's public-key (or signature?). Anything you need to point at you'd either just give its hash, or hash + author-public-key. This way you completely eliminate this goofy filesystem hierarchy. Everything else is embed it in the Record. Lexicons/Collections are just a field in the Record. Reverse looking up the hash to find what it is, also a separate problem.
- evbogue 9mo agoYes. SSB and ANProto do this. We actually can simply link to a hash of a pubkey+signature which opens to a timestamped hashlink to a record. Everything is a hash lookup this way and thus all nodes can store data.
- danabramov 9mo agoI'm not sure I understand your proposal. Do you taking my example (a Twitter post) and showing how it would be stored in your system?
- geokon 9mo agoSure, you'd have something like: {:record {:person-key **public-key** :type :twitter-post :message "My friend {:person-key **danabramov-public-key**} suggested I make this on this HN post {:link **record-of-hn-post-hash**}. Look at his latest post {:link **danabramov-newtwitter-post-hash** :person-key **danabramov-public-key**} it's very cool!"} :hash **hash-of-the-record** :signature **signature-by-author**} So everything is self-contained. The other features you'd build on top of this basic primitive - Getting the @danabramov username would be done by having some lookup service that does person-key->username. You could have several. Usernames can be changed with the service.. But you can have your own map if you want, or infer it from github commits :)) There are some interesting ideas about usernames about. How this is done isn't specified by the Record - Lexicon is also done separately. This is some validation step that's either done by a consumer app/editor of the record or by a server which distributes records (could be based on the :type or something else). Such a server can check if you have less than 300 graphemes and reject the record if it fails. How this is done isn't specified by the Record - Collection.. This I think is just organizational? How this is done isn't specified by the Record. It's just aggregating all records of the same type from the same author I guess? - Hashes.. they can point at anything. You can point at a webpage or an image or another record (where you can indicate the author). For dynamic content you'd need to point at webpage that points at a static URL which has the dynamic content. You'd also need to have a hash->content mapping. How this is done isn't specified by the Record This kind of setup makes the Record completely decoupled from rest of the "stack". It becomes much more of independent moveable "file" (in the original sense that you have at the top) than the interconnected setup you end up with at the end.
- skybrian 9mo agoThis article goes into a lot of detail, more than is really needed to get the point across. Much of that could have been moved to an appendix? But it's a great metaphor. Someone should write a user-friendly file browser for PDS's so you can see it for yourself. I'll add that, like a web server that's just serving up static files, a Bluesky PDS is a public filesystem. Furthermore it's designed to be replicated, like a Git repo. Replicating the data is an inherent part of how Bluesky works. Replication is out of your control. On the bright side, it's an automatic backup. So, much like with a public git repo, you should be comfortable with the fact that anything you put there is public and will get indexed. Random people could find it in a search. Inevitably, AI will train on it. I believe you can delete stuff from your own PDS but it's effectively on your permanent record. That's just part of the deal. So, try not to put anything there that you'll regret. The best you could do is pick an alias not associated with your real name and try to use good opsec, but that's perilous.
- DustinBrett 9mo agoI think that is the general style of overreacted.io posts.
- seridescent 9mo ago> Someone should write a user-friendly file browser for PDS's so you can see it for yourself. https://pdsls.dev/ https://pdsls.dev/ can serve this purpose IMO :) it's a pretty neat app, open source, and is totally client-side edit: whoops, pdsls is already mentioned at the end of the article
- danabramov 9mo agoMy goal with writing is generally to move things out of my head in the shape that they existed in my head. If it's useful but too long, I trust other people to pick what they find valuable, riff on it, and so on. >Someone should write a user-friendly file browser for PDS's so you can see it for yourself. You can skip to the end of the article where I do a few demos: https://overreacted.io/a-social-filesystem/#up-in-the-atmosphere https://overreacted.io/a-social-filesystem/#up-in-the-atmosp.... I suggest a file manager there: >Open https://pdsls.dev https://pdsls.dev. [...] It’s really like an old school file manager, except for the social stuff. And yes, the paradigm is essentially "everyone is a scraper".
- noelwelsh 9mo agoThis, Local-first Software [1], the Humane Web Manifesto [2], etc. make me optimistic that we're moving away from the era of "you are the product" dystopian enshittification to a more user-centric world. Here's hoping. [1]: https://www.inkandswitch.com/essay/local-first/ https://www.inkandswitch.com/essay/local-first/ [2]: https://humanewebmanifesto.com/ https://humanewebmanifesto.com/
- pegasus 9mo agoIndeed. And we can get inspired and involved in bringing about that better world.
- sneak 9mo agoLosing private keys is much more common than losing domains.
- danabramov 9mo agoYes, which is why by default, key management is done by your hosting. You log into your host with login/password or whatever mechanism your host supports. Adding your own emergency rotational key in case your hosting goes rogue is supported, but is a separate thing and not required for normal usage. I'd like this to be more ergonomical though.
- clnhlzmn 9mo agoSeems similar to remoteStorage [0]. What happened to that anyway? [0]: https://remotestorage.io/ https://remotestorage.io/
- danabramov 9mo agoThis doesn't look similar to me. remoteStorage seems aimed at apps that don't aggregate data across users. AT aims to solve aggregation, which is when many users own their own data, but what you want to display is something computed from many of them. Like social media or even HN itself.
- Vinnl 9mo agoremoteStorage is still occasionally getting updates. https://solidproject.org https://solidproject.org is a somewhat newer, similar project backed by Tim Berners-Lee. (With its own baggage.) I think of those projects as working relatively well for private data, but public data is kinda awkward. ATProto is the other way around: it has a lot of infra to make public data feasible, but private data is still pretty awkward. It's a lot more popular though, so maybe has a bigger chance of solving those issues? Alternatively, Bluesky keeps its own extensions for that, and starts walling those bits off more and more as the VCs amp up the pressure. That said, I know very little about Bluesky, so this speculation might all be nonsense.
- imglorp 9mo agoYes Solid is another idea that's clearly the right thing, especially now. It might mlnit be easy to convince everyone they need it though, and the economics will be uphill given the trillions in entrenched, incumbent opposition.
- jrm4 9mo agoThe more I read and consider Bluesky and this protocol, the more pointless -- and perhaps DANGEROUS -- I find the idea. It really feels like no one is addressing the elephant in the room of; okay, someone who makes something like this is interested in "decentralized" or otherwise bottom-up ish levels of control. Good goal. But then, when you build something like this, you're actually helping build a perfect decentralized surveillance record. This why I say that most of Mastodon's limitations and bugs in this regard (by leaving everything to the "servers") are actually features. The ability to forget and delete et al is actually important, and this makes that HARDER. I'm just kind of like, JUST DO MASTODONS MODEL, like email. It's better and the kinks are more well thought about and/or solved.
- plagiarist 9mo agoShouldn't the ability to forget and delete content that was ever public on the internet be considered fictional anyway?
- bee_rider 9mo agoThis seems like tensions between normal/practical and “opsec” style privacy thinking… Really, we can never be sure anything that gets posted on the internet won’t be captured by somebody outside our control. So, if we want to be full paranoid, we should act like it will be. But practically lots of people have spent a long time posting their opinions carelessly on the internet. Just protected by the fact that nobody really has (or had) space to back up every post or time to look at them too carefully. The former has probably not been the case for a long time (hard drives are cheap), and the latter is possibly not true anymore in the LLM era. To some extent maybe we should be acting like everything is being put into a perfect distributed record. Then, the fact that one actually exists should serve as a good reminder of how we ought to think of our communications, right?
- jrv 9mo agoExactly. Anything that's ever been public on the internet is never really gone anyways, and it's unsafe to assume so. This is similar to publishing a website or a blog post. Plus, from a practical (non-opsec) point of view, you can delete items (posts, likes, reposts, etc.) on ATProto, and those items will disappear from whatever ATProto app you are using - usually even live. You need to dive into the protocol layer to still see deleted items.
- nonethewiser 9mo agoBut how do you get people to actually want this? This stuff is pretty niche even within tech.
- danabramov 9mo agoBluesky is not huge, but 40M users is not nothing either. You don't get people to want this, you just try to build better products. The hope is that this enables us all to build better products by making them more interoperable by default. Whether this pans out remains to be seen.
- demux 9mo agoI also don't think the average user gets the value of the protocol yet. Most of those users were looking for a new, more politically palatable home but with the same features as Twitter. The new generation of apps on the protocol will be vital in showing users what's possible. IMO the two most valuable features at a practical level are: - social graph portability, which might look like having an onboarding experience that bootstraps your community on that app - lexicon cross compatibility, i.e. your data from app A shows up in a contextually relevant spot in app B. Or app B writes records that show up in app A. This is pretty key to get right because it might confuse or anger users if they aren't condition to expect it. Once the average user groks these features though, I'd be surprised if they voluntarily switch back to the standard corpo apps that eventually exit to some company who tries to monetize the shit out of every feature.
- heyitsaamir 9mo agoI think most people do want this. They want to own their data. If you ask someone if they post on IG, if they should own that, or IG, they'll tell you it's them. The hard problem IMO is how do you incentivize companies from adopting this since walled gardens helps reduce competition.
- __MatrixMan__ 9mo agoWe want more control over data that we've created, and more control over data that's about us. I'm not sure either of these concepts align well with "ownership" though. Property and data are concepts that don't mix. Language nitpicking aside... you subvert the walls of their gardens and aggregate the walled-off data without the walls, so users face a choice not between: - facebook - everything else but instead between - facebook and everything else - just facebook But that approach only works if we can solve the "data I created" problems in a way that doesn't also require us to acknowledges facebook's walls.
- motoxpro 9mo agoI've always thought walled gardens are the effect of consumer preferences, not the cause. The effect of the internet (everything open to everyone) was to create smaller pockets around a specific idea or culture. Just like you have group chats with different people, thats what IG and Snap are. Segmentation all the way down. I am so happy that my IG posts arent available on my HN or that my IG posts arent being easily cross posted to a service I dont want to use like truth social. If you want it to be open, just post it to the web. I think I don't really understand the benefit of data portability in the situation. It feels like in crypto when people said I want to use my Pokemon in game item in Counterstrike (or any game) like, how and why would that even be valuable without the context? Same with a Snap post on HN or a HN post on some yet-to-be-created service.
- dameis 9mo ago>I am so happy that my IG posts arent available on my HN or that my IG posts arent being easily cross posted to a service I dont want to use like truth social. ATProto apps don't automatically work like this and don't support all types of "files" by default. The app's creator has to built support for a specific "file type". My app https://anisota.net https://anisota.net supports both Bluesky "files" and Leaflet "files", so my users can see Bluesky posts, Leaflet posts, and Anisota posts. But this is because I've designed it that way. Anyone can make a frontend that displays the contents of users PDSs. Here's an example... Bluesky Post on Bluesky: https://bsky.app/profile/dame.is/post/3m36cqrwfsm24 https://bsky.app/profile/dame.is/post/3m36cqrwfsm24 Bluesky Post on Anisota: https://anisota.net/profile/dame.is/post/3m36cqrwfsm24 https://anisota.net/profile/dame.is/post/3m36cqrwfsm24) Leaflet post on Leaflet: https://dame.leaflet.pub/3m36ccn5kis2x https://dame.leaflet.pub/3m36ccn5kis2x Leaflet post on Anisota: https://anisota.net/profile/dame.is/document/3m36ccn5kis2x https://anisota.net/profile/dame.is/document/3m36ccn5kis2x I also have a little side project called Aturi that helps provide "universal links" so that you can open ATProto-based content on the client/frontend of your choice: https://aturi.to/anisota.net https://aturi.to/anisota.net
- verdverm 9mo agoExcept that a lot of the app builders in ATProto seem to think the protocol was designed to make their lives easier when bootstrapping their network from Bluesky userbase. (Imo, that is a perverse interpretation, it's about user choice, which they are effectively taking away from me by auto importing and writing to my Bsky graph) re: the debates on reusing follows from Bluesky in other apps instead of their own
- eduction 9mo agoUnpopular opinion: this should be done with xml, not json. XML can have types, be self describing, and be extended (the X in XML). That said it’s a very elegant way to describe AT protocol.
- danabramov 9mo agoI'd be curious to see what that would look like!
- doctorflan 9mo agoI was hoping this was literally just going to be some safe version of a BBS/Usenet sort of filesharing that was peer-based king of like torrents, but just simple and straightforward, with no porn, infected warez, randomware, crypto-mining, racist/terrorist/nazi/maga/communist/etc. crap, where I could just find old computing magazines, homebrew games, recipes, and things like that. Why can’t we have nice things? I guess that’s what Internet Archive is for.
- skeledrew 9mo agoI've been thinking of this for some time, conceptually, but perhaps from a more fundamental angle. I think the idea of "files" is pretty dated and can be thrown out. Treat everything as data blobs (inspired by PerKeep[0]) addressed by their hashes and many of the issues described in the article just aren't even a thing. If it really makes sense, or for compatibility sake, relevant blobs can be exposed through a filesystem abstraction. Also, users don't really want apps. What users want are capabilities. So not Bluesky, or YouTube for example, but the capability to easily share a life update with interested parties, or the capability to access yoga tutorial videos. The primary issue with apps is that they bundle capabilities, but many times particular combinations of capabilities are desired, which would do well to be wired together. Something in particular that's been popping up fairly often for me is I'm in a messaging app, and I'd like to lookup certain words in some of the messages, then perhaps share something relevant from it. Currently I have to copy those words over to a browser app for that lookup, then copy content and/or URL and return to the messaging app to share. What I'd really love is the capability to do lookups in the same window that I'm chatting with others. Like it'd be awesome if I could embed browser controls alongside the message bubbles with the lookup material, and optionally make some of those controls directly accessible to the other part(y|ies), which may even potentially lead to some kind of adhoc content collaboration as they make their own updates. It's time to break down all these barriers that keep us from creating personalized workflows on demand. Both at the intra-device level where apps dominate, and at the inter-device level where API'd services do. [0] https://perkeep.org/ https://perkeep.org/
- danabramov 9mo agoI'm using filesystem more as a metaphor than literally. I picked this metaphor because "apps" are many-to-many to "file formats". I found "file format" to be a very powerful analogy for lexicons so I kind of built everything else in the explanation around that. You can read https://atproto.com/specs/repository https://atproto.com/specs/repository for more technical details about the repository data structure: The repository data structure is content-addressed (a Merkle-tree), and every mutation of repository contents (eg, addition, removal, and updates to records) results in a new commit data hash value (CID). Commits are cryptographically signed, with rotatable signing keys, which allows recursive validation of content as a whole or in part. Repositories and their contents are canonically stored in binary DAG-CBOR format, as a graph of data objects referencing each other by content hash (CID Links). Large binary blobs are not stored directly in repositories, though they are referenced by hash (CID). Re: apps, I'd say AT is actually post-app to some extent because Lexicons aren't 1:1 to apps. You can share Lexicons between apps and I totally can see a future where the boundaries are blurring and it's something closer to what you're describing.
- James_K 9mo agoAT Proto seems very overengineered. We already have websites with RSS feeds, which more or less covers the publishing end in a way far more distributed and reliable than what AT offers. Then all you need is a kind of indexer to provide people with notifications and discovery and you're done. But I suppose you can't sell that to shareholders because real decentralised technology probably isn't going to turn as much of a profit as a Twitter knockoff with a vague decentralised vibe to it that most users don't understand or care about.
- danabramov 9mo agoWhy so much cynicism? The people working there genuinely care about this stuff. Maybe you disagree with technical decisions but why start by projecting your fantasies about their motivations? RSS is OK for what it does, but it isn't realtime, isn't signed, and doesn't support arbitrary structured data. Whereas AT is signed, works with any application-defined data structures, and lets you aggregate over millions of users in real time with subsecond end-to-end latency.
- James_K 9mo agoRSS is signed because it's served over HTTPS which involves an SSL certificate exchange. It is also real time. You fetch the feed and get the most up to date content on it. You can host any XML document you want on your website and there are W3C recommended standards that allow the integration of rich data structures within web pages. The sole interesting part of ATProto is the aggregation part, which can also be solved in the same way with existing web technologies, that is a web indexer. Google allows you to set up notifications whenever a link is created to your website, which is essentially the unique offering of AT. None of the ideas it presents are new, but they are all incompatible with existing Web technologies. That is why I dislike it. I already have a website. I already have a set of RSS feeds that I follow. I do not want another website hosted on a strange and convoluted service in a bajillion different JSON records when there is a folder of XML on my computer called “my-website” which I can copy to any server and have all I need. Yet, in the name of a federated internet, the BSky folks have herded people away from the real distributed internet and into a crappy Twitter knockoff. Perhaps there are some business-unaware programmers very passionate about the idea of a distributed web and I commend them for that but it already exists and, in every real aspect, they are harming it. If this ATProto thing is successful in its goals, it will cause much inconvenience to me and greatly diminish my ability to publish my works on the internet.
- EGreg 9mo agoAs someone who explicitly designed social protocols since 2011, who met Tim Berners-Lee and his team when they were building SOLID (before he left MIT and got funded to turn it into a for-profit Inrupt) I can tell you that files are NOT the best approach. (And neither is SPARQL by the way, Tim :) SOLID was publishing ACLs for example as web resources. Presumably you’d manage all this with CalDAV-type semantics. But one good thing did come out of that effort. Dmitri Zagidulin, the chief architect on the team, worked hard at the W3C to get departments together to create the DID standard (decentralized IDs) which were then used in everything from Sidetree Protocol (thanks Dan Buchner for spearheading that) to Jack Dorsey’s “Web5”. Having said all this… what protocol is better for social? Feeds. Who owns the feeds? Well that depends on what politics you want. Think dat / hypercore / holepunch (same thing). SLEEP protocol is used in that ecosystem to sync feeds. Or remember scuttlebutt? Stuff like that. Multi-writer feeds were hard to do and abandoned in hypercore but you can layer them on top of single-writer. That’s where you get info join ownership and consensus. ps: Dan, if you read this, visit my profile and reach out. I would love to have a discussion, either privately or publicly, about these protocols. I am a huge believer in decentralized social networking and build systems that reach millions of community leaders in over 100 countries. Most people don’t know who I am and I’m happy w that. Occasionally I have people on my channel to discuss distributed social networking and its implications. Here are a few: Ian Clarke, founder of Freenet, probably the first decentralized (not just federated) social network: https://www.youtube.com/watch?v=JWrRqUkJpMQ https://www.youtube.com/watch?v=JWrRqUkJpMQ Noam Chomsky, about Free Speech and Capitalism (met him same day I met TimBL at MIT) https://www.youtube.com/watch?v=gv5mI6ClPGc https://www.youtube.com/watch?v=gv5mI6ClPGc Patri Friedman, grandson of Milton Friedman on freedom of speech and online networks https://www.youtube.com/watch?v=Lgil1M9tAXU https://www.youtube.com/watch?v=Lgil1M9tAXU
- danabramov 9mo agoTo be clear, I'm using files in a relatively loose sense to focus on the "apps : formats are many-to-many" angle. AT does not literally implement a full filesystem. As the article progresses, I restrict some freedoms in the metaphor (no directories except collections, everything is JSON, etc). If you're interested in the actual low-level repository format, it is described here: https://atproto.com/specs/repository https://atproto.com/specs/repository
- yladiz 9mo agoI know this is somewhat covered in another comment, but, the concepts described in the post could have been reduced quite a bit, no offense Dan. While I like the writing generally, I would consider writing and then letting it sit for a few days, rereading, and then cutting chaff (editing). This feels like a great first draft but without feedback, and could have greatly benefited from an editing process, and I think using the argument that you want to put out something for others to take and refine isn’t really a strong one… a bit more time and refinement could have made a big difference here (and given you have a decently sized audience I would keep in mind).
- danabramov 9mo agoFrom my perspective, there is no chaff. I've already the read the entire thing from top to bottom over 20 times (as I usually do with my writing), I've done several full edit passes, and I've removed everything inessential that I could find. The rest is what I wanted to be included into this article. I know my style is verbose but I try to include enough details to substantiate the argument at the level that I feel confident it fully stands for itself. If others find something useful in it, I trust that they can riff on those bits or simplify.
- pcthrowaway 9mo agoFor the commenters who have made similar comments, I'd be curious to hear what they think could be cut. I suspect different readers will have different opinions on this, which means it's probably a good thing you didn't make cuts.
- lanyard-textile 9mo agoThere is not much actionable here, as well intentioned as your comment is. It's like saying this MR could use some work but not citing a specific example.
- ahussain 9mo agoIt seems like the biggest downside of this world is iteration speed. If the AT instagram wants to add a new feature (i.e posts now support video!) then can they easily update their "file format"? How do they update it in a way that is compatible with every other company who depends on the same format, without the underlying record becoming a mess?
- danabramov 9mo agoThat's a great question! Adding new features is usually not a problem because you can always add optional fields and extend open unions. So, you just change `media: Link | Picture | unknown` to `media: Link | Picture | Video | unknown`. You can't remove things true, so records do get some deprecated fields. Re: updating safely, the rule is that you can't change which records it would consider valid after it gets used in the wild. So you can't change whether some field is optional or required, you can only add new optional fields. The https://github.com/bluesky-social/goat https://github.com/bluesky-social/goat tool has a linting command that instantly checks whether your changes pass the rules. In general it would be nice if lexicon tooling matures a bit, but I think with time it should get really good because there's explicit information the tooling can use. If you have to make a breaking change, you can make a new Lexicon. It doesn't have to cause tech debt because you can make all your code deal with a new version, and convert it during ingestion.
- skybrian 9mo agoAre these just guidelines or is this enforced in some way? I guess readers could validate and skip anything that doesn't match their schema.
- extraduder_ire 9mo agoMost apps reading records will validate a record against the schema for that type. e.g. there's nothing stopping you from making a app.bsky.feed.post record with more than 300 graphemes in the "text" field, but that post won't appear in the "official" app/website because it fails schema validation. Similarly, there's nothing stopping you from adding another field in your post. It'll just get ignored because the app you're using doesn't know about it. e.g. posts bridged from mastodon by bridgy have an extra field containing the full original post's text, which you can display in your app if desired. reddwarf.app does this with these posts. Here's the schema for bluesky posts, if you're interested: https://pdsls.dev/at://did:plc:4v4y5r3lwsbtmsxhile2ljac/com.atproto.lexicon.schema/app.bsky.feed.post#schema https://pdsls.dev/at://did:plc:4v4y5r3lwsbtmsxhile2ljac/com....
- theturtletalks 9mo agoPOSSE and AT Protocol can be understood as interoperable marketplaces. Platforms like Reddit and Instagram already function this way: the product is user content, the payment is attention, and the platform’s cut is ads or behavioral data. Dan argues that this structure is not inevitable. If social data is treated as something people own and store themselves, applications stop being the owners of social graphs and become interfaces that read from user-controlled data instead. I am working on a similar model for commerce. Sellers deploy their own commerce logic such as orders, carts, and payments as a hosted service they control, and marketplaces integrate directly with seller APIs rather than hosting sellers. This removes platform overhead, lowers fees, and shifts ownership back to the people creating value, turning marketplaces into interoperable discovery layers instead of gatekeepers.
- deleted 9mo ago[deleted]
- dharmatech 9mo agoWas going to ask this but then I found openship in your profile. Will check it out. Thanks!
- dharmatech 9mo agoThis is a nice introduction: https://openship.org/about https://openship.org/about
- ivell 9mo agoYou may want to take a look at India's ONDC (https://github.com/ONDC-Official https://github.com/ONDC-Official).
- sfdlkj3jk342a 9mo agoSounds interesting, but I can't find any links to an actual live B2C storefront..
- 9mo ago
- jadbox 9mo agoHow do people view AT Protocol vs Nostr? Why choose one over the other? Which has a better chance at replacing X?
- danabramov 9mo agoI found this helpful: https://shreyanjain.net/2024/07/05/nostr-and-atproto.html https://shreyanjain.net/2024/07/05/nostr-and-atproto.html
- LoganDark 9mo agoI did a double take at "DID as identity" because Dissociative Identity Disorder shares the same acronym
- bschmidt999 9mo ago[dead]
- swyx 9mo ago> Apps may come and go, but files stay—at least, as long as our apps think in files. yes: https://www.swyx.io/data-outlasts-code-but https://www.swyx.io/data-outlasts-code-but all lasting work is done in files/data (can be parsed permissionlessly, still useful if partially corrupted), but economic incentives keep pushing us to keep things in code (brittle, dies basically when one of maintainer|buildtools|hardware substrate dies). when standards emerge (forcing code to accept/emit data) that is worth so much to a civilization. a developer ecosystem tipping the incentive scales such that companies like the Googl/Msft/OpenAI/Anthropics of the world WANT to contribute/participate in data standards rather than keep things proprietary is one of the most powerful levers we as a developer community collectively hold. (At the same time we shoudl also watch out for companies extending/embracing/extinguishing standards... although honestly outside of Chrome I struggle to think of a truly successful example)
- danabramov 9mo agoNice to see you :) I didn't know the "indirection" law, that's funny.
- zahlman 9mo agoIndeed. My first reaction was: > Files are the source of truth—the apps would reflect whatever’s in your folder. Now that the "app" is a web site that supports itself with advertising revenue, it has no incentive whatsoever to work this way.
- danabramov 9mo agoI think it's on open social apps to show that they're actually meaningfully better products, and that is possible because they're open. With luck, this may lead to an ecosystem where it's worth staying compatible and interoperable, and where users scoff if someone is trying to break it, and where users have an easy way to walk away. I know this sounds super idealistic but this did essentially happen with open source over a long time. At some point, people were just as skeptical of open source as we might be about open social.
- 9mo ago
- hollowonepl 9mo agoInteresting concept for all new social platforms that already live in federated, distributed environments that share communication protocols and communication data formats. I bet more difficult to push existing commercial platforms to anyhow consider. That would make marketing tools to manage social communications and posting across popular social media, much easier. Never the less Social Marketing tools have already invented we similar analogy just to make control over own content and feedback across instances and networks. We still live in a world where some would say BSKY some would say Mastodon is the future… while everybody still has facebook and instagram and youngsters tik tok too. Those are closed platforms where only tools to hack them, not standards persist
- christophilus 9mo agoI’ve been reading “The Unix Programming Environment”. It’s made me realize how much can be accomplished with a few basic tools and files (mostly plain text). I want to spend some time thinking of what a modern equivalent would look like. For example, what would Slack look like if it was file (and text) oriented and UNIXy? Well, UNIX had a primitive live chat in the form of live inter-user messaging. I’d love to see a move back to simpler systems that composed well.
- zahlman 9mo agoUnix gave the world a lot of good ideas about architecture, but I think it really hamstrung itself by treating all the data as plain text and resisting the idea of having any kind of structured, formatted data passed within a pipeline. It's nice to be able to serialize to something human-readable and -editable, but constantly re-parsing and re-formatting it becomes a real pain.
- itmitica 9mo agoTo share is to lose control. You can't undo, even once shared, it can't be undone. You can't retract a published novel. You can't retract a broadcast music or show. What makes you think you can do it over internet?
- danabramov 9mo agoI don't think my article makes any claims that one can undo sharing. What I'm saying is that we benefit collectively from being able to untether data from applications. It's the same logic as https://stephango.com/file-over-app https://stephango.com/file-over-app but applied to the aggregating web applications.
- itmitica 9mo agoShare and you can't control what everyone else does with the data. Hence, untether is a non concept. In the sense that that data has now lost it's uniqueness. You can untether all you want. Copies of it will be subjected to any kind of tethering and transformation. So, if your point was about control, forget it.
- camgunz 9mo agoI'm skeptical of these kind of like, self-describing data models. Like, I generally like at proto--because I like IPFS--but I think the whole "just add a lexicon for your service and bickety bam, clients appear" is a leap too far. For example, gaze upon dev.ocbwoy3.crack.defs [0] and dev.ocbwoy3.crack.alterego [1]. If you wanted to construct a UI around these, realistically you're gonna need to know wtf you're building (it's a twitter/bluesky clone); there simply isn't enough information in the lexicons to do a good job. And the argument can't be "hey you published a lexicon and now people can assume your data validates", because validation isn't done on write, it's done on read. So like, there really is no difference between this and like, looking up the docs on the data format and building a client. There are no additional guarantees. Maybe there's an argument for moving towards some kind of standardization, but... do we really need that? Like are we plagued by dozens of slightly incompatible scrobbling data models? Even if we are, isn't this the job of like, an NPM library and not a globally replicated database? Anyway, I appreciate that, facially, at proto is trying to address lock in. That's not easy, and I like their solution. But I don't think that's anywhere near the biggest problem Twitter had. Just scanning the Bluesky subreddit, there's still problems like too much US politics and too many dick pics. It's good to know that some things just never change I guess. [0]: https://lexicon.garden/lexicon/did:plc:s7cesz7cr6ybltaryy4meb6y/dev.ocbwoy3.crack.defs https://lexicon.garden/lexicon/did:plc:s7cesz7cr6ybltaryy4me... [1]: https://lexicon.garden/lexicon/did:plc:s7cesz7cr6ybltaryy4meb6y/dev.ocbwoy3.crack.alterego https://lexicon.garden/lexicon/did:plc:s7cesz7cr6ybltaryy4me...
- danabramov 9mo agoNot sure I fully get you... In your example, isn't the problem that nobody cares about this data? So there is no motivation to build a client. Whereas if these were beloved notes or minisites or whatever that got wiped out by the latest acquisition (e.g. see https://bento.me/ https://bento.me/ shutting down), people would know exactly what those are, and there would be incentive for someone to compete for the userbase. E.g. Blento (https://blento.app/ https://blento.app/) is atproto Bento that I only saw a couple of days ago. But the cool thing is that if it shuts down, not only someone else can set it up again (it's open source), but they're also gonna be able to render all of the users' existing content. I think that's a meaningful step forward for this use case. Yes, there's gonna be tons of stuff on the network that's too niche, but then there's no harm in it either. Whereas wherever there is enough interest, someone can step in and provide the code for the data.
- air217 9mo agonostr protocol and the client/relay model is one simple way to separate apps (clients) from the data (relays)
- echoangle 9mo agoIs there anything stopping me from backdating my own records? Since the createdAt is just an arbitrary field, I can just write whatever I want in there, right? Is there a way for the viewing application to verify when the record was created (and not modified since), maybe by looking at the mentioned signing?
- danabramov 9mo agoYou can indeed backdate records. Since the application knows when it has first seen (i.e. indexed) your record, it can decide what to do with that information. If there's a difference, the Bluesky app, for example, shows its own indexing time, but also shows a separate panel saying the post is backdated to some other date. Other apps could choose to show something else. It is possible to create links asserting a specific version by making a "strong ref" which includes content hash.
- diceduckmonk 9mo agoGit is the API. Github/Gitlab would be a provider of the filesystem. The problem is app developers like Google want to own your files.
- sroerick 9mo agoInteresting - I just spent all day on this on an app which I'm using. My architecture is a little different (probably worse). The app lives on a single OpenBSD server. All user data is stored in /srv/app/[user]. Authentication is done by accessing OpenBSD Auth helper functions. Users can access their data through the UI normally. Or they can use a web based filesystem browser to edit their data files. Or, alternately, they can ssh into the server and have full access to their files with all the advantages this entails. Hopefully, this raises the ceiling a bit for what power users of the system can accomplish. I wanted to unify the OS ecosystem and the web app ecosystem and play around with the idea of what happens if those things aren't separate. I'm sure I'm introducing all kinds of security concerns which I'm not currently aware of. Another commenter brought up Perkeep, which I think is very interesting. Even though I love Plan 9 conceptually, I do sort of wonder if "everything is a file" was a bit of a wrong turn. If I had my druthers, I think building on top of an OS which had DB and blob storage as the primary concept would be interesting and perhaps better. If anybody cares, it's POOh stack, Postgres, OCAML, OpenBSD, AND htmx
- nonethewiser 9mo agoIronically, DID is the perfect vehicle for age verification.
- hahahahhaah 9mo agoI have always thought open file format > open source. My ideal web everyone has their own web file storage (get from anywhere e.g. email provider) and web apps use that to store things. Team collab etc. built on top of that e.g. sharing a file means share ann accept edits type flow. Everyone owns their file.
- viraptor 9mo agoI like the write-up of this idea. It's well presented. But I'd change one aspect: "We could leave author: 'dril' in the JSON but this is unnecessary too." - kind of. What the post lacks is the record of the identity at the time. What the user's username and the avatar was at the time can change the meaning of the post entirely. To really preserve the message, you need to reference what the displayed identity was used to post it - not just the account id. There's a number of famous accounts that do it continuously. For example popehat today is "Fucking Bitch Hat" but will change to something else soon that may be related to the current events.
- danabramov 9mo agoI think most people's mental model is that they should be able to change their handle / display name / avatar freely, and their posts would display the new versions. So those aren't a part of the post itself. That said, you could create an AT app that displays a version of the post using the profile at the time. You'd just need to index all profile changes into a local database, and then query that database for the "profile at that time" to display the post. So what you're describing is possible—it just requires a different aggregation. The source of truth, however, should be denormalized and reflect most recent data.
- japanuspus 9mo ago> Identity -- This is a difficult problem. My hope is that in 5 years, I will not have anything in my feeds that have not been signed in a way that I can assign a trust level. Here in the Nordics, we are already seeing messaging apps such as [hudd] that require government issued ID to sign in. I want this to spread to everything from podcasts and old-school journalism to the soccer-club newsletter, so that I can always connect a piece of information back to a responsible source. [hudd]: (https://about.hudd.dk/ https://about.hudd.dk/))
- wink 9mo agoSo you're simply not interested in reading any random website by random people who don't see a benefit of establishing any form of trust, especially if should not be connected to their official government IDs? Or to put it differently: Where should this come from, and which issuer would you trust? And why should anyone else agree with you that this is good?
- johnecheck 9mo agoTrust is subjective! Let's establish trust in each other rather than rely on one-size-fits-all solutions. Personally, I trust my friends, family, and some public figures and institutions to varying degrees. I want to see social experiences that reflect that.
- theshrike79 9mo agoI trust person A, person A trusts B When I browse to random site, I can see in my browser that A's first level contact trusts this site. Now I can make a decision based on the amount of trust I have on A. Maybe after exploring the site I can mark specific pages or the whole domain as trusted, so people in my network can see the same. On a larger level I might trust the Country of Finland, who will only mark their official sites as Trusted. This way I instantly know if I'm on an official site or something pretending to be one.
- trizuz 9mo ago[dead]
- articsputnik 9mo agoAmazing write-up. Owing to your data, a standardized protocol (such as the AT Protocol) is so great! It's like markdown, everything is basically built on files (but cleverly architected so that it works decentralized across the web too!). I wrote a little along the same lines on «Well Being in Times of Algorithms» at https://www.ssp.sh/blog/well-being-algorithms/ https://www.ssp.sh/blog/well-being-algorithms/ (or HN: https://news.ycombinator.com/item?id=46352747 https://news.ycombinator.com/item?id=46352747).
- black_puppydog 9mo agoThe premise of this article resonates so much with me! I didn't see the angle on ATproto coming, and frankly this description of it is the first that makes me want to dig into it a bit. The issue of "file-less" computing has been bothering me a lot. It's worst on iOS, where apple are really pushing hard to have users never ever think in "files". Closely followed by MacOS and only then android, imho. "That thing over there? That's not a file! That's a photo! Very different thing, that. No, you can't use $app to view it or share it, sorry. And if you want to copy it, you have to go through our export functionality which is buggy and strips all kind of info and generally works best through iCloud. My mind's insistance that behind all the flashy apps must he a backing store, and that it's most likely file based, makes it even more infuriating when trying to get unprocessed photos off a relative's iPhone or such.
- aembleton 9mo agoWhy would we need to store the createdAt value in a file? The filesystem already stores this information. We could just store the text which would mean no Json would be needed.
- danabramov 9mo agoI'm using "filesystem" a bit loosely here. The important parallel I was going for was "file format" as interface between apps (= lexicons being an interface between social apps). If you want details on the actual data structures, check https://atproto.com/specs/repository https://atproto.com/specs/repository.
- voidUpdate 9mo agoHow does this system determine the amount of likes a post has? Since there is no back reference on a post to people who have liked it, don't you have to iterate over every single person, iterate over their likes, to see if one of them is the post you are viewing, and add all them up?
- galactus 9mo agoIn the ATProto architecture, this function is handled by the AppView, which monitors the full network and produces the corresponding aggregates.
- danabramov 9mo agoYes. I describe this in this part of the article: https://overreacted.io/a-social-filesystem/#:~:text=One%20challenge%20with,like%20vote_count. https://overreacted.io/a-social-filesystem/#:~:text=One%20ch... It's basically event sourcing. You listen to the data you care about from the network and update the local index (DB). There are also tools like Tap (https://docs.bsky.app/blog/introducing-tap https://docs.bsky.app/blog/introducing-tap) that do the plumbing work and let you backfill automatically.
- voidUpdate 9mo agoAh, I'm sorry, I somehow skipped over that bit entirely. Need more coffee, I think
- extraduder_ire 9mo agoThere's also software called Constellation[0] from microcosm that exclusively collects "backlinks", which are items which link to a certain thing. e.g. follow records for an account, reposts/replies/likes on a post. If you click on the Backlinks tab when examining something on pdsls.dev you can see a list of them. It can be self hosted and is much lighter than running a full appview. The author has a public endpoint for it that's keeping up with the whole network running on a single raspberry pi on his home network. 0: https://www.microcosm.blue/ https://www.microcosm.blue/
- extraduder_ire 9mo agoYou mention having a "self" record in the app.bsky.actor.profile lexicon to store profile information, is there any reason to have more records of that type in your repository? I've seen a few people make other ones when examining their accounts with pdsls, but they seem to be there for "just because I can" reasons.
- danabramov 9mo agoDifferent apps can have different notions of a profile so you'd probably have one per app.
- xk3 9mo agoMakes me think of Syncthing... I wrote a wrapper for it that attempts to make it easier for people to use: https://github.com/chapmanjacobd/syncweb-py https://github.com/chapmanjacobd/syncweb-py https://github.com/chapmanjacobd/syncweb-ui https://github.com/chapmanjacobd/syncweb-ui Unfortunately, mesh storage systems are very different conceptually so it is difficult for people to think about permissions and access. You can bolt on something familiar but then it really limits the usefulness of mesh storage and you may as well just be using HTTP servers.
- dannyfritz07 9mo agoIs there a community or forum where questions about ATProtocol can be asked? Can you can spread your identity across multiple PDS repositories? Was thinking about creating a PDS that can store binary blobs to download, but not all PDS would like that amount of binary data stored. UPDATE: found it: https://discourse.atprotocol.community https://discourse.atprotocol.community