4 ms·

Genuinely curious, how would you build it differently? Would you stick with atproto's indexing model and try to simplify it, or would you use another approach?

by pfraze 1mo ago

James_K 1mo ago
You send a request to a web indexer with a page you want added to the network, it scans your page for outgoing links and sends an email to the owners of any domains who are also on the service. One can imagine making it a little more featureful, for instance indexing OPML blogrolls which would allow you to see whom a person follows and https://microformats.org/wiki/h-entry https://microformats.org/wiki/h-entry which would allow outgoing links to be categorised (like, reply, etc) so notifications filtered. With that, it would be possible to add a front-end site imitating one of the popular social media paradigms (Reddit-style or Twitter-style being the most obvious). If there's one genuine design decision I would give for it beyond what is basically a cobbling together of existing interfaces, it would be to charge users per page uploaded. Likely very detrimental to growing the service, but I think one of the simplest ways to weed out spam and junk. A real problem with many web services is that the receiver of the message pays for it in terms of attention, where in other mediums the sender has to pay. Given that uploading crap is basically free, that's all you get. Increasing the cost of upload would weed out those endless AI summaries and lists of affiliate links. The issue with ATProto is that it operates on too many layers. You can see lacking in what I have described here the concept of durable authorship, but this is a property of content not how the content is distributed. Perhaps someone will invent a standard way to sign HTML documents, in which case you could base user accounts on that instead of DNS. AT enforces this centrally but it does not need to. It is walling itself off from the common and decentralised software ecosystem of the web for no good reason.
- pfraze 1mo ago
  Yeah that'd be interesting to try. If you extend the endpoints and the vocabularies around RSS/OPML enough, you could likely create a fairly robust dataset replication protocol. Microformats are a decent schema basis, but you might want to build upon them as well. If you want to make the users' sites reusable as a datastore across applications, you could introduce an oauth flow to enable 3rd party writes. You might also want to add content signatures so you can verify the data authenticity from third parties (e.g. to support the equivalent of reposts).
  - James_K 1mo ago
    You would repost a page by linking to it from your own. As for uploading content, that is outside of the scope of an indexer. The user hosts their data with some hosting service, or independently. I believe signatures are already handled by XHTML which allows you to add XML signatures to documents, but regular HTML is sadly lacking here thought it would not be too hard to extend (or to just use the XML serialisation of HTML5). I think it is generally bad to implement new features like this on the part of the indexer. It should just keep track of an existing web of documents rather than creating it's own format and walled garden.
    - pfraze 1mo ago
      > You would repost a page by linking to it from your own. Ah right, of course. Fair enough. I think Dave Winer is trying some similar ideas, though I haven't looked at it beyond knowing he's trying to extend RSS to support these capabilities.
      - James_K 1mo ago
        We already have those capabilities. My website has reposts on it. I just link to someone else's content from my Atom feed instead of something hosted on my own site. Then I attach a https://validator.w3.org/feed/docs/atom.html#category https://validator.w3.org/feed/docs/atom.html#category node indicating that those are reposts.
- inigyou 1mo ago
  Isn't that how it already works but with different protocols that you don't like? You (a PDS) send a request to an indexer (relay) with a page (activity) you want added to the network.
  - James_K 1mo ago
    The difference is one uses W3C recommended web interfaces and the other does not. I make blog posts to my website using HTML, anyone can open them using a web browser. I can host those pages one of many static hosting services with no need for a special protocol or anything like that. AT on the other hand reïnvents this entire thing. Can you type the AT URI of a resource into the browser and have it open? I do not believe you can. So if I want to publish HTML content I now need both my regular web server and an AT PDS running on my server. There exists, as far as I'm aware, no application that can view HTML hosted on the AT network so I have to produce all of my blog posts in an agnostic format or publish them to no one, or I can use one of the services that automatically posts a link to that HTML as a BSky post. So you have all of that architecture and what it accomplishes is posting a link to content hosted elsewhere. Why, instead of creating an indexer that works with existing formats, did they choose to create one intentionally incompatible with 99.999% of web content? Because in practice it's not a web protocol, it's not a serious thing. It's just a whole bunch of bullshit for nerds to make blog posts about that serves as the back end of a Twitter clone. It is standards proliferation. It adds nothing and fragments existing efforts. Instead of contributing to the web ecosystem, it subtracts from it. It breaks the important rules of software development, to do one thing and do it well, to use existing solutions. Everything it does is duplicated, there's a new way to sign documents, a new way to host content, a new data format, a new exchange protocol, a new way to serve documents, a new way to send notifications, etc. It's not a tenable way of doing things. If every person looking to make a web indexer created an entire separate web ecosystem, that would not be sustainable. They're clearly not intending to replace the Web, so what are they doing? It's just a complicated way to store tweets. And the silliest part is that, if I recall correctly, they don't even have decentralised network yet since only they can issue PLC DIDs.
    - inigyou 1mo ago

Genuinely curious, how would you build it differently? Would you stick with atproto's indexing model and try to simplify it, or would you use another approach?

by pfraze 1mo ago

Genuinely curious, how would you build it differently? Would you stick with atproto's indexing model and try to simplify it, or would you use another approach?

James_K 1mo ago

You send a request to a web indexer with a page you want added to the network, it scans your page for outgoing links and sends an email to the owners of any domains who are also on the service. One can imagine making it a little more featureful, for instance indexing OPML blogrolls which would allow you to see whom a person follows and https://microformats.org/wiki/h-entry https://microformats.org/wiki/h-entry which would allow outgoing links to be categorised (like, reply, etc) so notifications filtered. With that, it would be possible to add a front-end site imitating one of the popular social media paradigms (Reddit-style or Twitter-style being the most obvious). If there's one genuine design decision I would give for it beyond what is basically a cobbling together of existing interfaces, it would be to charge users per page uploaded. Likely very detrimental to growing the service, but I think one of the simplest ways to weed out spam and junk. A real problem with many web services is that the receiver of the message pays for it in terms of attention, where in other mediums the sender has to pay. Given that uploading crap is basically free, that's all you get. Increasing the cost of upload would weed out those endless AI summaries and lists of affiliate links. The issue with ATProto is that it operates on too many layers. You can see lacking in what I have described here the concept of durable authorship, but this is a property of content not how the content is distributed. Perhaps someone will invent a standard way to sign HTML documents, in which case you could base user accounts on that instead of DNS. AT enforces this centrally but it does not need to. It is walling itself off from the common and decentralised software ecosystem of the web for no good reason.

pfraze 1mo ago
Yeah that'd be interesting to try. If you extend the endpoints and the vocabularies around RSS/OPML enough, you could likely create a fairly robust dataset replication protocol. Microformats are a decent schema basis, but you might want to build upon them as well. If you want to make the users' sites reusable as a datastore across applications, you could introduce an oauth flow to enable 3rd party writes. You might also want to add content signatures so you can verify the data authenticity from third parties (e.g. to support the equivalent of reposts).
- James_K 1mo ago
  You would repost a page by linking to it from your own. As for uploading content, that is outside of the scope of an indexer. The user hosts their data with some hosting service, or independently. I believe signatures are already handled by XHTML which allows you to add XML signatures to documents, but regular HTML is sadly lacking here thought it would not be too hard to extend (or to just use the XML serialisation of HTML5). I think it is generally bad to implement new features like this on the part of the indexer. It should just keep track of an existing web of documents rather than creating it's own format and walled garden.
  - pfraze 1mo ago
    > You would repost a page by linking to it from your own. Ah right, of course. Fair enough. I think Dave Winer is trying some similar ideas, though I haven't looked at it beyond knowing he's trying to extend RSS to support these capabilities.
    - James_K 1mo ago
      We already have those capabilities. My website has reposts on it. I just link to someone else's content from my Atom feed instead of something hosted on my own site. Then I attach a https://validator.w3.org/feed/docs/atom.html#category https://validator.w3.org/feed/docs/atom.html#category node indicating that those are reposts.
inigyou 1mo ago
Isn't that how it already works but with different protocols that you don't like? You (a PDS) send a request to an indexer (relay) with a page (activity) you want added to the network.
- James_K 1mo ago
  The difference is one uses W3C recommended web interfaces and the other does not. I make blog posts to my website using HTML, anyone can open them using a web browser. I can host those pages one of many static hosting services with no need for a special protocol or anything like that. AT on the other hand reïnvents this entire thing. Can you type the AT URI of a resource into the browser and have it open? I do not believe you can. So if I want to publish HTML content I now need both my regular web server and an AT PDS running on my server. There exists, as far as I'm aware, no application that can view HTML hosted on the AT network so I have to produce all of my blog posts in an agnostic format or publish them to no one, or I can use one of the services that automatically posts a link to that HTML as a BSky post. So you have all of that architecture and what it accomplishes is posting a link to content hosted elsewhere. Why, instead of creating an indexer that works with existing formats, did they choose to create one intentionally incompatible with 99.999% of web content? Because in practice it's not a web protocol, it's not a serious thing. It's just a whole bunch of bullshit for nerds to make blog posts about that serves as the back end of a Twitter clone. It is standards proliferation. It adds nothing and fragments existing efforts. Instead of contributing to the web ecosystem, it subtracts from it. It breaks the important rules of software development, to do one thing and do it well, to use existing solutions. Everything it does is duplicated, there's a new way to sign documents, a new way to host content, a new data format, a new exchange protocol, a new way to serve documents, a new way to send notifications, etc. It's not a tenable way of doing things. If every person looking to make a web indexer created an entire separate web ecosystem, that would not be sustainable. They're clearly not intending to replace the Web, so what are they doing? It's just a complicated way to store tweets. And the silliest part is that, if I recall correctly, they don't even have decentralised network yet since only they can issue PLC DIDs.
  - inigyou 1mo ago