4 ms·

How would you practically implement robust backlinks in a distributed hypertext system to begin with?

DaiPlusPlus 6y ago
Technically they exist in HTML+HTTP via the (mispelled) `Referer` header. It sucks that cross-site Referer sharing is seemingly being phased out by browsers in the name of user-privacy (as referer URIs do often contain PII), but there are more cynical explanations afoot which I won’t go into. Another approach is having a central link-repository that works as an overlay on the fully distributed web - this could be provided by search-engines. Another approach is a distributed link database. I’m sure some clever DHT system could be used to allow practically anyone to participate without needing nodes to hold a copy of the full database (a-la Bitcoin), or even one that allows directly-relevant nodes (e.g. a website’s own node) to stores only it’s own bi-di links and not waste resources on unaffiliated content. Of course, all of this assumes that spam, trolling, griefing, etc don’t exist :) For all the talk WWW gets about being a distributed system... it isn’t: it’s more of a federation of systems united only by their common use of HTML+HTTP and (hopefully) RESTful resource architecture. A true distributed WWW would be some horrible mix of IPFS, BitTorrent and Ethereum. I think that’s how Roko's Basilisk will come into existence...
- kmeisthax 6y ago
  Okay, "decentralized hypertext", not distributed. Same problem still exists: servers that run an HTTP service would need to accept and track backlink pings. This was done a decade and change ago with Movable Type's trackbacks feature and that was, as you guessed, a massive spam vector. You can, of course, construct a backlink system by indexing pages. That's basically how Google works and is probably the only use case I can think of where backlinks help more than they hurt. In fact, your distributed link database is basically half of what you'd need to build a search engine. The other half being, of course, an oppressive antispam system. The only reason Google works at all is because they frequently adjust their ranking algorithms or shove them into piles of linear algebra nobody can comprehend. No clue how you do that in a distributed manner, though.
  - zozbot234 6y ago
    > This was done a decade and change ago with Movable Type's trackbacks feature and that was, as you guessed, a massive spam vector. This is not an insurmountable problem. You can choose to accept trackbacks only from sites that can point to endorsement by some mutually trusted 3rd party. This is ultimately how the Fediverse works - bad actors will get booted from the most popular networks and find themselves unable to 'push' content to the network. And modern federated-web standards include the "trackback" case, e.g. via WebMention.
    - DaiPlusPlus 6y ago
      > You can choose to accept trackbacks only from sites that can point to endorsement by some mutually trusted 3rd party. This is ultimately how the Fediverse works - bad actors will get booted from the most popular networks and find themselves unable to 'push' content to the network. I suppose this would have worked in the 2005-ish era when MovableType, Pingbacks/Linkbacks, and Web 2.0 was all the rage, but in today's internet culture and its very partisan politics (...of a distinctly American flavour...) permeating everywhere, the concept of "endorsement", even in a technical sense, can be framed as a political gesture, even if the basis of an endorsement can be defended on purely technical grounds. We saw this happening with the "Twitter Verified" badges controversy (as the identity-verification of an account is a purely technical and factual thing, but because Twitter chose to use the imagery-of-endorsement to indicate verified accounts it led to... well, that's another discussion). My point being that even if a pingback/linkback/trackback system can accurately filter-out (commercial) spam, maintainers of those websites would likely still want to curate pingbacks anyway - which means still having to actively moderate/audit that, and who wants to spend their time doing that?
  - DaiPlusPlus 6y ago
    > No clue how you do that in a distributed manner, though. Distributed blockchains (Bitcoin, Ethereum, etc) solved this with Proof-of-Work, which is derived from the concept of an "e-mail postage stamp" to fight spam by forcing SMTP senders to compute an expensive hash. Similarly, if each backlink "cost" a PoW unit to earn Googlejuice it would probably be a useful metric again. ...or not: because that means indirectly placing a dollar-amount on a backlink, and well-funded marketing companies are more-than-willing to shell-out for that. Just an idea though... Anyone want to start a "Xanadu 2: Electric Boogaloo" with a PoW algorithm?
- andai 6y ago
  You misspelled misspell. (Sorry.)
  - gumby 6y ago
    Ha, trying to Americanify "misspelt".
- selfhoster11 6y ago
  > but there are more cynical explanations afoot which I won’t go into. Can you expand on this?
drewcoo 6y ago
Do we have robust forward links or is this a burden only for back links?
- gumby 6y ago
  No, we get 404s from broken forward links, either because the target went away, or because of a typo in the link. I can link to any page I like without having to talk to or get permission from anyone at all. While the person who manages the target of that link can decide on their own how to respond when the link is dereferenced. Allowing this to happen, and making 404 perfectly acceptable was, in retrospect, an excellent idea.
  - 082349872349872 6y ago
    Are there any screen-sharing VNC-like things that allow jumping via links from one server to another (the way that HTML links allow jumping from one document to another)? I'm sure it'd be a security/usability nightmare, but wonder if it might be an interesting nightmare.
    - gumby 6y ago
      Given that dereferencing a URI can result in program execution rather than simple transmission of a document (or, if you prefer, you can ssh to a host and then ssh thither) I’d say yes. A host is a pretty lightweight concept these days (so lightweight and often ephemeral that we don’t use the term much and rarely even give them names). CL/DM was written in the mainframe era.
      - 082349872349872 6y ago
        Yeah, as for program execution, I was specifically thinking of VNC servers that are the equivalent of CGI in that they produce "displays" that never occurred in any framebuffer. Hyperlinks would allow networks of this kind of virtual VNC...
baxrob 6y ago
From trying to read about this years ago, I always remember the "Ent" and the "Enfilade" being Ted Nelson's abstractions for supporting bi-directional linking https://en.wikipedia.org/wiki/Enfilade_(Xanadu) https://en.wikipedia.org/wiki/Enfilade_(Xanadu)
slightwinder 6y ago
Give each document a unique identifier linked to a namespace (the domain or website-url) and let the crawlers do their work. A bit more optimizations here and there might be benefitial, like special support from browsers, a versionized linking-descriptopn of your namespace to allow local building of the real ressource-endpoint and such things.

Technically they exist in HTML+HTTP via the (mispelled) `Referer` header. It sucks that cross-site Referer sharing is seemingly being phased out by browsers in the name of user-privacy (as referer URIs do often contain PII), but there are more cynical explanations afoot which I won’t go into. Another approach is having a central link-repository that works as an overlay on the fully distributed web - this could be provided by search-engines. Another approach is a distributed link database. I’m sure some clever DHT system could be used to allow practically anyone to participate without needing nodes to hold a copy of the full database (a-la Bitcoin), or even one that allows directly-relevant nodes (e.g. a website’s own node) to stores only it’s own bi-di links and not waste resources on unaffiliated content. Of course, all of this assumes that spam, trolling, griefing, etc don’t exist :) For all the talk WWW gets about being a distributed system... it isn’t: it’s more of a federation of systems united only by their common use of HTML+HTTP and (hopefully) RESTful resource architecture. A true distributed WWW would be some horrible mix of IPFS, BitTorrent and Ethereum. I think that’s how Roko's Basilisk will come into existence...

kmeisthax 6y ago
Okay, "decentralized hypertext", not distributed. Same problem still exists: servers that run an HTTP service would need to accept and track backlink pings. This was done a decade and change ago with Movable Type's trackbacks feature and that was, as you guessed, a massive spam vector. You can, of course, construct a backlink system by indexing pages. That's basically how Google works and is probably the only use case I can think of where backlinks help more than they hurt. In fact, your distributed link database is basically half of what you'd need to build a search engine. The other half being, of course, an oppressive antispam system. The only reason Google works at all is because they frequently adjust their ranking algorithms or shove them into piles of linear algebra nobody can comprehend. No clue how you do that in a distributed manner, though.
- zozbot234 6y ago
  > This was done a decade and change ago with Movable Type's trackbacks feature and that was, as you guessed, a massive spam vector. This is not an insurmountable problem. You can choose to accept trackbacks only from sites that can point to endorsement by some mutually trusted 3rd party. This is ultimately how the Fediverse works - bad actors will get booted from the most popular networks and find themselves unable to 'push' content to the network. And modern federated-web standards include the "trackback" case, e.g. via WebMention.
  - DaiPlusPlus 6y ago
    > You can choose to accept trackbacks only from sites that can point to endorsement by some mutually trusted 3rd party. This is ultimately how the Fediverse works - bad actors will get booted from the most popular networks and find themselves unable to 'push' content to the network. I suppose this would have worked in the 2005-ish era when MovableType, Pingbacks/Linkbacks, and Web 2.0 was all the rage, but in today's internet culture and its very partisan politics (...of a distinctly American flavour...) permeating everywhere, the concept of "endorsement", even in a technical sense, can be framed as a political gesture, even if the basis of an endorsement can be defended on purely technical grounds. We saw this happening with the "Twitter Verified" badges controversy (as the identity-verification of an account is a purely technical and factual thing, but because Twitter chose to use the imagery-of-endorsement to indicate verified accounts it led to... well, that's another discussion). My point being that even if a pingback/linkback/trackback system can accurately filter-out (commercial) spam, maintainers of those websites would likely still want to curate pingbacks anyway - which means still having to actively moderate/audit that, and who wants to spend their time doing that?
- DaiPlusPlus 6y ago
  > No clue how you do that in a distributed manner, though. Distributed blockchains (Bitcoin, Ethereum, etc) solved this with Proof-of-Work, which is derived from the concept of an "e-mail postage stamp" to fight spam by forcing SMTP senders to compute an expensive hash. Similarly, if each backlink "cost" a PoW unit to earn Googlejuice it would probably be a useful metric again. ...or not: because that means indirectly placing a dollar-amount on a backlink, and well-funded marketing companies are more-than-willing to shell-out for that. Just an idea though... Anyone want to start a "Xanadu 2: Electric Boogaloo" with a PoW algorithm?
andai 6y ago
You misspelled misspell. (Sorry.)
- gumby 6y ago
  Ha, trying to Americanify "misspelt".
selfhoster11 6y ago
> but there are more cynical explanations afoot which I won’t go into. Can you expand on this?

drewcoo 6y ago

Do we have robust forward links or is this a burden only for back links?

gumby 6y ago
No, we get 404s from broken forward links, either because the target went away, or because of a typo in the link. I can link to any page I like without having to talk to or get permission from anyone at all. While the person who manages the target of that link can decide on their own how to respond when the link is dereferenced. Allowing this to happen, and making 404 perfectly acceptable was, in retrospect, an excellent idea.
- 082349872349872 6y ago
  Are there any screen-sharing VNC-like things that allow jumping via links from one server to another (the way that HTML links allow jumping from one document to another)? I'm sure it'd be a security/usability nightmare, but wonder if it might be an interesting nightmare.
  - gumby 6y ago
    Given that dereferencing a URI can result in program execution rather than simple transmission of a document (or, if you prefer, you can ssh to a host and then ssh thither) I’d say yes. A host is a pretty lightweight concept these days (so lightweight and often ephemeral that we don’t use the term much and rarely even give them names). CL/DM was written in the mainframe era.
    - 082349872349872 6y ago
      Yeah, as for program execution, I was specifically thinking of VNC servers that are the equivalent of CGI in that they produce "displays" that never occurred in any framebuffer. Hyperlinks would allow networks of this kind of virtual VNC...

baxrob 6y ago

From trying to read about this years ago, I always remember the "Ent" and the "Enfilade" being Ted Nelson's abstractions for supporting bi-directional linking https://en.wikipedia.org/wiki/Enfilade_(Xanadu) https://en.wikipedia.org/wiki/Enfilade_(Xanadu)

slightwinder 6y ago

Give each document a unique identifier linked to a namespace (the domain or website-url) and let the crawlers do their work. A bit more optimizations here and there might be benefitial, like special support from browsers, a versionized linking-descriptopn of your namespace to allow local building of the real ressource-endpoint and such things.