9 ms·

Dat: A P2P hypermedia protocol with public-key-addressed file archives

149 points by pcr910303 7y ago

dangoor 7y ago
Can Dat really be described as a "new" protocol 3 years in?
- dang 7y ago
  We can take 'new' out of the title above, though I'm reminded of a second-hand clothing store in my home town which used to be called "New 2 You".
  - dangoor 7y ago
    Yeah, thanks! I'm totally on board with this link being here and would love for more people to learn about Dat.
- api 7y ago
  Yes, as it's still under heavy development and has not yet seen heavy usage. This mentality that things either flip instantly to mass adoption or are "old" needs to die. All the easy stuff that can be done in 6-12 months has already been done. Anything worth doing today is going to take a minimum of 1-2 years of R&D unless it's nothing more than a packaging and polish/branding of something already in existence.
  - dangoor 7y ago
    > This mentality that things either flip instantly to mass adoption or are "old" needs to die. That's honestly not where I'm coming from, though. I think Dat is a cool project and I've been aware of it from, likely, close to the beginning. The scale between "new" and "old" is not binary, nor is it necessarily related to "mass adoption". There are people who still treat React as though it's "the new hotness", despite the fact that it's been out for 6 years and is widely adopted. I'll grant that different people will have different thresholds for what they see as "new". I just personally think Dat has gone beyond "new" and is in a phase of maturation. It's even got a browser with deep support!
rzzzt 7y ago
"How Dat Works" linked from the page is still a great visualization of all the bits that make up the protocol: - https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/ - https://news.ycombinator.com/item?id=20363813 https://news.ycombinator.com/item?id=20363813
- dang 7y ago
  Also https://news.ycombinator.com/item?id=20811936 https://news.ycombinator.com/item?id=20811936, a related project from 2019.
JakeAl 7y ago
Kind of stagnant, but I liked the IDEA of what was being done with the Beaker Browser using the DAT protocol to run a self-hosted P2P web. I just wish someone would take the project to completion/usability. It would really disrupt things. https://beakerbrowser.com https://beakerbrowser.com
- karissa 7y ago
  They are working on a new version, it's almost done. It's going to be a lot better! https://github.com/beakerbrowser/beaker/pull/1435 https://github.com/beakerbrowser/beaker/pull/1435
- Taek 7y ago
  Skynet is a new protocol that has been working on something like this. You can see some of the webapps built on Skynet here: https://skynethub.io/_B19P18DEI4Y_37wg6yXtfulEq2U8AcIO_lWM7sKncC1rw/index.html#/apps/all https://skynethub.io/_B19P18DEI4Y_37wg6yXtfulEq2U8AcIO_lWM7s... Main project website: https://siasky.net https://siasky.net
  - karissa 7y ago
    I don't see a github repository or any link to the open source code there. Is it open source?
    - soupbowl 7y ago
      I found this on their website: https://github.com/NebulousLabs/skynet-webportal https://github.com/NebulousLabs/skynet-webportal
  - koeng 7y ago
    Is there anyway to pin skynet files using storage rather than Sia, or a way to download skynet files using the command line (independent of a curl request from a portal)?
    - Taek 7y ago
karissa 7y ago
Here are all the projects that are using components from the dat team over the years: https://dat.foundation/explore/projects/ https://dat.foundation/explore/projects/ and https://cobox.cloud/ https://cobox.cloud/
rasengan0 7y ago
i swung and missed :-( https://github.com/datproject/dat/issues/1008 https://github.com/datproject/dat/issues/1008
rapnie 7y ago
Nice. Aral Balkan of small-tech.org is having plans with Dat for Tincan and Site.js: https://small-tech.org/research-and-development/ https://small-tech.org/research-and-development/
Already__Taken 7y ago
Wouldn't a protocol like this be a killer feature in the next package manager to solve problems npm had? Then you've just got a discoverability issue without the whole infrastructure headache on top to maintain.
duskwuff 7y ago
Conceptually, this sounds very similar to Freenet; a DAT URL seems analogous to a Freenet SSK. What I don't see is anything analogous to a USK -- there's no obvious way for an author to distribute an update to content they have published. It's also unclear how much (if any!) privacy this protocol provides to content publishers or requestors -- the use of discovery keys only provides protection for users requesting content which an eavesdropper has no knowledge of.
- rakoo 7y ago
  There is no equivalent to an USK because dat doesn't give direct access to old revisions. What you get is always the latest. The old content is still there though, and you can access it, just not in an "easy" manner: https://docs.dat.foundation/docs/faq#does-dat-store-version-history https://docs.dat.foundation/docs/faq#does-dat-store-version-.... The dat url references a given private key, and that's about it in terms of privacy. Transfer is done between two endpoints on the normal internet, so neither peer is "hidden". It's a shame but the dat content is spread in many places and it's hard to get access to all the documentations. The most impressive and interesting part of the project right now is probably Beaker, you should have a look: https://beakerbrowser.com/ https://beakerbrowser.com/
  - duskwuff 7y ago
    > There is no equivalent to an USK because dat doesn't give direct access to old revisions. What you get is always the latest. How do you guarantee that you're getting the latest content, though? If a peer has a cached copy of an older revision, wouldn't you end up with that instead, since there's no way to distinguish between them? > The dat url references a given private key, and that's about it in terms of privacy. Transfer is done between two endpoints on the normal internet, so neither peer is "hidden". What concerns me here is that, from what I'm reading, it seems like any client on the local network could eavesdrop on mDNS requests to determine what content other clients are viewing. Worse, a client could announce itself with the discovery key for a well-known piece of content to be notified when any other client, anywhere, requests that content. This is a worse privacy model than unencrypted HTTP. Are you aware of any plans to mitigate this?
StavrosK 7y ago
I have failed time and again to grasp how exactly Dat works. IPFS is easy for me to grok; your content is hash-addressable, thus immutable. Can someone explain in a sentence or two how Dat handles content?
- mattlondon 7y ago
  I think the difference here is that the URL in dat is not a guarantee of immutability - it is just a public key and the author of the dat is free to update the content at any time without changing the key (there is a change history though). I am not an expert on this - just a casual passing interest so might have it wrong. This explains more: https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/
  - StavrosK 7y ago
    Oh hmm, thanks. I've seen that link, but I'm not interested in getting into the byte-level, I just want a high-level overview. A URL being a public key that you can update if you have the private key makes sense, thank you. Do you know how propagation/distribution happens? With IPFS, whoever accesses your content serves it for some time.
    - mattlondon 7y ago
      I believe the distribution is largely the same concept, i.e. you get a copy of the data from somewhere, then other nodes that are also looking for the same data might discover that you have it as well and so you might then serve it on to those other nodes too.
- rakoo 7y ago
  Bittorrent is immutable, Dat is the mutable version of Bittorrent. You create an archive, that is identified with a cryptographic pubkey. You add files in it, dat stores some metadata in it. You give the archive's id to a friend, who starts retrieving the metadata, and can then see there are files; he can download files as he pleases. Syncing can also be realtime, so he gets the new content as soon as you put it. Only you, the holder of the cryptographic privkey, can add content to the archive. Crypto is used to sign all content, so there's no doubt it was legitimately written by you. Since the id doesn't move, it is possible for multiple peers to inter-connect and exchange data as needed in a swarm fashion, even if you're offline
mattlondon 7y ago
When I last looked at dat some time ago I seem to recall that there was no way to revoke a dat or set a TTL-type field. Can dat forget? E.g. If I created a dat, that was it and the data in the dat was then potentially out there forever and ever and ever in the distributed network. There was no way to tell a client "this dat is only good for 90 seconds/minutes/hours/days/years/decades/etc - after that please drop/delete". I know there is this desire in certain circles for Blockchain-stylee "everything is stored for ever and we can cryptographically prove every single byte all the way back to the dinosaurs!" sentiment, but I am not sure that really jives with a distributed data/Web publishing protocol (at least in my mind) - I want to be able to reliably "delete" something. If I know that anything I ever publish will be irrevocably around forever, it has a chilling effect on what I choose to do and publish with dat (plus maybe legal challenges? IANAL but e.g. GDPR? Right to be forgotten?) E.g would we actually gain anything if hypothetically Google supported dat and so then every single Google search result page ever generated was stored forever in a dat? Would future users benefit from storing decades of archived versions of Google search result pages for "Facebook" (because people search for "Facebook" then click the first result instead of just typing "facebook.com") or "weather" which then need to be endlessly duplicated around the network for the rest of eternity? Seems unlikely to be of any benefit to me - surely better to mark some of it as ephemeral and let the data naturally timeout and die? Does anyone know if that has changed with dat and data can now die? Or have I just misunderstood?
- tangent128 7y ago
  Data in a dat, just like data in a torrent, only lives as long as there are peers interested in seeding that particular dat; it's not a blockchain-style "everybody has to replicate everything". You can't unilaterally withdraw content somebody else is also seeding; the best you can do to my knowledge is publish a new version that replaces all the dat's content with a "please delete your history of this archive" message, but as with any distributed system you have no real way of knowing if that was respected.
aabbcc1241 7y ago
DAT is not new, I've been building stuff on top of it. Good to see more people are awared of it.

149 points by pcr910303 7y ago

dangoor 7y ago

Can Dat really be described as a "new" protocol 3 years in?

dang 7y ago
We can take 'new' out of the title above, though I'm reminded of a second-hand clothing store in my home town which used to be called "New 2 You".
- dangoor 7y ago
  Yeah, thanks! I'm totally on board with this link being here and would love for more people to learn about Dat.
api 7y ago
Yes, as it's still under heavy development and has not yet seen heavy usage. This mentality that things either flip instantly to mass adoption or are "old" needs to die. All the easy stuff that can be done in 6-12 months has already been done. Anything worth doing today is going to take a minimum of 1-2 years of R&D unless it's nothing more than a packaging and polish/branding of something already in existence.
- dangoor 7y ago
  > This mentality that things either flip instantly to mass adoption or are "old" needs to die. That's honestly not where I'm coming from, though. I think Dat is a cool project and I've been aware of it from, likely, close to the beginning. The scale between "new" and "old" is not binary, nor is it necessarily related to "mass adoption". There are people who still treat React as though it's "the new hotness", despite the fact that it's been out for 6 years and is widely adopted. I'll grant that different people will have different thresholds for what they see as "new". I just personally think Dat has gone beyond "new" and is in a phase of maturation. It's even got a browser with deep support!

rzzzt 7y ago

"How Dat Works" linked from the page is still a great visualization of all the bits that make up the protocol: - https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/ - https://news.ycombinator.com/item?id=20363813 https://news.ycombinator.com/item?id=20363813

dang 7y ago
Also https://news.ycombinator.com/item?id=20811936 https://news.ycombinator.com/item?id=20811936, a related project from 2019.

JakeAl 7y ago

Kind of stagnant, but I liked the IDEA of what was being done with the Beaker Browser using the DAT protocol to run a self-hosted P2P web. I just wish someone would take the project to completion/usability. It would really disrupt things. https://beakerbrowser.com https://beakerbrowser.com

karissa 7y ago
They are working on a new version, it's almost done. It's going to be a lot better! https://github.com/beakerbrowser/beaker/pull/1435 https://github.com/beakerbrowser/beaker/pull/1435
Taek 7y ago
Skynet is a new protocol that has been working on something like this. You can see some of the webapps built on Skynet here: https://skynethub.io/_B19P18DEI4Y_37wg6yXtfulEq2U8AcIO_lWM7sKncC1rw/index.html#/apps/all https://skynethub.io/_B19P18DEI4Y_37wg6yXtfulEq2U8AcIO_lWM7s... Main project website: https://siasky.net https://siasky.net
- karissa 7y ago
  I don't see a github repository or any link to the open source code there. Is it open source?
  - soupbowl 7y ago
    I found this on their website: https://github.com/NebulousLabs/skynet-webportal https://github.com/NebulousLabs/skynet-webportal
- koeng 7y ago
  Is there anyway to pin skynet files using storage rather than Sia, or a way to download skynet files using the command line (independent of a curl request from a portal)?
  - Taek 7y ago

karissa 7y ago

Here are all the projects that are using components from the dat team over the years: https://dat.foundation/explore/projects/ https://dat.foundation/explore/projects/ and https://cobox.cloud/ https://cobox.cloud/

rasengan0 7y ago

i swung and missed :-( https://github.com/datproject/dat/issues/1008 https://github.com/datproject/dat/issues/1008

rapnie 7y ago

Nice. Aral Balkan of small-tech.org is having plans with Dat for Tincan and Site.js: https://small-tech.org/research-and-development/ https://small-tech.org/research-and-development/

Already__Taken 7y ago

Wouldn't a protocol like this be a killer feature in the next package manager to solve problems npm had? Then you've just got a discoverability issue without the whole infrastructure headache on top to maintain.

duskwuff 7y ago

Conceptually, this sounds very similar to Freenet; a DAT URL seems analogous to a Freenet SSK. What I don't see is anything analogous to a USK -- there's no obvious way for an author to distribute an update to content they have published. It's also unclear how much (if any!) privacy this protocol provides to content publishers or requestors -- the use of discovery keys only provides protection for users requesting content which an eavesdropper has no knowledge of.

rakoo 7y ago
There is no equivalent to an USK because dat doesn't give direct access to old revisions. What you get is always the latest. The old content is still there though, and you can access it, just not in an "easy" manner: https://docs.dat.foundation/docs/faq#does-dat-store-version-history https://docs.dat.foundation/docs/faq#does-dat-store-version-.... The dat url references a given private key, and that's about it in terms of privacy. Transfer is done between two endpoints on the normal internet, so neither peer is "hidden". It's a shame but the dat content is spread in many places and it's hard to get access to all the documentations. The most impressive and interesting part of the project right now is probably Beaker, you should have a look: https://beakerbrowser.com/ https://beakerbrowser.com/
- duskwuff 7y ago
  > There is no equivalent to an USK because dat doesn't give direct access to old revisions. What you get is always the latest. How do you guarantee that you're getting the latest content, though? If a peer has a cached copy of an older revision, wouldn't you end up with that instead, since there's no way to distinguish between them? > The dat url references a given private key, and that's about it in terms of privacy. Transfer is done between two endpoints on the normal internet, so neither peer is "hidden". What concerns me here is that, from what I'm reading, it seems like any client on the local network could eavesdrop on mDNS requests to determine what content other clients are viewing. Worse, a client could announce itself with the discovery key for a well-known piece of content to be notified when any other client, anywhere, requests that content. This is a worse privacy model than unencrypted HTTP. Are you aware of any plans to mitigate this?

StavrosK 7y ago

I have failed time and again to grasp how exactly Dat works. IPFS is easy for me to grok; your content is hash-addressable, thus immutable. Can someone explain in a sentence or two how Dat handles content?

mattlondon 7y ago
I think the difference here is that the URL in dat is not a guarantee of immutability - it is just a public key and the author of the dat is free to update the content at any time without changing the key (there is a change history though). I am not an expert on this - just a casual passing interest so might have it wrong. This explains more: https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/
- StavrosK 7y ago
  Oh hmm, thanks. I've seen that link, but I'm not interested in getting into the byte-level, I just want a high-level overview. A URL being a public key that you can update if you have the private key makes sense, thank you. Do you know how propagation/distribution happens? With IPFS, whoever accesses your content serves it for some time.
  - mattlondon 7y ago
    I believe the distribution is largely the same concept, i.e. you get a copy of the data from somewhere, then other nodes that are also looking for the same data might discover that you have it as well and so you might then serve it on to those other nodes too.
rakoo 7y ago
Bittorrent is immutable, Dat is the mutable version of Bittorrent. You create an archive, that is identified with a cryptographic pubkey. You add files in it, dat stores some metadata in it. You give the archive's id to a friend, who starts retrieving the metadata, and can then see there are files; he can download files as he pleases. Syncing can also be realtime, so he gets the new content as soon as you put it. Only you, the holder of the cryptographic privkey, can add content to the archive. Crypto is used to sign all content, so there's no doubt it was legitimately written by you. Since the id doesn't move, it is possible for multiple peers to inter-connect and exchange data as needed in a swarm fashion, even if you're offline

mattlondon 7y ago

When I last looked at dat some time ago I seem to recall that there was no way to revoke a dat or set a TTL-type field. Can dat forget? E.g. If I created a dat, that was it and the data in the dat was then potentially out there forever and ever and ever in the distributed network. There was no way to tell a client "this dat is only good for 90 seconds/minutes/hours/days/years/decades/etc - after that please drop/delete". I know there is this desire in certain circles for Blockchain-stylee "everything is stored for ever and we can cryptographically prove every single byte all the way back to the dinosaurs!" sentiment, but I am not sure that really jives with a distributed data/Web publishing protocol (at least in my mind) - I want to be able to reliably "delete" something. If I know that anything I ever publish will be irrevocably around forever, it has a chilling effect on what I choose to do and publish with dat (plus maybe legal challenges? IANAL but e.g. GDPR? Right to be forgotten?) E.g would we actually gain anything if hypothetically Google supported dat and so then every single Google search result page ever generated was stored forever in a dat? Would future users benefit from storing decades of archived versions of Google search result pages for "Facebook" (because people search for "Facebook" then click the first result instead of just typing "facebook.com") or "weather" which then need to be endlessly duplicated around the network for the rest of eternity? Seems unlikely to be of any benefit to me - surely better to mark some of it as ephemeral and let the data naturally timeout and die? Does anyone know if that has changed with dat and data can now die? Or have I just misunderstood?

tangent128 7y ago
Data in a dat, just like data in a torrent, only lives as long as there are peers interested in seeding that particular dat; it's not a blockchain-style "everybody has to replicate everything". You can't unilaterally withdraw content somebody else is also seeding; the best you can do to my knowledge is publish a new version that replaces all the dat's content with a "please delete your history of this archive" message, but as with any distributed system you have no real way of knowing if that was respected.

aabbcc1241 7y ago

DAT is not new, I've been building stuff on top of it. Good to see more people are awared of it.