4 ms·

Conceptnet looks really cool, I'm going to dig into that later tonight. Dat's really similar to BitTorrent when it comes to availability; it doesn't do anythin

by pfraze 9y ago

Conceptnet looks really cool, I'm going to dig into that later tonight. Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon. Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and I can help you get started.

rspeer 9y ago
Should I just try using the tools at https://datproject.org/ https://datproject.org/ ? It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-sharing the files to myself.) And I guess a Dat URL shouldn't really be an authoritative entry point referred to in a paper, as it would just point to one version of the data with no context. Maybe Dat plus Zenodo could do the trick eventually.
- filiwickers 9y ago
  We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. I'd recommend checking back then (though some, such as CLI, are released already on npm). Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat compared to http urls. > Maybe Dat plus Zenodo could do the trick eventually. Yes, exactly!! Dat supports http publishing right now, so you can run `dat sync --http` on a server and it'll live publish files from the source to an http site. We are working on also supporting http downloads. The idea is you publish to Zenodo, including the SLEEP metadata, dat can then clone the files over http (including content verification) or via the peer network, i.e. `dat clone zenodo.org/record/439922`. We are super excited for the http downloading because it'll allow dat to store files on any data repository with a good api and a http file backend. We've been talking with the Dataverse folks on how to accomplish this there and have an eye on others such as Zenodo.
  - lgierth 9y ago
    > We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. Awesome, congratulations! :):) For distributed/p2p software these are just amazing times to be alive :)

by pfraze 9y ago

rspeer 9y ago

Should I just try using the tools at https://datproject.org/ https://datproject.org/ ? It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-sharing the files to myself.) And I guess a Dat URL shouldn't really be an authoritative entry point referred to in a paper, as it would just point to one version of the data with no context. Maybe Dat plus Zenodo could do the trick eventually.

filiwickers 9y ago
We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. I'd recommend checking back then (though some, such as CLI, are released already on npm). Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat compared to http urls. > Maybe Dat plus Zenodo could do the trick eventually. Yes, exactly!! Dat supports http publishing right now, so you can run `dat sync --http` on a server and it'll live publish files from the source to an http site. We are working on also supporting http downloads. The idea is you publish to Zenodo, including the SLEEP metadata, dat can then clone the files over http (including content verification) or via the peer network, i.e. `dat clone zenodo.org/record/439922`. We are super excited for the http downloading because it'll allow dat to store files on any data repository with a good api and a http file backend. We've been talking with the Dataverse folks on how to accomplish this there and have an eye on others such as Zenodo.
- lgierth 9y ago
  > We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. Awesome, congratulations! :):) For distributed/p2p software these are just amazing times to be alive :)