3 ms·
Dat – Distributed Dataset Synchronization and Versioning
- pfraze 9y agoWe use Dat in Beaker[1] to host sites and files from the user's device. It's a pretty interesting protocol. It's developed by Code for Science[2], a 501(c)(3) led by Max Ogden[3] and with protocol dev led by Mafintosh[4]; their mission is to help with archival of science and civic data. Some interesting properties: 1. It uses a BitTorrent-style of swarm, but primarily to sync signed append-only logs, which are in fact flattened Merkle Trees (similar to Certificate Transparency). The Dat archives are addressed by public keys. The tree is used to enforce the append-only constraint by making it easy to detect if the history has been changed by the author. 2. The "Secret Sharing" feature. The public key of a Dat archive is hashed before querying or announcing on the discovery network, and then the traffic is encrypted using the public key as a symmetric key. This has the effect of hiding the content from the network, and thus making the public key of a Dat a "read capability": you have to know the key to access its files. There's a reference implementation in JS available at https://github.com/datproject/dat-node https://github.com/datproject/dat-node, and a fair number of tools being built around it. 1 https://beakerbrowser.com/ https://beakerbrowser.com/ 2 https://datproject.org/ https://datproject.org/ 3 https://twitter.com/denormalize https://twitter.com/denormalize 4 https://twitter.com/mafintosh https://twitter.com/mafintosh
- nerdponx 9y agoI wish Beaker had picked a different name. It collides with the Beaker Notebook, a Jupyter alternative that unfortunately never seemed to gain traction but had some really killer features that Jupyter has yet to pick up, especially the ability to mix Python, R, and Julia cells in the same notebook.
- pfraze 9y agoYeah I regret that. We'll consider a rename at some point. Here's their project: http://beakernotebook.com/ http://beakernotebook.com/
- alexvoda 9y agoWill you keep the laboratory glassware theme? Flask is already used ( http://flask.pocoo.org/ http://flask.pocoo.org/ ) Maybe Retort? Funnel? Or maybe a proper name? Berzelius? Erlenmeyer would be difficult for people to pronounce.
- foota 9y agoIt also collides with the beaker framework
- nwmcsween 9y agoWasn't dat originally going to be a part of ipfs or was it the browser? What are the reasons for dat vs ipfs?
- hobofan 9y agoContrary to IPFS dat is more focused on what can be achieved right now while sacrificing true decentralisation, and is more opinonated with providing in its use case with providing things like versioning.
- tbv 9y agoWhat do you think it sacrifices? IPFS is more focused on static blob addressing while Dat focuses on data sources, and in that sense, there’s a single authority over a dataset. But I see that as a positive, since mutability is pretty valuable. TBH, I'm not sure how a site can work without mutability.
- GhotiFish 9y agoa dynamic site can't work without mutability, ipfs can't deal with mutability, so ipfs comes with ipns, which allows you to statically reference content that might change.
- tbv 9y agoYeah, IPFS + IPNS is effectively on par with Dat archive mutability, but mutability being built-in to Dat + its verifiable history log is particularly well-suited to building peer-to-peer websites
- vertex-four 9y agoOf course, the design of IPNS makes it impossible to prove that you've got the latest version of a name's value, and makes it relatively easy to attack. I don't know if Dat has the same issue, I haven't looked at it.
- draw_down 9y agoMax Ogden and Mafintosh are incredibly talented and productive. Awe-inspiring to see the things they make.
- fiatjaf 9y agoNice, are you sure they aren't going to change the protocol to something totally new and with radically different features in maybe two months.
- rspeer 9y agoAs someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been getting the data -- building it requires about 100 GB of raw data or 15 GB of computed data that can be imported into PostgreSQL. I once tried git-annex. It turned out not to be a good choice -- its tools were flaky, its usage patterns confusing, it leaves a permanent record of your mistakes in configuring data sources, and it was very hard to convince to use ordinary HTTP downloads instead of trying to get read-write access to S3 (which wouldn't work for anyone but me). Now I have weird branches and remotes in my repositories, and weird data in my S3 buckets, that I can't get rid of in case someone tries to use git-annex in a way I told them would work. After that I just went with distributing the data with plain HTTP downloads from S3. I wish I could do better than this. The only semblance of versioning is putting the date in the URL, and also people in Asia tell me that the build fails because their downloads from us-east-1 get interrupted. Oh, and if I ever stop paying for S3, everything will break. If I tried making data reproducible with Dat, would it be safe to promise people that they could use Dat to get the data? Even if in the future I don't like Dat anymore? For instance, do I have to commit to hosting the data somewhere? If not, who does? Does it disappear when people lose interest, like BitTorrent? [1] http://conceptnet.io http://conceptnet.io
- pfraze 9y agoConceptnet looks really cool, I'm going to dig into that later tonight. Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon. Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and I can help you get started.
- rspeer 9y agoShould I just try using the tools at https://datproject.org/ https://datproject.org/ ? It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-sharing the files to myself.) And I guess a Dat URL shouldn't really be an authoritative entry point referred to in a paper, as it would just point to one version of the data with no context. Maybe Dat plus Zenodo could do the trick eventually.
- skybrian 9y agoIt seems like Dat has some usability quirks that might take some getting used to: - You can publish new versions to a URL until you somehow forget the private key, and then it's fixed forever, so long as people hang onto copies. - There's nothing to prevent people from passing around a URL with a version in it. So, although it looks like the author has some control, this is an illusion; publishing is irrevocable and anything published could go viral. (This is generally true of making copies, but it's the opposite of Snapchat.) - Suppose someone chooses to publish a private key? Is it a world-writable URL? Hmm.
- pfraze 9y agoAll true, though if you leak the private key, what will happen is that (due to lack of strict consensus between the leaked-key users) conflicting updates will be published, causing a detectable split history. That's a corruption event. At the moment that would result in each leaked-key user maintaining a different history, with different peers only downloading the updates from the leak-author they happen to receive data from first. But in the future what will happen, once we get to writing the software for it, is the corruption event will be detected and recorded by all possible peers, freezing the dat from receiving future updates.
- maxogden 9y agoGreat analysis. We anticipate that in order to fix these three usability issues around trust we will need to provide a centralized identity provider in the future. This would also address privacy issues especially regarding leaking what dats you are accessing. The design philosophy around Dat is to start from the end of the completely decentralized spectrum but be flexible in letting the application choose the tradeoffs as they move more towards centralized components.
- skybrian 9y agoGood to know. It would be good to figure out key rotation for Dat URL providers since this probably has to be built into the protocol. Any thoughts on integrating with keybase? I like keybase's model where you have device-specific keys. But this would probably make moving a Dat URL provider to a different machine trickier. This all assumes that well-known Dat URL's become an important thing to preserve (they are published in papers, etc) even though they are very user-unfriendly, even more than IP addresses. A naming system on top of them would make key rotation a non-issue (rotate Dat URL's instead) and you could completely replace or remove the history, sort like a git rebase. But that loses other nice properties of the system? I suppose irrevocability is something we deal with in git repos all the time. Although you can do a rebase locally, once a commit is accepted by a popular project, they're unlikely to let you remove it from history. The review process makes it unlikely that any really embarrassing mistake would be accepted, so this seems ok in practice.