15 ms·
Dat: non-profit, secure, and distributed package manager for data
- deleted 9y ago[deleted]
- colordrops 9y agoDat being short for data, would a package for the state assemblies be "Dat Ass"?
- squaredpants 9y agoAstute.
- artorias 9y agobest HN comment of the year
- noobermin 9y agoI recall at least two other "data package managers" out there being shared on HN a couple of months back.
- FabioFleitas 9y agoYou happen to remember which ones? Would be interested in checking them out.
- filiwickers 9y agoThe other one I know of is datahub [1], which used to be frictionless data. It is from the same group that built CKAN. [1] https://datahub.io/ https://datahub.io/
- noobermin 9y agoI was thinking it would take a while to look back, but the new "upvoted submissions" tool is more useful than I thought. The project was Quilt[0] and here[1] is the comments on it. The other project I saw referenced in the comments was Pachyderm[2]. It looks like Dat indeed was referenced in the comments, and the main difference is Dat is distributed while Quilt is centralized (like github)...and Pachyderm is more like git. [0] https://quiltdata.com/ https://quiltdata.com/ [1] https://news.ycombinator.com/item?id=14771406 https://news.ycombinator.com/item?id=14771406 [2] http://www.pachyderm.io/ http://www.pachyderm.io/ EDIT: Annnd...looking up, I see others have already referenced quilt in other comments here.
- pfraze 9y agoDat is both a protocol and a toolset. The protocol is basically an improved version of BitTorrent that supports changes and versioning (here's a short page on its technical merits [1]). The toolset is a commandline and some desktop apps, and we use it in the Beaker Browser. Dat couldn't be made by nicer people [2]. It's a non-profit led by Max Ogden [3] and the protocol work is led by Mafintosh [4], both of whom are pretty well known in the nodejs community. The project started as a way for academics to work on tabular datasets, but they pretty quickly found out that academics more often want to work on unstructured (or custom-structured) files. So Max recruited Mafintosh and they started working on the p2p protocol, which focused on improving archival and data-sharing flows within labs. There's a pretty simple CLI you can install from NPM (npm i -g dat). Give it a try. Also, they need help securing grants, so if you have a talent or a connection in that area, definitely get in touch with them. They're doing good work. 1. https://beakerbrowser.com/docs/inside-beaker/ https://beakerbrowser.com/docs/inside-beaker/ 2. https://datproject.org/team https://datproject.org/team 3. https://twitter.com/denormalize https://twitter.com/denormalize 4. https://twitter.com/mafintosh https://twitter.com/mafintosh
- SOLAR_FIELDS 9y agoI’ve contributed to a couple of Mafintosh’s projects. the dude is impressively prolific and writes good code. Nice to see him get some recognition here.
- zitterbewegung 9y agoSo how does this compare with quilt? From what I see 1. Quilt is for profit while Dat is non-profit. 2. Dat has ~20 datasets that are public Quilt has 50+ that are public 3. Dat is on a shared network while Quilt is hosted on a centralized server 4. Both of them offer version control and hosting. Quilt has private hosting for a fee. Dat seems to have only public hosting 5.Quilt is funded by YC. Dat is funded by non profits. 6. Quilt has a Python interface while Dat has one in Javascript I understand who Quilt is targeting but I'm having trouble understanding who Dat is targeting?
- tbv 9y agoI'm one of the creators of the Beaker browser[1] and the reason we use Dat is that as a p2p protocol, it offers a lot of neat properties, including making datasets more resilient. As long as one peer on the network is hosting a dataset, it will be reachable, even if the original author has stopped hosting it. I won't speak authoritatively on behalf of the Dat team, but I believe one of their goals is to make it difficult for public scientific datasets to be lost, and data living on a centralized server is particularly vulnerable to that. 1. https://github.com/beakerbrowser/beaker https://github.com/beakerbrowser/beaker
- kevinSuttle 9y agoWhy dat or quilt and not blockchain?
- tbv 9y agoA blockchain is an over-engineered solution to the problems we’re trying to solve. Blockchains provide shared global state. We don’t need that. https://beakerbrowser.com/docs/inside-beaker/other-technologies.html#why-not-a-blockchain https://beakerbrowser.com/docs/inside-beaker/other-technolog...
- jhoechtl 9y agoA blockchain is a rather weak database in itself. However to store pointers into a dht like Dat would be fine
- filiwickers 9y agoHi everyone, I'm one of the core contributors to Dat, @joeahand. Happy to answer any questions. It's an interesting time to see this posted because I've been working on a new datproject.org site recently =). Dat Project started with a focus on increasing access to research & public data. To support the data tools we built a peer-to-peer protocol. People are doing some really cool stuff on top of Dat (such as Beaker Browser), we're really excited about it and want to make sure to support all the neat use cases. We'll be launching an updated site soon to highlight more of the work around the protocol and what the community is building. Our main use case will still be data management but most of what you see on the current site will shift to a new domain.
- batbomb 9y agoHow are you going to beat Globus for research data of appreciable size?
- filiwickers 9y agoWe'd love to work with them! We really like what they've done and we have a few partners that use Globus. Underneath all the transfers in Globus use GridFTP. Using Dat could help distribute bandwidth and speed up transfers. It'll also add version control for free, which (I think) Globus does not have yet.
- rakoo 9y agoI've glanced at the docs before already, and dat seemed super cool except for one little thing: it seems that there can only be one source that is allowed to modify the dataset. Is this by design or am I missing something ?
- tbv 9y agoThat’s correct. But the protocol author mafintosh is currently working on a major new release which introduces multi-writer capabilities.
- baldfat 9y ago
- noobermin 9y agoI can think of one use for this. I run computational plasma physics simulations for a living, and while some of us have gotten better about sharing code, sharing simulation results for published papers would be beneficial. Will have to think about this.
- pfraze 9y agoYou should jump on #dat or get in touch with someone on the team via twitter. They'd be happy to help you get setup.
- sandGorgon 9y agoThis is very cool, but I think you need to go down to fine-grained permissions to be truly effective. Otherwise , it is pretty much bittorrrent with a private tracker. For example, I should be able to give unique URLs (for the same data) to different users and expire one but continue for the other,etc.
- watson 9y agoEverything you put into dat is encrypted and unique key pairs are used for each shared item. Only the people you share the public key with can access the data. That doesn't address the expiry use-case, but it allows you to completely control who have access to what. Does that address your fine-grained permission needs? Edit: The keys are very short (64 bytes), so they can easily be copy/pasted, tweeted and what have you :)
- sandGorgon 9y agoI was not able to express properly - sorry about that. You generate one key pair for one shared item. I'm talking about multiple key pairs for each shared item, so that I can give access to individual users for the same data and revoke them when necessary. Fundamentally if you don't have that, then your case is trivially solved by a private bittorrrent tracker. This is the fundamental difference between things like Quilt and bittorrrent.
- watson 9y agoOne other crucial difference between dat and bittorrent is that dat allows you to update datasets. In bittorrent, you can't change/add files once you've shared your torrent
- sandGorgon 9y agothe DHT mutable data BEP-46 is proposing to take care of that. http://www.libtorrent.org/dht_store.html http://www.libtorrent.org/dht_store.html previous HN discussion - https://news.ycombinator.com/item?id=12257065 https://news.ycombinator.com/item?id=12257065
- erikb 9y agonaive me's first thought would be: Why not use git? Can someone with more experience in this area explain?
- jhoechtl 9y agoThe git toolset is, out of the box, weak on tabular data. Also out of the box large file support requires one central repository. None of that is true for Dat
- icebraining 9y agoout of the box large file support requires one central repository. How so? As far as I know, git doesn't treat large files any differently than small ones.
- watson 9y agoI'll try to answer your question, but it's been a while since I last looked at it, so I might get some/most of this wrong - so don't shoot me: As far as I know git isn't good at storing binary data. Git depends on line breaks to be able to diff and make change sets. If you store a binary file in git and make an update to it - even though that update only changed 1 byte, the entire new version of the file is stored again. Dat uses Rabin fingerprinting to intelligently slice binary files into chunks that are less likely to change. That make dat a lot more efficient at storing, versioning, and syncing videos, images, and other large binary files.
- icebraining 9y agoConceptually, git doesn't use change-sets, each commit is a snapshot of all the files in the current version. For storage and transmission efficiency, though, it can store and send them in packfiles, which use delta compression based on LibXDiff - which uses Rabin's fingerprint algorithm (as well as other algorithm by Joshua P. MacDonald) for binary files.
- rspeer 9y ago
- e12e 9y agoThe p2p and security aspects look very interesting[1] - apart from the obvious network effect/existing user base and tooling - are there any reasons to not prefer dat to bittorrent for all the things? It looks like an interesting way to store/share backups and server images - easily scaling bandwidth and availability with the nerd to spin up new instances, or shifting across data centers? Maybe also as an apt back-end similar to: https://wiki.debian.org/DebTorrent https://wiki.debian.org/DebTorrent [1] https://docs.datproject.org/security https://docs.datproject.org/security
- tbv 9y agoTo draw a quick contrast between Dat and BitTorrent, BitTorrent magnet links are static, meaning if you change the content, you get an entirely new magnet link. Dat archives (a networked directory, essentially) are mutable, so you can publish modifications at a consistent address. I’m not familiar with DebTorrent, but if you’re interested to learn more about the innards of Dat, this post by pfraze is a good place to start: https://beakerbrowser.com/2017/06/19/cryptographically-secure-change-feeds.html https://beakerbrowser.com/2017/06/19/cryptographically-secur...
- spraak 9y agoThere was a startup I came across recently doing about the same thing... I don't recall the name.
- Fnoord 9y agoDo you mean Quilt Data, Inc with the product Quilt? Its being mentioned all over this thread.
- rambojazz 9y agoIt looks cool, but I don't completely understand what this is about. Is it a way to share files over a P2P network? Isn't this basically the same as Torrent or IPFS?
- watson 9y agoIt's very similar to bittorrent but with a few key differences. For one, bittorrent doesn't allow you to update/add files in a dataset once you've shared the torrent. Dat does. Dat also has versioning built in. IPFS seems to me to be a bit over-engineered whereas dat is a lot more simple/low level - something that suits my way of working really well.
- rambojazz 9y ago> For one, bittorrent doesn't allow you to update/add files in a dataset once you've shared the torrent. Dat does. Dat also has versioning built in. Could this be used as a "distributed archive" to store any kind of information? Or does Dat only store some type of data? For example, could I have a "Dat dataset" containing a local Git repo, and every time I update my repo the new files are distributed to whoever is "watching" my dataset?
- filiwickers 9y agoDat allows you to store any type of data. Yes, I think you could do what you're suggesting (though I don't follow 100%). You can use it like you would Dropbox, except only one person can write at the moment. Under the hood, there are two implementations: hyperdrive and hypercore. Hyperdrive is a filesystem for storing any kinds of files. Hypercore can store any kind of data and is really great for streaming data (hyperdrive is built on top of it).
- rambojazz 9y ago> Yes, I think you could do what you're suggesting (though I don't follow 100%). I was thinking of sharing a Git directory, directly from my computer instead of using any centralized provider such as GitHub/NotABug. > Hypercore can store any kind of data and is really great for streaming data is there any web "bridge" available. I'm thinking in particular about streaming data over the P2P network, but also accessible on the web (for example for streaming a video).
- subcosmos 9y agoCan the fetch command be "get-dat"?... Please?!
- Fnoord 9y agoYou could easily make an alias.
- baldfat 9y agoThis seems like my setup but I use Resilio (Bittorrent Sync) with encryption. I don't need version control for my data but you get archives if you want one.
- megamindbrian 9y agohttps://blog.quiltdata.com/its-time-to-manage-data-like-source-code-3df04cd312b8 https://blog.quiltdata.com/its-time-to-manage-data-like-sour...