12 ms·
Hyperdrive v10 – a peer-to-peer filesystem
- pfraze 6y agoHyperdrive is part of the peer-to-peer networking stack that Beaker Browser uses for websites. Today is release day and this tech update is the first announcement.
- billme 6y agoWhat is your relationship with Hyperdrive & the Beaker Browser?
- errnoh 6y agoColor me interested. I've been wanting to use IPFS for similar use case but there's always something that annoys me or documentation that's lacking. While reading Hyperdrive/Hyperswarm docs I actually managed to find everything I wanted to know without much effort (most of all it seems like setting up this on private network should be doable)
- jstanley 6y agoTotal noob question, but: Last time Dat was on HN I tried to follow the "simple chat application" tutorial[0], but got stuck at the stage where 2 instances were supposed to automatically discover each other because they only intermittently managed to actually discover each other. Will this new version of Hyperdrive improve this? Or is it something completely different? [0] https://docs.dat.foundation/docs/kappa https://docs.dat.foundation/docs/kappa
- andrewosh 6y agoThe new version has a totally revamped networking stack that should be a lot faster and more reliable. Part of it includes a new UDP holepunching algorithm that's had a super high success rate for us in practice. There are some diagrams describing this on the Hypercore Protocol [0] site, and the repo's can be found here. [0] https://hypercore-protocol.org/#hyperswarm https://hypercore-protocol.org/#hyperswarm [1] https://github.com/hyperswarm/hyperswarm https://github.com/hyperswarm/hyperswarm
- mafintosh 6y agoThe new holepunching algorithm also runs distributed on the our DHT, meaning any peer can help you holepunch to other peers. We also talk a bit more about this on the hypercore website
- rakoo 6y agoCongratulations to the team ! 2 questions: - What is the difference between dat and hyperdrive ? - I see hyperdrive manages hyper:// urls -- what about dat:// urls ? Will they be still managed ?
- pfraze 6y agoThanks! - "Hyperdrive" was formerly the internal data-structure name of Dat archives. With this release, the team decided to rename the protocol from Dat to Hypercore Protocol. Subsequently, "Dat Archives" are now "Hyperdrives." The Dat community will post some updates about this soon. - The hyperdrive-daemon does not support dat:// URLs. I don't know what the future of dat:// URLs will be but Beaker is phasing them out with a converter tool.
- aspenmayer 6y agoAnother comment which is now gone asked what your relation to the Beaker project is, and how Hyperdrive is related to the Beaker browser?
- pfraze 6y agoJust replied there - Beaker uses Hyperdrive, it's basically the source of its novel features. The Beaker team works on Hyperdrive (the Hypercore Protocol) but we maintain a separate org at https://github.com/hypercore-protocol https://github.com/hypercore-protocol
- myu701 6y agoI'm not able to view the link due to HTTPS errors. The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
- pfraze 6y agoThat's odd, what browser?
- noidesto 6y agoWhat are the main differences between this and Syncthing?
- pfraze 6y agoI'm not deeply familiar with Syncthing but here are some differences based on what I do know: - Hyperdrive includes mass-publishing as a usecase, so it uses public-key URLs and a bandwidth-sharing mechanism among its active peers (like BitTorrent) - Hyperdrive is built on a general-purpose protocol called Hypercore which is a signed append-only log. These logs can be used for other datastructures. Some examples [1] [2] [1] Kappa-core, a general-purpose db built on the logs https://github.com/kappa-db/kappa-core https://github.com/kappa-db/kappa-core [2] Cabal, a chat network https://cabal.chat/ https://cabal.chat/
- choeger 6y ago> written in Node.js Yeah. No, thanks.
- dsun179 6y agoWow. What a way to describe the tradeoff of a programming language. No, thanks
- cxr 6y agoJavaScript is a programming language. NodeJS is not. It's the substrate that the NPM ecosystem runs on. choeger's comment very well may have not been made with a distinction in mind (and I agree that it was a low effort, low signal comment), but the distinction exists. choeger seems to be using Firefox. Firefox is implemented using an Emacs-like architecture. The Firefox codebase includes millions of lines of JS, and that's been the case since before the language was JITted. I ran Firefox 1.0, 2.5, and 2.0 on an 800MHz PIII with 128 MB (later 192 MB, wow-wee!) of RAM. The JS in the Firefox codebase wasn't written in the NodeJS style, and if it were, that would have made it a non-starter. Kneejerk JS haters are tiring, but so is the conflation of JS and the NodeJS ecosystem (by NodeJS supporters and NodeJS haters alike). (Side note: if you insist on giving attention to folks engaged in low-value discussions, try to elevate the conversation to a level that makes it actually worth having.)
- RussianCow 6y agoThese days, using JavaScript outside of the browser sort of implies Node.js, in the same way that using Python implies that you're using CPython, unless you specify otherwise. In theory your distinction makes sense, but in practice, very few people are using anything but Node.js for backend JavaScript systems (at least for new projects).
- cxr 6y agoOkay. Let's consider your comment within the context of the discussion taking place. What's your point?
- carapace 6y agoOh dude, I'm sorry but I read > Hyperdrive is a POSIX-compatible filesystem implementation, written in Node.js And just cracked up! Thanks, I needed that.
- andrewosh 6y agoFair enough -- updated it to POSIX-like. The main point there being that it should be as straightforward to use as Node's fs module.
- yjftsjthsd-h 6y agoI don't think that was the humor. Node is, from one point of view, a piece of a browser engine that's been extracted out. And you wrote a filesystem in it. It's like writing a webapp in x86_64 assembly - you can do it, obviously, but it can be viewed as an odd juxtaposition.
- deleted 6y ago[deleted]
- 0x006A 6y agodoes it scale now? previously i was not able to import large folders (several TB) or many files (+100k files)
- mafintosh 6y agoYep! That is one of the things we’ve worked the hardest on. Completely new indexing structure, using an append-only hash trie which scales really well. We’ve tested it with many big datasets including importing all of Wikipedia as files in a single folder. Worked like a charm :)
- 0x006A 6y agothis one? https://dumps.wikimedia.org/other/static_html_dumps/current/en/ https://dumps.wikimedia.org/other/static_html_dumps/current/... how long does it take to import it?
- mafintosh 6y agoI think it was that one yes. Can’t remember the exact time it took, as we ran it over a couple of days due to some unrelated computer issues.
- deleted 6y ago[deleted]
- alphydan 6y agoCould this be used to create a kbfs (keybase file system) competitor?
- mafintosh 6y agoYea totally. The FUSE support is already very close to being elexactly that. We’ve been using it internally during our beta as an in office P2P dropbox using the mounts feature
- rakoo 6y agoIn kbfs when I create a shared drive between you and me, the space is automatically available in read and write for both of us. Can this be done directly on top of hyperdrive or do we need to modify how hyperdrive (or the daemon I suppose) work ?
- mafintosh 6y agoWe can mount each others drives and collab that way. For a full “union” mount experience we still have some research to do but we are working on it. The mount setup is really good though and fully p2p
- jhoechtl 6y agoCan it be that you moved from the early days of dat which was written in Go afaik to primarily node/js? Why the move?
- pfraze 6y agoI think dat has always been node/js
- tangent128 6y agoThey may be thinking of IPFS, which is written in Go.
- Mizza 6y agoBeen following this for years, congratulations on your release. I looked at dat for building P2P services, but found that ZeroNet was actually more capable for "real" services - maybe now is the time to reevaluate that, especially given the improvements to large (both deep and wide) archives and the new hole punching. Can you please mention your thoughts on: - Discoverability of content in Hyperswarm (DHT Search/"Superpeers"/???) - What happens to the old DEP proposals? (There is a critical feature that I need for my service that's still an open DEP proposal! https://github.com/datprotocol/DEPs/issues/61 https://github.com/datprotocol/DEPs/issues/61) - What uses cases do you have in mind for this service? Thanks!
- andrewosh 6y agoThanks Mizza! Not sure if I'm answering your first point, but Hyperswarm is baked into the Hyperdrive daemon, so daemon users should have their drives swarmed/available automatically. There are a few CLI commands to toggle this behavior too, in case you don't want to add your drive's key to the DHT. The Hypercore Protocol org's creating a similar proposal repo called HYP [0] (we couldn't resist the name), scoped tightly to the core protocol. We're still solidifying the proposals plan, but yours would add lots of value, so we don't want to lose track of it. Since your proposal is about peer identifiers, you might like that Hyperswarm uses the Noise protocol [0] to handshake each connection, and the Noise key can be used as a stable peer ID. As for use-cases, check out Beaker (launched today too). Paul's made a whole bunch of example applications that take advantage of Hyperdrive features, like drive mounts. I'm personally really interested in using Beaker to make personal document indexers + search (kinda like an amped-up Dropbox), but that's for another post! [0] https://noiseprotocol.org/ https://noiseprotocol.org/
- andrewosh 6y agoIn the flurry of the launch I got all my links wrong. The HYP repo can be found here: https://github.com/hypercore-protocol/hyp https://github.com/hypercore-protocol/hyp
- telamohn 6y agoCheck out the dependency hyper-protocol, specifically the extension messaging, custom-messaging has been around for a while allowing you to define your own protocol-extensions and peer behaviour. hyper-protocol-v7 came with an option for doing authenticated peer handshakes as well.
- 0x006A 6y agowhat version of node.js does it require? i get this with v10.19.0 from Ubuntu 20.04: node_modules/hyperdrive-daemon/node_modules/hyperdrive-daemon-client/bin/commands/create.js:8 static usage = 'create [path]' ^ SyntaxError: Unexpected token =
- imhoguy 6y agoDoes it support private clustering and encryption? My use case is to have shared volumes between servers, laptops but without all the networking hassle (VPNs, opening firewalls).
- mafintosh 6y agoIt uses Noise to encrypt all transport connections and using the api you can whitelist which peers (by their public key) you want to replicate with. This API is not yet exposed through the daemon but it’s in the modules. Hyperdrives also have a built in capability system where you have to know the public key of the drive to download it from a peer, so if you only share the key with yourself no one else can access it. Finally using the modules you can build almost any kind of networking you’d imagine but that part requires more work on your side obviously.
- deleted 6y ago[deleted]
- hinkley 6y agoMy go-to example for 'private cloud' (which as soon as we hit the Trough of Disillusionment on this Hype Train, we will all be talking about) is having a set of people in your social or family group maintaining a file share for your semi-private photos and videos. You don't have to worry about a flood or tornado taking out all the photos of first birthdays, or Grandma when she was younger than you are now, and you also don't broadcast family dynamics, whereabouts (people have been robbed when social media made it clear they were not at home) or pictures of minors. You, your uncle who Moved to the City and your cousin by marriage who wants to be a game designer all set up a file server and share the photos (your cousin is gonna throttle traffic while he's playing CoD or Dota of course, which it turns out he does all the time but at least he's an extra backup copy). I don't know how you keep that one relative from uploading funny things they found on the internet that keep trying to install spyware or back doors though, but I suppose you'd have that problem now.
- beckingz 6y ago
- q3k 6y ago> Hyperdrive is a POSIX-like filesystem implementation How POSIX-like is it? From a filesystem I would expect some more rigorousness than just stating this without any extra information, especially as implementing an actual, to-spec, fully distrubuted POSIX file system is known to be a difficult problem and is generally solved by not wanting a full POSIX-compatible implementation instead (ie. no locking, append-only files, etc.).
- BoumTAC 6y agoI've been following dat since a few years and I've been hearing about multiwriter since a few years but not anymore. Is it still something you are working on ?
- mafintosh 6y agoYes we are continuously exploring and research this. The mount support is our first stepping stone towards this. See the union mounts section of the post
- rklaehn 6y agoCongrats for the release. Mostly familiar with IPFS, but learned about DAT at dtn.is last year. The hypercore protocol is pretty well designed and documented. How would hyperdrive deal with the following scenario: you got a large dataset such as wikipedia. Lots of people have browsed it, but most of them only have a tiny fraction. How do yo know which peers to connect to to get a particular bit (offset?) you are interested in? The DHT only tells you which nodes participate in the hypercore, not what they have in detail, right?
- mafintosh 6y agoThe peers gossip using compressed bitfields in regards to what data they have. These bitfields are super small so we can pack quite a bit of information. At the moment we don't do anything special in regards to discovery, but as we scale that's something we want to investigate. Since everything is running on append-only logs we can group the data into sections quite easily so there is some easy wins we can do there with announcing to the dht that you have data in a specific region.
- rklaehn 6y agoSo do you opportunistically gather info about what other peers have via this gossip protocol, or just when you need something? I had looked at https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/ , but I don't remember anything about a gossip protocol or a peer building a "view of the world". Is that new?
- mafintosh 6y agoIt's only between the peers in your subset of the swarm for now. They exchange a series of WANT and HAVE messages where they subscribe to the sections of each others logs they are interested in. We are working on expanding this scheme so peers can help discover peers that have the section you are looking for. Due to the compressed bitfields these section are quite large. In most cases using a few kilobytes you can share WANT/HAVE for millions of blocks
- obiefernandez 6y agoHow soon until this can replace Dropbox for my little distributed working group consisting of 3 people wanting to share a single file system?
- andrewosh 6y agoOur little team has been doing just this, both for dogfooding and because it works well for us. Take a look at the "mounts" section of the blog post where we describe a group pattern you can set up. You can create a group directory called "team-drive", then mount each user's drive into the group. If you're using FUSE, this will feel similar to Dropbox, but with one directory per-user. We're starting with these kinds of simple mounts, and brainstorming ways to extend them soon.
- deleted 6y ago[deleted]
- grizzles 6y agoHow do you handle merge conflicts? Since you aren't calling it a blockchain I take it that Hypercore is targeted at the trusted (writers) and semi-trusted (readers) use case?
- mafintosh 6y agoHypercore is a single writer append-only log. The website has a bit more info about how it works, but's basically a merkle log signed by a private key / public key scheme. We build collaborative data structures by combining multiple Hypercores. Hyperdrives builds a p2p filesystem on top of Hypercore for a single writer. Using mounts you can mount other peoples drives so merge conflicts don't happen since there is no overlapping writes. We are working on a union mount approach as well for overlapping drives (we talk a bit about this in the post)
- grizzles 6y agoSo it sounds a bit like you've replicated git. Do you agree that for the collaborative data structures side of things (like the chat app) users of the hypercore-protocol will likely run into clock trust problems? PS I'm a big fan of your work/repos.
- mafintosh 6y agoYea you have to trust the original writers atm. I have some ideas for reducing this trust in the future through some consensus schemes but nothing fully baked yet. Def something I wanna hit tho, so we can get better security in something like a massively distributed chat system.
- rodolphoarruda 6y agoWould it be possible to build a service that resembles email using this technology? Like dropping a message file within a local folder and making it appear on someone else's local folder?
- andrewosh 6y ago@pfrazee threw together a demo app called PaulMail [1] that you can view in Beaker [0]. It's very simple and is just a proof-of-concept (messages are unencrypted, for example), but it does hint at possibilities there. [0] https://beakerbrowser.com https://beakerbrowser.com [1] hyper://1bc1faf01a22270fb5698a60e63ef7a596ad976457e6d9914a8fd56d87281917/
- pfraze 6y agoYour recipients also need to be in your beaker address book for it work. It's a very rough PoC.
- rklaehn 6y agoQuick note: it would be nice to ship the hyperdrive daemon as something more self-contained than a npm package. I tried to install it and got some c compile errors. Probably my nodejs version is too old, but you want to have the absolute minimum of friction for users to install it. Beaker comes as an appimage for linux, which worked flawlessly at the first try. Maybe do that for the hyperdrive daemon as well?
- EGreg 6y agoI really liked Dat (thanks, Knight Foundation). And I think it gets a ton of things right. However, I wish it had two things that we need for our purposes... can anyone chime in and say how AND IF they can be accomplished with the current Dat and hypercore? 1. Consensus. If I submit conflicting updates and sign both, how does the swarm resolve what is the latest state? 2. Migrating the swarm. If all the machines in a swarm get corrupted, can I migrate to a totally new swarm?
- sktrdie 6y agoThis is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System (https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. This is similar techniques behind Hyperdrive I believe just again, using standard tools and tech that exists and highly optimized: https://www.sqlite.org/vfs.html https://www.sqlite.org/vfs.html Every time a new version of the SQLite db is published (say by wikipedia), the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated. I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 Again not against redoing things better, but why not use existing proven tech for certain parts of the tool?
- xvector 6y agoThat is pretty clever!
- guerrilla 6y agoSeems like somone should make a frontend for what you just said, especially that last part which would get annoying to do manually.
- fenwick67 6y agoThe big asterisk is, this only works if your database never changes.
- rrdharan 6y agoNot sure if the parent edited their post after you stated this but note they explained a technique to accommodate database updates / changes / edits.
- MayeulC 6y agoInteresting, thanks. Not clear to me when reading the announcement is: will hyperdrive replace dat? From reading the comments here, it seems that the answer is "yes". Now, onto the content, you touch on de-duplication. I am quite concerned with the cost associated to updating a large file. Is something like rolling hashes investigated, to chunk files independently of their size? I guess it is, given you seem to be working hard on de-duplication. But then, that kind of trick best works on uncompressed data, which is inefficient for transmission. Is data compressed before transmission? Whole chunks, or whole files at a time? Ahead of time? Interactively based on what the peer needs? The trade-offs are many, and complex to investigate. I'm wondering if this could be used as an OS image, like OSTree does? And lastly, I did not get if multiple peers having the same private key identity could modify the structure simultaneously. What would happen? Also, nodejs gave me a kneejerk reaction that may be unwarranted, but that's quite a huge dependency to pull in for something that wants to be a ubiquitous building block. Does it have a C API? Also, twitter, discord, github (node to a lesser extent)... it seems somewhat ironic to build a ultimate decentralized filesystem while relying on these hypercentralized offerings, and I am afraid it could turn some contributors off.
- rklaehn 6y agoI kinda share your kneejerk reaction about nodejs for foundational tools, even though I must admit that dat/hyperdrive works surprisingly well for a js tool. But dat/hyperdrive is a well-documented protocol, and there are several ports to other languages. I am very interested in the rust port, but I am not sure in what state it is: https://datrs.yoshuawuyts.com/ https://datrs.yoshuawuyts.com/ I have seen some tweets about progress being made on this. Does anybody know more?
- mafintosh 6y agoThe rust port of Hypercore has been very active recently and they are making good progress. Part of the latest Hypercore release was to move some of the transport crypto to be easier to port to other languages such as rust. The wire protocol works now: https://github.com/Frando/hypercore-protocol-rs https://github.com/Frando/hypercore-protocol-rs and the community is active in #datrs on freenode
- contravariant 6y agoSo regarding the multi-writer problem, exactly how far away is this from implementing a central (bare) git repository? Seems to me that all you need is some kind of way to take care of authentication and possibly some way to make git's (mostly) append-only data-structure play nice with Hyperdrives'. I realise I'm probably missing some details, so my question is which details?
- ohvigrinia 6y agoWould there be an advantage to implementing this using WebRTC as part of the networking, so you could use it in browser? I don't know much about this space, but always been interested in some sort of web socket/WebRTC p2p fs.
- hutzlibu 6y agoThis sounds good! But I would not call it finished, as of today, apparently only one person can make changes to the filesystem. That does limit the use cases. "In v10, we don't go all the way to a general multi-writer solution; solving multi-writer scalably, without incurring major performance penalties or confusing UX, remains a research question for us. " .. so, give them some support, so they can solve this.
- czei002 6y agoSharing an append only log can be quite treacherous for users that are unaware of it. For example, when accidentally including some confidential stuff. A not so security aware person may think a quick delete fixes it and depending on the situation this might event be true, but its still in the log... Not familiar with Hyperdrive so please somebody correct me if I am wrong and this case is handled.
- ZoomZoomZoom 6y agoWould Hyperswarm be useful for setting up mesh-like Wireguard networks automatically? We need some solution for this ASAP - IPv6 not coming, CGNATs everywhere.
- gyrgtyn 6y agoThat'd be zerotier or tailscale
- gyrgtyn 6y agoCan you limit the synced size on clients (like an LRU cache) ?
- seigel 6y agoGreat work. Planning to read about it. I, and I am sure others, are continually intrigued about this space and I hold out hope that the definitive solution appears soon (along with the solution to micropayments !!). On a slightly related note, is anyone interested in having a discussion about how to layer on top of all these "drive" systems a `HDFS` like drive that would have n block replication across different sources, and trying to interpret any given source (dropbox, hyperdrive, google drive, etc..) blocks to make sense of what is being stored there would render the person confused? :)
- seigel 6y agoOne more question... How effective would this be for a `network questionable` environment, if I were let's say doing a mobile app and wanted to incorporate this type of solution in for an offline first type of experience?
- andrewosh 6y agoHyperdrive (and the stack it's built on more generally) works great offline. All the drives you own are still editable without the network, and changes will sync when you come online. As for 'network questionable' and mobile, here are a few of the other projects building on Hypercore that have made those a priority [0] [1] [2]. The Hyperdrive daemon as it's currently built wouldn't fare too well in a bandwidth and/or battery constrained environment (wasn't designed for that), but a mobile solution is definitely on our radar. [0] https://cabal.chat/ https://cabal.chat/ [1] https://www.digital-democracy.org/mapeo/ https://www.digital-democracy.org/mapeo/ [2] https://github.com/consento-org/mobile https://github.com/consento-org/mobile