4 ms·

We decided to take a year to go heads down and rework a lot of stuff, so we did get publicly stagnant but we're fulltime on the next version and should release

by pfraze 7y ago

We decided to take a year to go heads down and rework a lot of stuff, so we did get publicly stagnant but we're fulltime on the next version and should release a public beta soon. I tweet a lot about our progress if you can sort through my sillier tweets (@pfrazee). Past year has had a lot of improvements - Protocol moved to a hole-punching DHT for peer discovery (hyperswarm). - Protocol now scales # of writes and # of files much better. We were able to put a Wikipedia export, which is millions of files in 2 flat dirs, into a single "drive" and get good read performance. This performance bump came from an indexing structure that's built into every log entry (hypertrie). - Protocol now supports "mounting" which is a way to basically symlink drives onto each other. Good composition tool, esp useful for mounting deps in a /vendor directory. - Browser now has a builtin editor that splits the screen for live editing. Feels similar to hackmd. - Browser added a bash-like terminal for working with the protocol's filespace. It's glued to the current page so you can drive around the web using `cd`. - Browser added a basic identity system. Every user has a profile site that's created automatically and maintains an address book of other users. - We built out application tooling a fair amount. It's fairly easy to build multi-user apps now, where previously it was a bit of rocket surgery. Some of the year was spent prototyping ideas and throwing them away as well. A bit inefficient, but helped us learn.

grng3r 7y ago
I've been reading about dat and doing some basic projects for about a year in my spare time, stagnating a bit lately as am finishing my degree. As I remember there was some talk about Python and Rust implementations at the time, is that still under development as well? I am not as enthusiastic with Js but really like the protocol and the idea behind it.... Thanks
- pfraze 7y ago
  Yeah I do know there's a Rust impl that people are working on
koeng 7y ago
I’ve been thinking of deploying dat for Nanopore DNA sequencing, but recall something about dat not being great at handling that kind of data. Does the newest version of dat handle large files well (10gb)? Does it handle tons of files nested in a few directories well? Are there any issues I should know about there? What is the command line support like for multi-writer? Do you have any metrics for how much Dat is currently being used? Thanks!
- pfraze 7y ago
  > Does the newest version of dat handle large files well (10gb)? Large files work fine but currently any change to a file rewrites it in its entirety. That will mean history will be large until the GC kicks in, and any file that's modified has to be redownloaded in its entirety. The team spent a fair amount of time looking at a solution to partial file-updates which works like inodes. They ultimately decided it was too difficult to pull off for now. > Does it handle tons of files nested in a few directories well? Yep, no issues there > What is the command line support like for multi-writer? We're still deciding on how to handle multi-writer. It's a priority for us after the upcoming stable release. > Do you have any metrics for how much Dat is currently being used? Nothing concrete atm. If I had to guess, it'd be no more than 1k.
  - karissa 7y ago
    > Does the newest version of dat handle large files well (10gb)? I'd be curious if anyone wanted to try it ;) https://github.com/mafintosh/hyperdrive/blob/master/index.js#L441 https://github.com/mafintosh/hyperdrive/blob/master/index.js... > What is the command line support like for multi-writer? There is an experimental multiwriter CLI using hyperdrive and kappa-db (github.com/kappa-db) https://cobox.cloud https://cobox.cloud
  - ianopolous 7y ago
    We've solved efficient partial file updates in Peergos, which is built on ipfs. Happy to talk you through our data structures if you're interested. The key ones are cryptree and Merkle-champs.
ignoramous 7y ago
Thanks. Has anyone by any chance benchmarked hypertrie against u/marknadal's [0] gun.js? Speaking of which, the gun.js network hosts the internet-archive (archive.org) [1] which is likely to be bigger than the English Wikipedia (?). Cloudflare runs an IPFS gateway. Are there organizations of similar size dat is looking to onboard or has onboard-ed? --- A few questions: 1a. Re: hole-punching: Do you support clients on LTE (or, networks behind CG-NAT)? Is hole-punching just for the DHT connections (which seems to operate over UDP4) or for Feed (hypercore) connections, too (which default to TCP?). 1b. Are there plans to use WebRTC (like WebTorrent) instead of TCP or UDP sockets? 1c. In case of TURN relays, how long do you intend to support traffic through your servers? It might get cost-prohibitive at some point. Are (will) TURN relays (be) deployed worldwide to combat latencies? 2. Are hyperswarm and hyperdrive different example usages for hypercore or are they complementary? If so, how are those projects related, as in, is hyperdrive built on top of hyperswarm? 3. The project is spread across disparate GitHub accounts-- Some top-level packages in mafintosh's account and some in hyperswarm's, for instance. Is this intentional? 4. I saw references to merkle-trees, flat-trees, kademlia, HAMT, bitfields... is there a documentation on why these data structures are used? I can guess kademlia is for the DHT and merkle-trees for some form of entropy management. Are such implementation details documented (looking for something similar to gun.js [2] or redis [3] docs)? 5a. Can you please point out a few major differences to gun.js and ProtocolLab's IPFS? 5b. Conversely, what are some use-cases for which dat really outshines other such protocols? [0] https://news.ycombinator.com/item?id=15818856 https://news.ycombinator.com/item?id=15818856 [1] https://news.ycombinator.com/item?id=17685682 https://news.ycombinator.com/item?id=17685682 [2] https://gun.eco/docs/RAD https://gun.eco/docs/RAD [3] https://news.ycombinator.com/item?id=15384396 https://news.ycombinator.com/item?id=15384396
m1sta_ 7y ago
Once a read key becomes public, it's always public right? I appreciate that arguably that's always the case.

by pfraze 7y ago

grng3r 7y ago

I've been reading about dat and doing some basic projects for about a year in my spare time, stagnating a bit lately as am finishing my degree. As I remember there was some talk about Python and Rust implementations at the time, is that still under development as well? I am not as enthusiastic with Js but really like the protocol and the idea behind it.... Thanks

pfraze 7y ago
Yeah I do know there's a Rust impl that people are working on

koeng 7y ago

I’ve been thinking of deploying dat for Nanopore DNA sequencing, but recall something about dat not being great at handling that kind of data. Does the newest version of dat handle large files well (10gb)? Does it handle tons of files nested in a few directories well? Are there any issues I should know about there? What is the command line support like for multi-writer? Do you have any metrics for how much Dat is currently being used? Thanks!

pfraze 7y ago
> Does the newest version of dat handle large files well (10gb)? Large files work fine but currently any change to a file rewrites it in its entirety. That will mean history will be large until the GC kicks in, and any file that's modified has to be redownloaded in its entirety. The team spent a fair amount of time looking at a solution to partial file-updates which works like inodes. They ultimately decided it was too difficult to pull off for now. > Does it handle tons of files nested in a few directories well? Yep, no issues there > What is the command line support like for multi-writer? We're still deciding on how to handle multi-writer. It's a priority for us after the upcoming stable release. > Do you have any metrics for how much Dat is currently being used? Nothing concrete atm. If I had to guess, it'd be no more than 1k.
- karissa 7y ago
  > Does the newest version of dat handle large files well (10gb)? I'd be curious if anyone wanted to try it ;) https://github.com/mafintosh/hyperdrive/blob/master/index.js#L441 https://github.com/mafintosh/hyperdrive/blob/master/index.js... > What is the command line support like for multi-writer? There is an experimental multiwriter CLI using hyperdrive and kappa-db (github.com/kappa-db) https://cobox.cloud https://cobox.cloud
- ianopolous 7y ago
  We've solved efficient partial file updates in Peergos, which is built on ipfs. Happy to talk you through our data structures if you're interested. The key ones are cryptree and Merkle-champs.

ignoramous 7y ago

Thanks. Has anyone by any chance benchmarked hypertrie against u/marknadal's [0] gun.js? Speaking of which, the gun.js network hosts the internet-archive (archive.org) [1] which is likely to be bigger than the English Wikipedia (?). Cloudflare runs an IPFS gateway. Are there organizations of similar size dat is looking to onboard or has onboard-ed? --- A few questions: 1a. Re: hole-punching: Do you support clients on LTE (or, networks behind CG-NAT)? Is hole-punching just for the DHT connections (which seems to operate over UDP4) or for Feed (hypercore) connections, too (which default to TCP?). 1b. Are there plans to use WebRTC (like WebTorrent) instead of TCP or UDP sockets? 1c. In case of TURN relays, how long do you intend to support traffic through your servers? It might get cost-prohibitive at some point. Are (will) TURN relays (be) deployed worldwide to combat latencies? 2. Are hyperswarm and hyperdrive different example usages for hypercore or are they complementary? If so, how are those projects related, as in, is hyperdrive built on top of hyperswarm? 3. The project is spread across disparate GitHub accounts-- Some top-level packages in mafintosh's account and some in hyperswarm's, for instance. Is this intentional? 4. I saw references to merkle-trees, flat-trees, kademlia, HAMT, bitfields... is there a documentation on why these data structures are used? I can guess kademlia is for the DHT and merkle-trees for some form of entropy management. Are such implementation details documented (looking for something similar to gun.js [2] or redis [3] docs)? 5a. Can you please point out a few major differences to gun.js and ProtocolLab's IPFS? 5b. Conversely, what are some use-cases for which dat really outshines other such protocols? [0] https://news.ycombinator.com/item?id=15818856 https://news.ycombinator.com/item?id=15818856 [1] https://news.ycombinator.com/item?id=17685682 https://news.ycombinator.com/item?id=17685682 [2] https://gun.eco/docs/RAD https://gun.eco/docs/RAD [3] https://news.ycombinator.com/item?id=15384396 https://news.ycombinator.com/item?id=15384396

m1sta_ 7y ago

Once a read key becomes public, it's always public right? I appreciate that arguably that's always the case.