7 ms·

Nebulus: An IPFS-Less IPFS

143 points by skogard 5y ago

adkadskhj 5y ago
This looks cool! I'm writing an immutable data store, with my own hashes as IDs; I wonder how viable it would be to use IPFS CID's as my IDs? I expect the trouble there would be how the data is chunked. If IPFS disagrees about the chunking we would have different CIDs. .. also i'd need to implement all of this myself to get it in Rust. Hmm.
- Jayschwa 5y ago
  https://github.com/ipfs-rust https://github.com/ipfs-rust Maybe some of the building blocks you need can be found in there somewhere.
  - handrous 5y ago
    Not implementing the core of the protocol in something portable-library-friendly has been a real drag on IPFS. It's made it, in practice, more of a program than a protocol. The counter-argument is that they'd not have made so much progress in the first place, in the short list of languages that qualify as portable-library-friendly.
nynx 5y ago
Two things: 1) I wonder why this wasn't written in a language that can compile to wasm, so it could be used on both native and js without reimplementing. 2) Can someone contrast hypercore and IPFS?
- pfraze 5y ago
  Hypercore distributes signed log-based structures addressed by pubkey. It embeds indexes in each log entry to make this perform well; for instance Hyperbee is a b-tree over a log. (See https://github.com/hypercore-protocol/p2p-indexing-and-search https://github.com/hypercore-protocol/p2p-indexing-and-searc... for how that works.) IPFS distributes content-addressed blobs which interlink to form DAGs. Mutability is supported by pubkey (or similar) pointers to recent hashes. I work with Hypercore. It’s often a matter of preference between them.
  - skogard 5y ago
    I mentioned this in above thread, but I am thinking of ways to mash up Nebulus with Hypercore (Because the content addressable aspect has been unbundled from the networking aspect of IPFS, I think there will be some interesting ways to deliver IPFS over Hypercore). Will keep the hypercore community posted when I make some progress, and thank you for all your work with Hypercore. I use Hypercore for several of my projects.
    - Taek 5y ago
      What's the performance like for hypercore? How long does it take for updates and new data to propagate the network? What type of throughput do you get for uploads and downloads?
- skogard 5y ago
  > 1) I wonder why this wasn't written in a language that can compile to wasm, so it could be used on both native and js without reimplementing. The most straightforward answer is, I extracted this out of an existing project I've been working on: https://rarepress.org/ https://rarepress.org/ I had to build a system that makes use of IPFS without having to publish everything to the public network, and this was where the idea came from. But the "IPFS-Less IPFS" turned out to be much more powerful idea than I originally thought, so decided to turn it into its own open source project. That said, I think it shouldn't be difficult to implement this using Go at some point if this approach becomes popular. For now I wanted to keep the initiative minimal so I can move fast. > 2) Can someone contrast hypercore and IPFS? This is a great question in this context. Hypercore is more focused on the networking aspect, but IPFS is not just a networking protocol but also a way to represent files using immutable identifiers (content addressable file system). What I just did with Nebulus is unbundle the "file system" from the "network" so we can use IPFS in more flexible manner. Speaking of Hypercore, because Nebulus unbundles the network aspect of IPFS from the content addressable file system aspect, it is possible to use ANY network transport to replicate IPFS files, which includes Hypercore.
capableweb 5y ago
This entire project can be replaced in two different ways with just IPFS. You don't have to run IPFS to be able to add files or retrieve them (if you've added them on the same computer), so simply add the files without running the daemon and you're already "Offline and Private" and "IPFS without the Network". If that's not enough, you can also run your IPFS daemon with `--offline` and it won't connect to any network, giving you another way of using IPFS offline and privately. If you want to go one step further, when adding files with the IPFS CLI, you can add the flag `--only-hash` and IPFS will only hash the content without actually writing anything to disk, making it even more "offline" and "private". All in all, this seems to be a project whose author missed to read the documentation for IPFS as everything mentioned is already supported in go-ipfs, the main IPFS implementation. Edit: the title "IPFS-less IPFS" is fun too, since nebulus seems to include ipfs-core as a dependency, not really IPFS-less then I'd say :) https://github.com/skogard/nebulus/blob/480d43dc22ccd949c6ae2776131845ba94aac3a1/package.json#L15 https://github.com/skogard/nebulus/blob/480d43dc22ccd949c6ae...
- skogard 5y ago
  Hi there, just want to clarify that Nebulus is not a replacement for IPFS or competes with IPFS. In fact it internally uses all the features you just mentioned. In fact it's even powered by JS-IPFS and makes use of the "--only-hash" features and other features you mentioned. I needed a programmatic way to do all these things (as well as many other user friendly features that the IPFS raw protocol will never provide) and the existing IPFS libraries don't provide a way to do so, so I built it and have been using it from another project, and it was so useful that I decided to extract it out of that project into a standalone one. Just hope others find it useful as well. Thank you.
  - whyrusleeping 5y ago
    This is really great stuff, thank you for making this!
  - IncRnd 5y ago
    So, it isn't an IPFS-less IPFS but a layer on top of IPFS?
thesausageking 5y ago
Very confused by the name. I thought this was related to the Nebulous Labs (the startup behind the Sia storage network) and their work around IPFS: https://blog.sia.tech/skynet-summer-2021-update-86ed8db21eae https://blog.sia.tech/skynet-summer-2021-update-86ed8db21eae
- dennis-tra 5y ago
  A couple of days ago I’ve released an IPFS DHT crawler and named it Nebula [0] if I knew that this was such a crowded place… [0] https://github.com/dennis-tra/nebula-crawler https://github.com/dennis-tra/nebula-crawler
chejazi 5y ago
Reminds me of Camlistore, one of the earlier content-addressable personal storage systems, by Brad Fitzpatrick. Now called Perkeep apparently https://perkeep.org/ https://perkeep.org/
CGamesPlay 5y ago
What I really want is a private, secure IPFS. I do want the network functionality, but I want it to exist only between my nodes. I know I can pin encrypted blobs, but I want something that handles this transparently. Is this possible somewhere?
- mtnGoat 5y ago
  run IPFS node in --offline so it doesnt connect to unwanted peers and then stick it behind a firewall so access is controlled?
- handrous 5y ago
  Yeah. You'll need to run your own bootstrap node(s) and create a custom key for the swarm. IIRC that's about all it takes. A bootstrap node is just any node that's (more or less) permanently reachable on the public Internet. Same as lots of tech like this, IPFS needs them for initial peer discovery and (in many cases, and I'm guessing in IPFS' case as well) NAT-breaking, plus as a route of last resort.
- yusefnapora 5y ago
  You can run a private network by configuring all your nodes with a pre-shared "swarm key" and running your own bootstrap nodes. There's some documentation here: https://github.com/ipfs/go-ipfs/blob/release-v0.9.0/docs/experimental-features.md#private-networks https://github.com/ipfs/go-ipfs/blob/release-v0.9.0/docs/exp...
- white-flame 5y ago
  Yes, Freenet. IPFS is basically Freenet with all the anonymity and privacy stripped out, and claiming it's some new innovation. Freenet allows connecting only to peers, and is the same sort of p2p hash-based storage cache. It's about as old as Tor.
qbasic_forever 5y ago
What's the use-case for this kind of usage of IPFS? Is it just a generic content addressable store for local use? Like perhaps use it to compile and install packages that you symlink into your system path a bit like DIY nix?
- chalst 5y ago
  Any application using content addressing can make use of at least some IPFS technologies. Sharing the contents of /nix/store on a local network or VPN is something I've been thinking about recently.
- skogard 5y ago
  I wrote a dedicated section in the documentation just for this, it can be pretty powerful: https://docs.nebulus.dev/#/?id=use-cases https://docs.nebulus.dev/#/?id=use-cases
  - argvargc 5y ago
    Looks great, thank you for presenting it so thoughtfully too. Do I read correctly to mean Nebulus could potentially be used as a replacement for p2p file xfer, such as per wormhole? Any caveats with such a usecase? (Also, just FYI there's a one-letter typo with the first sub-title under API)
kordlessagain 5y ago
It would be cool to hook this up to the lightning network.

143 points by skogard 5y ago

This looks cool! I'm writing an immutable data store, with my own hashes as IDs; I wonder how viable it would be to use IPFS CID's as my IDs? I expect the trouble there would be how the data is chunked. If IPFS disagrees about the chunking we would have different CIDs. .. also i'd need to implement all of this myself to get it in Rust. Hmm.

Jayschwa 5y ago
https://github.com/ipfs-rust https://github.com/ipfs-rust Maybe some of the building blocks you need can be found in there somewhere.
- handrous 5y ago
  Not implementing the core of the protocol in something portable-library-friendly has been a real drag on IPFS. It's made it, in practice, more of a program than a protocol. The counter-argument is that they'd not have made so much progress in the first place, in the short list of languages that qualify as portable-library-friendly.

nynx 5y ago

Two things: 1) I wonder why this wasn't written in a language that can compile to wasm, so it could be used on both native and js without reimplementing. 2) Can someone contrast hypercore and IPFS?

pfraze 5y ago
Hypercore distributes signed log-based structures addressed by pubkey. It embeds indexes in each log entry to make this perform well; for instance Hyperbee is a b-tree over a log. (See https://github.com/hypercore-protocol/p2p-indexing-and-search https://github.com/hypercore-protocol/p2p-indexing-and-searc... for how that works.) IPFS distributes content-addressed blobs which interlink to form DAGs. Mutability is supported by pubkey (or similar) pointers to recent hashes. I work with Hypercore. It’s often a matter of preference between them.
- skogard 5y ago
  I mentioned this in above thread, but I am thinking of ways to mash up Nebulus with Hypercore (Because the content addressable aspect has been unbundled from the networking aspect of IPFS, I think there will be some interesting ways to deliver IPFS over Hypercore). Will keep the hypercore community posted when I make some progress, and thank you for all your work with Hypercore. I use Hypercore for several of my projects.
  - Taek 5y ago
    What's the performance like for hypercore? How long does it take for updates and new data to propagate the network? What type of throughput do you get for uploads and downloads?
skogard 5y ago
> 1) I wonder why this wasn't written in a language that can compile to wasm, so it could be used on both native and js without reimplementing. The most straightforward answer is, I extracted this out of an existing project I've been working on: https://rarepress.org/ https://rarepress.org/ I had to build a system that makes use of IPFS without having to publish everything to the public network, and this was where the idea came from. But the "IPFS-Less IPFS" turned out to be much more powerful idea than I originally thought, so decided to turn it into its own open source project. That said, I think it shouldn't be difficult to implement this using Go at some point if this approach becomes popular. For now I wanted to keep the initiative minimal so I can move fast. > 2) Can someone contrast hypercore and IPFS? This is a great question in this context. Hypercore is more focused on the networking aspect, but IPFS is not just a networking protocol but also a way to represent files using immutable identifiers (content addressable file system). What I just did with Nebulus is unbundle the "file system" from the "network" so we can use IPFS in more flexible manner. Speaking of Hypercore, because Nebulus unbundles the network aspect of IPFS from the content addressable file system aspect, it is possible to use ANY network transport to replicate IPFS files, which includes Hypercore.

capableweb 5y ago

This entire project can be replaced in two different ways with just IPFS. You don't have to run IPFS to be able to add files or retrieve them (if you've added them on the same computer), so simply add the files without running the daemon and you're already "Offline and Private" and "IPFS without the Network". If that's not enough, you can also run your IPFS daemon with `--offline` and it won't connect to any network, giving you another way of using IPFS offline and privately. If you want to go one step further, when adding files with the IPFS CLI, you can add the flag `--only-hash` and IPFS will only hash the content without actually writing anything to disk, making it even more "offline" and "private". All in all, this seems to be a project whose author missed to read the documentation for IPFS as everything mentioned is already supported in go-ipfs, the main IPFS implementation. Edit: the title "IPFS-less IPFS" is fun too, since nebulus seems to include ipfs-core as a dependency, not really IPFS-less then I'd say :) https://github.com/skogard/nebulus/blob/480d43dc22ccd949c6ae2776131845ba94aac3a1/package.json#L15 https://github.com/skogard/nebulus/blob/480d43dc22ccd949c6ae...

skogard 5y ago
Hi there, just want to clarify that Nebulus is not a replacement for IPFS or competes with IPFS. In fact it internally uses all the features you just mentioned. In fact it's even powered by JS-IPFS and makes use of the "--only-hash" features and other features you mentioned. I needed a programmatic way to do all these things (as well as many other user friendly features that the IPFS raw protocol will never provide) and the existing IPFS libraries don't provide a way to do so, so I built it and have been using it from another project, and it was so useful that I decided to extract it out of that project into a standalone one. Just hope others find it useful as well. Thank you.
- whyrusleeping 5y ago
  This is really great stuff, thank you for making this!
- IncRnd 5y ago
  So, it isn't an IPFS-less IPFS but a layer on top of IPFS?

thesausageking 5y ago

Very confused by the name. I thought this was related to the Nebulous Labs (the startup behind the Sia storage network) and their work around IPFS: https://blog.sia.tech/skynet-summer-2021-update-86ed8db21eae https://blog.sia.tech/skynet-summer-2021-update-86ed8db21eae

dennis-tra 5y ago
A couple of days ago I’ve released an IPFS DHT crawler and named it Nebula [0] if I knew that this was such a crowded place… [0] https://github.com/dennis-tra/nebula-crawler https://github.com/dennis-tra/nebula-crawler

chejazi 5y ago

Reminds me of Camlistore, one of the earlier content-addressable personal storage systems, by Brad Fitzpatrick. Now called Perkeep apparently https://perkeep.org/ https://perkeep.org/

CGamesPlay 5y ago

What I really want is a private, secure IPFS. I do want the network functionality, but I want it to exist only between my nodes. I know I can pin encrypted blobs, but I want something that handles this transparently. Is this possible somewhere?

mtnGoat 5y ago
run IPFS node in --offline so it doesnt connect to unwanted peers and then stick it behind a firewall so access is controlled?
handrous 5y ago
Yeah. You'll need to run your own bootstrap node(s) and create a custom key for the swarm. IIRC that's about all it takes. A bootstrap node is just any node that's (more or less) permanently reachable on the public Internet. Same as lots of tech like this, IPFS needs them for initial peer discovery and (in many cases, and I'm guessing in IPFS' case as well) NAT-breaking, plus as a route of last resort.
yusefnapora 5y ago
You can run a private network by configuring all your nodes with a pre-shared "swarm key" and running your own bootstrap nodes. There's some documentation here: https://github.com/ipfs/go-ipfs/blob/release-v0.9.0/docs/experimental-features.md#private-networks https://github.com/ipfs/go-ipfs/blob/release-v0.9.0/docs/exp...
white-flame 5y ago
Yes, Freenet. IPFS is basically Freenet with all the anonymity and privacy stripped out, and claiming it's some new innovation. Freenet allows connecting only to peers, and is the same sort of p2p hash-based storage cache. It's about as old as Tor.

qbasic_forever 5y ago

What's the use-case for this kind of usage of IPFS? Is it just a generic content addressable store for local use? Like perhaps use it to compile and install packages that you symlink into your system path a bit like DIY nix?

chalst 5y ago
Any application using content addressing can make use of at least some IPFS technologies. Sharing the contents of /nix/store on a local network or VPN is something I've been thinking about recently.
skogard 5y ago
I wrote a dedicated section in the documentation just for this, it can be pretty powerful: https://docs.nebulus.dev/#/?id=use-cases https://docs.nebulus.dev/#/?id=use-cases
- argvargc 5y ago
  Looks great, thank you for presenting it so thoughtfully too. Do I read correctly to mean Nebulus could potentially be used as a replacement for p2p file xfer, such as per wormhole? Any caveats with such a usecase? (Also, just FYI there's a one-letter typo with the first sub-title under API)

kordlessagain 5y ago

It would be cool to hook this up to the lightning network.