5 ms·

Iroh: A New Implementation of IPFS

115 points by dannyobrien 4y ago

transfire 4y ago
I’m confused. The write speeds are slower?
- transfire 4y ago
  Yes.
mdaniel 4y ago
> We are targeting the end of October 2022 for an initial version, heh, they better hurry :-D but seriously, I hope this cures some of the oft-reported perf woes with go-ipfs
- jandrese 4y ago
  Yes. I experimented with the go IPFS client but stopped using it after noticing that even a fairly small share was burning up 75% of a CPU core all of the time.
  - b_fiive 4y ago
    fully agreed. I dropped down to start this project because there's no reason this should be the case.
  - Thaxll 4y ago
    Not sure how the Go implementation is but by design IPFS is a very slow protocol.
  - diegocg 4y ago
    How is Rust or any other language supposed to help with this? Unless there is a serious design problem in the current go codebase, removing GC overhead will not by itself solve the problem of some software having to run all the time
    - lijogdfljk 4y ago
      Yea... i'm a huge fan of Rust and it is my primary language these days, but despite my many complaints with Go.. it's not bad. In fact it's pretty solid. I can't imagine the primary slowness in ipfs-go will be solved by ipfs-rust, at least on language alone. Some, sure, but not primary.. i would think. Do i think Rust could have a better, more generic library with code that is far more concise than Go? Yes.. but that's not likely the cause for ipfs-go's issues. I can definitely see a chance for a rewrite producing better code though. As much as i'd love to give Rust that victory, i just can't imagine Rust is _that_ big of contributor in this specific case.
      - rklaehn 4y ago
        We are going to first try to reach feature parity with the go implementation in the areas we care about. But it is quite clear that there are some drastic improvements needed at the protocol level to get the desired performance. We are going to experiment with such improvements, and try to work with protocol labs to standardise them if/when we come up with a good solution.
    - wongarsu 4y ago
      I'm not sure people are claiming that the choice of programming language is the deciding factor? All the website and the comments are saying is "the current client (which happens to be written in go) is slow, maybe this rewrite will be faster". I'm not sure it's fair to read it as an attack on go, or an endorsement of rust.
    - b_fiive 4y ago
      hey I work on the project. This isn't a "re-write in rust" thing, we intend to iterate on the protocol itself to drive performance improvements. nearly all of our team are veterans of the IPFS ecosystem who want to see IPFS evolve to be more performant. We obviously can't change the overall performance of the network overnight, but we do control interop between nodes that run iroh, where we plan to ship fast-paths that we can propose as spec-level changes to the protocol once proven in the wild. Our initial research indicates the thing that needs the most attention isn't the DHT (a commonly-cited source of slowness), but the data transfer protocol: bitswap. We plan to tackle that in the coming months.
- b_fiive 4y ago
  I work on the project, we cut a first release today!
faitswulff 4y ago
This library paves the way for Zuko, an impetuous young interplanetary computation system.
- lcnPylGDnU4H9OF 4y ago
  In case someone is confused by this comment: https://en.wikipedia.org/wiki/Avatar:_The_Last_Airbender https://en.wikipedia.org/wiki/Avatar:_The_Last_Airbender (The character Iroh is the uncle of the character Zuko.)
- superkuh 4y ago
  Gotta wait till someone figures out how to implement proof of honor.
  - evilos 4y ago
    Powered by hot leaf juice.
- pcthrowaway 4y ago
  Zuko will boast 50% more honour than Iroh.
vfclists 4y ago
- ummonk 4y ago
  I normally disagree with people criticizing websites for focusing on style instead of having an ugly 2000s interface, but I have to agree with you on this. This website is extremely difficult to navigate.
- mattlondon 4y ago
  The use of the "fi" ligature on a fixed-width font is also extremely jarring.
- tptacek 4y ago
  Please don't complain about tangential annoyances—things like article or website formats, name collisions, or back-button breakage. They're too common to be interesting. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
nocobot 4y ago
looking forward to playing with this! FYI: the github link (when clicking from the docs, not the landing page) is broken
- b_fiive 4y ago
  thank you. Will fix :)
dignifiedquire 4y ago
Hey folks, I am one of the founders behind this project, happy to answer any questions you might have about it. We have an initial release out since earlier today: https://github.com/n0-computer/iroh/releases/tag/v0.1.0 https://github.com/n0-computer/iroh/releases/tag/v0.1.0 but we are still very early, so be gentle :)
- Gigachad 4y ago
  Project looks great. I have been following IPFS from a distance for a while now but it still seems like it’s not really ready to use for much. I was investigating tech that would allow you to build a decentralised, censorship resistant forum but it seemed extremely difficult to run a real website in IPFS. What use cases do you think work well today and are there any prominent projects doing anything real with the tech?
  - tazard 4y ago
    What do you mean a real website? What difficulties do you see? Lots of sites are hosted on IPFS today
- 3np 4y ago
  As kubo/go-ipfs are notoriously finicky to load-balance/proxy and being able to run anonymously over Tor/I2P are just short of explicit non-goals to the point of neglecting open PRs for years - can we expect a different approach from Iroh?
- capableweb 4y ago
  Is this project also paid for by Protocol Labs, like the initial IPFS implementation? Or is it a completely separate organisation? Is n0 somehow receiving grants from Protocol Labs to fund this effort? If not, how is the organisation supposed to survive long-term?
- supermatt 4y ago
  Was interested in the mobile support, but link is dead: https://iroh.computer/docs/platforms/mobile https://iroh.computer/docs/platforms/mobile
encryptluks2 4y ago
What IPFS lacks that I think would be ideal is a distributed file system. Let's say that Wikipedia shares some data, and I only have 100GB of space but want to help them out, then being able to dedicate 100GB of storage to help Wikipedia or whoever else would be awesome. Also, if I have a complete file or backup on multiple computers being able to immediately start sharing that across multiple servers without having to use like double the storage to also share it with IPFS.
- mdaniel 4y ago
  > ideal is a distributed file system Well, some of that is anti the way IPFS CID hashing works: every edit would result in a new CID for (e.g. https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System) so your local pin of /ipfs/68106339799a6170efed62807e5c245d becomes orphaned when the next edit makes /ipfs/1ba72a5504f558ade3e784d9d349543c the most recent revision. I'm aware of /ipns but unless there was going to be an /ipns entry for every single page then it doesn't become as much of a distributed filesystem as you might think. Or, I guess put another way: it's really great for a distributed filesystem that doesn't change very often. I would expect archive.org would get more benefit from having IPFS seeds than anything on the Internet that is dynamically generated content I was actually sad that they don't already generate torrents for all their collections (or at least it didn't appear that they do)
  - stoplying1 4y ago
    Er, it's not that hard. Think about git repos, they have refs that point to a tip that you traverse. This would work the same way. And in fact CID and chunking is what makes this possible... Ipfs is in fact ripe and perfect for some of these sorts of things and I highly suspect we're going to see more innovation in this space, in applications that are actually pushing the envelope, now that there's a sold rust impl
    - mdaniel 4y ago
      > Er, it's not that hard. You must be my boss; he loves to say that expression on zoom meetings where he's not responsible for the implementation > refs that point to a tip that you traverse Oh, ok, I guess you'll go to /ipns/wikipedia.org and just keep clicking links until you arrive at /wiki/InterPlanetary_File_System then. I think someone tried to prove the "6 degrees of Wikipedia," so yeah, how hard can it be? I'm ashamed I got baited into even replying to this
stoplying1 4y ago
Holy thank god. These things deserve to be written in Rust (see also, matrix-rust-sdk and the entire ecosystem rebasing on it). And ipfs-go and the existing ipfs ecosystem, I'm just so disappointed. They've had so much money floating around, so many filecoin adjacent projects with ipfs-adjacent insiders and yet so much low hanging fruit that seems unrealized or locked up in poc nodejs projects. This is probably the single best thing to happen for ipfs in 3+ years. Bravo, I actually have hope in ipfs again. The messaging on the site makes me think I'm not alone in feeling this, and that excites me even more.
- Iwan-Zotow 4y ago
  I'm voting for Zig - after all, Zig is now self-hosting
  - bobbylarrybobby 4y ago
    Doesn’t Zig still allow for C-style memory bugs like dangling pointers?
- TeeWEE 4y ago
  What’s wrong with ipfs in Golang: https://github.com/ipfs/kubo https://github.com/ipfs/kubo IPFS runs on Golang. This implementation is not going to change anything.
  - kevincox 4y ago
    What do you mean "IPFS runs on Golang"? IPFS is a protocol that had official implementations in Golang and JavaScript. I'm glad to see another implementation. After trying to run go-ipfs (no Kubo) for years I gave up. The implemention was incredibly low quality. It had tons of bugs, a very quirky CLI and API, and performs horribly. With any decent amount of data it would thrash like crazy for no clear reason. I'm really glad to see another implementation. It may bring me back to running my own node. I happen to like Rust and despise Golang but that isn't a big deal, just raises the chance that I can contribute. The go-ipfs code that I read was low quality even go Go.
hestefisk 4y ago
Not to be facetious, but what are serious use cases / uses of IPFS in the wild? I usually use it for downloading books from Libgen… I’d be curious to hear from people who use it for less illicit stuff. I think it’s a wonderful project and good to have more than one implementation.
- capableweb 4y ago
  I've used it in the past to distribute updates to a cluster of servers sharing the same egress/ingress. At one point, the amount of servers made it so updating the software on the servers saturated the ingress and bottlenecked on bandwidth. Used IPFS to distribute the updates in a P2P fashion instead so all the servers could share the data between themselves. Rolling out updates went from taking minutes to seconds. Could have used torrents for the same effect basically, but felt for trying something novel and worked out fine in the end.

115 points by dannyobrien 4y ago

transfire 4y ago

I’m confused. The write speeds are slower?

transfire 4y ago
Yes.

mdaniel 4y ago

> We are targeting the end of October 2022 for an initial version, heh, they better hurry :-D but seriously, I hope this cures some of the oft-reported perf woes with go-ipfs

jandrese 4y ago
Yes. I experimented with the go IPFS client but stopped using it after noticing that even a fairly small share was burning up 75% of a CPU core all of the time.
- b_fiive 4y ago
  fully agreed. I dropped down to start this project because there's no reason this should be the case.
- Thaxll 4y ago
  Not sure how the Go implementation is but by design IPFS is a very slow protocol.
- diegocg 4y ago
  How is Rust or any other language supposed to help with this? Unless there is a serious design problem in the current go codebase, removing GC overhead will not by itself solve the problem of some software having to run all the time
  - lijogdfljk 4y ago
    Yea... i'm a huge fan of Rust and it is my primary language these days, but despite my many complaints with Go.. it's not bad. In fact it's pretty solid. I can't imagine the primary slowness in ipfs-go will be solved by ipfs-rust, at least on language alone. Some, sure, but not primary.. i would think. Do i think Rust could have a better, more generic library with code that is far more concise than Go? Yes.. but that's not likely the cause for ipfs-go's issues. I can definitely see a chance for a rewrite producing better code though. As much as i'd love to give Rust that victory, i just can't imagine Rust is _that_ big of contributor in this specific case.
    - rklaehn 4y ago
      We are going to first try to reach feature parity with the go implementation in the areas we care about. But it is quite clear that there are some drastic improvements needed at the protocol level to get the desired performance. We are going to experiment with such improvements, and try to work with protocol labs to standardise them if/when we come up with a good solution.
  - wongarsu 4y ago
    I'm not sure people are claiming that the choice of programming language is the deciding factor? All the website and the comments are saying is "the current client (which happens to be written in go) is slow, maybe this rewrite will be faster". I'm not sure it's fair to read it as an attack on go, or an endorsement of rust.
  - b_fiive 4y ago
    hey I work on the project. This isn't a "re-write in rust" thing, we intend to iterate on the protocol itself to drive performance improvements. nearly all of our team are veterans of the IPFS ecosystem who want to see IPFS evolve to be more performant. We obviously can't change the overall performance of the network overnight, but we do control interop between nodes that run iroh, where we plan to ship fast-paths that we can propose as spec-level changes to the protocol once proven in the wild. Our initial research indicates the thing that needs the most attention isn't the DHT (a commonly-cited source of slowness), but the data transfer protocol: bitswap. We plan to tackle that in the coming months.
b_fiive 4y ago
I work on the project, we cut a first release today!

faitswulff 4y ago

This library paves the way for Zuko, an impetuous young interplanetary computation system.

lcnPylGDnU4H9OF 4y ago
In case someone is confused by this comment: https://en.wikipedia.org/wiki/Avatar:_The_Last_Airbender https://en.wikipedia.org/wiki/Avatar:_The_Last_Airbender (The character Iroh is the uncle of the character Zuko.)
superkuh 4y ago
Gotta wait till someone figures out how to implement proof of honor.
- evilos 4y ago
  Powered by hot leaf juice.
pcthrowaway 4y ago
Zuko will boast 50% more honour than Iroh.

vfclists 4y ago

ummonk 4y ago
I normally disagree with people criticizing websites for focusing on style instead of having an ugly 2000s interface, but I have to agree with you on this. This website is extremely difficult to navigate.
mattlondon 4y ago
The use of the "fi" ligature on a fixed-width font is also extremely jarring.
tptacek 4y ago
Please don't complain about tangential annoyances—things like article or website formats, name collisions, or back-button breakage. They're too common to be interesting. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

nocobot 4y ago

looking forward to playing with this! FYI: the github link (when clicking from the docs, not the landing page) is broken

b_fiive 4y ago
thank you. Will fix :)

dignifiedquire 4y ago

Hey folks, I am one of the founders behind this project, happy to answer any questions you might have about it. We have an initial release out since earlier today: https://github.com/n0-computer/iroh/releases/tag/v0.1.0 https://github.com/n0-computer/iroh/releases/tag/v0.1.0 but we are still very early, so be gentle :)

Gigachad 4y ago
Project looks great. I have been following IPFS from a distance for a while now but it still seems like it’s not really ready to use for much. I was investigating tech that would allow you to build a decentralised, censorship resistant forum but it seemed extremely difficult to run a real website in IPFS. What use cases do you think work well today and are there any prominent projects doing anything real with the tech?
- tazard 4y ago
  What do you mean a real website? What difficulties do you see? Lots of sites are hosted on IPFS today
3np 4y ago
As kubo/go-ipfs are notoriously finicky to load-balance/proxy and being able to run anonymously over Tor/I2P are just short of explicit non-goals to the point of neglecting open PRs for years - can we expect a different approach from Iroh?
capableweb 4y ago
Is this project also paid for by Protocol Labs, like the initial IPFS implementation? Or is it a completely separate organisation? Is n0 somehow receiving grants from Protocol Labs to fund this effort? If not, how is the organisation supposed to survive long-term?
supermatt 4y ago
Was interested in the mobile support, but link is dead: https://iroh.computer/docs/platforms/mobile https://iroh.computer/docs/platforms/mobile

encryptluks2 4y ago

What IPFS lacks that I think would be ideal is a distributed file system. Let's say that Wikipedia shares some data, and I only have 100GB of space but want to help them out, then being able to dedicate 100GB of storage to help Wikipedia or whoever else would be awesome. Also, if I have a complete file or backup on multiple computers being able to immediately start sharing that across multiple servers without having to use like double the storage to also share it with IPFS.

mdaniel 4y ago
> ideal is a distributed file system Well, some of that is anti the way IPFS CID hashing works: every edit would result in a new CID for (e.g. https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System) so your local pin of /ipfs/68106339799a6170efed62807e5c245d becomes orphaned when the next edit makes /ipfs/1ba72a5504f558ade3e784d9d349543c the most recent revision. I'm aware of /ipns but unless there was going to be an /ipns entry for every single page then it doesn't become as much of a distributed filesystem as you might think. Or, I guess put another way: it's really great for a distributed filesystem that doesn't change very often. I would expect archive.org would get more benefit from having IPFS seeds than anything on the Internet that is dynamically generated content I was actually sad that they don't already generate torrents for all their collections (or at least it didn't appear that they do)
- stoplying1 4y ago
  Er, it's not that hard. Think about git repos, they have refs that point to a tip that you traverse. This would work the same way. And in fact CID and chunking is what makes this possible... Ipfs is in fact ripe and perfect for some of these sorts of things and I highly suspect we're going to see more innovation in this space, in applications that are actually pushing the envelope, now that there's a sold rust impl
  - mdaniel 4y ago
    > Er, it's not that hard. You must be my boss; he loves to say that expression on zoom meetings where he's not responsible for the implementation > refs that point to a tip that you traverse Oh, ok, I guess you'll go to /ipns/wikipedia.org and just keep clicking links until you arrive at /wiki/InterPlanetary_File_System then. I think someone tried to prove the "6 degrees of Wikipedia," so yeah, how hard can it be? I'm ashamed I got baited into even replying to this

stoplying1 4y ago

Holy thank god. These things deserve to be written in Rust (see also, matrix-rust-sdk and the entire ecosystem rebasing on it). And ipfs-go and the existing ipfs ecosystem, I'm just so disappointed. They've had so much money floating around, so many filecoin adjacent projects with ipfs-adjacent insiders and yet so much low hanging fruit that seems unrealized or locked up in poc nodejs projects. This is probably the single best thing to happen for ipfs in 3+ years. Bravo, I actually have hope in ipfs again. The messaging on the site makes me think I'm not alone in feeling this, and that excites me even more.

Iwan-Zotow 4y ago
I'm voting for Zig - after all, Zig is now self-hosting
- bobbylarrybobby 4y ago
  Doesn’t Zig still allow for C-style memory bugs like dangling pointers?
TeeWEE 4y ago
What’s wrong with ipfs in Golang: https://github.com/ipfs/kubo https://github.com/ipfs/kubo IPFS runs on Golang. This implementation is not going to change anything.
- kevincox 4y ago
  What do you mean "IPFS runs on Golang"? IPFS is a protocol that had official implementations in Golang and JavaScript. I'm glad to see another implementation. After trying to run go-ipfs (no Kubo) for years I gave up. The implemention was incredibly low quality. It had tons of bugs, a very quirky CLI and API, and performs horribly. With any decent amount of data it would thrash like crazy for no clear reason. I'm really glad to see another implementation. It may bring me back to running my own node. I happen to like Rust and despise Golang but that isn't a big deal, just raises the chance that I can contribute. The go-ipfs code that I read was low quality even go Go.

hestefisk 4y ago

Not to be facetious, but what are serious use cases / uses of IPFS in the wild? I usually use it for downloading books from Libgen… I’d be curious to hear from people who use it for less illicit stuff. I think it’s a wonderful project and good to have more than one implementation.

capableweb 4y ago
I've used it in the past to distribute updates to a cluster of servers sharing the same egress/ingress. At one point, the amount of servers made it so updating the software on the servers saturated the ingress and bottlenecked on bandwidth. Used IPFS to distribute the updates in a P2P fashion instead so all the servers could share the data between themselves. Rolling out updates went from taking minutes to seconds. Could have used torrents for the same effect basically, but felt for trying something novel and worked out fine in the end.