4 ms·

In recent years there has been a large growth of alternative IPFS implementations. Partly because of different needs, partly because of unhappiness with the per

In recent years there has been a large growth of alternative IPFS implementations. Partly because of different needs, partly because of unhappiness with the performance and reliability of the go-ipfs implementation. At my previous company actyx.com we wrote our own ipfs implementation in rust called ipfs-embed ( https://github.com/ipfs-rust/ipfs-embed https://github.com/ipfs-rust/ipfs-embed ) that serves as the storage for our distributed event sourcing system. It is used in production and is reliable. The user of an Actyx deployment will not notice in any way that they are using ipfs in the background. There are also some other interesting developments involving ipfs and rust. See a list of implementations here: https://docs.ipfs.tech/basics/ipfs-implementations/ https://docs.ipfs.tech/basics/ipfs-implementations/

_hl_ 4y ago
Wow, is there a tech blog where we can read more about the design and why it was made that way? Sounds absolutely fascinating!
- rklaehn 4y ago
  I will talk about it at IPFS camp 2022. I hope it will be recorded. Here is my talk about actyx from IPFS camp 2019: https://www.youtube.com/watch?v=Qzu0xtCT-R0 https://www.youtube.com/watch?v=Qzu0xtCT-R0 . It does not go into much technical details, and it does not describe the current architecture but a previous iteration though.
stavros 4y ago
This isn't compatible with go-ipfs, right? I can't run this as a node and connect to IPFS the network?
- rklaehn 4y ago
  ipfs-embed is using rust-libp2p and official protocols like gossipsub. So for that it is fully compatible with the ipfs network. At least as compatible as you can get when the specs are sometimes vague. Bitswap has a compat mode, but when two ipfs-embed versions talk to each other they use a different protocol. Note that ipfs-embed is made for a different use case than the global ipfs. It is typically used by private swarms where there are a smaller number of nodes. So e.g. we don't use the DHT at all. We also found that go-ipfs compat is not that useful for our use case, so the compat layer was mainly done as a service to the community and might not work with the latest go-ipfs / kubo.
  - stavros 4y ago
    Makes sense, thanks! I'm not sure what not using the DHT entails, does that mean you can't discover content? That can't be it, so how does that happen? Is it just gossip?
    - rklaehn 4y ago
      Yes, pretty much. Basically the nodes of the swarm (around 100 at most) gossip events to each other. That is also how they learn about the presence of the other nodes. They try to stay connected as well as the network topology allows. Then when you want some bulk data via bitswap they just ask their current peers. Either the emitter of the data is a current peer, then it obviously has it. Or eventually one of your peers will replicate the data, and then you will get it with some delay. The latter happens frequently if a part of the swarm is behind a NAT. You have some delay, but eventually everybody gets all events.

_hl_ 4y ago

Wow, is there a tech blog where we can read more about the design and why it was made that way? Sounds absolutely fascinating!

rklaehn 4y ago
I will talk about it at IPFS camp 2022. I hope it will be recorded. Here is my talk about actyx from IPFS camp 2019: https://www.youtube.com/watch?v=Qzu0xtCT-R0 https://www.youtube.com/watch?v=Qzu0xtCT-R0 . It does not go into much technical details, and it does not describe the current architecture but a previous iteration though.

stavros 4y ago

This isn't compatible with go-ipfs, right? I can't run this as a node and connect to IPFS the network?

rklaehn 4y ago
ipfs-embed is using rust-libp2p and official protocols like gossipsub. So for that it is fully compatible with the ipfs network. At least as compatible as you can get when the specs are sometimes vague. Bitswap has a compat mode, but when two ipfs-embed versions talk to each other they use a different protocol. Note that ipfs-embed is made for a different use case than the global ipfs. It is typically used by private swarms where there are a smaller number of nodes. So e.g. we don't use the DHT at all. We also found that go-ipfs compat is not that useful for our use case, so the compat layer was mainly done as a service to the community and might not work with the latest go-ipfs / kubo.
- stavros 4y ago
  Makes sense, thanks! I'm not sure what not using the DHT entails, does that mean you can't discover content? That can't be it, so how does that happen? Is it just gossip?
  - rklaehn 4y ago
    Yes, pretty much. Basically the nodes of the swarm (around 100 at most) gossip events to each other. That is also how they learn about the presence of the other nodes. They try to stay connected as well as the network topology allows. Then when you want some bulk data via bitswap they just ask their current peers. Either the emitter of the data is a current peer, then it obviously has it. Or eventually one of your peers will replicate the data, and then you will get it with some delay. The latter happens frequently if a part of the swarm is behind a NAT. You have some delay, but eventually everybody gets all events.