10 ms·

Design and evaluation of IPFS: a storage layer for the decentralized web

193 points by dennis-tra 4y ago

truth_seeker 4y ago
> The content retrieval process across all regions takes 2.90 s, 4.34 s, and 4.74 s in the 50th, 90th, and 95th percentiles Good improvement over the years but still a long way to go feel it even soft real time. Not sure these servers are using on the fly gzip compression before sending over network but they should consider adding compression feature at file or block level natively in "ipfs add" command. There was an interesting paper "Hadoop on IPFS" (around year 2016-17). I hope these continuous improvement will play good role in making big data and analytics decentralised before it hits v1.0 https://s3-ap-southeast-2.amazonaws.com/scott-brisbane-thesis/decentralising-big-data-processing.pdf https://s3-ap-southeast-2.amazonaws.com/scott-brisbane-thesi...
- IanCal 4y ago
  I'd like to use it for analytics. One issue I hit was when I was testing things out for that use I added a file to several pinning services and the result wasn't available via my node or the ipfs main gateway for hours.
morsch 4y ago
I keep confusing IPFS with the (much older) I2P. Just in case someone else makes that mistake.
wespiser_2018 4y ago
I'd like to image a world with globally addressable devices, maybe that's through NAT hole punching or just IPv6 where we can share everything, but the way things are now, I really struggle to determine what's the legitimate use case for this technology if we ignore the interest in decentralization for it's own sake and just consider engineering tradeoffs like cost for performance and end user experience/ergonomics. As a storage layer, there are major challenges to adoption IPFS that have persisted almost a decade in to the project. At this level of partition comes at an incredible cost to availability, and from everything I read, the best practices for hosting user generating content still involve paying a service to "pin" your content to ensure it doesn't get dropped, so you still pay someone to host your data! So what I'd like to know, is why would I want to use IPFS to host anything, when better, more performant and cost effective alternatives exist, and IPFS doesn't guarantee a file is actually hosted? Like, are there words you can say to your boss to argue for IPFS as a rational choice in systems architecture? What is the use case here?
noasaservice 4y ago
Lets remind everyone: 1. IPFS attaches ALL network interfaces (internal and external) to your identity. 2. Tor is still "experimental" done by 3rd parties. https://flyingzumwalt.gitbooks.io/decentralized-web-primer/content/avenues-for-access/lessons/tor-transport.html https://flyingzumwalt.gitbooks.io/decentralized-web-primer/c... 3. Due to 1 and 2, any hosted content is EASILY trackable to a user's computer, even behind NATs. A machine cryptokey also helps cement that (but can be changed). This allows easy DDoS'ing of any and all endpoints hosting content you don't like. 4. It is trivial to ask the dHT for *who* has a certain content key, and get all (or the top 50?) computers hosting that content. (this matters with regards to "sensitive" content) 5. Running a node is still high cpu, ram, and network chattiness - so using a VPS to keep IPFS off your local network is still tenuous to run.
- px43 4y ago
  Yeah, doing crime on IPFS is a really bad idea. I talked with Juan Benet about this in maybe 2016 or so, and he said it's a feature, which is fair enough. Privacy is very much not the goal of IPFS, and never has been. Not only is it obvious who is hosting any given file, but if you listen actively you can usually see who is downloading any given file as well.
  - 7v3x3n3sem9vv 4y ago
    sounds like a potential privacy disaster of Google Facebook and others can just listen in on erroneously assumed privacy. I suppose the only reason Google/Facebook aren't pushing for it is it allows anyone to collect this data vs just the big players.
    - chriswarbo 4y ago
      > sounds like a potential privacy disaster of Google Facebook and others can just listen in on erroneously assumed privacy IPFS is an alternative to HTTP. Would you characterise Google's Web crawler as "listening in" and an "potential privacy disaster"? Are those paying for SEO "erroneously assum[ing] privacy"?
eminence32 4y ago
3 of the authors work for Protocol Labs (the company that develops IPFS), which is likely why the paper is able to analyze data from the ipfs.io gateways
purim 4y ago
IPFS is just too slow for it to be usable on a mass scale. It's a neat idea but unfortunately, p2p file storage is tough, you absolutely need a central model to scale up. Offering coins to cash out at casinos where the other side of the order book are people with unlimited supply of it. Doesn't work.
- yamtaddle 4y ago
  The fundamental problem with any of these p2p-hosting solutions is that most devices folks use these days are battery-powered. Any solution that requires a persistent connection or intermittent network wake-ups to achieve even semi-decent performance won't make it, and if you overcome that hurdle you still won't be able to count on most of the devices downloading content to also serve content. No-one's gonna sacrifice battery life to make the distributed Web dream happen. They seem a lot more useful for internal infra of hosting providers, though for IPFS in particular I expect the performance isn't consistent enough to be a suitable solution for most of those use cases.
  - tinalumfoil 4y ago
    Having used ipfs casually the big draw for me has been the gateways. Running a full ipfs node on client devices is mostly not practical, but I can choose a trusted (or not so trusted since I can check the hash) gateway and that can handle the heavy lifting for me. And even though the ipfs network is slow, the distributed nature lends itself to very heavy caching, so the gateway doesn't have to be any slower than if it was serving me its own static files. Plus the gateways provide compatibility to www. Given, things are far from ideal now. But looking over the water it's good to see Mastodon taking off and I think that's largely because you have the option of just choosing a single trusted provider ("instances") from which you can access the rest of the network. The trusted provider does the heavy lifting for you.
  - zozbot234 4y ago
    IPFS is built to work reasonably well with ephemeral connections. Sure, not everyone is interested in serving content but for those who are it seems to be a reasonable choice.
dartharva 4y ago
I only know IPFS from it being one of the enablers in downloading from Library Genesis, and man am I glad that it works so good for them..
- unmole 4y ago
  Is IPFS usable without the Cloudflare bridge?
  - px43 4y ago
    Yeah, there are something like four major public IPFS gateways, one run by Cloudflare, one run by Protocol Labs, or you can just run your own node, which is something that many thousands of people are doing.
    - a-user-you-like 4y ago
      Does Protocol labs use cloudflare?
      - rmorey 4y ago
        no, the ipfs.io gateway predates the cloudflare one
      - lapinot 4y ago
        no, ipfs.io seems to originate from protocol labs own network: https://bgp.he.net/AS40680 https://bgp.he.net/AS40680
- sixtyfourbits 4y ago
  It actually doesn't work so well, unfortunately. Picture this: 3 million books, at least one CID each (in practice it's often multiple, since the libgen collection uses a chunk size of 256kb). Section 3.1 of the paper talks about content publication - for each CID, a provider record is published on up to 20 different peers. Because the CIDs are derived from a high-quality hash function, they are evenly distributed. So this means that a node with a sufficient number of items ends up connecting to every single node on the network. For 3 million CIDs * 20 publication records, this means sending out 60 million publication records, every 12 hours, i.e. an average of 1388 publication records per second (assuming one CID per file, which is conservative). This is just to announce to the network "hi... just wanted to let you know I still have the same content I did yesterday". And every full replica of libgen is doing this. Another major flaw derives from the way bitswap works as discussed in Section 3.2, which states "before entering the DHT lookup, the requesting peer asks all peers it is already connected to for the desired CID". I'm not sure if that actually means all the machines the node has any type of connection to, or only those connections over which it is using bitswap. Even so, asking every peer you're connected to, even if it's only the subset for which you have a bitswap connection established, is inefficient. Compare this to bittorrent: - First there is a much coarser level of granularity; in the libgen case the torrents contain 1,000 books each, so there's much fewer announcements to the bittorrent DHT and trackers. Of course the tradeoff with this is that you can't look up the identifier of an individual book. Instead if you know the magnet link (which can be constructed from the torrent hash) and the name of the file within that torrent. - Secondly, a node hosting a large number of torrents (and a large number of active connections) will only send out want lists to the peers that it knows are also hosting that torrent. Peers also exchange have lists, and I think one or both can be represented as a bitfield for efficiency (rather than a list of CIDs/hashes). With bitswap you can end up asking every connected peer, just in case they have it. On a practical note, hosting the torrents is quite practical with adequate hardware, but even with fairly powerful machines IPFS (at least in it's current main implementation, go-ipfs aka Kubo) really struggles and can bring a machine to its knees, even when hosting only a portion of the full collection. In terms of scalability, bittorrent and IPFS are in completely different leagues. Scalability is the main reason the archive of papers from sci-hub (over 80 million) isn't available via IPFS yet, because it's just not going to be able to handle that at all in its current state. Having said all this, I should state my knowledge of the protocol and go-ipfs is incomplete, as I've only used it but not done any development work on it or dived deeply into the code. I'm happy to be corrected if I've misunderstood anything mentioned above. Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better. I don't have the answers as to how to achieve the granularity you get with IPFS vs bittorrent (which is a major point of difference, and something that sets IPFS apart in a significant way). But it's something that definitely has to be fixed to be truly capable of achieving it's stated goals.

193 points by dennis-tra 4y ago

> The content retrieval process across all regions takes 2.90 s, 4.34 s, and 4.74 s in the 50th, 90th, and 95th percentiles Good improvement over the years but still a long way to go feel it even soft real time. Not sure these servers are using on the fly gzip compression before sending over network but they should consider adding compression feature at file or block level natively in "ipfs add" command. There was an interesting paper "Hadoop on IPFS" (around year 2016-17). I hope these continuous improvement will play good role in making big data and analytics decentralised before it hits v1.0 https://s3-ap-southeast-2.amazonaws.com/scott-brisbane-thesis/decentralising-big-data-processing.pdf https://s3-ap-southeast-2.amazonaws.com/scott-brisbane-thesi...

IanCal 4y ago
I'd like to use it for analytics. One issue I hit was when I was testing things out for that use I added a file to several pinning services and the result wasn't available via my node or the ipfs main gateway for hours.

morsch 4y ago

I keep confusing IPFS with the (much older) I2P. Just in case someone else makes that mistake.

wespiser_2018 4y ago

I'd like to image a world with globally addressable devices, maybe that's through NAT hole punching or just IPv6 where we can share everything, but the way things are now, I really struggle to determine what's the legitimate use case for this technology if we ignore the interest in decentralization for it's own sake and just consider engineering tradeoffs like cost for performance and end user experience/ergonomics. As a storage layer, there are major challenges to adoption IPFS that have persisted almost a decade in to the project. At this level of partition comes at an incredible cost to availability, and from everything I read, the best practices for hosting user generating content still involve paying a service to "pin" your content to ensure it doesn't get dropped, so you still pay someone to host your data! So what I'd like to know, is why would I want to use IPFS to host anything, when better, more performant and cost effective alternatives exist, and IPFS doesn't guarantee a file is actually hosted? Like, are there words you can say to your boss to argue for IPFS as a rational choice in systems architecture? What is the use case here?

noasaservice 4y ago

Lets remind everyone: 1. IPFS attaches ALL network interfaces (internal and external) to your identity. 2. Tor is still "experimental" done by 3rd parties. https://flyingzumwalt.gitbooks.io/decentralized-web-primer/content/avenues-for-access/lessons/tor-transport.html https://flyingzumwalt.gitbooks.io/decentralized-web-primer/c... 3. Due to 1 and 2, any hosted content is EASILY trackable to a user's computer, even behind NATs. A machine cryptokey also helps cement that (but can be changed). This allows easy DDoS'ing of any and all endpoints hosting content you don't like. 4. It is trivial to ask the dHT for *who* has a certain content key, and get all (or the top 50?) computers hosting that content. (this matters with regards to "sensitive" content) 5. Running a node is still high cpu, ram, and network chattiness - so using a VPS to keep IPFS off your local network is still tenuous to run.

px43 4y ago
Yeah, doing crime on IPFS is a really bad idea. I talked with Juan Benet about this in maybe 2016 or so, and he said it's a feature, which is fair enough. Privacy is very much not the goal of IPFS, and never has been. Not only is it obvious who is hosting any given file, but if you listen actively you can usually see who is downloading any given file as well.
- 7v3x3n3sem9vv 4y ago
  sounds like a potential privacy disaster of Google Facebook and others can just listen in on erroneously assumed privacy. I suppose the only reason Google/Facebook aren't pushing for it is it allows anyone to collect this data vs just the big players.
  - chriswarbo 4y ago
    > sounds like a potential privacy disaster of Google Facebook and others can just listen in on erroneously assumed privacy IPFS is an alternative to HTTP. Would you characterise Google's Web crawler as "listening in" and an "potential privacy disaster"? Are those paying for SEO "erroneously assum[ing] privacy"?

eminence32 4y ago

3 of the authors work for Protocol Labs (the company that develops IPFS), which is likely why the paper is able to analyze data from the ipfs.io gateways

purim 4y ago

IPFS is just too slow for it to be usable on a mass scale. It's a neat idea but unfortunately, p2p file storage is tough, you absolutely need a central model to scale up. Offering coins to cash out at casinos where the other side of the order book are people with unlimited supply of it. Doesn't work.

yamtaddle 4y ago
The fundamental problem with any of these p2p-hosting solutions is that most devices folks use these days are battery-powered. Any solution that requires a persistent connection or intermittent network wake-ups to achieve even semi-decent performance won't make it, and if you overcome that hurdle you still won't be able to count on most of the devices downloading content to also serve content. No-one's gonna sacrifice battery life to make the distributed Web dream happen. They seem a lot more useful for internal infra of hosting providers, though for IPFS in particular I expect the performance isn't consistent enough to be a suitable solution for most of those use cases.
- tinalumfoil 4y ago
  Having used ipfs casually the big draw for me has been the gateways. Running a full ipfs node on client devices is mostly not practical, but I can choose a trusted (or not so trusted since I can check the hash) gateway and that can handle the heavy lifting for me. And even though the ipfs network is slow, the distributed nature lends itself to very heavy caching, so the gateway doesn't have to be any slower than if it was serving me its own static files. Plus the gateways provide compatibility to www. Given, things are far from ideal now. But looking over the water it's good to see Mastodon taking off and I think that's largely because you have the option of just choosing a single trusted provider ("instances") from which you can access the rest of the network. The trusted provider does the heavy lifting for you.
- zozbot234 4y ago
  IPFS is built to work reasonably well with ephemeral connections. Sure, not everyone is interested in serving content but for those who are it seems to be a reasonable choice.

dartharva 4y ago

I only know IPFS from it being one of the enablers in downloading from Library Genesis, and man am I glad that it works so good for them..

unmole 4y ago
Is IPFS usable without the Cloudflare bridge?
- px43 4y ago
  Yeah, there are something like four major public IPFS gateways, one run by Cloudflare, one run by Protocol Labs, or you can just run your own node, which is something that many thousands of people are doing.
  - a-user-you-like 4y ago
    Does Protocol labs use cloudflare?
    - rmorey 4y ago
      no, the ipfs.io gateway predates the cloudflare one
    - lapinot 4y ago
      no, ipfs.io seems to originate from protocol labs own network: https://bgp.he.net/AS40680 https://bgp.he.net/AS40680
sixtyfourbits 4y ago
It actually doesn't work so well, unfortunately. Picture this: 3 million books, at least one CID each (in practice it's often multiple, since the libgen collection uses a chunk size of 256kb). Section 3.1 of the paper talks about content publication - for each CID, a provider record is published on up to 20 different peers. Because the CIDs are derived from a high-quality hash function, they are evenly distributed. So this means that a node with a sufficient number of items ends up connecting to every single node on the network. For 3 million CIDs * 20 publication records, this means sending out 60 million publication records, every 12 hours, i.e. an average of 1388 publication records per second (assuming one CID per file, which is conservative). This is just to announce to the network "hi... just wanted to let you know I still have the same content I did yesterday". And every full replica of libgen is doing this. Another major flaw derives from the way bitswap works as discussed in Section 3.2, which states "before entering the DHT lookup, the requesting peer asks all peers it is already connected to for the desired CID". I'm not sure if that actually means all the machines the node has any type of connection to, or only those connections over which it is using bitswap. Even so, asking every peer you're connected to, even if it's only the subset for which you have a bitswap connection established, is inefficient. Compare this to bittorrent: - First there is a much coarser level of granularity; in the libgen case the torrents contain 1,000 books each, so there's much fewer announcements to the bittorrent DHT and trackers. Of course the tradeoff with this is that you can't look up the identifier of an individual book. Instead if you know the magnet link (which can be constructed from the torrent hash) and the name of the file within that torrent. - Secondly, a node hosting a large number of torrents (and a large number of active connections) will only send out want lists to the peers that it knows are also hosting that torrent. Peers also exchange have lists, and I think one or both can be represented as a bitfield for efficiency (rather than a list of CIDs/hashes). With bitswap you can end up asking every connected peer, just in case they have it. On a practical note, hosting the torrents is quite practical with adequate hardware, but even with fairly powerful machines IPFS (at least in it's current main implementation, go-ipfs aka Kubo) really struggles and can bring a machine to its knees, even when hosting only a portion of the full collection. In terms of scalability, bittorrent and IPFS are in completely different leagues. Scalability is the main reason the archive of papers from sci-hub (over 80 million) isn't available via IPFS yet, because it's just not going to be able to handle that at all in its current state. Having said all this, I should state my knowledge of the protocol and go-ipfs is incomplete, as I've only used it but not done any development work on it or dived deeply into the code. I'm happy to be corrected if I've misunderstood anything mentioned above. Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better. I don't have the answers as to how to achieve the granularity you get with IPFS vs bittorrent (which is a major point of difference, and something that sets IPFS apart in a significant way). But it's something that definitely has to be fixed to be truly capable of achieving it's stated goals.