13 ms·

IPFS, The Interplanetary File System, Simply Explained

261 points by acob 8y ago

sixhobbits 8y ago
https://web.archive.org/web/20181210191657/https://achainofblocks.com/2018/10/05/ipfs-interplanetary-file-system-simply-explained/ https://web.archive.org/web/20181210191657/https://achainofb...
mStreamTeam 8y ago
Can someone explain to me how IPFS is superior to BitTorrent from a usability perspective? I get all the additional functionality IPFS comes with, but it just feels too complex to get adopted. I can go out right now and start interacting with BitTorrent networks, but I don't even know where to get started with an IPFS system
- perfmode 8y ago
  There is a single global swarm.
  - silotis 8y ago
    I'm much more familiar with Bittorrent than IPFS, but as I understand it IPFS treats each object as its own swarm. This is quite the opposite of having a single global swarm. In Bittorrent terms it's as if each piece and the torrent itself are all separate swarms. What makes IFPS more global is that they have defined a set of standards for linking objects together to form potentially complex structures. The holy grail is to scale this up to create a whole new World Wide Web based on linked IFPS objects rather than linking to resources on specific hosts.
    - setr 8y ago
      Its still not clear to me if its true, but my impression from reading the whitepaper a couple years back was that you could fetch hashblocks for your target file, from any arbitrary node that happens to be serving it, regardless of what the original target was. Eg a block composed purely of white pixels can be equally shared across thousand of gifs, independent of the original file. Which if true would make it more of a global swarm Oddly, it would also imply you could potentially generate entirely new files out of the network; in an infinitely large network, you wouldnt even have to specifically upload a new file — the hashtree would be enough, and the network would find the pieces to compose it Never got anyone to confirm/deny my understanding of it, and its not clear to me why it wouldnt be the case, if it weren’t.
      - krrrh 8y ago
        Given that there would never be an infinitely large network, how likely is it that this sort of de-duplication would offer benefits? We use hashes to ensure data integrity because if a single bit is changed in the data the hash will reliably change. GIF compression already reduces "256 white pixels in a row" to a few bytes (other compression algorithms aren't so straightforward). Even with all the white gifs people are sharing online, you'd need vanishingly small block sizes for this strategy to achieve anything measurable, approaching the point where the hash doesn't just represent the block, but is the block.
        setr 8y ago
        It's more of it's not clear to me why you wouldn't implement it in this fashion, my impression from the paper was that it was implemented in this fashion, and as a side-effect of that implementation, this kind of de-duplication would be possible somewhat by accident. More specifically, its not clear to me what benefit exists to associating a block with its originating file; my understanding of bittorrent requires you to receive/send out the total list of blocks anyways (the .torrent file), as you're requesting the missing blocks (and at start, thats all blocks), and others are responding with blocks the have to offer (which may not be all blocks, since you can request from partially-finished peers aka leechers). In IPFS, you don't have an associated server (a tracker) to tell you who your peers are for a given file, so first you'll ask around for the list of blocks required for that file, and then you'll ask anyone who responded for your missing blocks, same as bittorrent. But you'll have to keep asking for new peers, since you aren't being updated by the server, and sending out the list of missing blocks all the same. It seems to me that there's not much benefit to requesting specifically the file, and then the blocks, rather than just requesting the blocks directly, when doing that updated request. And if you're asking for the blocks directly... then that kind of de-duplication would be a natural outcome. Also a spot where it might actually be impactful today -- Memes. And if its truly an interplanetary mindset, there might come a point long into the future where such a feature would be an actual feature, rather than an accident. But regardless, I only mean it as an implication of how I believe IPFS is implemented, not an intended benefit/feature of it. I specifically mean it as an easy way to point out where my understanding failed, if I'm wrong; or an easy way to verify my understanding is correct ;)
- dana321 8y ago
  https://d.tube https://d.tube is probably the best open example of ipfs thats fairly easy to use. Otherwise, its first and foremost for geeks, then other geeks implement apps for the normals using the technology.
  - acob 8y ago
    D Tube actually runs on top of Steem Blockchain and uses IPFS as a Filesystem. This is similar to how Tron will work.
    - acob 8y ago
      https://achainofblocks.com/2018/08/22/explaining-steem-blockchain-steemit-2018/ https://achainofblocks.com/2018/08/22/explaining-steem-block...
    - dana321 8y ago
      I know that, you don't need steem to view videos on the site, so its a good accessible example of an easy use site that uses ipfs.
  - rgardaphe 8y ago
    We (qri) have built a front-end webapp (free & open source) for publishing, sharing, and versioning datasets on IPFS. When you download qri, you're working on IPFS (distributed web). Still very geeky, but we're trying to make sure it's very simple to use for data scientists/analysts/researchers alike. here's more on us: https://qri.io https://qri.io & https://github.com/qri-io https://github.com/qri-io
- espadrine 8y ago
  BitTorrent started out without a DHT, and so requires metadata and a torrent file to start downloading. IPFS has a unique, global DHT as a matter of principle, and so, only requires the hash of the file to start downloading. Everyone gets the same bootstrap nodes, and then is part of the same DHT, with a common lookup mechanism relying on Kademlia. On the other hand, I feel like IPFS is not as advanced as dat in relevant ways. IPFS is not good for file edition, while dat has systems that simplify modification of even large files and merging of structured files.
  - antocv 8y ago
    Bittorrent transport layer, peer-to-peer is not encrypted, neither was IPFS last time I checked. Is it now? Has that changed? There is nothing stopping Bittorrent clients from agreeing on "this dht will be global", and similary, what stops IPFS clients/peers from forming a different DHT? Nothing as far as I know, not like scuttlebutt which has a transport-layer network-wide key. What about trackers, bittorrent can but does not work well without trackers, so its not really according to me, decentralized, since some peer is more equal than another peer. IPFS is meh, datproject is where its at.
    - kaffee 8y ago
      IPFS has perhaps 2,000 times the resources ($) of dat. (Raised via the filecoin sale.) Surely that must count for a little.
      - willio58 8y ago
        It should, but it might end up not.
      - antocv 8y ago
        Considering how much resources, github starts, and fanbois IPFS has, and yet the network layer is still shit, bandwidth not utilized to fullest. Instead of just reusing bittorrents piece-per-peer-exchange-protocol (that greedy not perfect algorithm, forgot its name) - which can saturate even high end links, they invented bitswap, full with bugs, then decided lets make filecoin instead, like a bartering engine of exchanging pieces. datproject also can saturate my 250mbit connection.
- user2426679 8y ago
  They're not mutually exclusive. There is a popular private tracker that uses IPFS to host screenshot previews, while serving the actual videos via torrent.
- StavrosK 8y ago
  IPFS is like if BitTorrent had a single swarm and the hash chose a set of files to download from that.
- jstanley 8y ago
  This should get you started: https://docs.ipfs.io/introduction/usage/ https://docs.ipfs.io/introduction/usage/ It's quick and easy. I recommend the browser extensions.
  - acob 8y ago
    Excellent resource, thank you for sharing.
- jchanimal 8y ago
  I've started using the `ipfs add` command to share files that are too big to email. The users can just download the file from a gateway like ipfs.io without installing anything, and the file is only hosted on my laptop.
- deleted 8y ago
  [deleted]
mattdemon 8y ago
Previous HN discussion on IPFS whitepaper and protocol https://news.ycombinator.com/item?id=16430742 https://news.ycombinator.com/item?id=16430742 Based on the review at https://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-content-addressed.html https://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-...
jonah-archive 8y ago
For interested Bay Area folks, we're hosting our monthly decentralized web meetup at the Internet Archive tomorrow night, and Jeremy Johnson from IPFS will be discussing some of their recent stuff. Details and tickets here: https://www.eventbrite.com/e/decentralized-web-meet-up-tickets-52509395014 https://www.eventbrite.com/e/decentralized-web-meet-up-ticke... (we ask for a small donation to cover food & facilities, but if this is a hardship for you, e-mail me and I can help out)
- acob 8y ago
  Wish I could make that. If someone gets a video, can you please post the link back on this thread?
- hinkley 8y ago
  > For interested Bay Area folks, we're hosting our monthly decentralized web meetup For some reason this reminded me of the scene from Stranger Than Fiction where she's teasing Harold, "Anarchists have a group? They assemble? Wouldn't that defeat the whole purpose?" (I don't suppose I know how you would handle a distributed meetup, unless IPFS has some facilities I'm unfamiliar with...)
  - aloisdg 8y ago
    By using a decentralized communication protocol like matrix? matrix: https://matrix.org/blog/home/ https://matrix.org/blog/home/
    - acct1771 8y ago
      Client: www.riot.im Not affiliated, just a fan plugging.
- rhizome 8y ago
  Jeremy Johnson from IPFS will be discussing some of their recent stuff. Heh, emphasis on "some," since he's only talking for 15min. ;)
pmarreck 8y ago
Are there any easy-to-use clients that interface with IPFS? Is it possible to put a site on it that does any kind of computation that is not frontend JS or that stores state in some fashion?
- rgardaphe 8y ago
  We (qri) have built a front-end webapp (free & open source) for publishing, sharing, and versioning datasets on IPFS. When you download qri, you're working on IPFS (distributed web). here's more on us: https://qri.io https://qri.io & https://github.com/qri-io https://github.com/qri-io
dymk 8y ago
IPFS "solves" the problem of distributing static files - basically, a CDN built on content addressable storage. But if you don't have nodes pinning your content, or you're going to loose that data. The article bills it as some alternative to HTTP, but it's a subset of functionality. You can't do dynamic pages with IPFS. There's no concept of cookies, so stateful sessions aren't a thing. Why do articles insist on comparing IPFS to "the web", whatever that means? It's a distributed filesystem. Not at all representative of most content on "the web".
- cjslep 8y ago
  The original "web" was built to serve files, with a special kind of markup called Hyper Text that could reference another file with a Hyper Link. Browsers today will still treat servers like a filesystem if it's old enough. It took a decade to get the JavaScript cruft and layers of browser features that basically makes the modern browser a native gaming-like app with a gigantic scripting engine and a fat pipe internet connection. It's possible iterations on IPFS could parallel the current web. But I am not a fortune teller. I just wanted to be sure we all understand where the web came from: serving files as if a gigantic connected filesystem.
  - zaarn 8y ago
    Browsers don't treat any server like a filesystem. If you see a directory index; that is generated purely on the server.
    - CompuHacker 8y ago
      In the process of researching and writing a one-line quip about FTP and Gopher, I found that Google Chrome will tell you not to enter credit card details or sensitive information over the insecure connection... to an FTP server. That being said, FTP though.
- jillesvangurp 8y ago
  The web is also started out as a distributed filesystem; it's just not replicated (unless you use a cdn). In the begining the web had no cookies, javascript, animated gifs, etc. So, ipfs adds resilience and scaling to the web. Using ipfs for serving up static content instead of a traditional http server can make a lot of sense. Long term browsers might learn to interface with ipfs directly and simply specify hashes of what to fetch in their dynamic content. Eventually that too might be content addressed. If you think about a modern docker based system, basically all you tell your server is an image name and a version tag that points to some content hash of the image it then goes and figures out what to run. Right now mostly you still get that stuff running on dedicated hardware provisioned ahead of time. But with e.g. lambda's that is less of a thing and the code gets provisioned to hardware just in time. Building a docker registry against ipfs probably isn't rocket science and it wouldn't surprise me if that hadn't been done already. Scripts in a browser are just files, so they could come from ipfs. Sessions are often ids or hashes identifying something in a DB. That could be a file in ipfs instead or in a distributed db on top of ipfs. Not saying all of that is a good idea but ipfs can be a bit more than just a file distribution mechanism.
momentoftop 8y ago
My excitement about IPFS is tempered by this issue: https://github.com/ipfs/go-ipfs/issues/3429 https://github.com/ipfs/go-ipfs/issues/3429 I was running IPFS on a VPS last year serving no content, and was consuming about half a gig a day in bandwidth. Is this a fundamental issue with the underlying concept?

261 points by acob 8y ago

https://web.archive.org/web/20181210191657/https://achainofblocks.com/2018/10/05/ipfs-interplanetary-file-system-simply-explained/ https://web.archive.org/web/20181210191657/https://achainofb...

mStreamTeam 8y ago

Can someone explain to me how IPFS is superior to BitTorrent from a usability perspective? I get all the additional functionality IPFS comes with, but it just feels too complex to get adopted. I can go out right now and start interacting with BitTorrent networks, but I don't even know where to get started with an IPFS system

perfmode 8y ago
There is a single global swarm.
- silotis 8y ago
  I'm much more familiar with Bittorrent than IPFS, but as I understand it IPFS treats each object as its own swarm. This is quite the opposite of having a single global swarm. In Bittorrent terms it's as if each piece and the torrent itself are all separate swarms. What makes IFPS more global is that they have defined a set of standards for linking objects together to form potentially complex structures. The holy grail is to scale this up to create a whole new World Wide Web based on linked IFPS objects rather than linking to resources on specific hosts.
  - setr 8y ago
    Its still not clear to me if its true, but my impression from reading the whitepaper a couple years back was that you could fetch hashblocks for your target file, from any arbitrary node that happens to be serving it, regardless of what the original target was. Eg a block composed purely of white pixels can be equally shared across thousand of gifs, independent of the original file. Which if true would make it more of a global swarm Oddly, it would also imply you could potentially generate entirely new files out of the network; in an infinitely large network, you wouldnt even have to specifically upload a new file — the hashtree would be enough, and the network would find the pieces to compose it Never got anyone to confirm/deny my understanding of it, and its not clear to me why it wouldnt be the case, if it weren’t.
    - krrrh 8y ago
      Given that there would never be an infinitely large network, how likely is it that this sort of de-duplication would offer benefits? We use hashes to ensure data integrity because if a single bit is changed in the data the hash will reliably change. GIF compression already reduces "256 white pixels in a row" to a few bytes (other compression algorithms aren't so straightforward). Even with all the white gifs people are sharing online, you'd need vanishingly small block sizes for this strategy to achieve anything measurable, approaching the point where the hash doesn't just represent the block, but is the block.
      - setr 8y ago
        It's more of it's not clear to me why you wouldn't implement it in this fashion, my impression from the paper was that it was implemented in this fashion, and as a side-effect of that implementation, this kind of de-duplication would be possible somewhat by accident. More specifically, its not clear to me what benefit exists to associating a block with its originating file; my understanding of bittorrent requires you to receive/send out the total list of blocks anyways (the .torrent file), as you're requesting the missing blocks (and at start, thats all blocks), and others are responding with blocks the have to offer (which may not be all blocks, since you can request from partially-finished peers aka leechers). In IPFS, you don't have an associated server (a tracker) to tell you who your peers are for a given file, so first you'll ask around for the list of blocks required for that file, and then you'll ask anyone who responded for your missing blocks, same as bittorrent. But you'll have to keep asking for new peers, since you aren't being updated by the server, and sending out the list of missing blocks all the same. It seems to me that there's not much benefit to requesting specifically the file, and then the blocks, rather than just requesting the blocks directly, when doing that updated request. And if you're asking for the blocks directly... then that kind of de-duplication would be a natural outcome. Also a spot where it might actually be impactful today -- Memes. And if its truly an interplanetary mindset, there might come a point long into the future where such a feature would be an actual feature, rather than an accident. But regardless, I only mean it as an implication of how I believe IPFS is implemented, not an intended benefit/feature of it. I specifically mean it as an easy way to point out where my understanding failed, if I'm wrong; or an easy way to verify my understanding is correct ;)
dana321 8y ago
https://d.tube https://d.tube is probably the best open example of ipfs thats fairly easy to use. Otherwise, its first and foremost for geeks, then other geeks implement apps for the normals using the technology.
- acob 8y ago
  D Tube actually runs on top of Steem Blockchain and uses IPFS as a Filesystem. This is similar to how Tron will work.
  - acob 8y ago
    https://achainofblocks.com/2018/08/22/explaining-steem-blockchain-steemit-2018/ https://achainofblocks.com/2018/08/22/explaining-steem-block...
  - dana321 8y ago
    I know that, you don't need steem to view videos on the site, so its a good accessible example of an easy use site that uses ipfs.
- rgardaphe 8y ago
  We (qri) have built a front-end webapp (free & open source) for publishing, sharing, and versioning datasets on IPFS. When you download qri, you're working on IPFS (distributed web). Still very geeky, but we're trying to make sure it's very simple to use for data scientists/analysts/researchers alike. here's more on us: https://qri.io https://qri.io & https://github.com/qri-io https://github.com/qri-io
espadrine 8y ago
BitTorrent started out without a DHT, and so requires metadata and a torrent file to start downloading. IPFS has a unique, global DHT as a matter of principle, and so, only requires the hash of the file to start downloading. Everyone gets the same bootstrap nodes, and then is part of the same DHT, with a common lookup mechanism relying on Kademlia. On the other hand, I feel like IPFS is not as advanced as dat in relevant ways. IPFS is not good for file edition, while dat has systems that simplify modification of even large files and merging of structured files.
- antocv 8y ago
  Bittorrent transport layer, peer-to-peer is not encrypted, neither was IPFS last time I checked. Is it now? Has that changed? There is nothing stopping Bittorrent clients from agreeing on "this dht will be global", and similary, what stops IPFS clients/peers from forming a different DHT? Nothing as far as I know, not like scuttlebutt which has a transport-layer network-wide key. What about trackers, bittorrent can but does not work well without trackers, so its not really according to me, decentralized, since some peer is more equal than another peer. IPFS is meh, datproject is where its at.
  - kaffee 8y ago
    IPFS has perhaps 2,000 times the resources ($) of dat. (Raised via the filecoin sale.) Surely that must count for a little.
    - willio58 8y ago
      It should, but it might end up not.
    - antocv 8y ago
      Considering how much resources, github starts, and fanbois IPFS has, and yet the network layer is still shit, bandwidth not utilized to fullest. Instead of just reusing bittorrents piece-per-peer-exchange-protocol (that greedy not perfect algorithm, forgot its name) - which can saturate even high end links, they invented bitswap, full with bugs, then decided lets make filecoin instead, like a bartering engine of exchanging pieces. datproject also can saturate my 250mbit connection.
user2426679 8y ago
They're not mutually exclusive. There is a popular private tracker that uses IPFS to host screenshot previews, while serving the actual videos via torrent.
StavrosK 8y ago
IPFS is like if BitTorrent had a single swarm and the hash chose a set of files to download from that.
jstanley 8y ago
This should get you started: https://docs.ipfs.io/introduction/usage/ https://docs.ipfs.io/introduction/usage/ It's quick and easy. I recommend the browser extensions.
- acob 8y ago
  Excellent resource, thank you for sharing.
jchanimal 8y ago
I've started using the `ipfs add` command to share files that are too big to email. The users can just download the file from a gateway like ipfs.io without installing anything, and the file is only hosted on my laptop.
deleted 8y ago
[deleted]

mattdemon 8y ago

Previous HN discussion on IPFS whitepaper and protocol https://news.ycombinator.com/item?id=16430742 https://news.ycombinator.com/item?id=16430742 Based on the review at https://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-content-addressed.html https://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-...

jonah-archive 8y ago

For interested Bay Area folks, we're hosting our monthly decentralized web meetup at the Internet Archive tomorrow night, and Jeremy Johnson from IPFS will be discussing some of their recent stuff. Details and tickets here: https://www.eventbrite.com/e/decentralized-web-meet-up-tickets-52509395014 https://www.eventbrite.com/e/decentralized-web-meet-up-ticke... (we ask for a small donation to cover food & facilities, but if this is a hardship for you, e-mail me and I can help out)

acob 8y ago
Wish I could make that. If someone gets a video, can you please post the link back on this thread?
hinkley 8y ago
> For interested Bay Area folks, we're hosting our monthly decentralized web meetup For some reason this reminded me of the scene from Stranger Than Fiction where she's teasing Harold, "Anarchists have a group? They assemble? Wouldn't that defeat the whole purpose?" (I don't suppose I know how you would handle a distributed meetup, unless IPFS has some facilities I'm unfamiliar with...)
- aloisdg 8y ago
  By using a decentralized communication protocol like matrix? matrix: https://matrix.org/blog/home/ https://matrix.org/blog/home/
  - acct1771 8y ago
    Client: www.riot.im Not affiliated, just a fan plugging.
rhizome 8y ago
Jeremy Johnson from IPFS will be discussing some of their recent stuff. Heh, emphasis on "some," since he's only talking for 15min. ;)

pmarreck 8y ago

Are there any easy-to-use clients that interface with IPFS? Is it possible to put a site on it that does any kind of computation that is not frontend JS or that stores state in some fashion?

rgardaphe 8y ago
We (qri) have built a front-end webapp (free & open source) for publishing, sharing, and versioning datasets on IPFS. When you download qri, you're working on IPFS (distributed web). here's more on us: https://qri.io https://qri.io & https://github.com/qri-io https://github.com/qri-io

dymk 8y ago

IPFS "solves" the problem of distributing static files - basically, a CDN built on content addressable storage. But if you don't have nodes pinning your content, or you're going to loose that data. The article bills it as some alternative to HTTP, but it's a subset of functionality. You can't do dynamic pages with IPFS. There's no concept of cookies, so stateful sessions aren't a thing. Why do articles insist on comparing IPFS to "the web", whatever that means? It's a distributed filesystem. Not at all representative of most content on "the web".

cjslep 8y ago
The original "web" was built to serve files, with a special kind of markup called Hyper Text that could reference another file with a Hyper Link. Browsers today will still treat servers like a filesystem if it's old enough. It took a decade to get the JavaScript cruft and layers of browser features that basically makes the modern browser a native gaming-like app with a gigantic scripting engine and a fat pipe internet connection. It's possible iterations on IPFS could parallel the current web. But I am not a fortune teller. I just wanted to be sure we all understand where the web came from: serving files as if a gigantic connected filesystem.
- zaarn 8y ago
  Browsers don't treat any server like a filesystem. If you see a directory index; that is generated purely on the server.
  - CompuHacker 8y ago
    In the process of researching and writing a one-line quip about FTP and Gopher, I found that Google Chrome will tell you not to enter credit card details or sensitive information over the insecure connection... to an FTP server. That being said, FTP though.
jillesvangurp 8y ago
The web is also started out as a distributed filesystem; it's just not replicated (unless you use a cdn). In the begining the web had no cookies, javascript, animated gifs, etc. So, ipfs adds resilience and scaling to the web. Using ipfs for serving up static content instead of a traditional http server can make a lot of sense. Long term browsers might learn to interface with ipfs directly and simply specify hashes of what to fetch in their dynamic content. Eventually that too might be content addressed. If you think about a modern docker based system, basically all you tell your server is an image name and a version tag that points to some content hash of the image it then goes and figures out what to run. Right now mostly you still get that stuff running on dedicated hardware provisioned ahead of time. But with e.g. lambda's that is less of a thing and the code gets provisioned to hardware just in time. Building a docker registry against ipfs probably isn't rocket science and it wouldn't surprise me if that hadn't been done already. Scripts in a browser are just files, so they could come from ipfs. Sessions are often ids or hashes identifying something in a DB. That could be a file in ipfs instead or in a distributed db on top of ipfs. Not saying all of that is a good idea but ipfs can be a bit more than just a file distribution mechanism.

momentoftop 8y ago

My excitement about IPFS is tempered by this issue: https://github.com/ipfs/go-ipfs/issues/3429 https://github.com/ipfs/go-ipfs/issues/3429 I was running IPFS on a VPS last year serving no content, and was consuming about half a gig a day in bandwidth. Is this a fundamental issue with the underlying concept?