20 ms·

"Even if my server goes down, as long as these files are pinned somewhere, anyone should be able to use that IPNS name at another gateway and see the blog." Th

"Even if my server goes down, as long as these files are pinned somewhere, anyone should be able to use that IPNS name at another gateway and see the blog." This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. And why would they? I understand that IPFS is supposed to prioritize connections based on seed/leach ratio like a typical torrent service, but I only torrent a few files at any given time and it's pretty trivial to set them to seed or disable manually; there's no way I'm going to make seed vs. don't seed decisions on every website I visit. So unless for some reason I specifically think to help share some webpage or other, it'll just get auto-deleted from my machine when it cycles out of the cache, as the automated stuff is what'll be keeping track of maintaining my seed/leech ratio, total disk usage for other people's content, upload limits, not leaving if you're one of the only seeders left, etc. In theory, IPFS could be even more prone to link-rot than the vanilla Web, depending on how many people try and actually depend on the decentralized hosting and end up having their files vanish once it's been a long enough time that nobody's still hosting them. And when there's so many different webpages, as opposed to just a few torrents, why would I think to pin any one thing in particular? Torrents can work based on charity and the need to maintain a particular seed/leech ratio; but I don't have the mental energy to bother deciding whether to be charitable about every dang website I visit. So this ends up meaning that IPFS works for popular, recent content, where there's enough people who have downloaded the content themselves recently enough that it's still in their cache to meaningfully take the load off the original host in serving that content. But you're not going to get dedicated long-term seeders of any particular site the way you do with highly-desirable files like pirate torrents. But generically, you will always need your own centralized server for any content you want to upload and make sure stays online long-term. As I understand it, this is sort of the problem Filecoin is trying to solve, but that has its own issues (it's hard to see how paying people to host your stuff on their own machines can ever be cost-competitive with paying AWS to do it).

MrEldritch 8y ago
Basically - why would anyone else have your files pinned somewhere?
- crankylinuxuser 8y ago
  Because the network can pay them, if you pay in a kiddy. Its called FileCoin. And its the answer to what the grandparent says.
- asdkhadsj 8y ago
  It's not just pinning though, is it? This is like saying "why would anyone host your file on BitTorrent", but the reality is that no one has to host your file on BitTorrent. Downloading naturally seeds[1]. Unless something has changed with the pitch, this is the same with IPFS. Not only do people seed while downloading, but it will (eventually) even seed locally first, such as within the same network. IPFS promises nothing, nor tries to, in the way of permanent archiving. It's about reducing congestion and gaining benefits of immutability. It could archive something in the sense that something popular is difficult to remove.. but that's definitely not a guarantee. [1]: ignoring aggressive non-sharing downloaders of course
  - stephengillie 8y ago
    [1]: ignoring human nature of course ftfy
  - MrEldritch 8y ago
    Yeah, and that's entirely true as an explanation for why IPFS is still valuable - if you have popular content other people have accessed recently or are currently accessing, it can in theory take a ton of load off of your server by letting those people share some of the load to new users. Incredibly powerful for DDoS protection/avoiding the "Hug of Death", and potentially greatly reducing hosting costs by reducing traffic to your particular server at precisely those moments that would be most expensive - when you'd otherwise be serving lots of requests for some newly popular bit of content. It's just a mistake to view IPFS as allowing for truly "decentralized" websites or as a decentralized file storage platform - unless you have the kind of content that makes other people want to follow the typical torrent model and actively long-term "seed" it by pinning, you'll still need to have your own personal central server to host the content on if and when nobody else is. Which, yes, IPFS has explicitly never promised that - but a lot of people seem to think it does.
pervycreeper 8y ago
The original IPFS whitepaper mentions Filecoin, so that is one possible incentive, but there are many other ways to incentivize people to keep files available. Constituting a superset of the current incentives for hosting files, at the very least.
zethraeus 8y ago
> (it's hard to see how paying people to host your stuff on their own machines can ever be cost-competitive with paying AWS to do it). The counterargument would be: there are cheaper ways to mine bitcoin than on aws infra. If you can incentivize that with money, you can incentivize storage with filecoin. You also get censorship resilience (and maybe even better distribution if all goes excellently).
panic 8y ago
For web pages, you could imagine a sort of universal "like" button which pins the page locally and bookmarks it. I think it's less about incentives and more about convenience -- people are happy to help out if it's not too much trouble for them.
- groby_b 8y ago
  People don't give a damn about helping you, and will unpin any site as soon as it becomes a resource drain. And over time, they will learn that's unpredictable, and nobody wants to be a site admin, and so nobody will pin things. I thought the last 20 years on the Internet have shown us that "humanity is noble, and good" isn't true, and we should assume "evil and/or lazy".
  - stephengillie 8y ago
    Inside every cynical person is a disappointed idealist. - George Carlin.
  - deleted 8y ago
    [deleted]
  - threeseed 8y ago
    Not evil and/or lazy so much as self serving.
  - deleted 8y ago
    [deleted]
  - MrEldritch 8y ago
    Yes, but keep in mind that, like BitTorrent, IPFS also includes a thing where nodes are given preferential treatment based on seed/leech ratio - so you're incentivized to help host content, and just the stuff you can share while downloading yourself might not be enough. So yes, they will "unpin any site as soon as it becomes a resource drain" unless they're feeling particularly charitable - but that's exactly how it works with torrents, too! Generally you set up rules like "seed until a maximum up/down ratio is reached for that file, then don't bother", "restrict uploads to a certain maximum speed and/or certain maximum amount of data per day", that kind of thing. I figure you'd basically have an automatically-managed cache of limited size that you'd use to hold the stuff you were liking-to-pin, and stuff that was judged to be no longer worth seeding would get kicked off, as would old/unpopular content if the cache had filled up and you were shuffling in new content. So unpinning anything that becomes a significant resource drain is also - at least in the relatively-short-to-medium-term - fully compatible with other people being willing to "like" your site to pin it, if not perpetually, than for an extended period of time.
fenwick67 8y ago
> it's hard to see how paying people to host your stuff on their own machines can ever be cost-competitive with paying AWS to do it That's a big part of Amazon's AWS offerings, hosting other people's files on their own machines. Heck, Amazon could spin up IPFS host nodes and start earning filecoin for themselves. The goal is that file hosting becomes competitive, cheaper and more available. You won't have to choose AWS or DigitalOcean, you just throw some change at it and anybody can host it.
- s73v3r_ 8y ago
  But, why would Amazon want to do that? Right now they get a lot of brand recognition from doing it themselves.
  - ric2b 8y ago
    Because they have idle machines and there would be money to be made.
benjaminjackman 8y ago
> This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. I think of it more like email, email doesn't work either unless you pay someone to host or manage your email situation (or setup your own). It's like email in that the protocols,(multihash stuff etc) is more like an RFC describing how the system works, and thus standardizes how files are shared on the internet. Other tools (browsers etc) can build on those primitives to have a better system of caching and tooling doesn't have to keep re-inventing the algorithmic wheel in terms of how to do this stuff. Maybe IPFS isn't the address files by the hash of their bytes protocol we end up with, but I think it's likely that we will end up with something(s) that do operate in that way and unify the disparate methods of managing that stuff we use today. So as for who hosts the files you pin, I would figure you'd pay someone to do that in the near term in the same way you'd pay for S3. In the future these providers could compete on price by offering features like p2p load balancing to offer cheaper prices, CDN integration for better perf. Then in a further off future something FileCoin[1] can move it to a decentralized system (we'll see how it plays out and even if we don't get there it's still better I think than the smorgasbord we deal with now). The cool thing is between all those transitions there should be minimal disruption in how your data is stored and accessible, and it should be common between a lot of different projects. To me the big open issue is that I think IPNS isn't what I would do for what I call the labeling problem, there i'd go with something more like git type of a system. You really need a way to move the content hashes out of the urls if you want them to be human readable, then the urls themselves (which really probably ought to be urn's) should assert their own immutability, allow namespacing and versioning. So I could say something like set-label 'mutable://blog.johnsmith.com' -> "content-hash|other-label" but have that action recorded into a blockchain or publicly accessible git repo like thing pointed at by DNS and have the name resolution system replacement be able to query that chain automagically when making requests (the browser for example is aware of this system and leverages it). Further for mutable urls (again probably ought to use urns) previous versions can be requested trivially with @version e.g. "mutable:blog.johnsmith.com@3" and so on. We can do all this now ad-hoc with http headers etc, but it's all disparate and not unified like it should be, for example it doesn't tie in meaningfully with your file system. If if did then my file system on my computer at the label level (not the inode implementation level) would also use the same scheme. I write to a folder there, it automatically updates the label tree of that folder into the naming system and it's all consistent. 1: My hunch is that AWS/Azure/etc will have economies of scale too hard to beat in terms of the actual hardware but they may be eclipsed / acquire a startup that does the user facing implementation of this.
chipotle_coyote 8y ago
This is a good description of IPFS for those of us who weren't aware of it before. But, I think if it works the way you're describing here (and I assume you have it right), then it's not going to fully succeed at being the "distributed web" that it desires to be for the reasons that you outline. Having said that, I think IPFS sounds like a really good idea -- oddly, it's a lot like what I envisioned in a science fiction novel, where the notion of data being replicated across hundreds of storage nodes is so taken for granted that the main character has trouble conceiving of the notion of data that has a "location" (that is, is stored on only one device). But to get there, it needs to be largely content-agnostic: if the data is out there on the network, then it's replicated across those hundreds of storage notes, regardless of popularity. IPFS proves that technology is basically already here in theory -- but in practice, I'm not sure it's feasible in terms of storage costs/requirements yet.
- MrEldritch 8y ago
  IPFS is basically a cross between BitTorrent, plus Git's datastructure for permanently and uniquely identifying particular versions of and updates to a given piece of content, plus a clever distributed hash table for keeping track of who has what content.
  - bordercases 8y ago
    Am I the only one who finds this prospect very exciting?
    - shliachtx 8y ago
      Definitely not, hence the enthusiasm with IPFS, and Cloudflare's recent decision to launch a IPFS gateway.
- ghthor 8y ago
  As a counter argument, your standard browser already caches a ton of stuff, might as well serve some of it back up if your upstream can support it. If you replace your browser cache with IPFS, and replace bookmarking with pinning you've got a functioning system. Your hosting requirements for content will likely go down, you get massive scale redundancy and reliability for free, no more slashdot effect and you might be able to claim 100% uptime due to just how unlikely it would be that someone hits your site while your root host is down and it's so low traffic that it's not bouncing around the network, even if it's just a standard consumer phylink. You could still get 'a hug of death ' but as the host you can just sit back and let the infrastructure work, distribute and recover, ala what it looks like to be the first seeder of a torrent. I predict this is unlikely in real world situations though, as for someone to share your site they'd likely have visited it first, thus your content host is likely to not be the only host. I'm an optimist, mostly because I think is stupid where we are at concerning self hosting from our homes.
kpcyrd 8y ago
You can rephrase this sentence: > it really doesn't necessarily lead to any kind of true decentralized <data storage> unless someone else has decided to <store your data> You might have an overly romantic idea of decentralization that doesn't necessarily align with the actual definition of decentralization. I would even argue that there is no solution for the idea you're suggesting. You can't have data that isn't stored anywhere. The permanent bit in ipfs is actually referring to something else: The data isn't guaranteed to be available at all times, but the link is guaranteed to point to the correct data. Slightly anecdotal, but a while ago there was a discussion about a pdf in #cjdns. Somebody had an ipfs link, but nobody was seeding it anymore. A few hours later somebody digged out a pdf from an old archive but wasn't sure if it's the correct one, so we ran `ipfs add file.pdf` and the ipfs link started working again.
- MrEldritch 8y ago
  To be clear: I was referring to an overly romantic idea of IPFS I had actually had at one point, and which I acquired because lots of other people were talking about IPFS that way. I am aware that IPFS does not actually do this, but a lot of people talk about it like it does, like IPFS means you can have websites without really needing to have/pay for any server yourself at all. (Which is, yes, technically sort of true, but only if visitors to your site reliably pin its content.)
  - StavrosK 8y ago
    You don't have to actually pin things. Merely visiting will cache and rehost the files, and they will only be GCed at some later time. All pinning does is disable the GC for those files, which, as of a few months ago, weren't getting GCed at all anyway, so visiting something was equivalent to pinning it.
    - MrEldritch 8y ago
      "So unless for some reason I specifically think to help share some webpage or other, it'll just get auto-deleted from my machine when it cycles out of the cache" - my original post above. I was thinking about that content effectively "linkrotting" away if nobody visits it for a while, say it's old posts on a blog or something. I suppose how big of an issue this really is comes down to how long stuff would tend to stay in that cache in a real, practical usage scenario. I guess I'd been assuming that it'd only be day or so, but it could be longer in practice, depending on how much space is actually allocated to the local cache and how much each individual bit of unique content takes up and how many such unique chunks of content will be downloaded per day, along with maybe some more complex factors besides first-in-first-out for deciding what to collect. (I elaborated a little more about this below, when I thought about it kinda working like an automated torrent manager that made sure you maintained a decent seed ratio, didn't exceed an upload data cap, etc. and shuffled stuff out when it was taking up too much space or had hit a target ratio and wasn't going to be seeded anymore, with some prioritization for "is there anyone else seeding this", "is this a highly-demanded bit of content", etc.)
ghthor 8y ago
One idea I have is to execute health checks to the network from different points in ipfs node graph. For example you could add your file to your localnode(connected) then spawn some geographically distributed ephemeral modes to request the file. Once all the your emphermal nodes have recieved they wipe and you take your localnode(offline). Now the ephemeral nodes heartbeat, at some interval t, to request the file with the original source offline. If some N of them don't receive the file in some timeout, reconnect your localnode. In theory this would keep your file alive within the network for some x*t timespan, the question is what is that timespan and what's the cost of running the heartbeats.... I'm actually going to run this experiment and I'll reply here with my results.
antiffan 8y ago
This feels a bit to me like saying: "electric cars still have to charge their batteries somewhere, and they aren't necessarily being charged with renewable energy." Electric cars provide the capability to power a car with renewable energy. Powering the grid with electric energy is a separate problem. IPFS provides the capability of distribution. Incentivizing pinning is indeed a separate problem, but providing this capability feels like a huge step forward to me.
- bordercases 8y ago
  Likewise if the process of distribution was automated there would be people complaining about opt-in being a waste of their own resources for content they don't care about, or something.
  - zeta0134 8y ago
    Exactly this. I don't understand why this is being thrown about like it's a bad thing; I gladly pinned all of Wikipedia, but it's also a huge amount of data and I made sure I was prepared to follow through before I asked my IPFS node to do it. If there's content out there that has earned my respect, I'll pin it gladly just to help the authors out, especially if they ask me politely. To be fair, some of this is automated. If you browse using your own node, it does maintain a small cache, which helps with the load distribution when something gets really popular. This blog post is a good example; despite literally being the #1 item on HN, it's still up; that's the global IPFS cache at work. It's slower than the average site, sure, but it didn't outright vanish, even though the author clearly is using minimal resources to actually host it. I think that's awesome.
  - adrianN 8y ago
    More realistically it would have the same complains as services like Freenet. You are in legal limbo if you just host random stuff and it's not too unlikely to get all your equipment confiscated.
ChrisClark 8y ago
I know every mention of this seems to get downvoted because it's a cryptocurrency, but that's what the second half of the IPFS plan is, FileCoin. They've found a way to incentivize storing files by allowing people to 'mine' FileCoin. Mining in this sense isn't a waste of energy, or staking of coins. It's proving you have available space and proving you're making duplications of the data. This causes a race to the bottom for the pricing of storage while balancing it out with reliability. If you want to host your website on IPFS, pop it up on a domain through Cloudflare and set up a bid for storage space.
- MrEldritch 8y ago
  The problem with FileCoin is that the math behind it - Proof-of-Spacetime and Proof-of-Replication - just ... doesn't exist yet. That's what the delay has been, trying to figure out how to actually turn those into usable cryptographic primitives. It's not clear that a practical, scalable implementation of the system described in the FileCoin paper is actually even possible. (There's been some progress on getting proof-of-replication working, but it's still early stages and I haven't seen anything on proof-of-spacetime.) Also, the economics of FileCoin don't make a lot of sense ... https://blog.dshr.org/2018/06/the-four-most-expensive-words-in.html https://blog.dshr.org/2018/06/the-four-most-expensive-words-...
  - ChrisClark 8y ago
    I've found they've been extremely quiet for the whole year, or longer. They did recently send out an email talking about their progress and showed some demos, but I just skipped over the email actually, didn't look into it to see if they talked about their POS 'Proof of [stuff]'. I just dug it up and they say Proof of Replication is going great. But don't show anything, they promise to open the code in the 'coming months' so people might see what progress has been done if any. No progress was mentioned for Proof of Spacetime. Edit: The link to their update, the only substantial one since the ICO last year I think... https://filecoin.io/blog/update-2018-q1-q2/ https://filecoin.io/blog/update-2018-q1-q2/
    - 8y ago
sliken 8y ago
Sure, IPFS isn't a great archive of uninteresting content that needs to be kept forever. But imagine the normal thing is to give 1GB of cache to IPFS. Granted if a planet worth of 1GB cache's isn't enough to save your content then it's not interesting enough.
- MrEldritch 8y ago
  The thing I'm worried about is niche content that is very interesting/useful, but only rarely - the kind of stuff that old Web1.0 personal sites, blogs, forum threads, and university-faculty webpages can be a treasure trove of, and which is especially prone to linkrot. It is precisely this content that a "permanent Web" would be of most interest, to me, about preserving. This is the "long tail" of content - any individual piece is only rarely interesting and only to some people, but in aggregate they actually make up a substantial fraction of the value of the Web to me. So if IPFS aspires to be the "permanent Web", but is virtually useless for preserving these, it hardly seems to qualify. With a few exceptions (for very large content that would require a ton of bandwidth to download and host, or to very complex content that requires preserving the structure of a whole large website to stay functional instead of individual pages, or for inherently server-side content), the kind of stuff that would be popular enough for someone to pin virtually never vanishes from the Web - even if links to it do break - because that's the kind of stuff that has almost certainly already been saved locally & reposted to other websites. And the stuff that's served unpinned from user caches is, very clearly, not permanent. So it's a "permanent Web" ... which is, at best, barely less fragile and prone to linkrot than the current web, in that if content is popular at one point and then the site layout changes or the original host goes down entirely it can still be kept up at the same link if someone was smart enough to save it locally, and specific versions of a website can be linked to specifically (even if nobody's necessarily hosting them anymore). But in all other respects, it is exactly as ephemeral as the current Web, and the fancy decentralized parts of it that are different than the current Web are among the most ephemeral parts, while the option to still have the boring old fragile centralized-server solutions where you host your own content personally are the durable ones.
  - roywiggins 8y ago
joshfraser 8y ago
While it remains to be seen whether Filecoin will be price-competitive with AWS, IPFS has already proven itself to be far more censorship-resistant than any centralized provider can hope to offer. https://news.ycombinator.com/item?id=15367531 https://news.ycombinator.com/item?id=15367531
mirimir 8y ago
In my experience, persistence is random. "Fast Data Transfer via Tor"[0] has persisted for ~20 months. But "Fast Data Transfer via Tor (Methods)"[1] was gone soon after I took the hosting IPFS node down. 0) https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZRuf4dutN/ https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZR... 1) https://ipfs.io/ipfs/QmSp8p6d3Gxxq1mCVG85jFHMax8pSBzdAyBL2jZxCcCLBL/ https://ipfs.io/ipfs/QmSp8p6d3Gxxq1mCVG85jFHMax8pSBzdAyBL2jZ...
ezoe 8y ago
To make the data live after the original node goes offline, you have to push your data to random nodes which also push to random nodes. Your node blindly accept data, store it and upload it. There are some P2P network implementations which did that and failed horribly bad on the end. A few abuser can DoS the network. Oh and there are illegal data and numbers. I'm rather extreme freedom-of-speech advocates myself. But even for me, there are some immoral data I will never help spreading it knowingly.
misrab 8y ago
Perhaps you're confusing IPFS and Filecoin? IPFS provides the storage and retrieval layer, while Filecoin allows for incentivised markets, to your point.
mmjaa 8y ago
>And why would they? Because they're reading the blog, and are interested in the content - i.e. because next generation IPFS-friendly browsers participate in the pinning, as they should. I mean, think about it: there is already a copy of this web page out there, multiply redundant, in the form of our browser cache. All it really needs for IPFS to be viable is for the browser vendors to give the user the means to make those cached files part of their contribution to the Internet...
- sterlind 8y ago
  That would be beautiful, and hopefully realizable with a browser extension, which would get glommed into the browser core (a la Brave and Firefox) once the technology is proven. But how do you solve NAT? Most end-user devices can't expose ports, and IPs are increasingly shared. Skype was uniquely successful there, but I doubt IPFS in its current form could make use of those tricks.
  - nybble41 8y ago
    > But how do you solve NAT? The solution to NAT is IPv6.
zaarn 8y ago
I think dat:// is a much more promising protocol over IPFS. The developers don't seem to delude themselves as much to the capabilities of the archive and there is no ICO. They advertise it as it is; Torrents with updates via public keys. ipfs:// on the other hand I hear most frequently handwaving away the problems like "who will pin unpopular content" or "how far does it scale" or "who will pay for gateway bandwidth once it gets big" or even "how to make content easily update" (I don't consider IPNS a good solution to that question)
- dancek 8y ago
  I was going to post just this. While I'm very skeptical that either of them will become popular, dat seems to be more suited for publishing websites and sharing large files. For a website it's nice that you share the whole site and are able to update the content while keeping the link (it's possible to tell if the content has changed, but there is no version history). And dat is already my first pick whenever I need to copy large files privately over internet. I blogged about my experience with dat a couple of months ago. Here's a link if anyone's interested. https://hannuhartikainen.fi/blog/dat-site/ https://hannuhartikainen.fi/blog/dat-site/ Since then I've used dat for copying files a couple of times. I haven't really browsed the dat:// web and I'd guess nobody has visited my website over that protocol (but then I don't have analytics and estimate my blog to have a dozen visitors a month).
- imtringued 8y ago
  >who will pay for gateway bandwidth once it gets big That's only a problem because IPFS isn't "big" enough for browsers to ship with an IPFS gateway integrated in them.
  - zaarn 8y ago
    IPFS will need very big gateways because until that happens IPFS needs to grow a lot. And even then I doubt browsers would just ship a P2P application to autostart with the browser to the consumer.
1_player 8y ago
Is it possible to set up an IPFS caching server, which stores any content from any point in time, up to some configurable percentage of disk space? Ideally with some kind of mechanism to store poorly seeded data. I don't know how IPFS works, but I'd like to contribute by setting up and forgetting an IPFS cache on the Internet to help the network grow.
- humblebee 8y ago
  Anytime a user adds content an IPFS daemon, provide messages are sent out to the closest peers containing the cid of the newly added content. The peers will store the record, the cid of the content, and who told them about it. The peers will not request and download the content. However, you could set this up on your own. It's not a configuration option though. You would either have to build up your own system using the ipfs libraries, or monitor the events from a daemon. Unfortunately there is a bug that is not allowing the information required to be logged correctly in the events, so the actual cid of the content is not exposed. If it where, you could look for the `handleAddProvider` or `handleGetProvider` operation through the event logs (`ipfs log tail`) and then inspect the object (`ipfs object stat`) to determine if you wanted to pin it. > Ideally with some kind of mechanism to store poorly seeded data. This would be a little more difficult, but you could attempt to query the dht to determine how many peers are already providers for the data (`ipfs dht findprovs`).
amelius 8y ago
> This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. Perhaps not, technically, but it will turn hosting into a commodity. And it will remove any difference between centralized and decentralized from the user's point of view. Therefore, it's a huge win.
decentralised 8y ago
>This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. This is why IPFS (w/ Filecoin), Storj and others use cryptoeconomic mechanism design and incentive structures to incentivise hosting / pinning data.
kkarakk 8y ago
From what I've understood, IPFS decentralization = if you have enough money to host , you have a voice. if enough people like what you say, you have a voice. if either of those fail, good luck & hope the door doesn't hit you on the way out.
maxZZzzz 8y ago
>it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files You can either pay somebody to pin the files. Or a public Organization like "archive.org" decides that a site is important enough to pin it. The typical surfer will seldom pin something but can keep low freq special interest sites alive. There is no fee lunch. With HTTP it has become hard to mirror a modern site. In IPFS it is a built in feature. Makes a better Internet.
bennofs 8y ago
Having a network-connected device that you own be always available may not be such a huge problem. Many home's use a router that is always connected: this router could also serve as IPFS content hoster! I view IPFS as a better distribution network, not a better host - in contrast to current's internet structure, if many people request the same content from me, they might only need to contact my server once and then share it between them thus reducing overall load. So the point of IPFS is not to store content for eternity, but to provide efficient distribution of content.

MrEldritch 8y ago

Basically - why would anyone else have your files pinned somewhere?

crankylinuxuser 8y ago
Because the network can pay them, if you pay in a kiddy. Its called FileCoin. And its the answer to what the grandparent says.
asdkhadsj 8y ago
It's not just pinning though, is it? This is like saying "why would anyone host your file on BitTorrent", but the reality is that no one has to host your file on BitTorrent. Downloading naturally seeds[1]. Unless something has changed with the pitch, this is the same with IPFS. Not only do people seed while downloading, but it will (eventually) even seed locally first, such as within the same network. IPFS promises nothing, nor tries to, in the way of permanent archiving. It's about reducing congestion and gaining benefits of immutability. It could archive something in the sense that something popular is difficult to remove.. but that's definitely not a guarantee. [1]: ignoring aggressive non-sharing downloaders of course
- stephengillie 8y ago
  [1]: ignoring human nature of course ftfy
- MrEldritch 8y ago
  Yeah, and that's entirely true as an explanation for why IPFS is still valuable - if you have popular content other people have accessed recently or are currently accessing, it can in theory take a ton of load off of your server by letting those people share some of the load to new users. Incredibly powerful for DDoS protection/avoiding the "Hug of Death", and potentially greatly reducing hosting costs by reducing traffic to your particular server at precisely those moments that would be most expensive - when you'd otherwise be serving lots of requests for some newly popular bit of content. It's just a mistake to view IPFS as allowing for truly "decentralized" websites or as a decentralized file storage platform - unless you have the kind of content that makes other people want to follow the typical torrent model and actively long-term "seed" it by pinning, you'll still need to have your own personal central server to host the content on if and when nobody else is. Which, yes, IPFS has explicitly never promised that - but a lot of people seem to think it does.

pervycreeper 8y ago

The original IPFS whitepaper mentions Filecoin, so that is one possible incentive, but there are many other ways to incentivize people to keep files available. Constituting a superset of the current incentives for hosting files, at the very least.

zethraeus 8y ago

> (it's hard to see how paying people to host your stuff on their own machines can ever be cost-competitive with paying AWS to do it). The counterargument would be: there are cheaper ways to mine bitcoin than on aws infra. If you can incentivize that with money, you can incentivize storage with filecoin. You also get censorship resilience (and maybe even better distribution if all goes excellently).

panic 8y ago

For web pages, you could imagine a sort of universal "like" button which pins the page locally and bookmarks it. I think it's less about incentives and more about convenience -- people are happy to help out if it's not too much trouble for them.

groby_b 8y ago
People don't give a damn about helping you, and will unpin any site as soon as it becomes a resource drain. And over time, they will learn that's unpredictable, and nobody wants to be a site admin, and so nobody will pin things. I thought the last 20 years on the Internet have shown us that "humanity is noble, and good" isn't true, and we should assume "evil and/or lazy".
- stephengillie 8y ago
  Inside every cynical person is a disappointed idealist. - George Carlin.
- deleted 8y ago
  [deleted]
- threeseed 8y ago
  Not evil and/or lazy so much as self serving.
- deleted 8y ago
  [deleted]
- MrEldritch 8y ago
  Yes, but keep in mind that, like BitTorrent, IPFS also includes a thing where nodes are given preferential treatment based on seed/leech ratio - so you're incentivized to help host content, and just the stuff you can share while downloading yourself might not be enough. So yes, they will "unpin any site as soon as it becomes a resource drain" unless they're feeling particularly charitable - but that's exactly how it works with torrents, too! Generally you set up rules like "seed until a maximum up/down ratio is reached for that file, then don't bother", "restrict uploads to a certain maximum speed and/or certain maximum amount of data per day", that kind of thing. I figure you'd basically have an automatically-managed cache of limited size that you'd use to hold the stuff you were liking-to-pin, and stuff that was judged to be no longer worth seeding would get kicked off, as would old/unpopular content if the cache had filled up and you were shuffling in new content. So unpinning anything that becomes a significant resource drain is also - at least in the relatively-short-to-medium-term - fully compatible with other people being willing to "like" your site to pin it, if not perpetually, than for an extended period of time.

fenwick67 8y ago

> it's hard to see how paying people to host your stuff on their own machines can ever be cost-competitive with paying AWS to do it That's a big part of Amazon's AWS offerings, hosting other people's files on their own machines. Heck, Amazon could spin up IPFS host nodes and start earning filecoin for themselves. The goal is that file hosting becomes competitive, cheaper and more available. You won't have to choose AWS or DigitalOcean, you just throw some change at it and anybody can host it.

s73v3r_ 8y ago
But, why would Amazon want to do that? Right now they get a lot of brand recognition from doing it themselves.
- ric2b 8y ago
  Because they have idle machines and there would be money to be made.

benjaminjackman 8y ago

> This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. I think of it more like email, email doesn't work either unless you pay someone to host or manage your email situation (or setup your own). It's like email in that the protocols,(multihash stuff etc) is more like an RFC describing how the system works, and thus standardizes how files are shared on the internet. Other tools (browsers etc) can build on those primitives to have a better system of caching and tooling doesn't have to keep re-inventing the algorithmic wheel in terms of how to do this stuff. Maybe IPFS isn't the address files by the hash of their bytes protocol we end up with, but I think it's likely that we will end up with something(s) that do operate in that way and unify the disparate methods of managing that stuff we use today. So as for who hosts the files you pin, I would figure you'd pay someone to do that in the near term in the same way you'd pay for S3. In the future these providers could compete on price by offering features like p2p load balancing to offer cheaper prices, CDN integration for better perf. Then in a further off future something FileCoin[1] can move it to a decentralized system (we'll see how it plays out and even if we don't get there it's still better I think than the smorgasbord we deal with now). The cool thing is between all those transitions there should be minimal disruption in how your data is stored and accessible, and it should be common between a lot of different projects. To me the big open issue is that I think IPNS isn't what I would do for what I call the labeling problem, there i'd go with something more like git type of a system. You really need a way to move the content hashes out of the urls if you want them to be human readable, then the urls themselves (which really probably ought to be urn's) should assert their own immutability, allow namespacing and versioning. So I could say something like set-label 'mutable://blog.johnsmith.com' -> "content-hash|other-label" but have that action recorded into a blockchain or publicly accessible git repo like thing pointed at by DNS and have the name resolution system replacement be able to query that chain automagically when making requests (the browser for example is aware of this system and leverages it). Further for mutable urls (again probably ought to use urns) previous versions can be requested trivially with @version e.g. "mutable:blog.johnsmith.com@3" and so on. We can do all this now ad-hoc with http headers etc, but it's all disparate and not unified like it should be, for example it doesn't tie in meaningfully with your file system. If if did then my file system on my computer at the label level (not the inode implementation level) would also use the same scheme. I write to a folder there, it automatically updates the label tree of that folder into the naming system and it's all consistent. 1: My hunch is that AWS/Azure/etc will have economies of scale too hard to beat in terms of the actual hardware but they may be eclipsed / acquire a startup that does the user facing implementation of this.

chipotle_coyote 8y ago

This is a good description of IPFS for those of us who weren't aware of it before. But, I think if it works the way you're describing here (and I assume you have it right), then it's not going to fully succeed at being the "distributed web" that it desires to be for the reasons that you outline. Having said that, I think IPFS sounds like a really good idea -- oddly, it's a lot like what I envisioned in a science fiction novel, where the notion of data being replicated across hundreds of storage nodes is so taken for granted that the main character has trouble conceiving of the notion of data that has a "location" (that is, is stored on only one device). But to get there, it needs to be largely content-agnostic: if the data is out there on the network, then it's replicated across those hundreds of storage notes, regardless of popularity. IPFS proves that technology is basically already here in theory -- but in practice, I'm not sure it's feasible in terms of storage costs/requirements yet.

MrEldritch 8y ago
IPFS is basically a cross between BitTorrent, plus Git's datastructure for permanently and uniquely identifying particular versions of and updates to a given piece of content, plus a clever distributed hash table for keeping track of who has what content.
- bordercases 8y ago
  Am I the only one who finds this prospect very exciting?
  - shliachtx 8y ago
    Definitely not, hence the enthusiasm with IPFS, and Cloudflare's recent decision to launch a IPFS gateway.
ghthor 8y ago
As a counter argument, your standard browser already caches a ton of stuff, might as well serve some of it back up if your upstream can support it. If you replace your browser cache with IPFS, and replace bookmarking with pinning you've got a functioning system. Your hosting requirements for content will likely go down, you get massive scale redundancy and reliability for free, no more slashdot effect and you might be able to claim 100% uptime due to just how unlikely it would be that someone hits your site while your root host is down and it's so low traffic that it's not bouncing around the network, even if it's just a standard consumer phylink. You could still get 'a hug of death ' but as the host you can just sit back and let the infrastructure work, distribute and recover, ala what it looks like to be the first seeder of a torrent. I predict this is unlikely in real world situations though, as for someone to share your site they'd likely have visited it first, thus your content host is likely to not be the only host. I'm an optimist, mostly because I think is stupid where we are at concerning self hosting from our homes.

kpcyrd 8y ago

You can rephrase this sentence: > it really doesn't necessarily lead to any kind of true decentralized <data storage> unless someone else has decided to <store your data> You might have an overly romantic idea of decentralization that doesn't necessarily align with the actual definition of decentralization. I would even argue that there is no solution for the idea you're suggesting. You can't have data that isn't stored anywhere. The permanent bit in ipfs is actually referring to something else: The data isn't guaranteed to be available at all times, but the link is guaranteed to point to the correct data. Slightly anecdotal, but a while ago there was a discussion about a pdf in #cjdns. Somebody had an ipfs link, but nobody was seeding it anymore. A few hours later somebody digged out a pdf from an old archive but wasn't sure if it's the correct one, so we ran `ipfs add file.pdf` and the ipfs link started working again.

MrEldritch 8y ago
To be clear: I was referring to an overly romantic idea of IPFS I had actually had at one point, and which I acquired because lots of other people were talking about IPFS that way. I am aware that IPFS does not actually do this, but a lot of people talk about it like it does, like IPFS means you can have websites without really needing to have/pay for any server yourself at all. (Which is, yes, technically sort of true, but only if visitors to your site reliably pin its content.)
- StavrosK 8y ago
  You don't have to actually pin things. Merely visiting will cache and rehost the files, and they will only be GCed at some later time. All pinning does is disable the GC for those files, which, as of a few months ago, weren't getting GCed at all anyway, so visiting something was equivalent to pinning it.
  - MrEldritch 8y ago
    "So unless for some reason I specifically think to help share some webpage or other, it'll just get auto-deleted from my machine when it cycles out of the cache" - my original post above. I was thinking about that content effectively "linkrotting" away if nobody visits it for a while, say it's old posts on a blog or something. I suppose how big of an issue this really is comes down to how long stuff would tend to stay in that cache in a real, practical usage scenario. I guess I'd been assuming that it'd only be day or so, but it could be longer in practice, depending on how much space is actually allocated to the local cache and how much each individual bit of unique content takes up and how many such unique chunks of content will be downloaded per day, along with maybe some more complex factors besides first-in-first-out for deciding what to collect. (I elaborated a little more about this below, when I thought about it kinda working like an automated torrent manager that made sure you maintained a decent seed ratio, didn't exceed an upload data cap, etc. and shuffled stuff out when it was taking up too much space or had hit a target ratio and wasn't going to be seeded anymore, with some prioritization for "is there anyone else seeding this", "is this a highly-demanded bit of content", etc.)

ghthor 8y ago

One idea I have is to execute health checks to the network from different points in ipfs node graph. For example you could add your file to your localnode(connected) then spawn some geographically distributed ephemeral modes to request the file. Once all the your emphermal nodes have recieved they wipe and you take your localnode(offline). Now the ephemeral nodes heartbeat, at some interval t, to request the file with the original source offline. If some N of them don't receive the file in some timeout, reconnect your localnode. In theory this would keep your file alive within the network for some x*t timespan, the question is what is that timespan and what's the cost of running the heartbeats.... I'm actually going to run this experiment and I'll reply here with my results.

antiffan 8y ago

This feels a bit to me like saying: "electric cars still have to charge their batteries somewhere, and they aren't necessarily being charged with renewable energy." Electric cars provide the capability to power a car with renewable energy. Powering the grid with electric energy is a separate problem. IPFS provides the capability of distribution. Incentivizing pinning is indeed a separate problem, but providing this capability feels like a huge step forward to me.

bordercases 8y ago
Likewise if the process of distribution was automated there would be people complaining about opt-in being a waste of their own resources for content they don't care about, or something.
- zeta0134 8y ago
  Exactly this. I don't understand why this is being thrown about like it's a bad thing; I gladly pinned all of Wikipedia, but it's also a huge amount of data and I made sure I was prepared to follow through before I asked my IPFS node to do it. If there's content out there that has earned my respect, I'll pin it gladly just to help the authors out, especially if they ask me politely. To be fair, some of this is automated. If you browse using your own node, it does maintain a small cache, which helps with the load distribution when something gets really popular. This blog post is a good example; despite literally being the #1 item on HN, it's still up; that's the global IPFS cache at work. It's slower than the average site, sure, but it didn't outright vanish, even though the author clearly is using minimal resources to actually host it. I think that's awesome.
- adrianN 8y ago
  More realistically it would have the same complains as services like Freenet. You are in legal limbo if you just host random stuff and it's not too unlikely to get all your equipment confiscated.

ChrisClark 8y ago

I know every mention of this seems to get downvoted because it's a cryptocurrency, but that's what the second half of the IPFS plan is, FileCoin. They've found a way to incentivize storing files by allowing people to 'mine' FileCoin. Mining in this sense isn't a waste of energy, or staking of coins. It's proving you have available space and proving you're making duplications of the data. This causes a race to the bottom for the pricing of storage while balancing it out with reliability. If you want to host your website on IPFS, pop it up on a domain through Cloudflare and set up a bid for storage space.

MrEldritch 8y ago
The problem with FileCoin is that the math behind it - Proof-of-Spacetime and Proof-of-Replication - just ... doesn't exist yet. That's what the delay has been, trying to figure out how to actually turn those into usable cryptographic primitives. It's not clear that a practical, scalable implementation of the system described in the FileCoin paper is actually even possible. (There's been some progress on getting proof-of-replication working, but it's still early stages and I haven't seen anything on proof-of-spacetime.) Also, the economics of FileCoin don't make a lot of sense ... https://blog.dshr.org/2018/06/the-four-most-expensive-words-in.html https://blog.dshr.org/2018/06/the-four-most-expensive-words-...
- ChrisClark 8y ago
  I've found they've been extremely quiet for the whole year, or longer. They did recently send out an email talking about their progress and showed some demos, but I just skipped over the email actually, didn't look into it to see if they talked about their POS 'Proof of [stuff]'. I just dug it up and they say Proof of Replication is going great. But don't show anything, they promise to open the code in the 'coming months' so people might see what progress has been done if any. No progress was mentioned for Proof of Spacetime. Edit: The link to their update, the only substantial one since the ICO last year I think... https://filecoin.io/blog/update-2018-q1-q2/ https://filecoin.io/blog/update-2018-q1-q2/
  - 8y ago

sliken 8y ago

Sure, IPFS isn't a great archive of uninteresting content that needs to be kept forever. But imagine the normal thing is to give 1GB of cache to IPFS. Granted if a planet worth of 1GB cache's isn't enough to save your content then it's not interesting enough.

MrEldritch 8y ago
The thing I'm worried about is niche content that is very interesting/useful, but only rarely - the kind of stuff that old Web1.0 personal sites, blogs, forum threads, and university-faculty webpages can be a treasure trove of, and which is especially prone to linkrot. It is precisely this content that a "permanent Web" would be of most interest, to me, about preserving. This is the "long tail" of content - any individual piece is only rarely interesting and only to some people, but in aggregate they actually make up a substantial fraction of the value of the Web to me. So if IPFS aspires to be the "permanent Web", but is virtually useless for preserving these, it hardly seems to qualify. With a few exceptions (for very large content that would require a ton of bandwidth to download and host, or to very complex content that requires preserving the structure of a whole large website to stay functional instead of individual pages, or for inherently server-side content), the kind of stuff that would be popular enough for someone to pin virtually never vanishes from the Web - even if links to it do break - because that's the kind of stuff that has almost certainly already been saved locally & reposted to other websites. And the stuff that's served unpinned from user caches is, very clearly, not permanent. So it's a "permanent Web" ... which is, at best, barely less fragile and prone to linkrot than the current web, in that if content is popular at one point and then the site layout changes or the original host goes down entirely it can still be kept up at the same link if someone was smart enough to save it locally, and specific versions of a website can be linked to specifically (even if nobody's necessarily hosting them anymore). But in all other respects, it is exactly as ephemeral as the current Web, and the fancy decentralized parts of it that are different than the current Web are among the most ephemeral parts, while the option to still have the boring old fragile centralized-server solutions where you host your own content personally are the durable ones.
- roywiggins 8y ago

joshfraser 8y ago

While it remains to be seen whether Filecoin will be price-competitive with AWS, IPFS has already proven itself to be far more censorship-resistant than any centralized provider can hope to offer. https://news.ycombinator.com/item?id=15367531 https://news.ycombinator.com/item?id=15367531

mirimir 8y ago

In my experience, persistence is random. "Fast Data Transfer via Tor"[0] has persisted for ~20 months. But "Fast Data Transfer via Tor (Methods)"[1] was gone soon after I took the hosting IPFS node down. 0) https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZRuf4dutN/ https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZR... 1) https://ipfs.io/ipfs/QmSp8p6d3Gxxq1mCVG85jFHMax8pSBzdAyBL2jZxCcCLBL/ https://ipfs.io/ipfs/QmSp8p6d3Gxxq1mCVG85jFHMax8pSBzdAyBL2jZ...

ezoe 8y ago

To make the data live after the original node goes offline, you have to push your data to random nodes which also push to random nodes. Your node blindly accept data, store it and upload it. There are some P2P network implementations which did that and failed horribly bad on the end. A few abuser can DoS the network. Oh and there are illegal data and numbers. I'm rather extreme freedom-of-speech advocates myself. But even for me, there are some immoral data I will never help spreading it knowingly.

misrab 8y ago

Perhaps you're confusing IPFS and Filecoin? IPFS provides the storage and retrieval layer, while Filecoin allows for incentivised markets, to your point.

mmjaa 8y ago

>And why would they? Because they're reading the blog, and are interested in the content - i.e. because next generation IPFS-friendly browsers participate in the pinning, as they should. I mean, think about it: there is already a copy of this web page out there, multiply redundant, in the form of our browser cache. All it really needs for IPFS to be viable is for the browser vendors to give the user the means to make those cached files part of their contribution to the Internet...

sterlind 8y ago
That would be beautiful, and hopefully realizable with a browser extension, which would get glommed into the browser core (a la Brave and Firefox) once the technology is proven. But how do you solve NAT? Most end-user devices can't expose ports, and IPs are increasingly shared. Skype was uniquely successful there, but I doubt IPFS in its current form could make use of those tricks.
- nybble41 8y ago
  > But how do you solve NAT? The solution to NAT is IPv6.

zaarn 8y ago

I think dat:// is a much more promising protocol over IPFS. The developers don't seem to delude themselves as much to the capabilities of the archive and there is no ICO. They advertise it as it is; Torrents with updates via public keys. ipfs:// on the other hand I hear most frequently handwaving away the problems like "who will pin unpopular content" or "how far does it scale" or "who will pay for gateway bandwidth once it gets big" or even "how to make content easily update" (I don't consider IPNS a good solution to that question)

dancek 8y ago
I was going to post just this. While I'm very skeptical that either of them will become popular, dat seems to be more suited for publishing websites and sharing large files. For a website it's nice that you share the whole site and are able to update the content while keeping the link (it's possible to tell if the content has changed, but there is no version history). And dat is already my first pick whenever I need to copy large files privately over internet. I blogged about my experience with dat a couple of months ago. Here's a link if anyone's interested. https://hannuhartikainen.fi/blog/dat-site/ https://hannuhartikainen.fi/blog/dat-site/ Since then I've used dat for copying files a couple of times. I haven't really browsed the dat:// web and I'd guess nobody has visited my website over that protocol (but then I don't have analytics and estimate my blog to have a dozen visitors a month).
imtringued 8y ago
>who will pay for gateway bandwidth once it gets big That's only a problem because IPFS isn't "big" enough for browsers to ship with an IPFS gateway integrated in them.
- zaarn 8y ago
  IPFS will need very big gateways because until that happens IPFS needs to grow a lot. And even then I doubt browsers would just ship a P2P application to autostart with the browser to the consumer.

1_player 8y ago

Is it possible to set up an IPFS caching server, which stores any content from any point in time, up to some configurable percentage of disk space? Ideally with some kind of mechanism to store poorly seeded data. I don't know how IPFS works, but I'd like to contribute by setting up and forgetting an IPFS cache on the Internet to help the network grow.

humblebee 8y ago
Anytime a user adds content an IPFS daemon, provide messages are sent out to the closest peers containing the cid of the newly added content. The peers will store the record, the cid of the content, and who told them about it. The peers will not request and download the content. However, you could set this up on your own. It's not a configuration option though. You would either have to build up your own system using the ipfs libraries, or monitor the events from a daemon. Unfortunately there is a bug that is not allowing the information required to be logged correctly in the events, so the actual cid of the content is not exposed. If it where, you could look for the `handleAddProvider` or `handleGetProvider` operation through the event logs (`ipfs log tail`) and then inspect the object (`ipfs object stat`) to determine if you wanted to pin it. > Ideally with some kind of mechanism to store poorly seeded data. This would be a little more difficult, but you could attempt to query the dht to determine how many peers are already providers for the data (`ipfs dht findprovs`).

amelius 8y ago

> This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. Perhaps not, technically, but it will turn hosting into a commodity. And it will remove any difference between centralized and decentralized from the user's point of view. Therefore, it's a huge win.

decentralised 8y ago

>This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files. This is why IPFS (w/ Filecoin), Storj and others use cryptoeconomic mechanism design and incentive structures to incentivise hosting / pinning data.

kkarakk 8y ago

From what I've understood, IPFS decentralization = if you have enough money to host , you have a voice. if enough people like what you say, you have a voice. if either of those fail, good luck & hope the door doesn't hit you on the way out.

maxZZzzz 8y ago

>it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files You can either pay somebody to pin the files. Or a public Organization like "archive.org" decides that a site is important enough to pin it. The typical surfer will seldom pin something but can keep low freq special interest sites alive. There is no fee lunch. With HTTP it has become hard to mirror a modern site. In IPFS it is a built in feature. Makes a better Internet.

bennofs 8y ago

Having a network-connected device that you own be always available may not be such a huge problem. Many home's use a router that is always connected: this router could also serve as IPFS content hoster! I view IPFS as a better distribution network, not a better host - in contrast to current's internet structure, if many people request the same content from me, they might only need to contact my server once and then share it between them thus reducing overall load. So the point of IPFS is not to store content for eternity, but to provide efficient distribution of content.