8 ms·
New improvements to IPFS Bitswap for faster container image distribution
- tyingq 7y ago"Web 2.0" Hadn't seen that gem for a while.
- jungong 7y ago3.0, no?
- 411111111111111 7y agopretty sure we're already at 4.0 with the Internet of Shit (IoT) and Mobile Internet, depending on the one using the buzzword but thats besides the point of the parent. web 2.0 hasn't really been mentioned in ages.
- anonsivalley652 7y agoYay. I last used ipfs for leeching abandonware around November. Although it had a tough time getting started and it would occasionally freeze up for several minutes, it worked well when it worked. It's seems to be getting better from when I first tried it.
- StavrosK 7y agoI wasn't aware of this use case, can you post a link/CID (if it's legal)?
- jude- 7y agoWould be curious to know how using IPFS for internal container distribution compares to using BitTorrent. IIRC BitTorrent has found similar uses in the past. Also, how well does BitSwap work when the underlying network is congested? Do IPFS nodes do any kind of congestion control?
- hiccuphippo 7y agoYou can't update a torrent, if the content changes you have to create a new one. IPNS helps with that. And you can't share pieces across different torrents, if some still have the old torrent, they share it separately from the new one even if the differences are minimal.
- jude- 7y agoThe article doesn't mention IPNS at all, nor does it talk about the need for mutating an image while it is being shared, so I'm not sure why you think IPNS is even desirable in this use-case?
- asdkhadsj 7y agoI would think because IPNS is IPFS.. and you asked how IPNS compares to BitTorrent. Maybe I misunderstand your question, but the reply seems totally on topic and a valid answer to your question.
- jude- 7y agoIPNS is a system that runs on top of IPFS. You do not need IPNS to use IPFS.
- zapita 7y agoThat's technically correct, but in practice the term "IPFS" is commonly used to refer to both ipfs and its optional feature ipns.
- Sargos 7y agoNot in my circles. IPFS is used a lot for storing and archiving files but IPNS is rarely used or mentioned. The only situations I've seen where IPNS would be useful, ENS was used instead as it's more reliable.
- 7y ago
- pjc50 7y agoNetflix use IPFS? That's quite an endorsement and makes me take it a lot more seriously.
- nolok 7y agoI don't disagree, but what they use it for, at what scale and at what priority matters a lot
- musingsole 7y agoRead the article? They were experimenting with using IPFS to distribute container images across AWS regions. And they do it at Netflix scale...
- k__ 7y agoDidn't they talk about distributing Docker containers for their developers? I mean, they have many devs, but that's still a few magnitudes below their number of viewers.
- Grustaf 7y ago”Netflix and IPFS began collaborating on ways to incorporate peer-to-peer services into Netflix’s developer tooling” Developer tooling is a pretty limited part of their traffic probably
- bastawhiz 7y agoThe volume of traffic doesn't really matter that much. Developer productivity is just as important as the service you're actually selling when you're the size of Netflix. If your N thousand engineers are suddenly unable to work, or slowed down by X%, that's a huge problem. Large companies treat (or should treat) developer tooling issues as seriously as application outages. If Netflix is using IPFS for anything worth mentioning, it's almost certainly substantive enough to be considered an endorsement.
- 7y ago
- js4ever 7y agoI'm shocked to see IPFS used to make something Faster...
- DaniFong 7y agoit's remarkable! a bootstrap technique to improve time coherence -- what an innovation!
- viraptor 7y agoI think that was a dig at ipfs's issues with real world usage where a lot of traffic until recently was wasted on metadata and every node used lots of bandwidth to gossip. Meanwhile the actual throughout on a non-tuned node was not great at all.
- StavrosK 7y agoHas this changed now?
- viraptor 7y agoI just realised that "recently" happened at the end of 2018 https://blog.ipfs.io/53-go-ipfs-0-4-18/ https://blog.ipfs.io/53-go-ipfs-0-4-18/ - oops. There was also a change later which turned off nodes being the middleman by default, but not sure which version. It's supposed to be much better these days, but I haven't tried it again in a few months.
- StavrosK 7y agoAh, that agrees with my experience, I noticed the IPFS node behaving much better some time in the last year.
- marshmellowtest 7y agoSaving Netflix's bandwith costs by sacrificing your privacy. IPFS and bittorrent don't do anything to protect the data you are uploading and your IP address. Case in point: https://iknowwhatyoudownload.com/en/peer/ https://iknowwhatyoudownload.com/en/peer/ Now every website you visit, any ad/tracker, any homecalling phone app can tell what movies and contents you watch and when you are at home. For years.
- knocte 7y agoI guess you mean "Making Netflix save bandwidth". You will get the same amount of data to watch the ninja turtles regardless if you use IPFS or not.
- marshmellowtest 7y agoYes. Edited to clarify.
- nine_k 7y agoI thought that ipfs is about high availability, fault tolerance, including some resistance against addressed censorship. It never looked like an anonymizing tool to me; did anybody advertise it as such?
- marshmellowtest 7y ago"resistance against addressed censorship" does not work at all when all your traffic is made public. People can be prosecuted or otherwise harassed for sharing contents on a P2P system. > It never looked like an anonymizing tool to me; did anybody advertise it as such? You are confusing "anonymizing" with "leaking a lot of information to the whole world". They constantly "forget" to tell people about the huge security impact.
- viraptor 7y agoIpfs helps you distribute content which may get taken down. It does not help you evade local police. For the second scenario, you want another layer which maintains secrecy. (Like the tor transport https://ipfs.io/ipfs/QmYKQvBsbYrRhdaGvQXcEoSam7s5gKVYULfRgNPzN5JM8N/IPFS-via-OnionCat.html https://ipfs.io/ipfs/QmYKQvBsbYrRhdaGvQXcEoSam7s5gKVYULfRgNP...)
- hinkley 7y ago> The node sends out a want for each CID to several peers in the session in parallel, because not all peers will have all blocks. If the node starts receiving a lot of duplicate blocks, it sends a want for each CID to fewer peers. If the node gets timeouts waiting for blocks, it sends a want for each CID to more peers. Trying to recall how the protocol works. Doesn’t this pattern of behavior mean that a lot of machines will end up with the beginning of a file and few will have the end? It sounds like the start of downloading would be very fast and the end would slow down while it hunts for a source May be why this is only 20% faster than Dockerhub.
- setr 7y agoI don't see why it'd be ordered for blocks received If I remember correctly, a node with a full file will send out the blocks in parallel, so leechers should receive blocks effectively in random order The only reason for some machines having only the start versus the end would be implementation-wise, where maybe you see the want list ordered, and the seeder responds by only shipping the first n blocks it reads in the want list If the seeder responds with random ordering, you'd avoid the problem of all leechers all having the same blocks
- tylersmith 7y agoFiles are not downloaded sequentially, they're chunked into blocks which are sent in parallel.
- hinkley 7y agoEach client downloads in random order, or all clients download in the same order?
- psKama 7y agoI really don't understand why there is a big hype towards IPFS which is still in development stage although, there are other options which are already out and working like Sia - Skynet which is not even getting a fraction of attention IPFS is getting.
- Taek 7y agoSkynet is barely two weeks old fwiw. As more people play around with it and see how strong it is I think it'll get a lot more attention. Lot of crypto projects are already planning to add support in the coming months.
- tln 7y ago"the container runtime can be modified to retrieve layers identified by their CIDs" How do you do this? Exercise for the reader? :) For the case of distributing containers in a datacenter with P2P, theres also this work: https://github.com/uber/kraken https://github.com/uber/kraken
- humblebee 7y agoThis might be related: https://github.com/docker/distribution/pull/2906 https://github.com/docker/distribution/pull/2906
- etaioinshrdlu 7y agoOne big optimization that could help in some cases for container platforms like Fargate is not downloading the entire image just to run the container. Instead read files (or even just blocks) from network storage on demand. This is basically how booting from a disk image works on most cloud platforms too.
- rochaporto_ 7y agoThat should be there soon : https://github.com/containerd/containerd/issues/3731 https://github.com/containerd/containerd/issues/3731
- miguelmota 7y agoIt's awesome to see these kind of improvements on IPFS. A while back as a side project I created an IPFS-backed docker registry which allows you to push and pull docker images from IPFS [0] [0] https://github.com/miguelmota/ipdr https://github.com/miguelmota/ipdr