8 ms·
Instant.io – Streaming file transfer over WebTorrent
- deleted 10y ago[deleted]
- jzelinskie 10y agoI work on some BitTorrent software and while it's a really cool protocol, it isn't designed to sequentially stream data. Some clients support streaming, but the act of prioritizing sequential chunks of data rather than chunks that are most likely to be unavailable in the future is bad behavior for the collective group of peers. I haven't personally given much thought to solving the problem of streaming, but I am surprised that the WebTorrent FAQ doesn't mention why they didn't take this opportunity to design a protocol that has more suitable trade-offs than BitTorrent. I'm getting mixed messaging; is their goal to connect the BitTorrent network with WebRTC or enable high quality P2P streaming via WebRTC?
- deleted 10y ago[deleted]
- alecco 10y agoWeird, a friend of mine used to stream with utorrent years ago without problems. Maybe one pause of a few seconds for a whole TV episode. The problem is not the protocol but the lack of features like subtitles. And torrent contents not being standard. All this can be easily scripted to cover most cases.
- batbomb 10y agoIf somebody was a genius, they'd write a bittorrent content delivery software that arbitrages those pauses with preloaded ads.
- Retr0spectrum 10y agoAnd then someone would come along a few days later with a client that doesn't display those ads.
- astro1138 10y agoYou'd still have to wait for the first block of data (piece in BitTorrent lingo) to complete.
- Retr0spectrum 10y agoI'd rather have a buffering symbol than an ad.
- milansuk 10y agoNice one! Plus you would not have to download ad.
- tomc1985 10y agoNo!
- Retr0spectrum 10y agoThe problems are not for the individual user, but for the rest of the group. By downloading blocks in sequential order, it guarantees that the last blocks will be rarer on average, which is bad for collective performance.
- joosters 10y agoIt doesn't have to be that bad. Users are likely to have more bandwidth available than is needed to stream the file (if they didn't, then the streaming will never work). Unlike simple HTTP streaming, the clients can use this spare bandwidth to download some blocks from the end of the file, even when streaming from near the start. So a sensible torrent streamer can still ensure that later blocks are not too rare.
- Retr0spectrum 10y agoI take your point. It's certainly not the end of the world, but it is sub-optimal.
- __s 10y agoThe increase in seeds from a more ergonomic protocol offsets the downside. Especially if the population has mostly the early blocks, but most of the population wants the early blocks, then there's room for much leech/leech sharing
- rawnlq 10y agoGuys guys hear me out ... the solution is to use "middle in" (not to be confused with middle out from Pied Piper). If you download from front and back simultaneously only the middle blocks would be scarce and you still get at least half your original streaming rate. And no weirdo will stop watching midway through a movie.
- Retr0spectrum 10y agoThat doesn't solve the problem, it just moves it around. A reasonable compromise would be to use any "spare" bandwidth to fetch random blocks.
- agumonkey 10y agoThe BitX program seems to handle subtitles outside of the container and protocol (some webservice I guess).
- daveguy 10y agoBitTorrent streams just dandy if you request/prioritize packets in order rather than random order. Most clients already have this capability.
- wyager 10y agoBut it's sometimes very bad for the network. There are a few papers out there about how if too many people stream vs random download it can lead to bad performance.
- nkohari 10y agoIs this necessarily true if all of the connected peers are also streaming? In my understanding, Bittorrent optimizes for the most in-demand chunks, and in a streaming context those would be the ones at the beginning of the video.
- swinglock 10y agoIt's actually the other way around. BitTorrent clients should prioritize that most rare pieces, there isn't really a notion of what the most in-demand pieces are. If you were to combine streaming with a BitTorrent protocol that prioritized the most in-demand pieces you would probably not be able to watch most videos to the end without pauses to buffer or maybe even at all. By prioritizing the most rare, even if all seeds left the swarm there's a much better chance swarm combined may still have all pieces and the torrent can still be finished.
- swinglock 10y agoIt works fine and is even supported in the official BitTorrent clients nowadays but it's correct that BitTorrent wasn't designed to stream and streaming was controversial in early BitTorrent. Clients are meant to prioritize the rarest pieces in the swarm first with some randomization thrown in. Downloading sequentially is bad for the swarms health but as it turns out doesn't seem to be bad enough that the protocol can't handle it, at least as long as there's a mix of streaming and classic clients or streaming clients use a combination of classic and streaming behavior.
- Longwelwind 10y agoWhat kind of trade-offs ? Genuinely interested, I know the basics of the torrent protocol, and I don't understand why the torrent protocol wouldn't work for streaming... I mean, you would just need to request the packets in order instead of randomly. It would be less efficient, sure, but it would work.
- TheSpiceIsLife 10y agoAnd for popular torrents, torrents with a lot of seeders who have at least okay bandwidth, it shouldn't matter. If a file can be downloaded in less time than it takes to use / consume / watch / listen to that file then there's space for innovation.
- the8472 10y ago> but it would work. Downloaders are only incentivized to give back data until they are done. Seeders are not really incentivized at all, so they can go away at any moment. So if everyone downloads sequentially and seeders go away, then you can end up getting stuck in a situation where everyone has the beginning of a file but nobody has the last parts. When you're streaming the protocol is not robust. With random order you have a robust protocol that just degrades in throughput if people are selfish. And with a webpage-bsed service being selfish is as easy as closing a tab.
- jjallen 10y agoYou can choose to not seed the second you begin downloading files with most torrent clients (at least that I've used), so the argument that torrenting ensures people share while their download is underway is weak. It's proof that seeders seed for the sake of seeding for the entire system to work.
- feross 10y agoExactly. In practice, modern torrent swarms have such an over-abundance of seeders that there is ample bandwidth for everyone. Consequently, the famed BitTorrent tit-for-tat algorithm as well as rarest-first piece selection strategy become a less important. Update: It's also noteworthy that BitTorrent Inc.'s official torrent client (as well as the largest player by marketshare), uTorrent, offers sequential downloading, as well as selective file downloading. And the BitTorrent network remains very healthy.
- j3097736 10y agoMost of the webtorrent streaming demos I've seen have a webseed fallback, so it's not really that important unless you're doing something completely distributed.
- feross 10y agoHi, creator of WebTorrent here. > [BitTorrent] isn't designed to sequentially stream data We’re working on improving the algorithm to switch back to a rarest-first strategy when there is not a high-priority need for specific pieces. In other words, when sufficient video is buffered, there’s no need to deviate from the normal piece selection algorithm. But the fact is that with the speed of today’s internet connections, the user is going to finish fully downloading the torrent in a fraction of the time it takes to view it, so they will still spend more time seeding than downloading. In practice, the only time that the rarest-first algorithm is important is on poorly-seeded torrents, or in the first few hours of a torrent being published when the ratio of seeders to leechers is really bad. I plan to keep improving the piece selection algorithm so that WebTorrent can be a good citizen. Also: you should note that not all WebTorrent users stream sequentially. That's just one option for downloading the data. Also: It's noteworthy that BitTorrent Inc.'s official torrent client (as well as the largest player by marketshare), uTorrent, offers sequential downloading, as well as selective file downloading. And the BitTorrent network remains very healthy. > why they didn't take this opportunity to design a protocol that has more suitable trade-offs than BitTorrent BitTorrent is the most successful, most widely-deployed P2P protocol in existence. It works really well. My goal with WebTorrent was to bring BitTorrent to the web in a way that interoperates with the existing torrent network. Re-inventing the protocol would have made WebTorrent fundamentally incompatible with existing clients and prevented adoption. The way we've done it is better. The wire protocol is exactly the same, but there's now a new way to connect to peers: WebRTC, in addition to the existing TCP and uTP. Also, re-inventing the protocol is a huge rabbit hole. There was already a lot of risk when I started the project -- will WebRTC get adopted by all the browser vendors? Will data channel stabilize and be performant? Is JavaScript fast enough to re-package MP4 videos on-the-fly for streaming playback with the MediaSource API? My thinking was: Why add inventing a new wire protocol and several algorithms to the table? Thanks for your thoughtful comment. Hope you'll give WebTorrent and our new desktop app, WebTorrent Desktop a try!
- the8472 10y ago> We’re working on improving the algorithm That sounds like you prioritized implementing streaming first over being a good citizen. > Also: It's noteworthy that BitTorrent Inc.'s official torrent client (as well as the largest player by marketshare), uTorrent, offers sequential downloading, a To my knowledge that is only available if the swarm condition allows and is not purely sequential. But that is second-hand knowledge, so I may be wrong. But either way, the default is rarest-first.
- deleted 10y ago[deleted]
- agumonkey 10y agoCool, a little more standalone than http://file.pizza http://file.pizza tests: https://instant.io/#74ce2f164e3d9ec5d5ee72c9aafc0cf5860e3d92 https://instant.io/#74ce2f164e3d9ec5d5ee72c9aafc0cf5860e3d92 https://instant.io/#c241674dc3b257637abfcb08203303fc25de007f https://instant.io/#c241674dc3b257637abfcb08203303fc25de007f
- amlib 10y agoTried to stream a 20GB mkv and it ate all my memory until the oom killer took over =(
- nashashmi 10y agoYup, It uses the browser's memory to transfer files. Until browser apps can be given permission to access the file system, this will be yhe case.
- EE84M3i 10y agoWe have this today, it's just that this site doesn't support it. Only place I've seen this used (other than thumbnailing images that are drag and dropped on imgur and whatnot) is on Mega. Spec: https://www.w3.org/TR/FileAPI/ https://www.w3.org/TR/FileAPI/ MDN: https://developer.mozilla.org/en-US/docs/Web/API/File_and_Directory_Entries_API/Introduction https://developer.mozilla.org/en-US/docs/Web/API/File_and_Di...
- eriknstr 10y agoThe MDN article mentions something which I have a question about; >The API doesn't give you access to the local file system, nor is the sandbox really a section of the file system. Instead, it is a virtualized file system that looks like a full-fledged file system to the web app. It does not necessarily have a relationship to the local file system outside the browser. >What this means is that a web app and a desktop app cannot share the same file at the same time. The API does not let your web app reach outside the browser to files that desktop apps can also work on. You can, however, export a file from a web app to a desktop app. For example, you can use the File API, create a blob, redirect an iframe to the blob, and invoke the download manager. Is it possible to "see" from JavaScript whether or not the download manager has downloaded the file completely, so that the file can be removed from FileAPI storage?
- feross 10y agoI think the spec you're looking for is the FileSystem API (https://developer.mozilla.org/en-US/docs/Web/API/File_and_Directory_Entries_API https://developer.mozilla.org/en-US/docs/Web/API/File_and_Di...), not the File API. Unfortunately, this is a non-standard API implemented only by Chrome. You have to use other APIs like IndexedDB and WebSQL (also deprecated and non-standard) to get a working solution in all browsers. This deficiency is really holding back the web.
- ckdarby 10y agoI see a lot of points about how this isn't exactly the best practice but I still don't follow why it isn't the best practice. Say you're already running a video sharing site and your servers are serving up all the content to the clients. So, you add your servers as seeders. The client comes in with support for webRTC, requests packets in order, gets your servers as seeders along with a couple other people watching the video and everyone goes along their merry way. The rare portions don't seem to be an issue because your servers are always seeds, always running, and already have the capacity to support all the demand. Is this not a win/win to reduce some bandwidth consumption?
- rakoo 10y agoAbsolutely, all the talk about rejecting streaming really concerns the "true" p2p swarms, where everybody can be a seeder and everybody can be a leecher, and there is only one "true" source, the original seeder. In those cases the peers can go down at any moment in time so it is very important for the swarm vitality that pieces be distributed as efficiently as possible. Your scenario is more or less the same as what we have today for those swarms that are comprised of many peers on the desktop and a few high-speed always-on seedboxes that already act like some kind of CDN. The more seeders there are, the better, in any situation. The question is whether the swarm we're talking about is whether you can expect some seeders to be relatively long-lived (in which case streaming is ok) or if we are in a free-for-all (in which case streaming is not). Not all swarms are of the first type, far from it.
- the8472 10y agoBittorrent is designed to not depend on such central, always-on servers. Avoiding piece availability bottlenecks is one of its robustness features. If you substitute built-in robustness with servers then yes, of course it will still work. But you're weakening the decentralized nature of the protocol by doing so.
- deleted 10y ago[deleted]
- mirimir 10y agoCool! But be aware WebRTC leaks public IP address for VPN users, and also leaks hashes of device IDs.[0] And in Chrome, it's very hard to block. This is a dangerous mix with talk of torrents :( [0] https://www.browserleaks.com/webrtc https://www.browserleaks.com/webrtc Edit: From feross I get that WebRTC no longer leaks ISP-assigned IPs when using VPNs.
- staticfish 10y agouBlock blocks webRTC leaking. it's also way better and faster than the crummy adblock extension.
- mirimir 10y agoWell, I use Firefox, and just disable WebRTC :) So does uBlock allow web torrents, while blocking webRTC leaks? I doubt it, because peers need to know public IP address. Unless you run a VPN client in the router, anyway.
- MrFlynn 10y agoI don't think it disables WebRTC, per se. I think uBlock prevents WebRTC applications from leaking your IP. Hangouts still works with WebRTC leaking disabled through uBlock, so that leads me to believe that's what's going on.
- mirimir 10y agoIf feross is right about WebRTC no longer leaking IP, then maybe it's not necessary to prevent leaks with uBlock?
- feross 10y ago> WebRTC leaks public IP address for VPN users This is incorrect. WebRTC data channels do not allow a website to discover your public IP address when there is a VPN in use. The WebRTC discovery process will just find your VPN's IP address and the local network IP address. Local IP addresses (e.g. 10.x.x.x or 192.168.x.x) can potentially be used to "fingerprint" your browser and identify across different sites that you visit, like a third-party tracking cookie. However, this is a separate issue than exposing your real public IP address, and it's worth noting that the browser already provides hundreds of vectors for fingerprinting you (e.g. your installed fonts, screen resolution, browser window size, OS version, language, etc.). If you have a VPN enabled, then WebRTC data channels will not connect to peers using your true public IP address, nor will it be reveled to the JavaScript running on the webpage. At one point in time, WebRTC did have an issue where it would allow a website to discover your true public IP address, but this was fixed a long time ago. This unfortunate misinformation keeps bouncing around the internet. There's now a spec that defines exactly which IP addresses are exposed with WebRTC. If you're interested in further reading, you can read the IP handling spec for yourself. https://tools.ietf.org/html/draft-ietf-rtcweb-ip-handling-01 https://tools.ietf.org/html/draft-ietf-rtcweb-ip-handling-01
- petre 10y agoNever worked for me. Tried to downliad Debian, no luck.
- caub 10y agowhy not using simply webrtc?
- stryk 10y agoI recently found this service: https://reep.io https://reep.io which, I believe, uses WebRTC to directly transfer between 2 browsers (they claim that after the initial 'handshake' they are out of the equation). I'm curious how it compares with Instant.io for simple file sharing use cases (example: send my mom a movie of my kids that is too large of a filesize to email, in a manner sufficient for a non-technical person to be able to easily receive, view, and save)
- arj 10y agoDoes anyone know if this works privately. Or if there is a good way to seed files only between friends?