8 ms·
WebWormHole: Send files quickly using WebRTC
- nerdbaggy 6y agoI love that it uses chunks and streaming to transfer the file. So many of these just try and load the entire file at once so you can’t transfer much.
- anderspitman 6y agoHaven't looked through the code yet. How does it handle stream backpressure with WebRTC?
- ryanar 6y agoWebRTC has built in mechanisms to check how many bytes have been buffered (RTCDataChannel.bufferedAmount), and you can register a low water mark (RTCDataChannel.bufferedamountlow) that fires an event handle when the buffered amount goes below that threshold (RTCDataChannel.onbufferedamountlow) where you can resume sending. WebRTC is sort of a combination of low-level and high-level APIs, but the ability to control backpressure ends up being very useful.
- anderspitman 6y agoThanks. Does bufferedAmount work better than it does on WebSockets[0], because I haven't had much luck with that. [0]: https://github.com/websockets/ws/issues/492 https://github.com/websockets/ws/issues/492
- _vbdg 6y agoInteresting note that this guy's choice of PAKE, Cpace, was chosen about a week ago by the CFRG for use in IETF protocols. Cpace is new, but that's a big vote of confidence for it.
- aspenmayer 6y agoNice spot, here's a link to the IETF draft spec for CPace mentioned. https://tools.ietf.org/id/draft-haase-cpace-01.html https://tools.ietf.org/id/draft-haase-cpace-01.html IETF post announcing the chosen candidates https://mailarchive.ietf.org/arch/msg/cfrg/LKbwodpa5yXo6VuNDU66vt_Aca8/ https://mailarchive.ietf.org/arch/msg/cfrg/LKbwodpa5yXo6VuND... Candidate selection process https://github.com/cfrg/pake-selection https://github.com/cfrg/pake-selection
- jedisct1 6y agoImplementation using libsodium https://github.com/jedisct1/cpace https://github.com/jedisct1/cpace
- mmm_grayons 6y agoHey, the libsodium guy! Thanks a million for your work on that; I've really enjoyed using it. I actually ran across this the day after the CFRG meeting and was happy to see a respected implementer had already written a C version. Would you say it's mature enough to use yet?
- sroussey 6y agoIs this in the wasm version?
- FiloSottile 6y agoIndeed! saljam asked me about a month ago what PAKE he should use with a Go implementation, and since there isn't a canonical one I put together a CPace implementation on top of ristretto255: filippo.io/cpace. It's a wonderfully simple algorithm when instantiated with a prime order group such as Ristretto. There are some implementation notes in the README: https://github.com/FiloSottile/go-cpace-ristretto255 https://github.com/FiloSottile/go-cpace-ristretto255
- gramakri 6y agofile.pizza is another similar project
- grishka 6y agoI once tried sending a 3 GB file through it, then kept wondering why my entire system became so sluggish. Turns out it loads the entire thing into RAM... The file didn't go through either. I hope this one isn't like that.
- f1refly 6y agoSame experience here. Every website I tried so far died when sending large files because it apparently loaded the whole thing into ram and then some. That might be fine on a system with 32gb memory, but my laptop with 8gb dies when trying to send a 7gb file.
- h-cobordism 6y agoDoes HTML5 even let you read a file without loading it all into memory? Edit: Looks like this is it. https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream https://developer.mozilla.org/en-US/docs/Web/API/ReadableStr... Edit 2: And yes, this is using it. https://github.com/saljam/webwormhole/blob/master/web/main.js#L96-L105 https://github.com/saljam/webwormhole/blob/master/web/main.j...
- anderspitman 6y agoYou can also do this manually (and on older browsers) by creating a FileReader and only loading new chunks after old ones have been transferred. With async APIs like WebSockets and WebRTC this typically requires implementing your own backpressure to avoid blowing up browser memory. See for example how omnistreams does it[0]. [0]: https://github.com/omnistreams/omnistreams-filereader-js/blob/master/index.mjs https://github.com/omnistreams/omnistreams-filereader-js/blo...
- lxgr 6y agoThat's great! Inconsistent large file support has also been what has prevented me from using the various predecessors to this so far.
- Sean-Der 6y agoThis is fantastic! Really nice work :) The nice thing about WebRTC is this works (pretty much) everywhere! Someone could throw up Python/Android/iOS/Go/Web/C++ Clients really easily. That is really exciting. Also just a HUGE fan of NAT Traversal/P2P in general. The less dependence we can have on others for sharing our data the better.
- saljam 6y agoThanks! There's already a Go client: https://github.com/saljam/webwormhole https://github.com/saljam/webwormhole
- Sean-Der 6y agoI was really excited to see you are using Pion (I am the creator) if there is anything I can do to make it better I am happy to help :) You should also join https://pion.ly/slack https://pion.ly/slack and share your project! I posted WebWormHole on https://twitter.com/_pion https://twitter.com/_pion also I see you are a Recurser (me as well!) I was Summer 2012 I think? Are you doing this as a project during your batch? I would really love to do a talk/deep dive on WebRTC as a talk sometime for Recursers, you should float the idea to Sonali and get me in :p
- saljam 6y agoPion is great. Thanks for making it! Definitely give a talk if you can. It's all remote currently!
- jackewiehose 6y ago> The nice thing about WebRTC is this works (pretty much) everywhere! Someone could throw up Python/Android/iOS/Go/Web/C++ Clients really easily. Huh, what did I miss?! How do you make a WebRTC-client in language-of-your-choice? The last time I checked for C++ I only found answers like "look at the chrome source". lol.
- Sean-Der 6y ago
- kevincox 6y agoIt is annoying that it removes the ID from the URL. It would be nice to bookmark my own code and I can just open it on multiple devices whenever I want to transfer a file. However I need to do a bit of gymnastics with the QR code to grab the URL.
- saljam 6y agoCodes are intentionally single use, to limit the bruteforce vector. And only two peers can connect any given time currently. It would be interesting to figure out how to make it work with more than 2 peers!
- waynenilsen 6y agoWe need a distributed db based on webrtc
- dezmou 6y agoCheckout gundb
- nomel 6y agoJust curious, why do we need this?
- waynenilsen 6y agoI would love to see zero barrier entry to creating a website (like twitter / name your favorite site that is not a static site) that can run on peoples laptops just by visiting a static javascript/html/css webpage. For this, you would need some decentralized backend storage solution akin to a sql or nosql database. IPFS is not a database, perhaps one could be built on top but then who pins the data and who runs the gateway? IPFS gateways are particularly confounding at this juncture. Until IPFS gets native browser support for pinning and gateways it cannot be the storage layer for a decentralized database. This kind of technology could help people take back the internet (imo) and webrtc goes a long way toward that goal. Even webrtc doesn't have the full promise of what I am suggesting however, it still requires a server that knows the IP of the other party so that you can directly connect (discovery). This would also need to be decentralized (somehow) perhaps only the discovery of peers would be done over a bootstrapped p2p DHT or something equivalent.
- folmar 6y agoIPFS probably has everything you need and a lot more
- api 6y agoIt's 2020, and people are elated to discover that it is possible to transfer a file directly between two systems on the Internet. True story: I was giving a guest lecture on network virtualization at UCI and demoing ZeroTier. One student came up afterwords and asked me how traffic could flow between systems without "a cloud." Evidently the idea that data could just go directly from point A to point B was utterly, completely foreign to the point that they weren't aware that the Internet could be used this way.
- _jal 6y agoI blame NAT. Treating end-users like second-class netizen consumers trained people to "need" the cloud to do perfectly normal peer-to-peer things.
- codethief 6y agoI'm not sure I would blame NAT. You can easily disable that. What you can't disable is the asymmetry of consumer internet connections (upload << download) and the fact that most consumer devices are not running (or connected to the internet) 24/7.
- pjc50 6y agoAlso security: if consumer PCs were open to the internet, they would be constantly getting breached in even bigger numbers.
- coopsmgoops 6y agoNice, I made one of these a few years ago http://passfiles.com http://passfiles.com Yours is a bit more polished than mine though. I didn't use QR codes either just good old fashioned urls.
- gccxsse 6y agoThere seem to be hundreds of these sites. They all do the same thing. Off the top of my head I can remember https://file.pizza https://file.pizza
- rakoo 6y agofile.pizza and friends work on webtorrent, which doesn't do end-to-end encryption EDIT: just learned that WebRTC actually does end-to-end encryption by default, so I'm wrong
- crazygringo 6y ago> ...it uses WebRTC to make the direct peer connections. This allows us to make use of WebRTC's NAT traversal tricks, as well as the fact that it can be used in browsers. But I'm assuming it can't break through all NAT routers, right? A good portion of people still won't be able to use this? A service usable by everyone would require STUN and TURN servers to be set up, no? Or has WebRTC made advances I'm unaware of?
- Sean-Der 6y agoI don't have any hard numbers, but I have heard ~85% ICE success rate with out TURN. But you are right, in some cases WebRTC will fail without TURN. Just no one wants to pay to run those servers :) I would love to see TCP hole punching in ICE, but it sounds like it is super hard to get right. Consumer internet does a lot better, lots of those failures come from Government/Military/Medical I bet.
- jackewiehose 6y agoYou always need a at least a STUN server and in my experience that 85% isn't remotely true. For example STUN-only never worked from mobile internet (only tested some german providers).
- algesten 6y agohow would TCP hole punching even work? TCP has state and a handshake. UDP doesn't.
- pkulak 6y agoVery nice! I'm assuming this is based on the wonderful "Magic Wormhole"? Is it actually using that program under the hood?
- psanford 6y agoNo, its doesn't use any of the same protocols that Magic Wormhole uses.
- jjice 6y agoI'm not as familiar with WebRTCPeerConnection as I'd like to be. Does it use the STUN server to get it's real IP and after that we can establish a completely peer to peer connection and now the webserver has no interaction with WebRTCPeer stream? If any of that is wrong, please enlighten me, I didn't realize peer to peer connections could be as simple as this.
- algesten 6y agothat's right. when establishing connection each side enumerates a bunch of "ice candidates". which is everything from your local LAN to outside NAT ip discovered via TURN servers (some restrictions due to privacy reasons). once the ice candidates are exchanged each side starts spraying the other with STUN messages to addresses ranked by "candidate pairs" that potentially could make a connection until one is found. this is simplified. there are mechanics like "trickle ice" and fallbacks to proxying via TURN servers. then there's the alleged idea this is all part of a webrtc "standard", which is laughable cause no browser follows the wild collection of RFCs that supposedly makes up the standard and the only reason any of it works is because there's a non written down general consensus of what's required.
- Sean-Der 6y agoThat is right! The only extra thing that STUN does is establish a hole punch. It isn't enough to just get your public hole, but you also do a temporary 'port forward' to the person that made the STUN request.
- strbean 6y agoLast I checked (long ago) there is the option for a TURN fallback as well.
- tptacek 6y agoThis is neat, but it seems like unlike with "real" Magic Wormhole, the server here can capture files by surreptitiously manipulating JS.
- Sean-Der 6y agoThere is a native client as well, you don't have to use JS!
- tptacek 6y agoNeat!
- saljam 6y agoAbsolutely true for the web interface if loaded from https://webwormhole.io https://webwormhole.io. I'm open for any more suggestions here! https://github.com/saljam/webwormhole/issues/13 https://github.com/saljam/webwormhole/issues/13 Someone mentioned the command line client. One can also build and serve the html/js/wasm from anywhere and it should still work, even with the same signalling server. It has pretty lax CORS for this reason.
- StavrosK 6y agoIPFS would be a solution here, since the files are content-addressed. You'd have to fetch them locally, since a gateway could still manipulate the content, but it's easier to find a gateway you trust.
- anderspitman 6y agoForgive my ignorance, but how would an IPFS gateway interfere here? If you have the hash of the js file you need, you can verify the gateway gives you the right one, correct? Or are you referring to the case where IPNS is used so the actually content at the address can change?
- 6y ago
- Naac 6y agoNeat, looks like a different backend and frontend implementation of the very similiar magic-wormhole[0] Now I wonder if anyone has made a web frontend of the original. [0] https://github.com/warner/magic-wormhole https://github.com/warner/magic-wormhole
- jimpick 6y agoI made a minimal web API for some experimental stuff I was trying: https://github.com/jimpick/magic-wormhole-service https://github.com/jimpick/magic-wormhole-service Not really hardened for production usage.
- jchook 6y agoWhy not generate the QR code client-side? Folks that care about P2P want to see zero HTTP requests to your server after loading the basic resources.
- 0xdeadbeefbabe 6y agoWho pays for the stun and turn servers?
- inglor 6y agoTurn servers are expensive but there are plenty of free SatUN servers. For example Google offers free stun servers.
- 0xdeadbeefbabe 6y agoHas the pandemic made ipv6 more popular somehow?
- tialaramex 6y agoOf course. IPv6 is much more widely deployed to home networks, whereas businesses tend to go out of their way to disable it. So when people are at home all over the world, whether that's the Christmas period or this present crisis, it bumps up IPv6 numbers slightly.
- jzer0cool 6y agoHad to look it up because I did not know: A STUN server is used to get an external network address. TURN servers are used to relay traffic if direct (peer to peer) connection fails. Could someone share practical usages of these commonly today to help me better grasp usage for these servers as a starting point?
- expialidocious 6y agoThe Twilio page you posted was good. I really like BlogGeek.me for WebRTC info. Here's an entry on the various servers involved. https://bloggeek.me/webrtc-server/ https://bloggeek.me/webrtc-server/
- weaksauce 6y agoi like the approach of encrypting locally, uploading to the cloud and sending the decryption key via a link. that's the way firefox send does it https://send.firefox.com https://send.firefox.com it's open source so you could run an instance of it if you wanted to.
- timvisee 6y agoFor the tech savy that like to use the CLI, checkout: https://github.com/timvisee/ffsend https://github.com/timvisee/ffsend
- tialaramex 6y agoThis is a very different (and equally valid) use case. It does mean somebody (in this case Mozilla) has to spend money on storage to deliver the service. Wormhole-style systems don't need to store the data because it's flowing from the sender to the recipient live.
- anderspitman 6y agoWhat makes this really interesting to me is that it uses a golang implementation of WebRTC on the server side. When I was playing with multiplayer networking in the browser ~1.5yr ago, there really didn't seem to be a lot of options for WebRTC servers. Great to see some progress here. Also, it sounds like it's using streaming rather than loading the entire file, which would give it an advantage over https://file.pizza https://file.pizza, which is what I usually recommend for p2p transfers. If you like these types of tools, but don't require p2p (or can't use it because of NAT), I'll also plug my own https://patchbay.pub https://patchbay.pub, which will let you do streamed transfers with nothing but curl.
- carlinmack 6y agothoughts on send.firefox.com?
- anderspitman 6y agoSeems like a solid option, and maybe better for recommending to people in general. Is it p2p? Nice thing about patchbay is pretty much all servers ship with a client (curl), so you don't have to worry about installing anything even for transferring large files. It's even lower friction than SSH/rsync with ssh agent because you don't have to type the destination path. Just run the receiver in the directory where you want the file to go. I used patchbay to transfer a 2GB earlier today. It looks like this: #sender curl https://patchbay.pub/anders/file.bin https://patchbay.pub/anders/file.bin --data-binary @file.bin #receiver curl https://patchbay.pub/anders/file.bin https://patchbay.pub/anders/file.bin > file.bin
- rakoo 6y agoI stumbled upon patchbay the other day and I thought it was really cool, now I just need to find a good excuse to toy with it. Just a heads up: the homepage still references the old index.html which you say in your blog article might have the wrong paths. Indeed looking at the docs all paths require a mode
- anderspitman 6y ago
- jonplackett 6y agoHow secure is this?
- quickthrower2 6y agoFrom the github repo (original caps): THIS PROJECT IS STILL IN EARLY DEVELOPMENT, USES EXPERIMENTAL CRYPTOGRAPHIC LIBRARIES, AND HAS NOT HAD ANY KIND OF SECURITY OR CRYPTOGRAPHY REVIEWS. IT MIGHT BE BROKEN AND UNSAFE.
- jonplackett 6y agoI’ll rephrase my question - how secure is this attempting to be?
- tialaramex 6y agoConceptually it's the same design as Magic Wormhole though all the technologies are different. It's just a PAKE then you do a file transfer encrypted with the key you agreed using the PAKE. PAKEs are very human friendly, they leverage a relatively weak secret (like "Monopoly Vegetable") that humans can deal with, to agree a good quality secret (like an effectively random 128-bit AES key) in such a way that both parties find out if the other party doesn't know the weak secret. Because humans are bored easily you can use rather weak secrets safely - it's a natural rate limit. An adversary who guesses almost right "Cluedo Animal?" only gets told they're wrong, and after maybe two or three more attempts the legitimate parties are annoyed and refuse to keep trying so their adversary is foiled. Machines wouldn't naturally use something like this because if a machine has a secure channel to another machine it can just move the 128-bit AES key, not waste time with some weaker human-memorable secret. This technology won't hide the IP addresses of those communicating A passive on-path adversary learns the size (perhaps not exactly but at least close) of the file transferred. And of course an active adversary can prevent the file transfer by spamming the service with nonsense.
- andred14 6y agoCome on it didn't work with a 5.5GB file :P
- andrewfromx 6y agoAnyone else get something like this? https://github.com/saljam/webwormhole/issues/27 https://github.com/saljam/webwormhole/issues/27 I was running `ww send ~/Downloads/myfile.bin` and my friend went to recv it from browser.
- mirimir 6y agoWebRTC is problematic if you're using a VPN service, with the VPN client running on the local machine. Quoting BrowserLeaks:[0] > IP address detection using JavaScript. Starting work on WebRTC API, the web browser communicates with the STUN server and shares information about local and public IP addresses even if you are behind NAT and use a VPN or Proxy. This tool will show if your real public IP is leaking out. However, if you run the VPN client on the router, there's no problem, because the local machine has no public IP address, just LAN and VPN interface addresses. 0) https://browserleaks.com/ https://browserleaks.com/
- Sean-Der 6y agoLocal IP leakage has been fixed! WebRTC uses mDNS candidates now, so there is nothing that shows your 'local IP' anymore. For 'Public IP' that sounds like a VPN configuration issue. Your WebRTC Agent should be routing the STUN requests through the VPN (and getting that public IP). But this effects all software/protocols, so don't think it is fair to ding WebRTC for this! [0] https://bloggeek.me/psa-mdns-and-local-ice-candidates-are-coming/ https://bloggeek.me/psa-mdns-and-local-ice-candidates-are-co...
- scruffups 6y agoLocal IP Leak SOLVED: Chrome (and maybe other browsers, too) no longer share local IP address. They share the mDNS address instead, which is often generated and registered locally by the browser. It's used only if the peers are on the same network. Else, it's useless but it used to be provided to all peers and malicious websites and they stopped doing that (speaking for those browsers that support mDNS) Public IP Leak in VPN SOLVED: In Chromium version 48+, you can set webRTCIPHandlingPolicy to default_public_interface_only which means that any VPN proxy will carry WebRTC media (over UPD if it supports UDP or else over TCP, which impacts quality of transmission) Your VPN provider just has to provide a Chrome extension to do the above or advise you to do that yourself. That way, the VPN's proxy IP address is what's visible to STUN, not the user's public IP address. There is also a more elaborate way around it, but the above should work.
- est 6y agoThis is nice and all but is it possible to make a few kilobytes single-file static HTML instead of 2.7MB https://webwormhole.io/util.wasm https://webwormhole.io/util.wasm ? I want to host it on my router.
- duke360 6y agoit happily loops forever tranferring an empty file :D
- neiman 6y agoIs there a size limit of on the files? This was the issue with all WebRTC file sending websites I saw so far.
- jo32 6y agoconsuming enormous cpu resources
- mgerb 6y agoSeems to work well sending images from my phone, but please let me select multiple at one time.
- harel 6y agoI've just used it. It's absolutely fantastic. Saved me from sending a 500MB file through a sharing service or having to create S3 temporary buckets or whatever other complicated method. Simple, works, perfect. Thank you!
- rmbeard 6y agoI'm curious how this compares with the DAT project https://docs.dat.foundation/docs/intro https://docs.dat.foundation/docs/intro I've had issues using that with networks throwing NAT errors, but need a secure P2P file solution for large data transfers. Wondering if this wil do the trick.
- rmbeard 6y agoSo, I tested this but it was blocked by our firewall.
- rpmuller 6y agoThis makes me ridiculously happy. Magic wormhole is one of those tools that works so well that you want to use it even when you don't need to. So happy to see something like this so I don't have to install wormhole on my wife or my son's computers to send them stuff.