5 ms·

Show HN: Gfile, a WebRTC based file transfer utility in Go

77 points by Antonito 7y ago

est 7y ago
> gfile is a WebRTC based file exchange software. > It allows to share a file directly between two computers, without the need of a third party. Can it be done with a single HTML file instead of Go? You know, which can be hosted anywhere even on Github pages.
- TomMarius 7y ago
  There is Go WASM runtime, isn't there?
  - Insanity 7y ago
    Yeah, you can compile to WASM since the last few versions of Go (1.10 or 1.11)!
- ggerganov 7y ago
  I actually did such kind of experiment recently [0]. As a bonus, the users do not have to copy/paste the SDP data manually - instead it is transmitted through sound! [0] https://ggerganov.github.io/wave-share https://ggerganov.github.io/wave-share
  - Antonito 7y ago
    That project is really cool! I'm planning to work on the SDP stuff at some point. It works well, but sharing two base64 encoded strings isn't very practical :/
- Sean-Der 7y ago
  The use case that excites me is transfer between servers/no X11. Just a single binary, no deps to worry about. They could also be air gapped, both behind NATs and other non-standard topologies. ICE will take care of all the hard problems :)
  - est 7y ago
    > transfer between servers A simple NC command would do. Or something like python -m http.server and curl
    - giancarlostoro 7y ago
      Heh can't tell you how many times I've used that python http server. Sure enough no matter the OS if Python is installed, it gives me the edge if I can access machines behind the same network.
    - Sean-Der 7y ago
      That wouldn’t work if they don’t have a direct connection (my main use case) I have a bunch of servers with no public IPs split between different regions.
- zimbatm 7y ago
  On a server it makes sense if there is no UI involved. Potentially the sender could output a URL that contains the SDP. That way the receiver can either be a CLI or a web browser.
- imetatroll 7y ago
  The page needs to run the appropriate javascript. That is all. Of course a STUN server at the least is required.
- Antonito 7y ago
  Yup, it can be done! Actually, it's one of my next objectives for it. I started to write a front-end client, with WebAssembly. It's not fully working yet, but you can take a look at it here: https://github.com/Antonito/gfile/tree/master/_client/web https://github.com/Antonito/gfile/tree/master/_client/web
kkapelon 7y ago
Honest question: what is the advantage over nc + compression utility of your choice? If you are just learning Golang, then congrats on shipping!
- Antonito 7y ago
  Hey! It can be useful if your transmitting data between two different clients on two different networks. In this case, you don't need to open ports on your firewall nor change your network configuration :) Thank you!
ignoramous 7y ago
Another similar service I've found useful is https://instant.io/ https://instant.io/ which does bit-torrent over WebRTC.
- Antonito 7y ago
  Didn't know about it, that's pretty cool!
knocte 7y ago
> It allows to share a file directly between two computers, without the need of a third party. I'm guessing it still depends on STUN and TURN servers?
- hardwaresofton 7y ago
  Yup -- https://github.com/Antonito/gfile/blob/v0.1.0/internal/session/session.go#L47 https://github.com/Antonito/gfile/blob/v0.1.0/internal/sessi... There's no TURN server specified, and I'm not 100% sure whether the one that's specified there (Google's) does both (so if STUN didn't work, would it fail or attempt TURN?).
  - lossolo 7y ago
    Google only provides STUN server. Providing free, performant TURN server would require HUGE amount of resources.
- Antonito 7y ago
  Yup, it still relies on a STUN server to get the connection's metadata. As hardwaresofton pointed out, I'm using Google's STUN server. Since it's only STUN, and not TURN, the actual transferred data does not go transit through it though.
  - hardwaresofton 7y ago
    Super late but thanks for clarifying this
  - davidcollantes 7y ago
    Is there no way to run this, without relying on a third party server/service?
hardwaresofton 7y ago
Despite how awesome WebRTC is, every time I see it my mind instantly goes to whether I understand STUN/TURN enough. They're part of the overall concept of ICE (Interactive Connection Establishment)[0], and STUN is a protocol for negotiated direct peer-to-peer connections[1] whereas TURN[2] is a relay mode that is used when a direct connection cannot be achieved for either side. When I saw the claim "without the need of a third party", it didn't fit with my knowledge of WebRTC so I went spelunking through the code and found that they do indeed use a STUN server[3], in particular google's "stun.l.google.com:19302". Just want to point out to those who might be relatively new to WebRTC, p2p mechanisms, etc -- it is surprisingly hard for two computers to talk to each other over the wide internet, even in this day and age. Even if you're behind the same router things aren't necessarily straight forward. [EDIT] - Also note that in the case of TURN, it is up to the application layer to encrypt communications[4]. [0]: https://en.wikipedia.org/wiki/Interactive_Connectivity_Establishment https://en.wikipedia.org/wiki/Interactive_Connectivity_Estab... [1]: https://en.wikipedia.org/wiki/STUN https://en.wikipedia.org/wiki/STUN [2]: https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_... [3]: https://github.com/Antonito/gfile/blob/v0.1.0/internal/session/session.go#L47 https://github.com/Antonito/gfile/blob/v0.1.0/internal/sessi... [4]: https://stackoverflow.com/questions/23085335/is-webrtc-traffic-over-turn-end-to-end-encrypted https://stackoverflow.com/questions/23085335/is-webrtc-traff...
- lossolo 7y ago
  > it is surprisingly hard for two computers to talk to each other over the wide internet, even in this day and age Only if both are behind NAT without forwarded ports and even then you can just use services like this[1] if you want to send files/data and it's super easy. > Even if you're behind the same router things aren't necessarily straight forward. I don't think this is true for most cases today. 1. https://justbeamit.com/ https://justbeamit.com/
siwatanejo 7y ago
I'm not a fan of Go. If anyone is interested in developing a Rust equivalent of this (all Rust with no C/C++ deps) I'm willing to pay for it, contact me at andrew.forsure@gmail.com (end result can be opensource)
- arghwhat 7y ago
  As someone who is personally a great fan of Rust, I have to say that this is a very defective attitude. If you're not writing it yourself (WebRTC is super simple, and this tool is basically the absolute bare-bone example of it—make a pipe and transfer something over it), why in the world would you care about what language it is written in beyond whether the language is safe, performant enough, and not a nightmare to execute on the target PC? If you're not making it, you don't get to pick the tools. Paying just to get decide tools for other people also seems like a really weird thing to do.
  - GoblinSlayer 7y ago
    Tools affect the result. If you ordered a tool in C, you will get more features than you wanted.
    - arghwhat 7y ago
      Tools, at a high level, do not affect what the end result is. Instead, it affects how end result is achieved. In real life, languages are more than tools, but as long as features/performance/behavior is acceptable, it still reduces to an implementation detail. (also, unless segfaults can be considered features, I don't see why C would get you anything extra.)
      - pjmlp 7y ago
        I guess black hats would consider memory corruption welcomed features.
  - hackerpacker 7y ago
    re: performance, as someone who isn't particularly enamored with rust or go (or any language), how is this thing only doing 16MBps local to local?!?
equalunique 7y ago
I wonder if I can use this to send ZFS snapshots over WebRTC
- kim0 7y ago
  Interesting, what's the use case here?
  - equalunique 7y ago
    Possibly a more robust protocol to send/recieve datasets, perhaps involving a Project-FiFo installation.
tpaschalis 7y ago
Thanks for this! I've wanted to learn more about WebRTC and Go, and digging through the source seems a good opportunity. To solve the same problem I've been using another [0] simple silly solution, but security is a concern. [0] https://github.com/tpaschalis/goof https://github.com/tpaschalis/goof
- Antonito 7y ago
  Glad you like it! If you think of any feature/improvements, feel free to open a issue/pull request! :)
maracuja-juice 7y ago
How about a tool like croc instead? https://github.com/schollz/croc https://github.com/schollz/croc
- qrv3w 7y ago
  croc currently doesn't do peer-to-peer with WebRTC, but I've actually been working on a branch that does. [1] Currently, croc uses a rendezvous server unless you happen to be on the same local network. Thanks to Antonito (and their usage of MIT license) I'll fork their code into croc so it can do WebRTC with a secure PAKE channel. [1]: https://github.com/schollz/croc/tree/v5 https://github.com/schollz/croc/tree/v5

77 points by Antonito 7y ago

est 7y ago

> gfile is a WebRTC based file exchange software. > It allows to share a file directly between two computers, without the need of a third party. Can it be done with a single HTML file instead of Go? You know, which can be hosted anywhere even on Github pages.

TomMarius 7y ago
There is Go WASM runtime, isn't there?
- Insanity 7y ago
  Yeah, you can compile to WASM since the last few versions of Go (1.10 or 1.11)!
ggerganov 7y ago
I actually did such kind of experiment recently [0]. As a bonus, the users do not have to copy/paste the SDP data manually - instead it is transmitted through sound! [0] https://ggerganov.github.io/wave-share https://ggerganov.github.io/wave-share
- Antonito 7y ago
  That project is really cool! I'm planning to work on the SDP stuff at some point. It works well, but sharing two base64 encoded strings isn't very practical :/
Sean-Der 7y ago
The use case that excites me is transfer between servers/no X11. Just a single binary, no deps to worry about. They could also be air gapped, both behind NATs and other non-standard topologies. ICE will take care of all the hard problems :)
- est 7y ago
  > transfer between servers A simple NC command would do. Or something like python -m http.server and curl
  - giancarlostoro 7y ago
    Heh can't tell you how many times I've used that python http server. Sure enough no matter the OS if Python is installed, it gives me the edge if I can access machines behind the same network.
  - Sean-Der 7y ago
    That wouldn’t work if they don’t have a direct connection (my main use case) I have a bunch of servers with no public IPs split between different regions.
zimbatm 7y ago
On a server it makes sense if there is no UI involved. Potentially the sender could output a URL that contains the SDP. That way the receiver can either be a CLI or a web browser.
imetatroll 7y ago
The page needs to run the appropriate javascript. That is all. Of course a STUN server at the least is required.
Antonito 7y ago
Yup, it can be done! Actually, it's one of my next objectives for it. I started to write a front-end client, with WebAssembly. It's not fully working yet, but you can take a look at it here: https://github.com/Antonito/gfile/tree/master/_client/web https://github.com/Antonito/gfile/tree/master/_client/web

kkapelon 7y ago

Honest question: what is the advantage over nc + compression utility of your choice? If you are just learning Golang, then congrats on shipping!

Antonito 7y ago
Hey! It can be useful if your transmitting data between two different clients on two different networks. In this case, you don't need to open ports on your firewall nor change your network configuration :) Thank you!

ignoramous 7y ago

Another similar service I've found useful is https://instant.io/ https://instant.io/ which does bit-torrent over WebRTC.

Antonito 7y ago
Didn't know about it, that's pretty cool!

knocte 7y ago

> It allows to share a file directly between two computers, without the need of a third party. I'm guessing it still depends on STUN and TURN servers?

hardwaresofton 7y ago
Yup -- https://github.com/Antonito/gfile/blob/v0.1.0/internal/session/session.go#L47 https://github.com/Antonito/gfile/blob/v0.1.0/internal/sessi... There's no TURN server specified, and I'm not 100% sure whether the one that's specified there (Google's) does both (so if STUN didn't work, would it fail or attempt TURN?).
- lossolo 7y ago
  Google only provides STUN server. Providing free, performant TURN server would require HUGE amount of resources.
Antonito 7y ago
Yup, it still relies on a STUN server to get the connection's metadata. As hardwaresofton pointed out, I'm using Google's STUN server. Since it's only STUN, and not TURN, the actual transferred data does not go transit through it though.
- hardwaresofton 7y ago
  Super late but thanks for clarifying this
- davidcollantes 7y ago
  Is there no way to run this, without relying on a third party server/service?

hardwaresofton 7y ago

Despite how awesome WebRTC is, every time I see it my mind instantly goes to whether I understand STUN/TURN enough. They're part of the overall concept of ICE (Interactive Connection Establishment)[0], and STUN is a protocol for negotiated direct peer-to-peer connections[1] whereas TURN[2] is a relay mode that is used when a direct connection cannot be achieved for either side. When I saw the claim "without the need of a third party", it didn't fit with my knowledge of WebRTC so I went spelunking through the code and found that they do indeed use a STUN server[3], in particular google's "stun.l.google.com:19302". Just want to point out to those who might be relatively new to WebRTC, p2p mechanisms, etc -- it is surprisingly hard for two computers to talk to each other over the wide internet, even in this day and age. Even if you're behind the same router things aren't necessarily straight forward. [EDIT] - Also note that in the case of TURN, it is up to the application layer to encrypt communications[4]. [0]: https://en.wikipedia.org/wiki/Interactive_Connectivity_Establishment https://en.wikipedia.org/wiki/Interactive_Connectivity_Estab... [1]: https://en.wikipedia.org/wiki/STUN https://en.wikipedia.org/wiki/STUN [2]: https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_... [3]: https://github.com/Antonito/gfile/blob/v0.1.0/internal/session/session.go#L47 https://github.com/Antonito/gfile/blob/v0.1.0/internal/sessi... [4]: https://stackoverflow.com/questions/23085335/is-webrtc-traffic-over-turn-end-to-end-encrypted https://stackoverflow.com/questions/23085335/is-webrtc-traff...

lossolo 7y ago
> it is surprisingly hard for two computers to talk to each other over the wide internet, even in this day and age Only if both are behind NAT without forwarded ports and even then you can just use services like this[1] if you want to send files/data and it's super easy. > Even if you're behind the same router things aren't necessarily straight forward. I don't think this is true for most cases today. 1. https://justbeamit.com/ https://justbeamit.com/

siwatanejo 7y ago

I'm not a fan of Go. If anyone is interested in developing a Rust equivalent of this (all Rust with no C/C++ deps) I'm willing to pay for it, contact me at andrew.forsure@gmail.com (end result can be opensource)

arghwhat 7y ago
As someone who is personally a great fan of Rust, I have to say that this is a very defective attitude. If you're not writing it yourself (WebRTC is super simple, and this tool is basically the absolute bare-bone example of it—make a pipe and transfer something over it), why in the world would you care about what language it is written in beyond whether the language is safe, performant enough, and not a nightmare to execute on the target PC? If you're not making it, you don't get to pick the tools. Paying just to get decide tools for other people also seems like a really weird thing to do.
- GoblinSlayer 7y ago
  Tools affect the result. If you ordered a tool in C, you will get more features than you wanted.
  - arghwhat 7y ago
    Tools, at a high level, do not affect what the end result is. Instead, it affects how end result is achieved. In real life, languages are more than tools, but as long as features/performance/behavior is acceptable, it still reduces to an implementation detail. (also, unless segfaults can be considered features, I don't see why C would get you anything extra.)
    - pjmlp 7y ago
      I guess black hats would consider memory corruption welcomed features.
- hackerpacker 7y ago
  re: performance, as someone who isn't particularly enamored with rust or go (or any language), how is this thing only doing 16MBps local to local?!?

equalunique 7y ago

I wonder if I can use this to send ZFS snapshots over WebRTC

kim0 7y ago
Interesting, what's the use case here?
- equalunique 7y ago
  Possibly a more robust protocol to send/recieve datasets, perhaps involving a Project-FiFo installation.

tpaschalis 7y ago

Thanks for this! I've wanted to learn more about WebRTC and Go, and digging through the source seems a good opportunity. To solve the same problem I've been using another [0] simple silly solution, but security is a concern. [0] https://github.com/tpaschalis/goof https://github.com/tpaschalis/goof

Antonito 7y ago
Glad you like it! If you think of any feature/improvements, feel free to open a issue/pull request! :)

maracuja-juice 7y ago

How about a tool like croc instead? https://github.com/schollz/croc https://github.com/schollz/croc

qrv3w 7y ago
croc currently doesn't do peer-to-peer with WebRTC, but I've actually been working on a branch that does. [1] Currently, croc uses a rendezvous server unless you happen to be on the same local network. Thanks to Antonito (and their usage of MIT license) I'll fork their code into croc so it can do WebRTC with a secure PAKE channel. [1]: https://github.com/schollz/croc/tree/v5 https://github.com/schollz/croc/tree/v5