10 ms·

SSH vs. OpenVPN for Tunneling

363 points by setra 9y ago

throwaway2048 9y ago
The overhead of layer 2/3 packet headers in OpenVPN did not cause such a huge slowdown, its much more likely that OpenVPN was operating in TCP mode, and thus subject to the tcp-in-tcp[1] problem with tunnel internal tcp sessions. Even an extraordinarily slow machine by modern standards can manage 100mbit openVPN without much of an issue. EDIT: did not see UDP results. There is still something else going on here, thats still much too slow for openVPN (without tweaking). [1] http://sites.inka.de/bigred/devel/tcp-tcp.html http://sites.inka.de/bigred/devel/tcp-tcp.html
- sitharus 9y ago
  Reading TFA both TCP and UDP OpenVPN modes were tested. UDP was about three times faster than TCP, but SSH was three times faster than that. Possibly the SSH encryption could have been more optimal for their use case?
ryanschneider 9y ago
Wow a little googling on this subject led me to sshuttle, which I've never heard of before but looks awesome, will try it out on Monday: http://sshuttle.readthedocs.io/en/stable/usage.html http://sshuttle.readthedocs.io/en/stable/usage.html Basically "vpn over ssh" (not really, but close enough).
- zzzcpan 9y ago
  Ssh has builtin vpn.
- Scarbutt 9y ago
  In case you don't know, ssh has built-in VPN support which may be enough for lots of use cases.
  - codefined 9y ago
    Both of the comments about VPN support within SSH seem to be heavily downvoted with no reason why. As far as I can see, their points are correct: > OpenSSH has built-in TUN/TAP support using -w<local-tun-number>:<remote-tun-number>. Here, a layer 3/point-to-point/ TUN tunnel is described. It is also possible to create a layer 2/ethernet/TAP tunnel.[0] Just with invalid terminology (s/ssh/OpenSSH) and the likelihood of the added slowdown mentioned in the article. [0] https://wiki.archlinux.org/index.php/VPN_over_SSH https://wiki.archlinux.org/index.php/VPN_over_SSH
    - lloeki 9y ago
      This is one of those Arch wiki pages that I find lacking, so I personally always refer to the man pages: man 1 ssh: -w local_tun[:remote_tun] Requests tunnel device forwarding with the specified tun(4) devices between the client (local_tun) and the server (remote_tun). The devices may be specified by numerical ID or the keyword ``any'', which uses the next available tunnel device. If remote_tun is not specified, it defaults to ``any''. See also the Tunnel and TunnelDevice directives in ssh_config(5). If the Tunnel directive is unset, it is set to the default tunnel mode, which is ``point-to-point''. man 5 ssh_config: Tunnel Request tun(4) device forwarding between the client and the server. The argument must be yes, point-to-point (layer 3), ethernet (layer 2), or no (the default). Specifying yes requests the default tunnel mode, which is point-to-point. TunnelDevice Specifies the tun(4) devices to open on the client (local_tun) and the server (remote_tun). The argument must be local_tun[:remote_tun]. The devices may be specified by numerical ID or the keyword any, which uses the next available tunnel device. If remote_tun is not specified, it defaults to any. The default is any:any. man 5 sshd_config: PermitTunnel Specifies whether tun(4) device forwarding is allowed. The argu- ment must be yes, point-to-point (layer 3), ethernet (layer 2), or no. Specifying yes permits both point-to-point and ethernet. The default is no. Independent of this setting, the permissions of the selected tun(4) device must allow access to the user.
- xrisk 9y ago
  Doesn't support UDP on macOS though.
  - tomxor 9y ago
    It doesn't support UDP in general AFAIK, i'm a frequent user of shuttle and find it indispensable, but if you need UDP it does fail, but then most VPN protocols only support a subset of IP layer protocols, e.g multicast doesn't generally work on any of them.
- XorNot 9y ago
  sshuttle is basically magic to me. Wherever someone with no idea about their threat model has deployed firewalls (ala every major business I've ever worked at) sshuttle is usually the key to being productive for me.
  - j_s 9y ago
    The stuff magic to me is the tools that speak both HTTPS + SSH or some other protocol, showing some normal site to anyone but those in the know. I've never had to use them since as you say few are that hardcore in their lockdowns, but it's nice to know they exist.
    - chx 9y ago
      I use SSLH for my poor man's VPN, I use redsocks but I guess I could use sshuttle. SSLH is really nice, easy to setup, easy on your resources.
- Retr0spectrum 9y ago
  I use sshuttle frequently, usually as a quick censorship bypass tool for specific sites. sshuttle also claims to have avoided the "TCP over TCP" problem. https://stackoverflow.com/questions/41427123/how-does-sshuttle-avoid-of-tcp-over-tcp-curse https://stackoverflow.com/questions/41427123/how-does-sshutt...
- chx 9y ago
  What's the difference between that and https://github.com/darkk/redsocks https://github.com/darkk/redsocks ? Edit: why the downvote? I thought I asked a legit question...?
  - aexaey 9y ago
    You can skip sshuttle's magic and do exactly the same thing manually with "ssh -D" + "iptables -j REDIRECT" + redsocks, but that's a lot moving parts.
  - jwilk 9y ago
    From the HN guidelines: Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
- noir_lord 9y ago
  If you pass --dns it tunnels that as well. Handy when you want to do something private on a machine you don't have a VPN setup on.
- rsync 9y ago
  "Wow a little googling on this subject led me to sshuttle, which I've never heard of before but looks awesome" sshuttle is indeed awesome - a viable VPN that requires nothing on the endpoint but a functioning ssh login (and python on the remote host). So any host, anywhere, with no configuration (and no permission required by the operators) that you can ssh into ... is now a VPN endpoint for you. Sibling comments in this thread are pointing out that ssh has this functionality already and that is true, but some configuration and maintenance (and understanding) is required. Not so with sshuttle. ALSO: We (rsync.net) sponsored the reworking of the ipfw target for sshuttle so that it now works properly on FreeBSD with the --dns option, etc. We also sponsored proper UDP tunneling for sshuttle on FreeBSD but I am not sure those patches are in a -release version of FreeBSD yet ... You need to use FreeBSD 11.0 (not 11.1) and you need to: git clone https://github.com/sshuttle/sshuttle.git cd sshuttle ; git checkout c746d6f7db3efbad6caddea76bdf916c46cf5c6e ... and that will get you a working sshuttle on FreeBSD with --dns support.
- nyolfen 9y ago
  i came to this thread to recommend sshuttle. it's indispensable and very well-performing, imo.
  - 3131s 9y ago
    I hadn't heard of it before, but it is beyond cool. I just installed it, ran the command, reconnected to my network and now it works. Amazing!
- GordonS 9y ago
  Wow, this does look amazing! A shame it's a bit more involved to get working with Windows, it's a pain to have to start a Linux VM to be able to use it :(
  - muxator 9y ago
    What about using cygwin instead of a Linux VM? I didn't have the time to try specifically this, bit normally you can reach really far with Cygwin. Plus, no need for root access, and the installation is natively "portable" (everything sits inside a single directory).
  - fpoling 9y ago
    If you use Windows and need to access in a browser some web services over ssh, then just run ssh -D and configure your browser to use PAC JavaScript file that redirects relevant host names to the ssh proxy.
    - GordonS 9y ago
      This is almost what I do today, only difference is that I use a proxy switcher plugin in Firefox, rather than using a PAC file.
- tomxor 9y ago
  I mention this every time VPNs come up... It's just so much better than any of the other VPN clients if you are only interested in TCP, I've been using it for over 6 years. The key difference between it and VPN protocols for anyone trying to compare is: TCP only, and TCP decompile/recompile with SSH in the middle. This has many advantages: speed (TCP over TCP is megga sucky obviously), server setup (basically none, you just need non-root access to an ssh server with python). It really is as simple as making an SSH connection on the command line and you can route all your TCP traffic immediately. It also has some really nice fine grained subnet routing features... On some strange occasions when I have needed it i have merged multiple remote subnets into the same virtual one one my machine by selecting ranges. It also has an auto discovery feature built in if you just want to route the specific IPs that exist on the other side. sshuttle is also in a number of package managers now, it's in apt at least.
  - Fnoord 9y ago
    Its great, but. It won't work on unrooted Android devices, and I doubt it will work on unrooted iOS devices. Not sure about ChromeOS.
- dexterdog 9y ago
  sshuttle hasn't had an actual release in 5 years according to the github project. If you're using a windows client I recommend bitvise tunnelier.
  - snowwolf 9y ago
    I suspect you are looking at the original repo. It has since been forked and has a few contributors now https://github.com/sshuttle/sshuttle https://github.com/sshuttle/sshuttle
- joveian 9y ago
  I've found that sshuttle with the --dns option still leaks dns queries occasionally so I set my web browser and curl to use SOCKS and run sshuttle to try to catch anything else.
ryan-c 9y ago
I found that OpenVPN benefited greatly from setting larger tunnel MTUs and then allowing the IP packets to be fragmented. I think it's encrypting per-packet which is somewhat inefficient.
staunch 9y ago
1. Doesn't look like there were runs of iperf over UDP and TCP, with simulated buffer sizes, so hard to know what the network connection is even capable of. 2. Also, how reliable is the network connection? Maybe it fluctuates every minute in usable bandwidth due to congestion or jitter. VMs are subject to noisy neighbor issues. 3. It looks like the ciphers are different, which could of course affect throughput greatly if one is not hardware accelerated or if its CPU bound. Worth checking CPU and other system resource usage for differences. 4. You could try OpenVPN with encryption and/or compression disabled entirely in UDP mode for best possible performance IIRC. This, at least, you would expect to beat an SSH tunnel.
- deleted 9y ago
  [deleted]
- stock_toaster 9y ago
  > 2. It looks like the cipher "AWS128-CTR" is being used for > SSH, I'm not even sure what that is. I'd look at CPU and > other system resource usage for differences. My guess is just a typo, for AES128-CTR.
- opmac 9y ago
  Regarding 4, sort of defeats the purpose of using OpenVPN no?
- zzzcpan 9y ago
  Port forwarding terminates TCP in the middle, it is supposed to be faster no matter what, no?
ajross 9y ago
This has been known forever. Stream-based packet tunneling absolutely does outperform packet wrapping in good network conditions, for the simple reason that small packets can be combined by the transport layer and avoid the overhead. The problem is that in the presence of any packet loss at all, every packet lost causes a stall of the whole tunnel until it gets retransmitted, which even in the best case recovery conditions requires two (three? I forget my SACK details) round trips. If one side is using traditional TCP retransmit timers it can be much, much longer.
- aexaey 9y ago
  Well-known unpredictable TCP-over-TCP performance would only explain OpenVPN/TCP results. But OP has posed also OpenVPN/UDP performance that is 2x...3x slower than SSH port forwarding, so I'd guess there is more to this story (see also andmarios's impromtu test results above).
- Buge 9y ago
  Your downside I don't think applies in this case. Because it's a single application wanting to make a single reliable connection. So if there was any packet loss at all it would have to stall the whole connection anyways regardless of whether it's wrapped in SSH, OpenVPN, or nothing.
- jacob019 9y ago
  I've been using a VPN over TCP ssh for a year. The end point is across the world. I get excellent bandwidth and no stalls. Latency is less than anticipated. The thoretical arguments against TCP/TCP do not match my experience.
  - ajross 9y ago
    Yeah, real world network paths, especially inter-data-center ones, generally do operate at exactly zero packet loss. Until they don't, alas.
andmarios 9y ago
There should be some catch in the setup. Maybe the CPUs he used are old and don't support hardware AES acceleration? I just run iperf3 over OpenVPN / UDP with AES-256-GCM between two servers in the same DC (but different rooms, through their public network, 1gE links) and got an average of 750MBits/sec. I don't have any special setting.
- deleted 9y ago
  [deleted]
- SomeStupidPoint 9y ago
  Out of curiosity -- what's your SSH tunneling rate in the same setup?
  - andmarios 9y ago
    Just tested. Ssh has indeed better performance, at around 870Mbit/sec. But the gap in my -anecdotal- testing is much smaller than OP's and maybe predictable due to the differences (as OP described) on how OpenVPN and ssh work.
    - deleted 9y ago
      [deleted]
rootw0rm 9y ago
fwiw OpenVPN has had more consistent performance over the years than SSH for me. my servers have always been half a world away tho. i can reliably saturate my 100mbit home connection over SSH now, so I'm happy.
gerdesj 9y ago
A very intelligent write up with one small, possibly fatal flaw. When you are comparing one thing to another and writing up your pearls of wisdom for the masses you need to control every variable (within reason). I can't see a LAN based test acting as a baseline.
- bartvk 9y ago
  > you need to control every variable Why? I plunk down a lot of stuff on my blog, not everything is up to that pretty high standard you mention.
throwaway613834 9y ago
How do you redirect DNS over SSH though?
- lpasselin 9y ago
  iirc man ssh has some info on ssh virtual private network
  - throwaway613834 9y ago
    IIRC that suffers from the TCP-through-TCP performance issue?
- deleted 9y ago
  [deleted]
- Fnoord 9y ago
  Sshuttle [1] takes care of that see --dns and --help and the documentation. You might also wanna disable IPv6 --disable-ipv6 Its basically a script which takes care of ssh, the routing, and firewalling. Its crossplatform, should work on Windows/Linux/macOS and perhaps more. [1] https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle [2] https://sshuttle.readthedocs.io/en/stable/ https://sshuttle.readthedocs.io/en/stable/
  - rsync 9y ago
    Just a clarification ... https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle is the original, now abandoned, sshuttle. Development was taken over by: https://github.com/sshuttle/sshuttle https://github.com/sshuttle/sshuttle ... which is the current, maintained version that you should track.
- xioxox 9y ago
  A SOCKS5 tunnel supports DNS. This is good for web browsing and is natively supported by ssh. You can even use it for individual programs using the tsocks command line wrapper.
nly 9y ago
One thing I hate about OpenVPN is the ridiculous configuration. I'm a big fan of tinc[0] personally, although you have to use 1.1pre releases to get half decent encryption, it's configuration is wonderful. [0] https://www.tinc-vpn.org/ https://www.tinc-vpn.org/
- proctor 9y ago
  tinc 1.1pre seems pretty good, but I do worry about the legacy fallback for encryption; There doesn't seem to be any way to e.g. --force-experimental to prevent encryption falling back to the old RSA mode.
antonios 9y ago
spiped would also be interesting to compare. Ridiculously easy to set up, and made by cperciva.
mirimir 9y ago
A huge advantage of vpn tunnels is that they stay up indefinitely. I've had openvpn links stay up for months. With ssh, you're lucky to get a few days. I've tried scripts and even autossh, and ssh has never been as reliable as openvpn or tinc.
- proctor 9y ago
  Does anyone have any experience with ipop/groupvpn[0]? It seems like an interesting alternative to tinc in that it can do turn/stun using libjingle to bust nat. I'm not clear on how well tested or secure it is though. [0] http://ipop-project.org/ http://ipop-project.org/
- perlgeek 9y ago
  Now I wonder if there's a way to combine mosh's automatic reconnecting with ssh tunneling somehow...
  - Fnoord 9y ago
    That's latency related. Mosh uses UDP only, sshuttle uses TCP only. AutoSSH plus some firewall scripts to ensure data only travels through the VPN would suffice.
deleted 9y ago
[deleted]
j_s 9y ago
Worth looking at ZeroTier (commercial - actually running a hardware IndieGoGo right now) and WireGuard (IPSec - 'The Next Generation' / 'Voyager' or whichever you thought was best) as modern options.
- jonathanoliver 9y ago
  We love ZeroTier. It's a great piece of software that makes layer 2 networking (and above) across WANs very simple and secure.
- pferde 9y ago
  I like that Wireguard is slowly sneaking into common use. Its performance seems promising. https://www.wireguard.com/performance/ https://www.wireguard.com/performance/
  - j_s 9y ago
    I believe it is headed for Linux kernel mainline soon-ish. https://news.ycombinator.com/item?id=15596963 https://news.ycombinator.com/item?id=15596963
jstewartmobile 9y ago
Didn't see any mention of MTU in the post. If the defaults were sub-optimal for OpenVPN, that alone would account for the reduced throughput.
hardwaresofton 9y ago
I recently had to set up a 3 computer SSH tunnel and wrote about it, maybe people will find it interesting: https://vadosware.io/post/ssh-tunneling-using-an-intermediary-computer/ https://vadosware.io/post/ssh-tunneling-using-an-intermediar... I first tried to setup OpenVPN and only wanted to use the the SSH tunnel for DNS purposes (I couldn't use normal DDNS tools), but in the end, I ended up running a SOCKS proxy over the tunnel and it was stunningly effective, enough that watching Netflix despite intercontinental round trips.
DrPhish 9y ago
I used to maintain an ipsec VPN for road warrior type scenarios, but I've found that mostly I just wanted to get a remote desktop on a computer within the network. So I've stopped mucking around with VPN and just published Apache Guacamole HTML5 VNC/RDP tunnels. No plugins, no clients except a modern web browser and good security with certbot+auto redirect to https+simple auth to keep badguys away from possible exploits in the guacamole login form. Configs can be as simple as the NoAuth plugin, all the way to full integration with eg. AD. I've implemented this at a number of businesses, and it does what almost everyone in the organization wants. It also supports things like printing (to a pdf that comes thru the browser as a download) and file transfer. Super easy to set up and is much more responsive than VNC or RDP thru ssh. The times when I actually want full IP access to a network from my laptop are so rare that it seemed silly to maintain IPSec. What need do others have for full-connectivity VPNs? Not site-to-site VPNs of course, but these ad-hoc on the road type ones. Honest question
- michaelmior 9y ago
  > It also supports things like printing (to a pdf that comes thru the browser as a download) Wow, didn't know that. Impressive!
- pcl 9y ago
  I VPN back to my home network when I’m abroad so I can watch US Netflix shows.
  - yeukhon 9y ago
    Curious but why?
    - iforgotpassword 9y ago
      Because he likes watching US Netflix shows.
    - aurelianito 9y ago
      Shows available on Netflix depend on location.
pwdisswordfish2 9y ago
An additional reason to choose OpenSSH over OpenVPN is that OpenVPN is dependent on OpenSSL. As such, it is potentially subject to past and future vulnerabilities OpenSSL. OpenSSH does not depend on OpenSSL.
qplex 9y ago
>"As long as you only need one TCP port forwarded, SSH is a much faster choice, because it has less overhead than SSH." OpenVPN is way more robust solution for tunneling though. I've found that UDP tunnels work sometimes better in when there is considerable packet loss and high latency (poor radio links).
whalesalad 9y ago
It’s really not an apples to apples comparison. For my teammates on the development team, yes an SSH tunnel is going to be easier for our needs due to the reduced overhead. But if I want to give my customer support team access to private web services in our production cluster, a VPN is the perfect solution. At FarmLogs, each of our Kube clusters has a bastion VPN host that puts you inside the VPC and handles DNS to Route53 (and the VPC) so <service>.farmlogs “just works” We use Foxpass to handle this with LDAP bridged to Google Apps so when an employee joins or leaves their VPN access is immediately updated.
k_vi 9y ago
SSH tunnel as proxy is easier and simpler to use if you have a VPS. Setup the tunnel: ssh -D 8080 vv@xx.xx.xx.xxx Change your system proxy settings: Socks proxy, 127.0.0.1, 8080 Done!
- foxhop 9y ago
  Yup, even works using putty.exe on Windows (http://www.foxhop.net/ssh-tunnel http://www.foxhop.net/ssh-tunnel)
  - fpoling 9y ago
    And with PAC JavaScript files one can redirect only specific hosts to ssh or redirect to different ssh connections.
unixhero 9y ago
Any love for Softether vpn here? I use it every day to connect gome and it just woeks. Always. It encapsulates ssh I think
jakobegger 9y ago
Using SSH Tunnels is not TCP over TCP. You have a TCP connection between local app and SSH client, then a second TCP connection between SSH client and SSH server, and a third TCP connection between SSH server and remote app. There is no TCP over TCP when you use SSH tunnels. Lost packets between SSH client and SSH server do not cause retransmits on either application side.
- erdewit 9y ago
  That's what I thought too, that there is just a byte stream being tunneled between SSH server and client and not a TCP packet stream. Btw I use unix sockets on client and server to avoid the local connection TCP overhead.
Yegorius 9y ago
I wonder why nobody has mentioned StrongSwan (IPSec+IKEv2) or ShadowSocks, which are both performant and very secure.
- Canada 9y ago
  Shadowsocks is not very secure at all. Static keys and naive crypto. But that doesn't matter much since its primary use is for obfuscation.
  - computerfriend 9y ago
    I would love to read more about Shadowsocks security, if you have a link.
INTPenis 9y ago
I use both actually, where the best tool is applicable. Some commenters have mentioned StrongSwan (or OpenSwan which is more known to me). I've actually seen OpenSwan perform better than Cisco VPN tunnels. And I love pointing that out to our networking guys since we pay nothing for OpenSwan.
foxhop 9y ago
If you are a road warrior (or you want to bypass security filters at work) you can securely proxy Firefox web and DNS traffic over a VPN using just SSH. All you need is an SSH server running at home or in the "cloud". I documented this a few years back: http://www.foxhop.net/ssh-tunnel http://www.foxhop.net/ssh-tunnel This is also a great way to access resources on the remote network, for example your router/modem setup page or Jenkins, or whatever else is running on that remote Network.
josteink 9y ago
They're different beasts with different use-cases. I rely on both, and like having both options around.
acd 9y ago
IPSec on Linux has better performance than OpenVPN due to multi core support. That said setting up VPN tends to be time consuming and error prone. But what if the ssh connection drops, what restarts the tunnel then? IPSec has restart mechanism for when the net drops which it will do.
khanjahanzaib27 9y ago
Cool Keep it up.
morpheuskafka 9y ago
[Zerotier](https://zerotier.com https://zerotier.com) is far nicer than all of them, clients (Win, macOS, GNU\Linux, OpenWrt, iOS, Android, NAS) are open-source and the hosted service allows up to 100 devices on one of unlimited networks for free. Because it emulates the ethernet layer, there is no client side configuration required and it can be used for non-IP packets (ex. DHCP). By enabling the default route permission on the client it can also be used as a gateway to the internet through a remote host on the network. All communications are e2e AES-256 with certificate-based access for private networks. Oh, and they also run a public earth network with unlimited members billed as "the global LAN party." It's a really convenient service, and the performance is good enough to be used between private clouds.
- mjcl 9y ago
  > non-IP packets (ex. DHCP) DHCP is IP/UDP.
  - morpheuskafka 9y ago
    True, I was thinking of NetBUEI/IPX/AppleTalk/etc.
- yjftsjthsd-h 9y ago
  I'm really tempted to use this, but I'm a little nervous about relying on a company for this, especially because it doesn't look like they should be profitable on their current offerings.
  - ac2u 9y ago
    Correct me if wrong, but I think that aside from their web based portal, everything else is open source. So you could control everything via API. So if they ever did go out of business, you could host it yourself and I'm sure by then an open source web portal would emerge to take the main site's place. (Not to mention that they'd probably just release the web app if they were going out of business). Hope it never comes to that though, I'd imagine their main bread and butter would be larger enterprise licenses.
sanbor 9y ago
There is a very simple test: fast.com. I tried fast.com over sshuttle and OpenVPN and OpenVPN was faster while sshuttle kind of collapsed.
sireat 9y ago
What's the fastest option when you do need UDP? Generally I just use SSH, but for gaming i've resorted to 160bit OpenVPN. Incidentally setting up a OpenVPN server on dd-wrt is a rather hellish experience.
saasproduct 9y ago
I use https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle but it's often too slow. I am wonder if their exist any go/rust equivalent of this project.
skarap 9y ago
`ip link set qlen 1000 dev tun0` (so it matches the eth0 queue, instead of being an order of magnitude smaller) on both ends of the tunnel, and you get 90% of direct connection speed from openvpn: $ iperf3 -c 10.1.0.2 Connecting to host 10.1.0.2, port 5201 [ 4] local 10.1.0.1 port 38274 connected to 10.1.0.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 3.12 MBytes 26.1 Mbits/sec 0 802 KBytes [ 4] 1.00-2.00 sec 22.3 MBytes 187 Mbits/sec 0 2.24 MBytes [ 4] 2.00-3.00 sec 21.4 MBytes 179 Mbits/sec 0 2.45 MBytes [ 4] 3.00-4.00 sec 23.3 MBytes 196 Mbits/sec 0 2.39 MBytes [ 4] 4.00-5.00 sec 23.4 MBytes 196 Mbits/sec 0 2.58 MBytes [ 4] 5.00-6.00 sec 21.5 MBytes 180 Mbits/sec 0 2.70 MBytes [ 4] 6.00-7.00 sec 23.5 MBytes 197 Mbits/sec 0 2.68 MBytes [ 4] 7.00-8.00 sec 22.6 MBytes 189 Mbits/sec 0 2.78 MBytes [ 4] 8.00-9.00 sec 22.4 MBytes 188 Mbits/sec 0 2.45 MBytes [ 4] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec 0 2.69 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 207 MBytes 174 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 207 MBytes 174 Mbits/sec receiver iperf Done.

363 points by setra 9y ago

throwaway2048 9y ago

The overhead of layer 2/3 packet headers in OpenVPN did not cause such a huge slowdown, its much more likely that OpenVPN was operating in TCP mode, and thus subject to the tcp-in-tcp[1] problem with tunnel internal tcp sessions. Even an extraordinarily slow machine by modern standards can manage 100mbit openVPN without much of an issue. EDIT: did not see UDP results. There is still something else going on here, thats still much too slow for openVPN (without tweaking). [1] http://sites.inka.de/bigred/devel/tcp-tcp.html http://sites.inka.de/bigred/devel/tcp-tcp.html

sitharus 9y ago
Reading TFA both TCP and UDP OpenVPN modes were tested. UDP was about three times faster than TCP, but SSH was three times faster than that. Possibly the SSH encryption could have been more optimal for their use case?

ryanschneider 9y ago

Wow a little googling on this subject led me to sshuttle, which I've never heard of before but looks awesome, will try it out on Monday: http://sshuttle.readthedocs.io/en/stable/usage.html http://sshuttle.readthedocs.io/en/stable/usage.html Basically "vpn over ssh" (not really, but close enough).

zzzcpan 9y ago
Ssh has builtin vpn.
Scarbutt 9y ago
In case you don't know, ssh has built-in VPN support which may be enough for lots of use cases.
- codefined 9y ago
  Both of the comments about VPN support within SSH seem to be heavily downvoted with no reason why. As far as I can see, their points are correct: > OpenSSH has built-in TUN/TAP support using -w<local-tun-number>:<remote-tun-number>. Here, a layer 3/point-to-point/ TUN tunnel is described. It is also possible to create a layer 2/ethernet/TAP tunnel.[0] Just with invalid terminology (s/ssh/OpenSSH) and the likelihood of the added slowdown mentioned in the article. [0] https://wiki.archlinux.org/index.php/VPN_over_SSH https://wiki.archlinux.org/index.php/VPN_over_SSH
  - lloeki 9y ago
    This is one of those Arch wiki pages that I find lacking, so I personally always refer to the man pages: man 1 ssh: -w local_tun[:remote_tun] Requests tunnel device forwarding with the specified tun(4) devices between the client (local_tun) and the server (remote_tun). The devices may be specified by numerical ID or the keyword ``any'', which uses the next available tunnel device. If remote_tun is not specified, it defaults to ``any''. See also the Tunnel and TunnelDevice directives in ssh_config(5). If the Tunnel directive is unset, it is set to the default tunnel mode, which is ``point-to-point''. man 5 ssh_config: Tunnel Request tun(4) device forwarding between the client and the server. The argument must be yes, point-to-point (layer 3), ethernet (layer 2), or no (the default). Specifying yes requests the default tunnel mode, which is point-to-point. TunnelDevice Specifies the tun(4) devices to open on the client (local_tun) and the server (remote_tun). The argument must be local_tun[:remote_tun]. The devices may be specified by numerical ID or the keyword any, which uses the next available tunnel device. If remote_tun is not specified, it defaults to any. The default is any:any. man 5 sshd_config: PermitTunnel Specifies whether tun(4) device forwarding is allowed. The argu- ment must be yes, point-to-point (layer 3), ethernet (layer 2), or no. Specifying yes permits both point-to-point and ethernet. The default is no. Independent of this setting, the permissions of the selected tun(4) device must allow access to the user.
xrisk 9y ago
Doesn't support UDP on macOS though.
- tomxor 9y ago
  It doesn't support UDP in general AFAIK, i'm a frequent user of shuttle and find it indispensable, but if you need UDP it does fail, but then most VPN protocols only support a subset of IP layer protocols, e.g multicast doesn't generally work on any of them.
XorNot 9y ago
sshuttle is basically magic to me. Wherever someone with no idea about their threat model has deployed firewalls (ala every major business I've ever worked at) sshuttle is usually the key to being productive for me.
- j_s 9y ago
  The stuff magic to me is the tools that speak both HTTPS + SSH or some other protocol, showing some normal site to anyone but those in the know. I've never had to use them since as you say few are that hardcore in their lockdowns, but it's nice to know they exist.
  - chx 9y ago
    I use SSLH for my poor man's VPN, I use redsocks but I guess I could use sshuttle. SSLH is really nice, easy to setup, easy on your resources.
Retr0spectrum 9y ago
I use sshuttle frequently, usually as a quick censorship bypass tool for specific sites. sshuttle also claims to have avoided the "TCP over TCP" problem. https://stackoverflow.com/questions/41427123/how-does-sshuttle-avoid-of-tcp-over-tcp-curse https://stackoverflow.com/questions/41427123/how-does-sshutt...
chx 9y ago
What's the difference between that and https://github.com/darkk/redsocks https://github.com/darkk/redsocks ? Edit: why the downvote? I thought I asked a legit question...?
- aexaey 9y ago
  You can skip sshuttle's magic and do exactly the same thing manually with "ssh -D" + "iptables -j REDIRECT" + redsocks, but that's a lot moving parts.
- jwilk 9y ago
  From the HN guidelines: Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
noir_lord 9y ago
If you pass --dns it tunnels that as well. Handy when you want to do something private on a machine you don't have a VPN setup on.
rsync 9y ago
"Wow a little googling on this subject led me to sshuttle, which I've never heard of before but looks awesome" sshuttle is indeed awesome - a viable VPN that requires nothing on the endpoint but a functioning ssh login (and python on the remote host). So any host, anywhere, with no configuration (and no permission required by the operators) that you can ssh into ... is now a VPN endpoint for you. Sibling comments in this thread are pointing out that ssh has this functionality already and that is true, but some configuration and maintenance (and understanding) is required. Not so with sshuttle. ALSO: We (rsync.net) sponsored the reworking of the ipfw target for sshuttle so that it now works properly on FreeBSD with the --dns option, etc. We also sponsored proper UDP tunneling for sshuttle on FreeBSD but I am not sure those patches are in a -release version of FreeBSD yet ... You need to use FreeBSD 11.0 (not 11.1) and you need to: git clone https://github.com/sshuttle/sshuttle.git cd sshuttle ; git checkout c746d6f7db3efbad6caddea76bdf916c46cf5c6e ... and that will get you a working sshuttle on FreeBSD with --dns support.
nyolfen 9y ago
i came to this thread to recommend sshuttle. it's indispensable and very well-performing, imo.
- 3131s 9y ago
  I hadn't heard of it before, but it is beyond cool. I just installed it, ran the command, reconnected to my network and now it works. Amazing!
GordonS 9y ago
Wow, this does look amazing! A shame it's a bit more involved to get working with Windows, it's a pain to have to start a Linux VM to be able to use it :(
- muxator 9y ago
  What about using cygwin instead of a Linux VM? I didn't have the time to try specifically this, bit normally you can reach really far with Cygwin. Plus, no need for root access, and the installation is natively "portable" (everything sits inside a single directory).
- fpoling 9y ago
  If you use Windows and need to access in a browser some web services over ssh, then just run ssh -D and configure your browser to use PAC JavaScript file that redirects relevant host names to the ssh proxy.
  - GordonS 9y ago
    This is almost what I do today, only difference is that I use a proxy switcher plugin in Firefox, rather than using a PAC file.
tomxor 9y ago
I mention this every time VPNs come up... It's just so much better than any of the other VPN clients if you are only interested in TCP, I've been using it for over 6 years. The key difference between it and VPN protocols for anyone trying to compare is: TCP only, and TCP decompile/recompile with SSH in the middle. This has many advantages: speed (TCP over TCP is megga sucky obviously), server setup (basically none, you just need non-root access to an ssh server with python). It really is as simple as making an SSH connection on the command line and you can route all your TCP traffic immediately. It also has some really nice fine grained subnet routing features... On some strange occasions when I have needed it i have merged multiple remote subnets into the same virtual one one my machine by selecting ranges. It also has an auto discovery feature built in if you just want to route the specific IPs that exist on the other side. sshuttle is also in a number of package managers now, it's in apt at least.
- Fnoord 9y ago
  Its great, but. It won't work on unrooted Android devices, and I doubt it will work on unrooted iOS devices. Not sure about ChromeOS.
dexterdog 9y ago
sshuttle hasn't had an actual release in 5 years according to the github project. If you're using a windows client I recommend bitvise tunnelier.
- snowwolf 9y ago
  I suspect you are looking at the original repo. It has since been forked and has a few contributors now https://github.com/sshuttle/sshuttle https://github.com/sshuttle/sshuttle
joveian 9y ago
I've found that sshuttle with the --dns option still leaks dns queries occasionally so I set my web browser and curl to use SOCKS and run sshuttle to try to catch anything else.

ryan-c 9y ago

I found that OpenVPN benefited greatly from setting larger tunnel MTUs and then allowing the IP packets to be fragmented. I think it's encrypting per-packet which is somewhat inefficient.

staunch 9y ago

1. Doesn't look like there were runs of iperf over UDP and TCP, with simulated buffer sizes, so hard to know what the network connection is even capable of. 2. Also, how reliable is the network connection? Maybe it fluctuates every minute in usable bandwidth due to congestion or jitter. VMs are subject to noisy neighbor issues. 3. It looks like the ciphers are different, which could of course affect throughput greatly if one is not hardware accelerated or if its CPU bound. Worth checking CPU and other system resource usage for differences. 4. You could try OpenVPN with encryption and/or compression disabled entirely in UDP mode for best possible performance IIRC. This, at least, you would expect to beat an SSH tunnel.

deleted 9y ago
[deleted]
stock_toaster 9y ago
> 2. It looks like the cipher "AWS128-CTR" is being used for > SSH, I'm not even sure what that is. I'd look at CPU and > other system resource usage for differences. My guess is just a typo, for AES128-CTR.
opmac 9y ago
Regarding 4, sort of defeats the purpose of using OpenVPN no?
zzzcpan 9y ago
Port forwarding terminates TCP in the middle, it is supposed to be faster no matter what, no?

ajross 9y ago

This has been known forever. Stream-based packet tunneling absolutely does outperform packet wrapping in good network conditions, for the simple reason that small packets can be combined by the transport layer and avoid the overhead. The problem is that in the presence of any packet loss at all, every packet lost causes a stall of the whole tunnel until it gets retransmitted, which even in the best case recovery conditions requires two (three? I forget my SACK details) round trips. If one side is using traditional TCP retransmit timers it can be much, much longer.

aexaey 9y ago
Well-known unpredictable TCP-over-TCP performance would only explain OpenVPN/TCP results. But OP has posed also OpenVPN/UDP performance that is 2x...3x slower than SSH port forwarding, so I'd guess there is more to this story (see also andmarios's impromtu test results above).
Buge 9y ago
Your downside I don't think applies in this case. Because it's a single application wanting to make a single reliable connection. So if there was any packet loss at all it would have to stall the whole connection anyways regardless of whether it's wrapped in SSH, OpenVPN, or nothing.
jacob019 9y ago
I've been using a VPN over TCP ssh for a year. The end point is across the world. I get excellent bandwidth and no stalls. Latency is less than anticipated. The thoretical arguments against TCP/TCP do not match my experience.
- ajross 9y ago
  Yeah, real world network paths, especially inter-data-center ones, generally do operate at exactly zero packet loss. Until they don't, alas.

andmarios 9y ago

There should be some catch in the setup. Maybe the CPUs he used are old and don't support hardware AES acceleration? I just run iperf3 over OpenVPN / UDP with AES-256-GCM between two servers in the same DC (but different rooms, through their public network, 1gE links) and got an average of 750MBits/sec. I don't have any special setting.

deleted 9y ago
[deleted]
SomeStupidPoint 9y ago
Out of curiosity -- what's your SSH tunneling rate in the same setup?
- andmarios 9y ago
  Just tested. Ssh has indeed better performance, at around 870Mbit/sec. But the gap in my -anecdotal- testing is much smaller than OP's and maybe predictable due to the differences (as OP described) on how OpenVPN and ssh work.
  - deleted 9y ago
    [deleted]

rootw0rm 9y ago

fwiw OpenVPN has had more consistent performance over the years than SSH for me. my servers have always been half a world away tho. i can reliably saturate my 100mbit home connection over SSH now, so I'm happy.

gerdesj 9y ago

A very intelligent write up with one small, possibly fatal flaw. When you are comparing one thing to another and writing up your pearls of wisdom for the masses you need to control every variable (within reason). I can't see a LAN based test acting as a baseline.

bartvk 9y ago
> you need to control every variable Why? I plunk down a lot of stuff on my blog, not everything is up to that pretty high standard you mention.

throwaway613834 9y ago

How do you redirect DNS over SSH though?

lpasselin 9y ago
iirc man ssh has some info on ssh virtual private network
- throwaway613834 9y ago
  IIRC that suffers from the TCP-through-TCP performance issue?
deleted 9y ago
[deleted]
Fnoord 9y ago
Sshuttle [1] takes care of that see --dns and --help and the documentation. You might also wanna disable IPv6 --disable-ipv6 Its basically a script which takes care of ssh, the routing, and firewalling. Its crossplatform, should work on Windows/Linux/macOS and perhaps more. [1] https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle [2] https://sshuttle.readthedocs.io/en/stable/ https://sshuttle.readthedocs.io/en/stable/
- rsync 9y ago
  Just a clarification ... https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle is the original, now abandoned, sshuttle. Development was taken over by: https://github.com/sshuttle/sshuttle https://github.com/sshuttle/sshuttle ... which is the current, maintained version that you should track.
xioxox 9y ago
A SOCKS5 tunnel supports DNS. This is good for web browsing and is natively supported by ssh. You can even use it for individual programs using the tsocks command line wrapper.

nly 9y ago

One thing I hate about OpenVPN is the ridiculous configuration. I'm a big fan of tinc[0] personally, although you have to use 1.1pre releases to get half decent encryption, it's configuration is wonderful. [0] https://www.tinc-vpn.org/ https://www.tinc-vpn.org/

proctor 9y ago
tinc 1.1pre seems pretty good, but I do worry about the legacy fallback for encryption; There doesn't seem to be any way to e.g. --force-experimental to prevent encryption falling back to the old RSA mode.

antonios 9y ago

spiped would also be interesting to compare. Ridiculously easy to set up, and made by cperciva.

mirimir 9y ago

A huge advantage of vpn tunnels is that they stay up indefinitely. I've had openvpn links stay up for months. With ssh, you're lucky to get a few days. I've tried scripts and even autossh, and ssh has never been as reliable as openvpn or tinc.

proctor 9y ago
Does anyone have any experience with ipop/groupvpn[0]? It seems like an interesting alternative to tinc in that it can do turn/stun using libjingle to bust nat. I'm not clear on how well tested or secure it is though. [0] http://ipop-project.org/ http://ipop-project.org/
perlgeek 9y ago
Now I wonder if there's a way to combine mosh's automatic reconnecting with ssh tunneling somehow...
- Fnoord 9y ago
  That's latency related. Mosh uses UDP only, sshuttle uses TCP only. AutoSSH plus some firewall scripts to ensure data only travels through the VPN would suffice.

deleted 9y ago

[deleted]

j_s 9y ago

Worth looking at ZeroTier (commercial - actually running a hardware IndieGoGo right now) and WireGuard (IPSec - 'The Next Generation' / 'Voyager' or whichever you thought was best) as modern options.

jonathanoliver 9y ago
We love ZeroTier. It's a great piece of software that makes layer 2 networking (and above) across WANs very simple and secure.
pferde 9y ago
I like that Wireguard is slowly sneaking into common use. Its performance seems promising. https://www.wireguard.com/performance/ https://www.wireguard.com/performance/
- j_s 9y ago
  I believe it is headed for Linux kernel mainline soon-ish. https://news.ycombinator.com/item?id=15596963 https://news.ycombinator.com/item?id=15596963

jstewartmobile 9y ago

Didn't see any mention of MTU in the post. If the defaults were sub-optimal for OpenVPN, that alone would account for the reduced throughput.

hardwaresofton 9y ago

I recently had to set up a 3 computer SSH tunnel and wrote about it, maybe people will find it interesting: https://vadosware.io/post/ssh-tunneling-using-an-intermediary-computer/ https://vadosware.io/post/ssh-tunneling-using-an-intermediar... I first tried to setup OpenVPN and only wanted to use the the SSH tunnel for DNS purposes (I couldn't use normal DDNS tools), but in the end, I ended up running a SOCKS proxy over the tunnel and it was stunningly effective, enough that watching Netflix despite intercontinental round trips.

DrPhish 9y ago

I used to maintain an ipsec VPN for road warrior type scenarios, but I've found that mostly I just wanted to get a remote desktop on a computer within the network. So I've stopped mucking around with VPN and just published Apache Guacamole HTML5 VNC/RDP tunnels. No plugins, no clients except a modern web browser and good security with certbot+auto redirect to https+simple auth to keep badguys away from possible exploits in the guacamole login form. Configs can be as simple as the NoAuth plugin, all the way to full integration with eg. AD. I've implemented this at a number of businesses, and it does what almost everyone in the organization wants. It also supports things like printing (to a pdf that comes thru the browser as a download) and file transfer. Super easy to set up and is much more responsive than VNC or RDP thru ssh. The times when I actually want full IP access to a network from my laptop are so rare that it seemed silly to maintain IPSec. What need do others have for full-connectivity VPNs? Not site-to-site VPNs of course, but these ad-hoc on the road type ones. Honest question

michaelmior 9y ago
> It also supports things like printing (to a pdf that comes thru the browser as a download) Wow, didn't know that. Impressive!
pcl 9y ago
I VPN back to my home network when I’m abroad so I can watch US Netflix shows.
- yeukhon 9y ago
  Curious but why?
  - iforgotpassword 9y ago
    Because he likes watching US Netflix shows.
  - aurelianito 9y ago
    Shows available on Netflix depend on location.

pwdisswordfish2 9y ago

An additional reason to choose OpenSSH over OpenVPN is that OpenVPN is dependent on OpenSSL. As such, it is potentially subject to past and future vulnerabilities OpenSSL. OpenSSH does not depend on OpenSSL.

qplex 9y ago

>"As long as you only need one TCP port forwarded, SSH is a much faster choice, because it has less overhead than SSH." OpenVPN is way more robust solution for tunneling though. I've found that UDP tunnels work sometimes better in when there is considerable packet loss and high latency (poor radio links).

whalesalad 9y ago

It’s really not an apples to apples comparison. For my teammates on the development team, yes an SSH tunnel is going to be easier for our needs due to the reduced overhead. But if I want to give my customer support team access to private web services in our production cluster, a VPN is the perfect solution. At FarmLogs, each of our Kube clusters has a bastion VPN host that puts you inside the VPC and handles DNS to Route53 (and the VPC) so <service>.farmlogs “just works” We use Foxpass to handle this with LDAP bridged to Google Apps so when an employee joins or leaves their VPN access is immediately updated.

k_vi 9y ago

SSH tunnel as proxy is easier and simpler to use if you have a VPS. Setup the tunnel: ssh -D 8080 vv@xx.xx.xx.xxx Change your system proxy settings: Socks proxy, 127.0.0.1, 8080 Done!

foxhop 9y ago
Yup, even works using putty.exe on Windows (http://www.foxhop.net/ssh-tunnel http://www.foxhop.net/ssh-tunnel)
- fpoling 9y ago
  And with PAC JavaScript files one can redirect only specific hosts to ssh or redirect to different ssh connections.

unixhero 9y ago

Any love for Softether vpn here? I use it every day to connect gome and it just woeks. Always. It encapsulates ssh I think

jakobegger 9y ago

Using SSH Tunnels is not TCP over TCP. You have a TCP connection between local app and SSH client, then a second TCP connection between SSH client and SSH server, and a third TCP connection between SSH server and remote app. There is no TCP over TCP when you use SSH tunnels. Lost packets between SSH client and SSH server do not cause retransmits on either application side.

erdewit 9y ago
That's what I thought too, that there is just a byte stream being tunneled between SSH server and client and not a TCP packet stream. Btw I use unix sockets on client and server to avoid the local connection TCP overhead.

Yegorius 9y ago

I wonder why nobody has mentioned StrongSwan (IPSec+IKEv2) or ShadowSocks, which are both performant and very secure.

Canada 9y ago
Shadowsocks is not very secure at all. Static keys and naive crypto. But that doesn't matter much since its primary use is for obfuscation.
- computerfriend 9y ago
  I would love to read more about Shadowsocks security, if you have a link.

INTPenis 9y ago

I use both actually, where the best tool is applicable. Some commenters have mentioned StrongSwan (or OpenSwan which is more known to me). I've actually seen OpenSwan perform better than Cisco VPN tunnels. And I love pointing that out to our networking guys since we pay nothing for OpenSwan.

foxhop 9y ago

If you are a road warrior (or you want to bypass security filters at work) you can securely proxy Firefox web and DNS traffic over a VPN using just SSH. All you need is an SSH server running at home or in the "cloud". I documented this a few years back: http://www.foxhop.net/ssh-tunnel http://www.foxhop.net/ssh-tunnel This is also a great way to access resources on the remote network, for example your router/modem setup page or Jenkins, or whatever else is running on that remote Network.

josteink 9y ago

They're different beasts with different use-cases. I rely on both, and like having both options around.

acd 9y ago

IPSec on Linux has better performance than OpenVPN due to multi core support. That said setting up VPN tends to be time consuming and error prone. But what if the ssh connection drops, what restarts the tunnel then? IPSec has restart mechanism for when the net drops which it will do.

khanjahanzaib27 9y ago

Cool Keep it up.

morpheuskafka 9y ago

[Zerotier](https://zerotier.com https://zerotier.com) is far nicer than all of them, clients (Win, macOS, GNU\Linux, OpenWrt, iOS, Android, NAS) are open-source and the hosted service allows up to 100 devices on one of unlimited networks for free. Because it emulates the ethernet layer, there is no client side configuration required and it can be used for non-IP packets (ex. DHCP). By enabling the default route permission on the client it can also be used as a gateway to the internet through a remote host on the network. All communications are e2e AES-256 with certificate-based access for private networks. Oh, and they also run a public earth network with unlimited members billed as "the global LAN party." It's a really convenient service, and the performance is good enough to be used between private clouds.

mjcl 9y ago
> non-IP packets (ex. DHCP) DHCP is IP/UDP.
- morpheuskafka 9y ago
  True, I was thinking of NetBUEI/IPX/AppleTalk/etc.
yjftsjthsd-h 9y ago
I'm really tempted to use this, but I'm a little nervous about relying on a company for this, especially because it doesn't look like they should be profitable on their current offerings.
- ac2u 9y ago
  Correct me if wrong, but I think that aside from their web based portal, everything else is open source. So you could control everything via API. So if they ever did go out of business, you could host it yourself and I'm sure by then an open source web portal would emerge to take the main site's place. (Not to mention that they'd probably just release the web app if they were going out of business). Hope it never comes to that though, I'd imagine their main bread and butter would be larger enterprise licenses.

sanbor 9y ago

There is a very simple test: fast.com. I tried fast.com over sshuttle and OpenVPN and OpenVPN was faster while sshuttle kind of collapsed.

sireat 9y ago

What's the fastest option when you do need UDP? Generally I just use SSH, but for gaming i've resorted to 160bit OpenVPN. Incidentally setting up a OpenVPN server on dd-wrt is a rather hellish experience.

saasproduct 9y ago

I use https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle but it's often too slow. I am wonder if their exist any go/rust equivalent of this project.

skarap 9y ago

`ip link set qlen 1000 dev tun0` (so it matches the eth0 queue, instead of being an order of magnitude smaller) on both ends of the tunnel, and you get 90% of direct connection speed from openvpn: $ iperf3 -c 10.1.0.2 Connecting to host 10.1.0.2, port 5201 [ 4] local 10.1.0.1 port 38274 connected to 10.1.0.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 3.12 MBytes 26.1 Mbits/sec 0 802 KBytes [ 4] 1.00-2.00 sec 22.3 MBytes 187 Mbits/sec 0 2.24 MBytes [ 4] 2.00-3.00 sec 21.4 MBytes 179 Mbits/sec 0 2.45 MBytes [ 4] 3.00-4.00 sec 23.3 MBytes 196 Mbits/sec 0 2.39 MBytes [ 4] 4.00-5.00 sec 23.4 MBytes 196 Mbits/sec 0 2.58 MBytes [ 4] 5.00-6.00 sec 21.5 MBytes 180 Mbits/sec 0 2.70 MBytes [ 4] 6.00-7.00 sec 23.5 MBytes 197 Mbits/sec 0 2.68 MBytes [ 4] 7.00-8.00 sec 22.6 MBytes 189 Mbits/sec 0 2.78 MBytes [ 4] 8.00-9.00 sec 22.4 MBytes 188 Mbits/sec 0 2.45 MBytes [ 4] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec 0 2.69 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 207 MBytes 174 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 207 MBytes 174 Mbits/sec receiver iperf Done.