6 ms·

Secure your code: OpenVPN in the cloud

66 points by trefn 16y ago

trun 16y ago
I was just looking for something like this, literally about 30 seconds before you posted it. Thanks!
tptacek 16y ago
I'm not totally wild about OpenVPN. If you asked me which I'd rather expose, SSH or OpenVPN, I'd have to flip a coin. In either case, it's easy to realize the "right" configuration, with a single "bastion" host (hate that word) that provides authorized clients with transparent access to servers. One thing you can say for SSH is that it gets a lot more audit attention than OpenVPN does. In either case, the problem I think most startups should move to solve early is multi-factor authentication. I have friends building things with Yubikeys, and others building things with SMS-based auth; either are better than straight passwords, which are (among other things) a policy nightmare.
- CrLf 16y ago
  OpenVPN can be configured to be stealthy (to start a connection, the first packet must be signed with a shared-secret, otherwise it doesn't respond), while SSH doesn't. And OpenVPN has the performance advantage (tunneling is done over UDP instead of SSH's TCP - TCP over TCP brings about a couple of performance problems, especially when there's congestion).
  - eli 16y ago
    Couldn't you just set up a port knocking for ssh (or anything else for that matter)?
  - tptacek 16y ago
    The whole stealth thing (and "port knocking" along with it) is wildly overrated. If your security depends on people not knowing you're running SSH or OpenVPN, you have bigger problems. If you want to get all cutesy, go hack SSH to add an extra round or two to OpenSSL's AES. That'll learn 'em.
    - dododo 16y ago
      it depends upon your threat model. there's probably a bug in some code on your system that's exploitable. maybe openvpn or ssh or its configuration (c.f., debian crypto bug). under this model, anything that increases the time til (inevitable, under this threat model) success, works. running ssh or openvpn on a different port or whatever to "hide" it, will help against mass scans for common exploits. it increases the work to find the problem. of course it won't really help against a concerted, targeted attack. but i'm pretty sure unskilled, mass attacks are more common than targeted attacks.
      - tptacek 16y ago
        The guy with the OpenVPN zero day probably isn't getting slowed down by your stealth configuration. At least, that's how my threat model works. If you believe in stuff like this, then do it right: require the attacker to know a secret algorithm (like "AES with 3 extra rounds") to talk to you.
        dododo 16y ago
        does the guy with zero day want to: 1. break into your machine? 2. break into some machine? if (1) then i certainly agree that none of this really matters (i apologise if that wasn't clear from my previous comment). on the other hand, if his objective is (2), and not everyone is using some of this obfuscation, then this separates you from the crowd for a little while (but my model is that as time goes on, you are inevitably exploited). i realise after your company was exploited it might make you feel like (1) is always the case but for many (2) is the salient threat.
        tptacek 16y ago
        Generally speaking, the people running around with OpenVPN and SSH zero day are looking to break into your machine. The people looking to break into any machine are either targeting Windows clientsides, or weeks-to-months-old web vulnerabilities. People with SSH zero-day are not, by and large, looking to burn those vulnerabilities by spraying them into every busybody's honey pot logs.
- acqq 16y ago
  Well, wasn't SSH de facto broken for almost two years on Debian distros, due to the bad random number generation bug?
  - tptacek 16y ago
    Everything crypto on Debian was tainted. OpenVPN also depends on OpenSSL.
- modoc 16y ago
  Multi-factor is a big deal, especially if you ever want to deal with credit cards at all. We looked at WiKID (open source two factor solution) but building a LDAP backed VPN solution around it was going to be a fair bit of work, so we ended up using PhoneFactor, which is a pretty cheap commercial offering using your cell phone instead of physical token. Big fan.
- theBobMcCormick 16y ago
  If they only thing you're planning to run over your VPN is SSH, then yeah, it's a coin toss whether you should run a VPN or just expose OpenSSH to the internet. But if you're planning on running anything else over your VPN, then I'd wildly disagree that the two are equivalent. Someone else already mentioned the performance advantage, but just the user convenience advantage alone would be a deciding factor for me.
  - tptacek 16y ago
    I'm considering the security of the two programs, not the convenience --- although I think SSH is more convenient than you give it credit for.
  - ams6110 16y ago
    You can tunnel other protocols over SSH.
    - theBobMcCormick 16y ago
      Well aware of that, but it can be a bit of a pain if you're tunneling more than a couple protocol or if your tunneling to more than a couple of hosts. Also, VPN let's you use the same configurations as you would if you were local to the machines, you don't have to use localhost, etc. And of course, there's also well known performance problems tunneling over SSH because you incur a double hit from the TCP backoff algorithm. Not to mention that you can't (easily) tunnel udp over SSH.... Come to think of it, there's no real comparison. SSH is a great remote shell protocol, but a shitty VPN protocol (not surprising since it's not * intended* to be a VPN protocol).
- apenwarr 16y ago
  If you would rather use ssh than OpenVPN but still want the convenience of "direct" connections to computers on the other side of the gateway host, and you want to avoid the tcp-over-tcp slowness problem, try my sshuttle project: http://github.com/apenwarr/sshuttle http://github.com/apenwarr/sshuttle
- ams6110 16y ago
  Yeah I'd go the ssh route myself. Nothing against OpenVPN, but the (or maybe another) big problem in this scenario is that people are using passwords to log into servers in the first place.
  - iuguy 16y ago
    OpenVPN supports certificate based auth but key-based SSH auth is considerably easier to do (and depending on your experience may be easier to manage). You can also restrict the commands that can be run so you can use things like Git or tunnel MySQL loopback connections without handing over the box.
ritonlajoie 16y ago
we are doing that in our production environment. We use different boxes (1 on US and 2 on EU), they are connected with openvpn to serv our different needs. Our personal development boxes are also connected to each others using OPENVPN and firewalled. All the internal code made for our app is using the VPN ips, and every application is listening on these IP only (not localhost nor 0.0.0.0) (applications are mysql/couchdb/git/apache) That's pretty great !
aguynamedben 16y ago
tinc is really good little secret I have found if you want a true mesh P2P secure overlay network. It's open source, has a good community in freenode #tinc, and really easy to install and use, just read the man page http://linux.die.net/man/5/tinc.conf http://linux.die.net/man/5/tinc.conf http://tinc-vpn.org/ http://tinc-vpn.org/
- pilif 16y ago
  can you tell me what the actual difference between OpenVPN and tinc is? I know OpenVPN very well and really like it. Why would I prefer tinc over OpenVPN (or the other way around)? In my opinion, they share a lot of the same features like running in userspace, easy configuration, ssl/certificate based authentication - just the same.
  - mike-cardwell 16y ago
    Front page of the project website: "Automatic full mesh routing. Regardless of how you set up the tinc daemons to connect to each other, VPN traffic is always (if possible) sent directly to the destination, without going through intermediate hop" OpenVPN doesn't do that, and it sounds very useful when multiple machines are involved
- tptacek 16y ago
  I strongly recommend not relying on the VPN written in C that nobody's heard of that uses a hand-rolled custom RSA authentication algorithm to authenticate connections for your perimeter security.
pmjordan 16y ago
The diagram has connection lines between all nodes; does this imply N² point-to-point OpenVPN connections, with a subnet for each connection?
- lobo-tuerto 16y ago
  That diagram was drawn like that just for exemplifying purposes. Just to illustrate what network interfaces you would use for communicating between server processes.
- trefn 16y ago
  The lines between the nodes were actually meant to show that those nodes use the Rackspace internal network (ServiceNet) for communication - see the colored labels on the diagram.
  - pmjordan 16y ago
    Ohh, I see, blue = unencrypted. That makes sense, thanks. So there really isn't any gain over using SSH with private key authentication with this?
    - mseebach 16y ago
      Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything). Second, getting a dump of a table from your MySQL DB with SSH+keys: ssh prod-db mysqldump what-i-need > file.sql exit scp prod-db:file.sql . With VPN (assuming the connection is running): mysqldump -h prod-db whatever > file.sql (I'm not saying it's neccesarily a good idea to du dumps of your production db, but that was the example that came to mind.) There's also stuff like instrumenting a running Tomcat with JVisualVM on a server in your remote environment (which, no, you can't easily do over SSH-forwarded ports.. also, portforwarding is annoying). Finally, VPN allows you to run internal services. A jabber-server was mentioned in the article, and perhaps a webserver with some reports/dashboard thing, cacti/nagios stuff etc. without worrying about setting up authentication and SSL for each. Each VPN user gets his own IP, so you can just do crude IP allow/deny if you need, or you can put the customer people on one subnet and allow them to access the reports server, and the devs/ops on another that can reach everything.
dugmartin 16y ago
We have been using OpenVPN to secure one of our Rackspace servers dedicated to development for the last couple of years. We also have ejabberd running on that box (just talking to the vpn interface). It works great for distributed development and the internal secured jabber IMs are great for transferring sensitive stuff like passwords between remote folks. We also have a persistent conference room for a shared "watercooler". The main problem we have is key management. Does anyone know of a good admin app for generating, storing and revoking keys?
- thwarted 16y ago
  I've been using tinyca2, which has a really nice gnome interface for managing a certificate authority. I'm really happy with it and it takes a lot of the pain out of CA and certificate management.
- trafficlight 16y ago
  Are you still generating keys by hand or have you found something that works a bit better? I've been looking around for an admin app as well. It would be even better if I could tie it into an intranet site as well.
- iuguy 16y ago
  Interesting. We ran ejabberd over SSL for years without needing a VPN. Most DVCS' support SSH in some form or another. You might find SSH key management a bit easier.
steve19 16y ago
My problem with OpenVPN is that you cannot access it from an iPhone.

66 points by trefn 16y ago

trun 16y ago

I was just looking for something like this, literally about 30 seconds before you posted it. Thanks!

I'm not totally wild about OpenVPN. If you asked me which I'd rather expose, SSH or OpenVPN, I'd have to flip a coin. In either case, it's easy to realize the "right" configuration, with a single "bastion" host (hate that word) that provides authorized clients with transparent access to servers. One thing you can say for SSH is that it gets a lot more audit attention than OpenVPN does. In either case, the problem I think most startups should move to solve early is multi-factor authentication. I have friends building things with Yubikeys, and others building things with SMS-based auth; either are better than straight passwords, which are (among other things) a policy nightmare.

CrLf 16y ago
OpenVPN can be configured to be stealthy (to start a connection, the first packet must be signed with a shared-secret, otherwise it doesn't respond), while SSH doesn't. And OpenVPN has the performance advantage (tunneling is done over UDP instead of SSH's TCP - TCP over TCP brings about a couple of performance problems, especially when there's congestion).
- eli 16y ago
  Couldn't you just set up a port knocking for ssh (or anything else for that matter)?
- tptacek 16y ago
  The whole stealth thing (and "port knocking" along with it) is wildly overrated. If your security depends on people not knowing you're running SSH or OpenVPN, you have bigger problems. If you want to get all cutesy, go hack SSH to add an extra round or two to OpenSSL's AES. That'll learn 'em.
  - dododo 16y ago
    it depends upon your threat model. there's probably a bug in some code on your system that's exploitable. maybe openvpn or ssh or its configuration (c.f., debian crypto bug). under this model, anything that increases the time til (inevitable, under this threat model) success, works. running ssh or openvpn on a different port or whatever to "hide" it, will help against mass scans for common exploits. it increases the work to find the problem. of course it won't really help against a concerted, targeted attack. but i'm pretty sure unskilled, mass attacks are more common than targeted attacks.
    - tptacek 16y ago
      The guy with the OpenVPN zero day probably isn't getting slowed down by your stealth configuration. At least, that's how my threat model works. If you believe in stuff like this, then do it right: require the attacker to know a secret algorithm (like "AES with 3 extra rounds") to talk to you.
      - dododo 16y ago
        does the guy with zero day want to: 1. break into your machine? 2. break into some machine? if (1) then i certainly agree that none of this really matters (i apologise if that wasn't clear from my previous comment). on the other hand, if his objective is (2), and not everyone is using some of this obfuscation, then this separates you from the crowd for a little while (but my model is that as time goes on, you are inevitably exploited). i realise after your company was exploited it might make you feel like (1) is always the case but for many (2) is the salient threat.
        tptacek 16y ago
        Generally speaking, the people running around with OpenVPN and SSH zero day are looking to break into your machine. The people looking to break into any machine are either targeting Windows clientsides, or weeks-to-months-old web vulnerabilities. People with SSH zero-day are not, by and large, looking to burn those vulnerabilities by spraying them into every busybody's honey pot logs.
acqq 16y ago
Well, wasn't SSH de facto broken for almost two years on Debian distros, due to the bad random number generation bug?
- tptacek 16y ago
  Everything crypto on Debian was tainted. OpenVPN also depends on OpenSSL.
modoc 16y ago
Multi-factor is a big deal, especially if you ever want to deal with credit cards at all. We looked at WiKID (open source two factor solution) but building a LDAP backed VPN solution around it was going to be a fair bit of work, so we ended up using PhoneFactor, which is a pretty cheap commercial offering using your cell phone instead of physical token. Big fan.
theBobMcCormick 16y ago
If they only thing you're planning to run over your VPN is SSH, then yeah, it's a coin toss whether you should run a VPN or just expose OpenSSH to the internet. But if you're planning on running anything else over your VPN, then I'd wildly disagree that the two are equivalent. Someone else already mentioned the performance advantage, but just the user convenience advantage alone would be a deciding factor for me.
- tptacek 16y ago
  I'm considering the security of the two programs, not the convenience --- although I think SSH is more convenient than you give it credit for.
- ams6110 16y ago
  You can tunnel other protocols over SSH.
  - theBobMcCormick 16y ago
    Well aware of that, but it can be a bit of a pain if you're tunneling more than a couple protocol or if your tunneling to more than a couple of hosts. Also, VPN let's you use the same configurations as you would if you were local to the machines, you don't have to use localhost, etc. And of course, there's also well known performance problems tunneling over SSH because you incur a double hit from the TCP backoff algorithm. Not to mention that you can't (easily) tunnel udp over SSH.... Come to think of it, there's no real comparison. SSH is a great remote shell protocol, but a shitty VPN protocol (not surprising since it's not * intended* to be a VPN protocol).
apenwarr 16y ago
If you would rather use ssh than OpenVPN but still want the convenience of "direct" connections to computers on the other side of the gateway host, and you want to avoid the tcp-over-tcp slowness problem, try my sshuttle project: http://github.com/apenwarr/sshuttle http://github.com/apenwarr/sshuttle
ams6110 16y ago
Yeah I'd go the ssh route myself. Nothing against OpenVPN, but the (or maybe another) big problem in this scenario is that people are using passwords to log into servers in the first place.
- iuguy 16y ago
  OpenVPN supports certificate based auth but key-based SSH auth is considerably easier to do (and depending on your experience may be easier to manage). You can also restrict the commands that can be run so you can use things like Git or tunnel MySQL loopback connections without handing over the box.

ritonlajoie 16y ago

we are doing that in our production environment. We use different boxes (1 on US and 2 on EU), they are connected with openvpn to serv our different needs. Our personal development boxes are also connected to each others using OPENVPN and firewalled. All the internal code made for our app is using the VPN ips, and every application is listening on these IP only (not localhost nor 0.0.0.0) (applications are mysql/couchdb/git/apache) That's pretty great !

aguynamedben 16y ago

tinc is really good little secret I have found if you want a true mesh P2P secure overlay network. It's open source, has a good community in freenode #tinc, and really easy to install and use, just read the man page http://linux.die.net/man/5/tinc.conf http://linux.die.net/man/5/tinc.conf http://tinc-vpn.org/ http://tinc-vpn.org/

pilif 16y ago
can you tell me what the actual difference between OpenVPN and tinc is? I know OpenVPN very well and really like it. Why would I prefer tinc over OpenVPN (or the other way around)? In my opinion, they share a lot of the same features like running in userspace, easy configuration, ssl/certificate based authentication - just the same.
- mike-cardwell 16y ago
  Front page of the project website: "Automatic full mesh routing. Regardless of how you set up the tinc daemons to connect to each other, VPN traffic is always (if possible) sent directly to the destination, without going through intermediate hop" OpenVPN doesn't do that, and it sounds very useful when multiple machines are involved
tptacek 16y ago
I strongly recommend not relying on the VPN written in C that nobody's heard of that uses a hand-rolled custom RSA authentication algorithm to authenticate connections for your perimeter security.

pmjordan 16y ago

The diagram has connection lines between all nodes; does this imply N² point-to-point OpenVPN connections, with a subnet for each connection?

lobo-tuerto 16y ago
That diagram was drawn like that just for exemplifying purposes. Just to illustrate what network interfaces you would use for communicating between server processes.
trefn 16y ago
The lines between the nodes were actually meant to show that those nodes use the Rackspace internal network (ServiceNet) for communication - see the colored labels on the diagram.
- pmjordan 16y ago
  Ohh, I see, blue = unencrypted. That makes sense, thanks. So there really isn't any gain over using SSH with private key authentication with this?
  - mseebach 16y ago
    Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything). Second, getting a dump of a table from your MySQL DB with SSH+keys: ssh prod-db mysqldump what-i-need > file.sql exit scp prod-db:file.sql . With VPN (assuming the connection is running): mysqldump -h prod-db whatever > file.sql (I'm not saying it's neccesarily a good idea to du dumps of your production db, but that was the example that came to mind.) There's also stuff like instrumenting a running Tomcat with JVisualVM on a server in your remote environment (which, no, you can't easily do over SSH-forwarded ports.. also, portforwarding is annoying). Finally, VPN allows you to run internal services. A jabber-server was mentioned in the article, and perhaps a webserver with some reports/dashboard thing, cacti/nagios stuff etc. without worrying about setting up authentication and SSL for each. Each VPN user gets his own IP, so you can just do crude IP allow/deny if you need, or you can put the customer people on one subnet and allow them to access the reports server, and the devs/ops on another that can reach everything.

dugmartin 16y ago

We have been using OpenVPN to secure one of our Rackspace servers dedicated to development for the last couple of years. We also have ejabberd running on that box (just talking to the vpn interface). It works great for distributed development and the internal secured jabber IMs are great for transferring sensitive stuff like passwords between remote folks. We also have a persistent conference room for a shared "watercooler". The main problem we have is key management. Does anyone know of a good admin app for generating, storing and revoking keys?

thwarted 16y ago
I've been using tinyca2, which has a really nice gnome interface for managing a certificate authority. I'm really happy with it and it takes a lot of the pain out of CA and certificate management.
trafficlight 16y ago
Are you still generating keys by hand or have you found something that works a bit better? I've been looking around for an admin app as well. It would be even better if I could tie it into an intranet site as well.
iuguy 16y ago
Interesting. We ran ejabberd over SSL for years without needing a VPN. Most DVCS' support SSH in some form or another. You might find SSH key management a bit easier.

steve19 16y ago

My problem with OpenVPN is that you cannot access it from an iPhone.