5 ms·

WireGuard is just a transport protocol, so of course you could use it in place of SSL/TLS if you wanted. Interesting though, and I prefer it to SSL/TLS because

by api 5y ago

WireGuard is just a transport protocol, so of course you could use it in place of SSL/TLS if you wanted. Interesting though, and I prefer it to SSL/TLS because X509 certs suck.

tptacek 5y ago
WireGuard isn't really the interesting bit here, it's running TCP/IP over it in userland. You cannot straightforwardly do that with SSL/TLS, but it is in fact the API that WireGuard provides.
- sho_hn 5y ago
  This is where the article lost me a little bit. I (think I) technically got the part of running a TCP/IP stack in an unprivileged user process, so you don't have to elevate privilege for adding a network interface and using the host OS TCP/IP stack. And maybe that's already very cool. But: - What other benefits does it give you? - This isn't a new problem and presumably has prior best practices for mitigation. What is this replacing, what was the landscape like before this? What was the most similar already? - Have people been looking for a better solution like this for some time? - What's the over/under on the maintenance cost? You added in another TCP/IP stack to look after. You maybe save on static configuration and can make your system more dynamic. Pros, cons ... let's list them. The talks a bit about problems they were trying to address, but not in a way that clearly answered the above for me. It's of course valid to write a piece with a more informed audience in mind, but in something that aims to spread the virtues of an idea I think it could do more.
  - dolmen 5y ago
    The answer by tptacek with pros/cons is here: https://news.ycombinator.com/item?id=30277278 https://news.ycombinator.com/item?id=30277278
    - sho_hn 5y ago
      Thanks to both of you!
  - tptacek 5y ago
    This is a good question and part of the reason you didn't get a clear answer from the article is that I'm not sure if I have a clear answer. What I think user-mode TCP/IP gives us is the ability to build arbitrary services --- Postgres, Redis, SSH, network management, whatever --- without having to make infrastructure changes. We don't have to have some weird API or application proxy that knows what's running and who's allowed to run what. Instead, that's simply baked into the network, and flyctl, by dint of netstack, can just use it. If somebody comes up with a cool network service to plug flyctl (or any other tool someone wants to write) into, it will just work. But things like the maintenance cost, well, yeah, that's most of what the post is about. The maintenance cost was not especially low.† There's a natural inclination to read any post like this as a kind of brag, but I'm really just experimenting with trying to show the good with the bad here. User-mode TCP/IP is a weird choice! Nobody else I know of does it! It might have been the wrong choice! Even though I love it! † It's actually not low even right now; I'm spending the first half of the day deploying code that relays stats from Netlink on our gateways through our GraphQL API, so that flyctl can check WireGuard gateway health. That is not a thing we would be spending time on if we had just written an explicit proxy for Postgres or whatever, rather than providing a generic network transport.
    - sho_hn 5y ago
      That's cool, and I appreciate you sharing these experiments. I'm in automotive/embedded at the moment, and our daily battle is making decisions on how static vs. dynamic we want our system to be - static (e.g. resource allocations or baked-in scheduling decisions) makes it easier to reason about the system and provide guarantees, but generally lowers efficiency at runtime. Dynamic can make the system much better at serving a wide range of usage scenarios, but makes it harder to eliminate the risk of pathological cases. It's hard not to see things through that lens. The way to construct and run services you've described here to me is an interesting option on that type of axis, in the sense of where the costs/friction goes.
      - tptacek 5y ago
        I wish I'd thought of writing a simple pro/con list when I wrote this post. I'll think about that!
  - zaphar 5y ago
    The problem isn't new but the previous best practices involved giving the tool super user. Usually through an installation process. See most other VPNs. UserMode TCP/IP stacks aren't very common in practice. This is why what fly.io did is interesting.
    - touisteur 5y ago
      Or network namespaces? User-space tcp might also make it easier to do tcp checkpoint restore and container/app migration. Interesting write-up. I keep thinking of the nightmare of keeping up with the world of Internet middleboxens, broken net layer implementations and icmp hacks that the Linux kernel supports and makes 'just work'. The jump to usermode tcp seems interesting if you're not worried about that (and I've been watching the formal-proven ip/tcp stack space like a hawk for years), but I've been burned so many times with non standard stacks and 'oh you need to connect to that non-updated lynxos system and huh' or 'hah could you enable ecn or this obscure tcp option because... Legacy?'... And sometimes I need tc/netem and netlink and I don't know...
      - ignoramous 5y ago
        We embed LwIP in an Android app, and it doesn't even implement most of TCP/IP features let alone handle most of the protocol's quirks. But, it mostly works as expected, because of Postel's law on the other side (for ex, https://apenwarr.ca/log/20090222 https://apenwarr.ca/log/20090222)
derekzhouzhen 5y ago
It is not just that. WireGuard is TCP tunneled through UDP. A TCP connection tunneled through another layer of TCP will suck badly performance wise. The sliding window algorithm in both layers will fight each other.
qbasic_forever 5y ago
Replacing SSL/TLS with wireguard is cool but aren't you just going to run into the same headaches of rotating certificates/keys? No one is really going to rely on using the same wireguard key indefinitely, right?
- tptacek 5y ago
  It's pretty easy for us to rotate keys now, since new WireGuard peers are extremely cheap to bring up (part of the point of the post is that for most of the last year, that was the opposite of the case, and a new peer was a very painful thing to ask for). But rotating WireGuard keys with Fly.io makes about as much sense as rotating the OAuth2 API token `flyctl` uses (the token is strictly more powerful than the WireGuard key), and people generally don't do that.
  - ignoramous 5y ago
    Please write a bit about the secrets-infrastructure at fly.io! The cert store, the token store, the trade-offs, the protections around it (though, I'm sure we will judge you for it, especially if it isn't "secure enough" for any made up definition of "secure").

by api 5y ago

WireGuard is just a transport protocol, so of course you could use it in place of SSL/TLS if you wanted. Interesting though, and I prefer it to SSL/TLS because X509 certs suck.

tptacek 5y ago

WireGuard isn't really the interesting bit here, it's running TCP/IP over it in userland. You cannot straightforwardly do that with SSL/TLS, but it is in fact the API that WireGuard provides.

sho_hn 5y ago
This is where the article lost me a little bit. I (think I) technically got the part of running a TCP/IP stack in an unprivileged user process, so you don't have to elevate privilege for adding a network interface and using the host OS TCP/IP stack. And maybe that's already very cool. But: - What other benefits does it give you? - This isn't a new problem and presumably has prior best practices for mitigation. What is this replacing, what was the landscape like before this? What was the most similar already? - Have people been looking for a better solution like this for some time? - What's the over/under on the maintenance cost? You added in another TCP/IP stack to look after. You maybe save on static configuration and can make your system more dynamic. Pros, cons ... let's list them. The talks a bit about problems they were trying to address, but not in a way that clearly answered the above for me. It's of course valid to write a piece with a more informed audience in mind, but in something that aims to spread the virtues of an idea I think it could do more.
- dolmen 5y ago
  The answer by tptacek with pros/cons is here: https://news.ycombinator.com/item?id=30277278 https://news.ycombinator.com/item?id=30277278
  - sho_hn 5y ago
    Thanks to both of you!
- tptacek 5y ago
  This is a good question and part of the reason you didn't get a clear answer from the article is that I'm not sure if I have a clear answer. What I think user-mode TCP/IP gives us is the ability to build arbitrary services --- Postgres, Redis, SSH, network management, whatever --- without having to make infrastructure changes. We don't have to have some weird API or application proxy that knows what's running and who's allowed to run what. Instead, that's simply baked into the network, and flyctl, by dint of netstack, can just use it. If somebody comes up with a cool network service to plug flyctl (or any other tool someone wants to write) into, it will just work. But things like the maintenance cost, well, yeah, that's most of what the post is about. The maintenance cost was not especially low.† There's a natural inclination to read any post like this as a kind of brag, but I'm really just experimenting with trying to show the good with the bad here. User-mode TCP/IP is a weird choice! Nobody else I know of does it! It might have been the wrong choice! Even though I love it! † It's actually not low even right now; I'm spending the first half of the day deploying code that relays stats from Netlink on our gateways through our GraphQL API, so that flyctl can check WireGuard gateway health. That is not a thing we would be spending time on if we had just written an explicit proxy for Postgres or whatever, rather than providing a generic network transport.
  - sho_hn 5y ago
    That's cool, and I appreciate you sharing these experiments. I'm in automotive/embedded at the moment, and our daily battle is making decisions on how static vs. dynamic we want our system to be - static (e.g. resource allocations or baked-in scheduling decisions) makes it easier to reason about the system and provide guarantees, but generally lowers efficiency at runtime. Dynamic can make the system much better at serving a wide range of usage scenarios, but makes it harder to eliminate the risk of pathological cases. It's hard not to see things through that lens. The way to construct and run services you've described here to me is an interesting option on that type of axis, in the sense of where the costs/friction goes.
    - tptacek 5y ago
      I wish I'd thought of writing a simple pro/con list when I wrote this post. I'll think about that!
- zaphar 5y ago
  The problem isn't new but the previous best practices involved giving the tool super user. Usually through an installation process. See most other VPNs. UserMode TCP/IP stacks aren't very common in practice. This is why what fly.io did is interesting.
  - touisteur 5y ago
    Or network namespaces? User-space tcp might also make it easier to do tcp checkpoint restore and container/app migration. Interesting write-up. I keep thinking of the nightmare of keeping up with the world of Internet middleboxens, broken net layer implementations and icmp hacks that the Linux kernel supports and makes 'just work'. The jump to usermode tcp seems interesting if you're not worried about that (and I've been watching the formal-proven ip/tcp stack space like a hawk for years), but I've been burned so many times with non standard stacks and 'oh you need to connect to that non-updated lynxos system and huh' or 'hah could you enable ecn or this obscure tcp option because... Legacy?'... And sometimes I need tc/netem and netlink and I don't know...
    - ignoramous 5y ago
      We embed LwIP in an Android app, and it doesn't even implement most of TCP/IP features let alone handle most of the protocol's quirks. But, it mostly works as expected, because of Postel's law on the other side (for ex, https://apenwarr.ca/log/20090222 https://apenwarr.ca/log/20090222)

derekzhouzhen 5y ago

It is not just that. WireGuard is TCP tunneled through UDP. A TCP connection tunneled through another layer of TCP will suck badly performance wise. The sliding window algorithm in both layers will fight each other.

qbasic_forever 5y ago

Replacing SSL/TLS with wireguard is cool but aren't you just going to run into the same headaches of rotating certificates/keys? No one is really going to rely on using the same wireguard key indefinitely, right?

tptacek 5y ago
It's pretty easy for us to rotate keys now, since new WireGuard peers are extremely cheap to bring up (part of the point of the post is that for most of the last year, that was the opposite of the case, and a new peer was a very painful thing to ask for). But rotating WireGuard keys with Fly.io makes about as much sense as rotating the OAuth2 API token `flyctl` uses (the token is strictly more powerful than the WireGuard key), and people generally don't do that.
- ignoramous 5y ago
  Please write a bit about the secrets-infrastructure at fly.io! The cert store, the token store, the trade-offs, the protections around it (though, I'm sure we will judge you for it, especially if it isn't "secure enough" for any made up definition of "secure").