13 ms·

JIT WireGuard

485 points by Lwrless 3y ago

irjustin 3y ago
For everyone else, I'll shamelessly use this to plug Netmaker[0]. Not affiliated, just a satisfied guy that needs access to private AWS VPCs across multiple accounts and would love to see them be more widely adopted. [0] https://www.netmaker.io/ https://www.netmaker.io/
- deleted 3y ago
  [deleted]
- porker 3y ago
  Is Netmaker like Tailscale? From their site I'm unclear what the distinguishing factor is.
  - thunfischbrot 3y ago
    Roughly, yes. Netmaker has a self-hostable server though. With tailscsle of course the 3rd-party headscale is available. Netbird also seems promising. See https://github.com/cedrickchee/awesome-wireguard https://github.com/cedrickchee/awesome-wireguard for more alternatives.
    - computeinbrain 3y ago
      Yes. Netbird is very recommendable: https://netbird.io/ https://netbird.io/
      - sneak 3y ago
        ...until you read the app privacy label. This app collects a lot of your data for being privacy software. Tell me again why p2p privacy software needs to phone home? https://apps.apple.com/us/app/netbird-p2p-vpn/id6469329339 https://apps.apple.com/us/app/netbird-p2p-vpn/id6469329339
        AnarchismIsCool 3y ago
        Using it in prod right now, not a fan. It will gleefully create a root ssh tunnel for you through its daemon if the box on the website is checked.
- denkmoon 3y ago
  Can’t you do that “AWS native” with private link or vpc peering? I’m a noob with these so I don’t understand the benefit of netmaker
  - irjustin 3y ago
    The goal isn't to make the networks seem like one and connect resources across accounts, which is what those products do. My goal is to access private resources via SSH bastion/jump machines in a specific account. There's a few ways to do this in AWS, but all of them are more costly by a pretty wide margin.
    - vasco 3y ago
      AWS VPN is pretty cost effective, we used if for a few years with a multi account setup. And it's pretty much zero work.
      - apitman 3y ago
        Unless you're transferring a lot of data, such as video streaming or copying files. Looks like $90/TB for egress.
    - seany 3y ago
      You can forward ssh through ssm, and dump that into your ssh config file. Works pretty nicely with some of the sso automation for the cli that's around these days.
      - rmccue 3y ago
        Is this using SSM’s raw port forwarding support? From what I’ve seen, their protocols seem to lack binary safety (we get weird encoding issues).
        zbentley 3y ago
        Without knowing the specifics of your situation, I would slightly suspect client/configuration if you’re encountering encoding issues. In my experience, integrating ssh and ssm is quite stable (provided you’re using OpenSSH and not a specific language client’s own implementation of the protocol).
- mtmk 3y ago
  Thanks, didn't know about this. I guess Netmaker (or similar) manage the keys for you which would make the admin a lot easier. In a previous job we setup and managed wg across a few Windows and Linux boxes using Ansible. It was OK but was getting a little messy in the end.
- deleted 3y ago
  [deleted]
- remram 3y ago
  What is it, a generic VPN platform? Similar to Tailscale etc? Their site is extremely vague.
rubatuga 3y ago
What's stopping the initial handshake packet from being replayed into the network stack? Seems like there would be no packets lost that way. Also, what is the purpose of checking for "udp[8] = 1" in the eBPF filter?
- zekica 3y ago
  udp[8] = 1 filters only handshake packets. Without it, data packets would also be sent to the userspace daemon. I'm not sure if initial handshakes can be replayed, but since WireGuard ignores unknown clients, it might be possible.
- tptacek 3y ago
  Nothing. It's a good idea. (As the sibling comment points out: the BPF filter snags just initiation packets, which is what we want; it's the WireGuard equivalent of sniffing for TCP SYNs to see connections starting).
- yencabulator 3y ago
  Yeah, sounds like an NFQUEUE helper that releases the packet after it has added the keys.
  - tptacek 3y ago
    Say more!
    - yencabulator 3y ago
      So NFQUEUE is an nftables verdict that puts the packet into a numbered queue going to userspace. A userspace process gets packets from the queue, and at some later point issues a verdict on each packet, which can be drop or allow the packet to pass. You can also pcap-style ask to see just first N bytes of the packet, to decrease overhead. Out of that, you can construct a userspace process that reads a packet from the queue, decrypts the noise initiation, requests configuration, adds wireguard config via netlink, and then releases the packet. And you can do that with multiple packets in flight concurrently. It also allows things like fail-open if the userspace process is broken or overwhelmed, which would be useful here (already in-kernel wg peers would keep working), and spreading load over multiple workers. https://netfilter.org/projects/nftables/manpage.html https://netfilter.org/projects/nftables/manpage.html (search for queue) https://wiki.nftables.org/wiki-nftables/index.php/Queueing_to_userspace https://wiki.nftables.org/wiki-nftables/index.php/Queueing_t... Here's a nice & simple Rust library for the userspace part, to give an idea of what the shape of the API is: https://docs.rs/nfq/latest/nfq/ https://docs.rs/nfq/latest/nfq/ Also, I'm available for contract work ;) Say hi to Ben from Tv.
RadiozRadioz 3y ago
[flagged]
- hug 3y ago
  Not yeeting them would appear to be a valid option, yes. You can leave them un-yote for as long as you wish.
  - teaearlgraycold 3y ago
    > un-yote :joy:
- denkmoon 3y ago
  Language is fluid :)
- Takennickname 3y ago
  Aren't you hip and with it?
  - ndsipa_pomu 3y ago
    They're so unhip, their trousers keep falling down
akira2501 3y ago
I always felt the disappointment of wireguard was wrapping it up into an opinionated network interface. It really should have been a generic "filter" that you could attach to any type of file handle. Then the configuration would be far less strongly coupled, less weirdly communicated to the kernel, and the status of your connection more immediately obvious. Plus, you could have wireguard files on your local or remote filesystems, or any character device, or named pipes if you felt like it. You could use a "jit" daemon to build tap or other interfaces for you, or just do it individually at the application layer. You could have pre registered keys with the kernel, or you could manage that directly, or generate them randomly. It's always been a weird smelling underspecified IPSEC clone to me, when it could have been so much more.
- mellutussa 3y ago
  But why don't use the nice smelling IPSEC if that ticks your boxes?
  - akira2501 3y ago
    It doesn't. It just foresaw the need to be able to dynamically configure tunnels on first connection and specified all of that. Which seems to me is a lot of what fly io has just mostly reimplemented here. In any case the point is I would prefer to just have the basic components available and let me piece them together however I want. Mostly to allow using the underlying technology in more contexts that it is currently available in.
    - medstrom 3y ago
      Heh, it really sounds like your needs would be better served with IPSec or something. WireGuard was born precisely because they saw that the whole problem making other existing solutions difficult to audit and insecure-in-practice was their thousand ways to configure. So they did the opposite. Low lines of code, few possibilities. In software you often choose between a small monolith and a big kitchen sink. Once you have 1 more need than the monolith covers, you have to go over to the kitchen sink.
d-z-m 3y ago
> We can install the peer as if we’re the initiator, and flyctl is the responder. The Linux kernel will initiate a WireGuard connection back to flyctl. This works; the protocol doesn’t care a whole lot who’s the server and who’s the client. We get new connections established about as fast as they can possibly be installed. Is this(effectively) adding a half round-trip to the handshake? i.e. 1. ->flyctl sends Initiation 2. <-peer is added via netlink(which causes new Initiation to be sent) 3. ->Response from flyctl
- OJFord 3y ago
  My reading was that both peers end up 'thinking' they initiated, but it doesn't matter. i.e. (3) either doesn't happen or just doesn't need to be waited for, or that they could even block (2-new initiation) and then it definitely wouldn't.
- saclark11 3y ago
  Pretty much, yes. If you imagine “Bob” has a policy that he can only converse with numbers in his address book, then you could think of it as: 1. -> Alice calls Bob 1.a. Bob does not pick up the call, but adds the number shown from caller ID to his address book 2. <- Bob calls the number (Alice) back 3. -> Alice picks up and they talk happily
niz4ts 3y ago
When I read this, I got a little too excited and thought they managed to get wireguard connections happening in the browser with webassembly (this isn't impossible, but the only attempt[0] I know of so far only works because of extra things tailscale has). It's an idea I've had for a project, but one I haven't had time to dedicate to (yet). In any case, really cool write-up! I wonder if they thought about making `flyctl` do a check with their API for any command that requires talking over wireguard to ensure the keys would be installed in the gateway. Since `flyctl` knows when the last command was run with it, it could do this only after some inactivity. And on the gateway machines, they'd just clean up any inactive peers with a cron (which they seem to be doing already). Not a solution as elegant as the one they reached (which is super cool), but I'm assuming the considerably lower effort would make it appealing. [0]: https://labs.leaningtech.com/blog/webvm-virtual-machine-with-networking-via-tailscale https://labs.leaningtech.com/blog/webvm-virtual-machine-with...
- apignotti 3y ago
  WebAssembly is not magic, the simple reality is that browsers do not expose low level socket interfaces, so they cannot connect to arbitrary services on the wider internet. We choose to use Tailscale since they allow WebSocket-based connections via their DERPs. It is interesting that, originally, DERPs were intended to be a solution for machines in extremely limited networking environment where nothing but HTTP is allowed. Turns out browsers are exactly one of those extremely limited networking environments.
  - apitman 3y ago
    > We choose to use Tailscale since they allow WebSocket-based connections via their DERPs. Some context that I wasn't initially aware of: apignotti is the CTO of Leaning Technologies, which is where the article GP linked is from.
- SparkyMcUnicorn 3y ago
  That's a separate blog post, and it definitely is pretty cool. https://tailscale.com/blog/ssh-console https://tailscale.com/blog/ssh-console https://www.npmjs.com/package/@tailscale/connect https://www.npmjs.com/package/@tailscale/connect
protoman3000 3y ago
> The problem you quickly run into to build this design is that Linux kernel WireGuard doesn’t have a feature for installing peers on demand. I don't seem to understand. You can add peers at runtime, e.g. https://serverfault.com/questions/1101002/wireguard-client-addition-without-restart https://serverfault.com/questions/1101002/wireguard-client-a... Can somebody clarify? EDIT: If I understand correctly, that step is already too late. They want to authenticate a peer before adding it to the interface in order to prevent stale entries on the interface. They thus put a eBPF filter in front of the interface and do the cryptokey-routing based association to an authorized counterpart by themselves. If it checks out, then they add the peer to the interface and remove it after a timeout.
- api 3y ago
  Seems to me they did this to avoid the alternative of running WG in user space. They wanted a feature the Linux kernel didn’t have to route by cryptographic address first but without leaving the kernel so they hacked it in.??? JIT Wireguard is a weird way to frame this. My mind went to “why? The performance bottleneck is the crypto and per client JIT won’t help with that.” I would have just gone user space. Use something like tokio-uring or glommio to get the performance. If you keep going in the kernel you are going to keep hitting limitations because Linux is not built to serve millions and millions of active tunnels. Even doing millions of TCP connections per kernel gets hairy sometimes. Every limitation will require a hack. Every hack will be some system config that has to be applied and managed. The tool chains for provisioning Linux metal boxes are vastly inferior to the tooling for developing apps and services and managing their config. Or am I stupid and misunderstanding?
  - vidarh 3y ago
    It does not seem like they need huge numbers of active tunnels per gateway. And JIT just as in "just in time" configuration of Wire guard. Once the configuration has been done, their stack stays out of it.
    - api 3y ago
justsomehnguy 3y ago
> and gateways with hundreds of thousands of peers that will never be used again My thoughts exactly as I was reading the first paragraphs. > Note that there’s no API call to subscribe for “incoming connection attempt” events. That’s OK! We can just make our own events. WireGuard connection requests are packets, and they’re easily identifiable, so we can efficiently snatch them with a BPF filter and a packet socket. Nice idea. > When we get an incoming initiation message, we have the 4-tuple address of the desired connection, including the ephemeral source port flyctl is using. We can install the peer as if we’re the initiator, and flyctl is the responder And this works behind NAT?
- chgs 3y ago
  If the packet goes back to the same ip/port and generated from the same ip/port it will work through nat.
- TheDong 3y ago
  > And this works behind NAT? Sure, UDP NAT only knows the 4-tuple (say {wggwd.fly.io, 12345, clientIP, 23456}). Any UDP packet, whether it's a new "initiator" udp packet, or a response to the outgoing initiation message, will look the exact same to any UDP NAT in the way since it only has the 4-tuple to go on, and the 4-tuple is the same.
  - justsomehnguy 3y ago
    The way it's written made me think it's flyctl sending it's own ip:port, which would be private ones behind the NAT. If it's the received packet source:port then it's, obviously, already translated.
  - ta1243 3y ago
    Presumably you could have a situation where there's deep-packet inspection on the traffic which would only allow a "handshake response" to come back through, and drop your attempt at an initialisation. I doubt that happens much, and I assume you'll fall back hapilly enough to the second time the "client" sends an init packet and then you simply respond with the response packet.
tinco 3y ago
While I generally agree with the idea that a direct HTTP request for a single point to point message would be more reliable than routing through a message queue, I'm a little bit surprised that there would be so many messages lost by NATS it had significant impact on their services. Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability?
- dilyevsky 3y ago
  > Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability? I believe if you're using core nats (not JetStream) there's no option for re-delivering like at all.
  - tinco 3y ago
    Well if that's what they did, then then using NATS over HTTP in that scenario is just switching a single point of failure for two single points of failure with no feedback on the second point, did they just pick it for the convenience of the NATS interface and only later realize their mistake?
    - emmanueloga_ 3y ago
      I’m guessing they realized late that core nats is a message broker without features like durability and at least once delivery. Jetstream was released less than a year ago (in nats 2.2) so perhaps by the time they made the switch it was not an option.
      - caleblloyd 3y ago
        NATS 2.2 with initial JetStream support was actually released 3 years ago.
- klabb3 3y ago
  I am also very curious to know more. I’m sure the NATS maintainers are as well. Their architecture is extremely intuitive and appealing to me, so I wonder where things went south. Nats has a lot of tunable parameters with Jetstream. For instance, an in-memory stream with a time-based duplicate detection window, both push and pull semantics, and configurable re-delivery and ack policies. The one thing where I can see an impedance mismatch is ephemeral single-message connections which I don’t think it’s built for. In either case, more details would be invaluable. (Fly folks, please consider sharing!)
dpeckett 3y ago
Might as well take the opportunity to shill one of my recent experimental projects, If you are interested in building Go apps that act as userspace WireGuard peers take a look at https://github.com/dpeckett/noisysockets https://github.com/dpeckett/noisysockets Based off the excellent work in done by wireguard-go but I've attempted to simplify and make things a lot more idiomatic for library use. I reckon building a service mesh out of it would be interesting, obviously supporting multiple languages would be hard but maybe you could implement a sockets API. Though it might be hard to compete performance wise with mTLS as I've not seen HW acceleration for WireGuards crypto yet. FWIW, I'm currently on the market for freelancing roles, so if you're interested in Golang freelancers in the high-speed/secure networking space, please reach out (email is in my profile).
- bscphil 3y ago
  This looks great, nice work! I have a dream of taking a userspace Wireguard project like this and gluing PAKE over a relay in front of it to exchange Wireguard keys, followed by holepunching and establishing a direct tunnel. Basically Magic Wormhole for arbitrary tunnels - and hopefully vastly improved performance for the file transfer case as well, rather than crapping out at 20-30 MB/s over long fat networks.
  - xyzzy_plugh 3y ago
    You're basically describing Tailscale.
    - mbreese 3y ago
      Maybe, but I read it as wormholes at the application level. Which would be awesome for me. I do a lot of work on remote servers for heavy data processing (large files). Sometimes I need to get these files back to my local computer, or just view figures I’ve generated using remote data. I have a small reverse tunnel daemon setup where I can (in my remote terminal) send arbitrary files or data back to my local computer. $ rtun send file.txt $ rtun view data.pdf It is a daemon that listens locally and when I setup SSH, a reverse tunnel is configured from one Unix socket to another. These are multiuser servers and Unix sockets handle authorization easily. The same setup could be handled similarly with a zmodem like process and aware terminal. However, this setup is a bit cumbersome and I can’t just run `ssh host`. If I could have the same setup with an in-app wireguard tunnel, with no other setup, it would be amazing.
sifatbinr 3y ago
[flagged]
mdavid626 3y ago
What does this mean? “We’ve gone a step beyond that: every time you run flyctl, our lovable, sprawling CLI, it conjures a TCP/IP stack out of thin air, with its own IPv6 address, and speaks directly to Fly Machines running on our networks.”
- makeworld 3y ago
  They're saying Wireguard is used.
  - Spivak 3y ago
    More than that. They're saying that they're running a userspace implementation of both the tcp/ip stack and wireguard. Your machine isn't a peer to the wireguard tunnel, only flyctl is.
    - dementik 3y ago
      Of course, your machine can be a peer also. You can create peers to your organization with `flyctl wireguard create`.
      - carl_dr 3y ago
        But that isn’t the case for all of the other uses of flyctl. You can’t make connections with other processes over the same connection flyctl makes when it’s doing stuff. That is what GP is saying.
- zrail 3y ago
  They wrote a blog post all about it: https://fly.io/blog/our-user-mode-wireguard-year/ https://fly.io/blog/our-user-mode-wireguard-year/
- ape4 3y ago
  Its hard for me to think of a sprawling CLI that I love.
- 3y ago
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
sneak 3y ago
Fun fact: the WireGuard macOS client application cannot work on macOS unless it's distributed via the App Store - Apple simply will not provide the required VPN entitlements for web/self distributed apps. You can use the commandline wg tools (which use a different OS API) but not the GUI ones. This means that WireGuard-the-org can't distribute the mac GUI app directly, and if you want to use the official WireGuard app (eg for accessing a VPN for privacy), somewhat ironically you have to ID yourself (which requires a phone number) and your computer's hardware serial to Apple first to download it via the App Store. It smacks of the totalitarian rms "right to read" essay - provide strong ID and hardware unique identifiers to be allowed to download and use privacy software on your own computer. https://www.wireguard.com/install/ https://www.wireguard.com/install/ While not directly relevant to fly.io, I figured this might be relevant in the context of Apple's other anticompetitive actions this week related to web-based native app distribution on iOS in the EU for the DMA. This issue with VPN apps has been true on macOS for years. I personally didn't know why WireGuard didn't offer direct downloads, but I emailed Jason Donenfeld a couple years ago and he let me know that Apple has been restricting these APIs in non-AppStore apps, which was news to me, as I didn't know that Apple had started any of their AppStore-only bullshit on macOS.
- Corrado 3y ago
  This is not entirely accurate. macOS now has System Extensions[0], which allow you to tie into the network to provide VPN services. This is what Tailscale uses in their non-App Store downloaded app. [0] https://developer.apple.com/system-extensions/ https://developer.apple.com/system-extensions/
  - sneak 3y ago
    https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_networking_networkextension https://developer.apple.com/documentation/bundleresources/en... It appears that they will now give out these entitlements for non-MAS apps, but only to members of their developer program. This means that you can't build one that will work on macOS without IDing yourself to Apple, so you still can't build it from scratch from source (as a non-developer-program person) and expect it to work (without disabling SIP).
tschumacher 3y ago
Sounds like a whole lot of effort to avoid a GraphQL request each time a flyctl client wants to connect.
- deleted 3y ago
  [deleted]
- sudhirj 3y ago
  Huh? They do make one to set it up. More a way to avoid having the public keys of every single client every loaded up into the wireguard kernel module on the gateway all the time.
  - tschumacher 3y ago
    My implicit suggestion was that clients make a GraphQL request not only before the first connection but before every connection. The gateway server can insert the keys into the kernel in response to an explicit GraphQL request instead of in response to some complicated packet sniffing.
    - rudasn 3y ago
      What would the payload of the grapphql request to fetch the wg config for that peer look like, when they don't know from which peer the request is coming from?
    - mrkurt 3y ago
      This needs to support any ol' wireguard client. We use it in `flyctl` but people also use it to create gateways so they can, eg, peer with VPCs.
chairmanwow1 3y ago
My startup used Fly for almost a year. The core feature of code to deployed code in less than a minute is beautiful. Spinning up / down new nodes for backfills takes seconds. But the company feels a little immature. Once our API server became unreachable in Fly for 48 hours. I'm not sure if it was my fault for getting config wrong or if they just had another "silent" failure. They have a "db" product, but it's "not managed postgres". Would get consistent disconnections from that. Just feels weird for them to add a top level noun in their cli for postgres and then limit the extent it's a feature they support. API access to their core service would frequently go down and leave us waiting to deploy new service fixes. I miss the deployment experience, but I'm frankly happier with Cloud Run on GCP. Just way fewer "surprises" and much more complete documentation.
- ZeroCool2u 3y ago
  Fly looks great to me, though I've never had a chance to use it. For what it's worth though, Cloud Run on GCP is one of my top 3 favorite infrastructure/deployment tools, so you're setting the bar pretty high.
  - icedchai 3y ago
    Cloud Run is pretty slick... services, batch jobs, easy to deploy, very flexible.
  - WuxiFingerHold 3y ago
    > so you're setting the bar pretty high Why would anyone choose a provider that is inferior (apart from toying)? I mainly read two kind of stories from fly.io. Their promotional, but well written and interesting technical blogs like this one and stories about issues with their services and miscommunication. So, despite liking their blogs I don't consider using it.
    - apitman 3y ago
      Part of the reason I stick with Fly.io is because I want a rock solid version of what they do to exist, and they're the most likely people to eventually get there. That said, I've had very few issues with their platform, and I don't think it's ever caused downtime for my (admittedly very small) service.
cushpush 3y ago
What a chart.
apitman 3y ago
Interesting that they're defaulting to tunneling WireGuard over WebSockets. Not great for performance but probably fine for the DevOpsy stuff flyctl is used for. This is something I've wondered about for the future of QUIC/HTTP3. There's a nonzero chance network operators will just block UDP on port 443 altogether rather than properly handling it.
- tptacek 3y ago
  You can absolutely use native WireGuard, including from `flyctl` (it's an option you can set). When UDP doesn't work, it doesn't work at all, and it's hard to debug, so our default is for the thing that we know will work. (I say this ruefully, having lost the argument about what our default should be.)
  - apitman 3y ago
    I'm assuming you decided doing some sort of happy eyeballs type thing isn't worth the complexity?
    - tptacek 3y ago
      Yeah; we're a public cloud platform, not a VPN service, so the key was just making sure everybody could drive our services. The "normie" way to do all of this stuff, at existing public clouds, is just HTTP anyways.
unixhero 3y ago
How might I take an arbitrary docker mized application and deploy it on Fly.io? Please take my money
- tptacek 3y ago
  cd directory/with/Dockerfile flyctl launch

485 points by Lwrless 3y ago

For everyone else, I'll shamelessly use this to plug Netmaker[0]. Not affiliated, just a satisfied guy that needs access to private AWS VPCs across multiple accounts and would love to see them be more widely adopted. [0] https://www.netmaker.io/ https://www.netmaker.io/

deleted 3y ago
[deleted]
porker 3y ago
Is Netmaker like Tailscale? From their site I'm unclear what the distinguishing factor is.
- thunfischbrot 3y ago
  Roughly, yes. Netmaker has a self-hostable server though. With tailscsle of course the 3rd-party headscale is available. Netbird also seems promising. See https://github.com/cedrickchee/awesome-wireguard https://github.com/cedrickchee/awesome-wireguard for more alternatives.
  - computeinbrain 3y ago
    Yes. Netbird is very recommendable: https://netbird.io/ https://netbird.io/
    - sneak 3y ago
      ...until you read the app privacy label. This app collects a lot of your data for being privacy software. Tell me again why p2p privacy software needs to phone home? https://apps.apple.com/us/app/netbird-p2p-vpn/id6469329339 https://apps.apple.com/us/app/netbird-p2p-vpn/id6469329339
      - AnarchismIsCool 3y ago
        Using it in prod right now, not a fan. It will gleefully create a root ssh tunnel for you through its daemon if the box on the website is checked.
denkmoon 3y ago
Can’t you do that “AWS native” with private link or vpc peering? I’m a noob with these so I don’t understand the benefit of netmaker
- irjustin 3y ago
  The goal isn't to make the networks seem like one and connect resources across accounts, which is what those products do. My goal is to access private resources via SSH bastion/jump machines in a specific account. There's a few ways to do this in AWS, but all of them are more costly by a pretty wide margin.
  - vasco 3y ago
    AWS VPN is pretty cost effective, we used if for a few years with a multi account setup. And it's pretty much zero work.
    - apitman 3y ago
      Unless you're transferring a lot of data, such as video streaming or copying files. Looks like $90/TB for egress.
  - seany 3y ago
    You can forward ssh through ssm, and dump that into your ssh config file. Works pretty nicely with some of the sso automation for the cli that's around these days.
    - rmccue 3y ago
      Is this using SSM’s raw port forwarding support? From what I’ve seen, their protocols seem to lack binary safety (we get weird encoding issues).
      - zbentley 3y ago
        Without knowing the specifics of your situation, I would slightly suspect client/configuration if you’re encountering encoding issues. In my experience, integrating ssh and ssm is quite stable (provided you’re using OpenSSH and not a specific language client’s own implementation of the protocol).
mtmk 3y ago
Thanks, didn't know about this. I guess Netmaker (or similar) manage the keys for you which would make the admin a lot easier. In a previous job we setup and managed wg across a few Windows and Linux boxes using Ansible. It was OK but was getting a little messy in the end.
deleted 3y ago
[deleted]
remram 3y ago
What is it, a generic VPN platform? Similar to Tailscale etc? Their site is extremely vague.

rubatuga 3y ago

What's stopping the initial handshake packet from being replayed into the network stack? Seems like there would be no packets lost that way. Also, what is the purpose of checking for "udp[8] = 1" in the eBPF filter?

zekica 3y ago
udp[8] = 1 filters only handshake packets. Without it, data packets would also be sent to the userspace daemon. I'm not sure if initial handshakes can be replayed, but since WireGuard ignores unknown clients, it might be possible.
tptacek 3y ago
Nothing. It's a good idea. (As the sibling comment points out: the BPF filter snags just initiation packets, which is what we want; it's the WireGuard equivalent of sniffing for TCP SYNs to see connections starting).
yencabulator 3y ago
Yeah, sounds like an NFQUEUE helper that releases the packet after it has added the keys.
- tptacek 3y ago
  Say more!
  - yencabulator 3y ago
    So NFQUEUE is an nftables verdict that puts the packet into a numbered queue going to userspace. A userspace process gets packets from the queue, and at some later point issues a verdict on each packet, which can be drop or allow the packet to pass. You can also pcap-style ask to see just first N bytes of the packet, to decrease overhead. Out of that, you can construct a userspace process that reads a packet from the queue, decrypts the noise initiation, requests configuration, adds wireguard config via netlink, and then releases the packet. And you can do that with multiple packets in flight concurrently. It also allows things like fail-open if the userspace process is broken or overwhelmed, which would be useful here (already in-kernel wg peers would keep working), and spreading load over multiple workers. https://netfilter.org/projects/nftables/manpage.html https://netfilter.org/projects/nftables/manpage.html (search for queue) https://wiki.nftables.org/wiki-nftables/index.php/Queueing_to_userspace https://wiki.nftables.org/wiki-nftables/index.php/Queueing_t... Here's a nice & simple Rust library for the userspace part, to give an idea of what the shape of the API is: https://docs.rs/nfq/latest/nfq/ https://docs.rs/nfq/latest/nfq/ Also, I'm available for contract work ;) Say hi to Ben from Tv.

RadiozRadioz 3y ago

[flagged]

hug 3y ago
Not yeeting them would appear to be a valid option, yes. You can leave them un-yote for as long as you wish.
- teaearlgraycold 3y ago
  > un-yote :joy:
denkmoon 3y ago
Language is fluid :)
Takennickname 3y ago
Aren't you hip and with it?
- ndsipa_pomu 3y ago
  They're so unhip, their trousers keep falling down

akira2501 3y ago

I always felt the disappointment of wireguard was wrapping it up into an opinionated network interface. It really should have been a generic "filter" that you could attach to any type of file handle. Then the configuration would be far less strongly coupled, less weirdly communicated to the kernel, and the status of your connection more immediately obvious. Plus, you could have wireguard files on your local or remote filesystems, or any character device, or named pipes if you felt like it. You could use a "jit" daemon to build tap or other interfaces for you, or just do it individually at the application layer. You could have pre registered keys with the kernel, or you could manage that directly, or generate them randomly. It's always been a weird smelling underspecified IPSEC clone to me, when it could have been so much more.

mellutussa 3y ago
But why don't use the nice smelling IPSEC if that ticks your boxes?
- akira2501 3y ago
  It doesn't. It just foresaw the need to be able to dynamically configure tunnels on first connection and specified all of that. Which seems to me is a lot of what fly io has just mostly reimplemented here. In any case the point is I would prefer to just have the basic components available and let me piece them together however I want. Mostly to allow using the underlying technology in more contexts that it is currently available in.
  - medstrom 3y ago
    Heh, it really sounds like your needs would be better served with IPSec or something. WireGuard was born precisely because they saw that the whole problem making other existing solutions difficult to audit and insecure-in-practice was their thousand ways to configure. So they did the opposite. Low lines of code, few possibilities. In software you often choose between a small monolith and a big kitchen sink. Once you have 1 more need than the monolith covers, you have to go over to the kitchen sink.

d-z-m 3y ago

> We can install the peer as if we’re the initiator, and flyctl is the responder. The Linux kernel will initiate a WireGuard connection back to flyctl. This works; the protocol doesn’t care a whole lot who’s the server and who’s the client. We get new connections established about as fast as they can possibly be installed. Is this(effectively) adding a half round-trip to the handshake? i.e. 1. ->flyctl sends Initiation 2. <-peer is added via netlink(which causes new Initiation to be sent) 3. ->Response from flyctl

OJFord 3y ago
My reading was that both peers end up 'thinking' they initiated, but it doesn't matter. i.e. (3) either doesn't happen or just doesn't need to be waited for, or that they could even block (2-new initiation) and then it definitely wouldn't.
saclark11 3y ago
Pretty much, yes. If you imagine “Bob” has a policy that he can only converse with numbers in his address book, then you could think of it as: 1. -> Alice calls Bob 1.a. Bob does not pick up the call, but adds the number shown from caller ID to his address book 2. <- Bob calls the number (Alice) back 3. -> Alice picks up and they talk happily

niz4ts 3y ago

When I read this, I got a little too excited and thought they managed to get wireguard connections happening in the browser with webassembly (this isn't impossible, but the only attempt[0] I know of so far only works because of extra things tailscale has). It's an idea I've had for a project, but one I haven't had time to dedicate to (yet). In any case, really cool write-up! I wonder if they thought about making `flyctl` do a check with their API for any command that requires talking over wireguard to ensure the keys would be installed in the gateway. Since `flyctl` knows when the last command was run with it, it could do this only after some inactivity. And on the gateway machines, they'd just clean up any inactive peers with a cron (which they seem to be doing already). Not a solution as elegant as the one they reached (which is super cool), but I'm assuming the considerably lower effort would make it appealing. [0]: https://labs.leaningtech.com/blog/webvm-virtual-machine-with-networking-via-tailscale https://labs.leaningtech.com/blog/webvm-virtual-machine-with...

apignotti 3y ago
WebAssembly is not magic, the simple reality is that browsers do not expose low level socket interfaces, so they cannot connect to arbitrary services on the wider internet. We choose to use Tailscale since they allow WebSocket-based connections via their DERPs. It is interesting that, originally, DERPs were intended to be a solution for machines in extremely limited networking environment where nothing but HTTP is allowed. Turns out browsers are exactly one of those extremely limited networking environments.
- apitman 3y ago
  > We choose to use Tailscale since they allow WebSocket-based connections via their DERPs. Some context that I wasn't initially aware of: apignotti is the CTO of Leaning Technologies, which is where the article GP linked is from.
SparkyMcUnicorn 3y ago
That's a separate blog post, and it definitely is pretty cool. https://tailscale.com/blog/ssh-console https://tailscale.com/blog/ssh-console https://www.npmjs.com/package/@tailscale/connect https://www.npmjs.com/package/@tailscale/connect

protoman3000 3y ago

> The problem you quickly run into to build this design is that Linux kernel WireGuard doesn’t have a feature for installing peers on demand. I don't seem to understand. You can add peers at runtime, e.g. https://serverfault.com/questions/1101002/wireguard-client-addition-without-restart https://serverfault.com/questions/1101002/wireguard-client-a... Can somebody clarify? EDIT: If I understand correctly, that step is already too late. They want to authenticate a peer before adding it to the interface in order to prevent stale entries on the interface. They thus put a eBPF filter in front of the interface and do the cryptokey-routing based association to an authorized counterpart by themselves. If it checks out, then they add the peer to the interface and remove it after a timeout.

api 3y ago
Seems to me they did this to avoid the alternative of running WG in user space. They wanted a feature the Linux kernel didn’t have to route by cryptographic address first but without leaving the kernel so they hacked it in.??? JIT Wireguard is a weird way to frame this. My mind went to “why? The performance bottleneck is the crypto and per client JIT won’t help with that.” I would have just gone user space. Use something like tokio-uring or glommio to get the performance. If you keep going in the kernel you are going to keep hitting limitations because Linux is not built to serve millions and millions of active tunnels. Even doing millions of TCP connections per kernel gets hairy sometimes. Every limitation will require a hack. Every hack will be some system config that has to be applied and managed. The tool chains for provisioning Linux metal boxes are vastly inferior to the tooling for developing apps and services and managing their config. Or am I stupid and misunderstanding?
- vidarh 3y ago
  It does not seem like they need huge numbers of active tunnels per gateway. And JIT just as in "just in time" configuration of Wire guard. Once the configuration has been done, their stack stays out of it.
  - api 3y ago

justsomehnguy 3y ago

> and gateways with hundreds of thousands of peers that will never be used again My thoughts exactly as I was reading the first paragraphs. > Note that there’s no API call to subscribe for “incoming connection attempt” events. That’s OK! We can just make our own events. WireGuard connection requests are packets, and they’re easily identifiable, so we can efficiently snatch them with a BPF filter and a packet socket. Nice idea. > When we get an incoming initiation message, we have the 4-tuple address of the desired connection, including the ephemeral source port flyctl is using. We can install the peer as if we’re the initiator, and flyctl is the responder And this works behind NAT?

chgs 3y ago
If the packet goes back to the same ip/port and generated from the same ip/port it will work through nat.
TheDong 3y ago
> And this works behind NAT? Sure, UDP NAT only knows the 4-tuple (say {wggwd.fly.io, 12345, clientIP, 23456}). Any UDP packet, whether it's a new "initiator" udp packet, or a response to the outgoing initiation message, will look the exact same to any UDP NAT in the way since it only has the 4-tuple to go on, and the 4-tuple is the same.
- justsomehnguy 3y ago
  The way it's written made me think it's flyctl sending it's own ip:port, which would be private ones behind the NAT. If it's the received packet source:port then it's, obviously, already translated.
- ta1243 3y ago
  Presumably you could have a situation where there's deep-packet inspection on the traffic which would only allow a "handshake response" to come back through, and drop your attempt at an initialisation. I doubt that happens much, and I assume you'll fall back hapilly enough to the second time the "client" sends an init packet and then you simply respond with the response packet.

tinco 3y ago

While I generally agree with the idea that a direct HTTP request for a single point to point message would be more reliable than routing through a message queue, I'm a little bit surprised that there would be so many messages lost by NATS it had significant impact on their services. Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability?

dilyevsky 3y ago
> Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability? I believe if you're using core nats (not JetStream) there's no option for re-delivering like at all.
- tinco 3y ago
  Well if that's what they did, then then using NATS over HTTP in that scenario is just switching a single point of failure for two single points of failure with no feedback on the second point, did they just pick it for the convenience of the NATS interface and only later realize their mistake?
  - emmanueloga_ 3y ago
    I’m guessing they realized late that core nats is a message broker without features like durability and at least once delivery. Jetstream was released less than a year ago (in nats 2.2) so perhaps by the time they made the switch it was not an option.
    - caleblloyd 3y ago
      NATS 2.2 with initial JetStream support was actually released 3 years ago.
klabb3 3y ago
I am also very curious to know more. I’m sure the NATS maintainers are as well. Their architecture is extremely intuitive and appealing to me, so I wonder where things went south. Nats has a lot of tunable parameters with Jetstream. For instance, an in-memory stream with a time-based duplicate detection window, both push and pull semantics, and configurable re-delivery and ack policies. The one thing where I can see an impedance mismatch is ephemeral single-message connections which I don’t think it’s built for. In either case, more details would be invaluable. (Fly folks, please consider sharing!)

dpeckett 3y ago

Might as well take the opportunity to shill one of my recent experimental projects, If you are interested in building Go apps that act as userspace WireGuard peers take a look at https://github.com/dpeckett/noisysockets https://github.com/dpeckett/noisysockets Based off the excellent work in done by wireguard-go but I've attempted to simplify and make things a lot more idiomatic for library use. I reckon building a service mesh out of it would be interesting, obviously supporting multiple languages would be hard but maybe you could implement a sockets API. Though it might be hard to compete performance wise with mTLS as I've not seen HW acceleration for WireGuards crypto yet. FWIW, I'm currently on the market for freelancing roles, so if you're interested in Golang freelancers in the high-speed/secure networking space, please reach out (email is in my profile).

bscphil 3y ago
This looks great, nice work! I have a dream of taking a userspace Wireguard project like this and gluing PAKE over a relay in front of it to exchange Wireguard keys, followed by holepunching and establishing a direct tunnel. Basically Magic Wormhole for arbitrary tunnels - and hopefully vastly improved performance for the file transfer case as well, rather than crapping out at 20-30 MB/s over long fat networks.
- xyzzy_plugh 3y ago
  You're basically describing Tailscale.
  - mbreese 3y ago
    Maybe, but I read it as wormholes at the application level. Which would be awesome for me. I do a lot of work on remote servers for heavy data processing (large files). Sometimes I need to get these files back to my local computer, or just view figures I’ve generated using remote data. I have a small reverse tunnel daemon setup where I can (in my remote terminal) send arbitrary files or data back to my local computer. $ rtun send file.txt $ rtun view data.pdf It is a daemon that listens locally and when I setup SSH, a reverse tunnel is configured from one Unix socket to another. These are multiuser servers and Unix sockets handle authorization easily. The same setup could be handled similarly with a zmodem like process and aware terminal. However, this setup is a bit cumbersome and I can’t just run `ssh host`. If I could have the same setup with an in-app wireguard tunnel, with no other setup, it would be amazing.

sifatbinr 3y ago

[flagged]

mdavid626 3y ago

What does this mean? “We’ve gone a step beyond that: every time you run flyctl, our lovable, sprawling CLI, it conjures a TCP/IP stack out of thin air, with its own IPv6 address, and speaks directly to Fly Machines running on our networks.”

makeworld 3y ago
They're saying Wireguard is used.
- Spivak 3y ago
  More than that. They're saying that they're running a userspace implementation of both the tcp/ip stack and wireguard. Your machine isn't a peer to the wireguard tunnel, only flyctl is.
  - dementik 3y ago
    Of course, your machine can be a peer also. You can create peers to your organization with `flyctl wireguard create`.
    - carl_dr 3y ago
      But that isn’t the case for all of the other uses of flyctl. You can’t make connections with other processes over the same connection flyctl makes when it’s doing stuff. That is what GP is saying.
zrail 3y ago
They wrote a blog post all about it: https://fly.io/blog/our-user-mode-wireguard-year/ https://fly.io/blog/our-user-mode-wireguard-year/
ape4 3y ago
Its hard for me to think of a sprawling CLI that I love.
3y ago

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

sneak 3y ago

Fun fact: the WireGuard macOS client application cannot work on macOS unless it's distributed via the App Store - Apple simply will not provide the required VPN entitlements for web/self distributed apps. You can use the commandline wg tools (which use a different OS API) but not the GUI ones. This means that WireGuard-the-org can't distribute the mac GUI app directly, and if you want to use the official WireGuard app (eg for accessing a VPN for privacy), somewhat ironically you have to ID yourself (which requires a phone number) and your computer's hardware serial to Apple first to download it via the App Store. It smacks of the totalitarian rms "right to read" essay - provide strong ID and hardware unique identifiers to be allowed to download and use privacy software on your own computer. https://www.wireguard.com/install/ https://www.wireguard.com/install/ While not directly relevant to fly.io, I figured this might be relevant in the context of Apple's other anticompetitive actions this week related to web-based native app distribution on iOS in the EU for the DMA. This issue with VPN apps has been true on macOS for years. I personally didn't know why WireGuard didn't offer direct downloads, but I emailed Jason Donenfeld a couple years ago and he let me know that Apple has been restricting these APIs in non-AppStore apps, which was news to me, as I didn't know that Apple had started any of their AppStore-only bullshit on macOS.

Corrado 3y ago
This is not entirely accurate. macOS now has System Extensions[0], which allow you to tie into the network to provide VPN services. This is what Tailscale uses in their non-App Store downloaded app. [0] https://developer.apple.com/system-extensions/ https://developer.apple.com/system-extensions/
- sneak 3y ago
  https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_networking_networkextension https://developer.apple.com/documentation/bundleresources/en... It appears that they will now give out these entitlements for non-MAS apps, but only to members of their developer program. This means that you can't build one that will work on macOS without IDing yourself to Apple, so you still can't build it from scratch from source (as a non-developer-program person) and expect it to work (without disabling SIP).

tschumacher 3y ago

Sounds like a whole lot of effort to avoid a GraphQL request each time a flyctl client wants to connect.

deleted 3y ago
[deleted]
sudhirj 3y ago
Huh? They do make one to set it up. More a way to avoid having the public keys of every single client every loaded up into the wireguard kernel module on the gateway all the time.
- tschumacher 3y ago
  My implicit suggestion was that clients make a GraphQL request not only before the first connection but before every connection. The gateway server can insert the keys into the kernel in response to an explicit GraphQL request instead of in response to some complicated packet sniffing.
  - rudasn 3y ago
    What would the payload of the grapphql request to fetch the wg config for that peer look like, when they don't know from which peer the request is coming from?
  - mrkurt 3y ago
    This needs to support any ol' wireguard client. We use it in `flyctl` but people also use it to create gateways so they can, eg, peer with VPCs.

chairmanwow1 3y ago

My startup used Fly for almost a year. The core feature of code to deployed code in less than a minute is beautiful. Spinning up / down new nodes for backfills takes seconds. But the company feels a little immature. Once our API server became unreachable in Fly for 48 hours. I'm not sure if it was my fault for getting config wrong or if they just had another "silent" failure. They have a "db" product, but it's "not managed postgres". Would get consistent disconnections from that. Just feels weird for them to add a top level noun in their cli for postgres and then limit the extent it's a feature they support. API access to their core service would frequently go down and leave us waiting to deploy new service fixes. I miss the deployment experience, but I'm frankly happier with Cloud Run on GCP. Just way fewer "surprises" and much more complete documentation.

ZeroCool2u 3y ago
Fly looks great to me, though I've never had a chance to use it. For what it's worth though, Cloud Run on GCP is one of my top 3 favorite infrastructure/deployment tools, so you're setting the bar pretty high.
- icedchai 3y ago
  Cloud Run is pretty slick... services, batch jobs, easy to deploy, very flexible.
- WuxiFingerHold 3y ago
  > so you're setting the bar pretty high Why would anyone choose a provider that is inferior (apart from toying)? I mainly read two kind of stories from fly.io. Their promotional, but well written and interesting technical blogs like this one and stories about issues with their services and miscommunication. So, despite liking their blogs I don't consider using it.
  - apitman 3y ago
    Part of the reason I stick with Fly.io is because I want a rock solid version of what they do to exist, and they're the most likely people to eventually get there. That said, I've had very few issues with their platform, and I don't think it's ever caused downtime for my (admittedly very small) service.

cushpush 3y ago

What a chart.

apitman 3y ago

Interesting that they're defaulting to tunneling WireGuard over WebSockets. Not great for performance but probably fine for the DevOpsy stuff flyctl is used for. This is something I've wondered about for the future of QUIC/HTTP3. There's a nonzero chance network operators will just block UDP on port 443 altogether rather than properly handling it.

tptacek 3y ago
You can absolutely use native WireGuard, including from `flyctl` (it's an option you can set). When UDP doesn't work, it doesn't work at all, and it's hard to debug, so our default is for the thing that we know will work. (I say this ruefully, having lost the argument about what our default should be.)
- apitman 3y ago
  I'm assuming you decided doing some sort of happy eyeballs type thing isn't worth the complexity?
  - tptacek 3y ago
    Yeah; we're a public cloud platform, not a VPN service, so the key was just making sure everybody could drive our services. The "normie" way to do all of this stuff, at existing public clouds, is just HTTP anyways.

unixhero 3y ago

How might I take an arbitrary docker mized application and deploy it on Fly.io? Please take my money

tptacek 3y ago
cd directory/with/Dockerfile flyctl launch