5 ms·

We recently moved our entire app deployment over to Fly and are mostly loving it, but one of the mildly janky features is hallpass. For instance, (1) connection

We recently moved our entire app deployment over to Fly and are mostly loving it, but one of the mildly janky features is hallpass. For instance, (1) connections often fail if you have X forwarding enabled (even if you did no specifial config on the machine), and (2) port forwarding doesn't work. While these aren't really a big deal since (1) you can just disable X forwarding in ssh_config and (2) port forwarding is unnecessary if you can tunnel in via Wireguard, it makes me wonder why a native SSH server isn't used with a small script to manage the required config changes. While on the topic, we're also eagerly awaiting improved autoscaling (e.g., more responsive, using additional metrics, and scaling down properly). I'd be really curious if you could leverage the more detailed access to instance-level metrics to implement some cool new queue-theoretic modeling: You know roughly how long it takes for an app to launch, you know the current request rate, and you know the time to service requests. You could apply a lightweight Markov model to predict the probability of a given queueing delay in each region within the average launch time and, if so, preemptively launch a new instance before queueing delays even occur. This could be configured to balance a client's tolerance for queueing delays with over-provisioning budget.

tptacek 5y ago
Hallpass is a truly trivial piece of code --- it might be less than 400 lines all in. All it really does is run certificate authentication off of a root cert we store in `_orgcert.internal` in DNS. If you like, you can roll a Dockerfile that runs OpenSSH directly on your internal network address (bind it to `fly-local-6pn`), and then use native WireGuard to talk to it. I've got a branch on hallpass that does port forwarding, but I never merge it, because you're right: using port forwarding on Fly.io is weird, because we already provide you direct access to any port you're exposing, and you can't talk to any of this stuff without WireGuard already. I think it would just confuse people more if I made port forwarding work.
michaeldwan 5y ago
As for autoscaling, our hands are tied as long as we're running on Nomad. Right now our autoscaler is nothing more than some ruby that loops over data from prometheus and changes counts in Nomad. It's slow and buggy, but worse we don't have control over where Nomad places VMs or which ones it stops when scaling down. We're working on a replacement for Nomad (called flyd) that gives us full control over VMs. Once apps are running on that we can do a lot of cool things. Better autoscaling is one, but I'm really excited about suspending idle VMs that our proxy wakes up on demand. That'll cover most use cases without forcing customers to worry about counts or blowing through a budget.
- atonse 5y ago
  I’d love to hear more about this move away from Nomad. We haven’t had too good a time with nomad, but not sure if it’s just our limited understanding. It doesn’t help that there are very few people out there that know it.
  - michaeldwan 5y ago
    We'll write about it when the time comes. To be fair, Nomad and Consul have served us well. Most of our troubles stem from abusing them in ways they weren't designed to handle.
    - atonse 5y ago
      That doesn’t surprise me. I think our biggest issue is not knowing it well enough and not being able to find people that know it. At that point it doesn’t matter how good the tech is.

tptacek 5y ago

Hallpass is a truly trivial piece of code --- it might be less than 400 lines all in. All it really does is run certificate authentication off of a root cert we store in `_orgcert.internal` in DNS. If you like, you can roll a Dockerfile that runs OpenSSH directly on your internal network address (bind it to `fly-local-6pn`), and then use native WireGuard to talk to it. I've got a branch on hallpass that does port forwarding, but I never merge it, because you're right: using port forwarding on Fly.io is weird, because we already provide you direct access to any port you're exposing, and you can't talk to any of this stuff without WireGuard already. I think it would just confuse people more if I made port forwarding work.

michaeldwan 5y ago

As for autoscaling, our hands are tied as long as we're running on Nomad. Right now our autoscaler is nothing more than some ruby that loops over data from prometheus and changes counts in Nomad. It's slow and buggy, but worse we don't have control over where Nomad places VMs or which ones it stops when scaling down. We're working on a replacement for Nomad (called flyd) that gives us full control over VMs. Once apps are running on that we can do a lot of cool things. Better autoscaling is one, but I'm really excited about suspending idle VMs that our proxy wakes up on demand. That'll cover most use cases without forcing customers to worry about counts or blowing through a budget.

atonse 5y ago
I’d love to hear more about this move away from Nomad. We haven’t had too good a time with nomad, but not sure if it’s just our limited understanding. It doesn’t help that there are very few people out there that know it.
- michaeldwan 5y ago
  We'll write about it when the time comes. To be fair, Nomad and Consul have served us well. Most of our troubles stem from abusing them in ways they weren't designed to handle.
  - atonse 5y ago
    That doesn’t surprise me. I think our biggest issue is not knowing it well enough and not being able to find people that know it. At that point it doesn’t matter how good the tech is.