5 ms·

The post mentions they are moving off nomad, interested to learn why and what is the replacement.

stryan 5y ago
> Rewriting a Rust program in Go is sacrilege, we know, but Go had the Consul libraries we needed, and, long term, that Go server is going to end up baked into our orchestration code, which is already in Go. This line from the post indicates to me that they're writing it themselves (since they wouldn't be worrying about being baked into orchestration code if it's Consul and Nomad). Still, I'd also be interested in what they're doing.
tptacek 5y ago
There are multiple reasons, some simple and some subtle. We don't totally have the bin-packing problem Nomad is designed to solve. We do have "users have fussy and meticulously specified ideas of how they want their apps to deploy", which is not really what Nomad is designed for; Nomad is designed to decouple your pool of compute from your applications, and to make it low-drama to deploy new applications to that pool of compute. But our users come with some level of drama built-in; it's an impedance mismatch. I like Nomad a lot. I think it's a better fit for what we're doing than Consul is (they're both great, but I think Consul is the bigger sticking point). You'd get different opinions from different people on our team, but I think Nomad's going to be around for awhile. But there are things we very much want to ship that we can't ship on Nomad. I would 100% use Nomad again if I had a somewhat complicated app I needed to deploy and couldn't use Fly.io for. (I'd use Consul too, for that matter.) Per the sibling comment: we are indeed writing a new orchestration system in Go, called `flyd`.
- schmichael 5y ago
  > we are indeed writing a new orchestration system in Go, called `flyd`. I know it's just a wild coincidence, but I couldn't help but be reminded of https://github.com/flynn/flynn https://github.com/flynn/flynn
  - tptacek 5y ago
    I'm friends with at least one Flynn person and I'm always wondering what they'd think of our weird architecture. There's a lot of interesting code in that repo!
- VectorLock 5y ago
  I'm curious what kind of things do you want to ship that you can't ship on Nomad?
  - mrkurt 5y ago
    The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs. We also need precise placement. We have people who want to run 8 VMs in London, 3 VMs in Los Angeles, and one VM in Sydney. They also want to do rolling deploys that revert on failure. Nomad's scheduling is not that precise. Databases too. Both Nomad and k8s have ways to do persistent workloads, but they rely heavily on the underlying infrastructure to "move" volumes between throwaway containers. We don't want that, we actually want something closer to old school VM provisioning than "orchestration".
    - dijit 5y ago
      Hiring? Sounds like a match for problems I find interesting.
      - mrkurt 5y ago
        Soon! We're polishing up a platform engineer role now. It'll appear on https://fly.io/jobs/ https://fly.io/jobs/
      - VectorLock 5y ago
        If you find these kind of DevOps problems challenging, I'm hiring. :D
    - schmichael 5y ago
      > The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs. Is this for scaling existing apps up (eg flyctl regions add ...) or for new deployments (flyctl deploy)? I suspect the former (scaling existing) as artifact distribution and deployment parameters like health checks seem like they prevent subsecond deployments (even ignoring the time for initial artifact uploading/building). Although my view is probably quite biased by Nomad having a much better chance at solving the former than the latter! (Also hi! I appreciate our chats in the past and feel free to reach out if you're willing to brain dump Nomad gripes on me. :) )
- joseph 5y ago
  I hope there will be a blog post about flyd! I love reading about what fly.io is doing.

> Rewriting a Rust program in Go is sacrilege, we know, but Go had the Consul libraries we needed, and, long term, that Go server is going to end up baked into our orchestration code, which is already in Go. This line from the post indicates to me that they're writing it themselves (since they wouldn't be worrying about being baked into orchestration code if it's Consul and Nomad). Still, I'd also be interested in what they're doing.

tptacek 5y ago

There are multiple reasons, some simple and some subtle. We don't totally have the bin-packing problem Nomad is designed to solve. We do have "users have fussy and meticulously specified ideas of how they want their apps to deploy", which is not really what Nomad is designed for; Nomad is designed to decouple your pool of compute from your applications, and to make it low-drama to deploy new applications to that pool of compute. But our users come with some level of drama built-in; it's an impedance mismatch. I like Nomad a lot. I think it's a better fit for what we're doing than Consul is (they're both great, but I think Consul is the bigger sticking point). You'd get different opinions from different people on our team, but I think Nomad's going to be around for awhile. But there are things we very much want to ship that we can't ship on Nomad. I would 100% use Nomad again if I had a somewhat complicated app I needed to deploy and couldn't use Fly.io for. (I'd use Consul too, for that matter.) Per the sibling comment: we are indeed writing a new orchestration system in Go, called `flyd`.

schmichael 5y ago
> we are indeed writing a new orchestration system in Go, called `flyd`. I know it's just a wild coincidence, but I couldn't help but be reminded of https://github.com/flynn/flynn https://github.com/flynn/flynn
- tptacek 5y ago
  I'm friends with at least one Flynn person and I'm always wondering what they'd think of our weird architecture. There's a lot of interesting code in that repo!
VectorLock 5y ago
I'm curious what kind of things do you want to ship that you can't ship on Nomad?
- mrkurt 5y ago
  The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs. We also need precise placement. We have people who want to run 8 VMs in London, 3 VMs in Los Angeles, and one VM in Sydney. They also want to do rolling deploys that revert on failure. Nomad's scheduling is not that precise. Databases too. Both Nomad and k8s have ways to do persistent workloads, but they rely heavily on the underlying infrastructure to "move" volumes between throwaway containers. We don't want that, we actually want something closer to old school VM provisioning than "orchestration".
  - dijit 5y ago
    Hiring? Sounds like a match for problems I find interesting.
    - mrkurt 5y ago
      Soon! We're polishing up a platform engineer role now. It'll appear on https://fly.io/jobs/ https://fly.io/jobs/
    - VectorLock 5y ago
      If you find these kind of DevOps problems challenging, I'm hiring. :D
  - schmichael 5y ago
    > The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs. Is this for scaling existing apps up (eg flyctl regions add ...) or for new deployments (flyctl deploy)? I suspect the former (scaling existing) as artifact distribution and deployment parameters like health checks seem like they prevent subsecond deployments (even ignoring the time for initial artifact uploading/building). Although my view is probably quite biased by Nomad having a much better chance at solving the former than the latter! (Also hi! I appreciate our chats in the past and feel free to reach out if you're willing to brain dump Nomad gripes on me. :) )
joseph 5y ago
I hope there will be a blog post about flyd! I love reading about what fly.io is doing.