5 ms·

I found debugging setup errors on fly to be incredibly frustrating You can't even ssh until the container started and they have some requirements on hostnames

I found debugging setup errors on fly to be incredibly frustrating You can't even ssh until the container started and they have some requirements on hostnames (0.0.0.0) At least a few of their examples are broken. The documentation could be better but it's pretty good Is it better than Aws? Yes of course, you need to spend way more to get a dx as bad as Aws. Setting up a VPS is a smoother experience than fly, though. Once it's running, it's lovely and reminds me of heroku.

biorach 4y ago
> You can't even ssh until the container started In fairness, I'm struggling to think of a way they could get you to ssh to a container that hasn't started yet > and they have some requirements on hostnames (0.0.0.0) Can you clarify?
tptacek 4y ago
Help me understand what you'd be SSH'ing into before your VM boots up? (I'm responsible for most of our SSH goop, so if there's an improvement to be made, I'm happy to make it). A reminder: we don't run "containers". We take containers, unpack them, and transform them into VMs. There's no host OS for you to access.
- indigodaddy 4y ago
  Maybe he means/wants some kind of “console” access for his “instance” ? Seems crazy as it takes I’m sure less than a second to boot, and if there is a problem booting up it’s probably not going to be a problem that a user/customer would be able to solve at console.
- mostly_lurks 4y ago
  Caveats: 1) this may not be what grandparent was trying to do and 2) I have limited experience debugging "regular" Docker, let alone firecracker or whatever Fly.io does. With that said, I succesfully deployed a simple Rust/Actix/Sqlite app with a Dockerfile to Fly.io. I then thought I'd try out litestream. For reasons I'm still not sure about, having `ENTRYPOINT ["litestream replicate … -exec myapp"]` resulted in immediate kernel panics.[1] As I was debugging, my instinct was to want to `fly ssh console` into a running container to see if running the same command from the shell produced any clues for further debugging. Though I understand this is the wrong mental model, the thought was something like "well, I know everything else works, I just need Fly to ignore this failing command so I can have a minute to poke around." To do this, I ended up just removing litestream from the ENTRYPOINT so the deploy would succeed, then I could SSH in and play around at the shell to see what was going on. Again, I have no idea whether this is the same sort of problem the other person was having, but for my case, what would be helpful is probably not changing how `fly ssh console` behaves, but perhaps some documentation of suggested debugging techniques in case your app is failing to start. [1] Putting the exact same command into a "start.sh" and making that the ENTRYPOINT worked fine so that's what I ended up doing.
  - tptacek 4y ago
    One of my biggest UX nits with Fly (I have no excuses, I have all the access I need to go fix this myself) is that we "kernel panic" when your entrypoint command fails. Of course, our kernel is not really panicking --- we've just run out of things for our `init` to manage, so it exits, and when pid 1 exits, so does the kernel. But you get the terrifying stack dump. We can clean this up, so that you get a clearer, simpler error ("your entrypoint exited, here's the exit code, there's nothing else for this VM to do so it's exiting, have a nice day"), and it's been on the docket for months. We'll get it done! We could conceivably add a flag for our `init` to hang around waiting for you to SSH in after your entrypoint exits. But that's clunky and complicated. Usually, you want your kernel to exit when your entrypoint fails, so that your service restarts! What you should do instead is push a container that has enough process supervision to hang around itself. Here's a doc: https://fly.io/docs/app-guides/multiple-processes/ https://fly.io/docs/app-guides/multiple-processes/
    - mostly_lurks 4y ago
      That all makes sense. I think context matters. Yes, when I have a service that's been up and running for some time, if that service fails, I want my service to restart. But when I'm trying to get a service running for the first time and I'm not sure that I have the right command in the entrypoint, the right arguments to that command, or the right supporting files in place, or the right libraries installed, or the right file permissions, …, well, then I don't want things to just blindly restart, I want a handle and some information so I can figure out why it isn't working. ETA: I recognize that your link to docs about running a supervisor addresses this problem. For me this raises some interesting questions. Like, I understand why Ben would implement `litestream exec` but maybe it would be better to steer users to a proper supervisor? Separately, what if it's the supervisor that's failing? Now I'm back to seeing kernel panics and not having error messages or a shell.
      - tptacek 4y ago
        Usually, when I'm debugging a container, I start with a `tail -f /dev/null` entrypoint, or something like that, and then just shell into it to run the real entrypoint to see if it's working.

biorach 4y ago

> You can't even ssh until the container started In fairness, I'm struggling to think of a way they could get you to ssh to a container that hasn't started yet > and they have some requirements on hostnames (0.0.0.0) Can you clarify?

tptacek 4y ago

Help me understand what you'd be SSH'ing into before your VM boots up? (I'm responsible for most of our SSH goop, so if there's an improvement to be made, I'm happy to make it). A reminder: we don't run "containers". We take containers, unpack them, and transform them into VMs. There's no host OS for you to access.

indigodaddy 4y ago
Maybe he means/wants some kind of “console” access for his “instance” ? Seems crazy as it takes I’m sure less than a second to boot, and if there is a problem booting up it’s probably not going to be a problem that a user/customer would be able to solve at console.
mostly_lurks 4y ago
Caveats: 1) this may not be what grandparent was trying to do and 2) I have limited experience debugging "regular" Docker, let alone firecracker or whatever Fly.io does. With that said, I succesfully deployed a simple Rust/Actix/Sqlite app with a Dockerfile to Fly.io. I then thought I'd try out litestream. For reasons I'm still not sure about, having `ENTRYPOINT ["litestream replicate … -exec myapp"]` resulted in immediate kernel panics.[1] As I was debugging, my instinct was to want to `fly ssh console` into a running container to see if running the same command from the shell produced any clues for further debugging. Though I understand this is the wrong mental model, the thought was something like "well, I know everything else works, I just need Fly to ignore this failing command so I can have a minute to poke around." To do this, I ended up just removing litestream from the ENTRYPOINT so the deploy would succeed, then I could SSH in and play around at the shell to see what was going on. Again, I have no idea whether this is the same sort of problem the other person was having, but for my case, what would be helpful is probably not changing how `fly ssh console` behaves, but perhaps some documentation of suggested debugging techniques in case your app is failing to start. [1] Putting the exact same command into a "start.sh" and making that the ENTRYPOINT worked fine so that's what I ended up doing.
- tptacek 4y ago
  One of my biggest UX nits with Fly (I have no excuses, I have all the access I need to go fix this myself) is that we "kernel panic" when your entrypoint command fails. Of course, our kernel is not really panicking --- we've just run out of things for our `init` to manage, so it exits, and when pid 1 exits, so does the kernel. But you get the terrifying stack dump. We can clean this up, so that you get a clearer, simpler error ("your entrypoint exited, here's the exit code, there's nothing else for this VM to do so it's exiting, have a nice day"), and it's been on the docket for months. We'll get it done! We could conceivably add a flag for our `init` to hang around waiting for you to SSH in after your entrypoint exits. But that's clunky and complicated. Usually, you want your kernel to exit when your entrypoint fails, so that your service restarts! What you should do instead is push a container that has enough process supervision to hang around itself. Here's a doc: https://fly.io/docs/app-guides/multiple-processes/ https://fly.io/docs/app-guides/multiple-processes/
  - mostly_lurks 4y ago
    That all makes sense. I think context matters. Yes, when I have a service that's been up and running for some time, if that service fails, I want my service to restart. But when I'm trying to get a service running for the first time and I'm not sure that I have the right command in the entrypoint, the right arguments to that command, or the right supporting files in place, or the right libraries installed, or the right file permissions, …, well, then I don't want things to just blindly restart, I want a handle and some information so I can figure out why it isn't working. ETA: I recognize that your link to docs about running a supervisor addresses this problem. For me this raises some interesting questions. Like, I understand why Ben would implement `litestream exec` but maybe it would be better to steer users to a proper supervisor? Separately, what if it's the supervisor that's failing? Now I'm back to seeing kernel panics and not having error messages or a shell.
    - tptacek 4y ago
      Usually, when I'm debugging a container, I start with a `tail -f /dev/null` entrypoint, or something like that, and then just shell into it to run the real entrypoint to see if it's working.