7 ms·

I've been using Flynn for a while now. Unfortunately there's always something that entirely bricks the cluster eventually. Sometimes a node will reboot unexpect

by corobo 10y ago

I've been using Flynn for a while now. Unfortunately there's always something that entirely bricks the cluster eventually. Sometimes a node will reboot unexpectedly (as happens with servers) and the cluster slowly implodes from there, sometimes I'll just get a monitoring alert that all the sites hosted on it have gone down for no apparent reason. They announced during their 1.0 release that Flynn is now production ready but I have to disagree at this time. I do honestly love the idea of running my own Heroku but it's just not stable enough and doesn't recover well if a node goes offline without manual intervention using the flynn-host fix command (and sometimes that command just chokes up completely). Next time it happens (when, not if, at this rate) I'll try to set aside some time to gather up the logs to help debug the problem but I'm using Flynn to avoid the sysadmin side of things and just build my product. In my opinion Flynn is tinker-ready, not production-ready. It's infuriating that it has these random problems because like I say I really _really_ love the product and desperately want to be able to use it reliably.

merb 10y ago
> Sometimes a node will reboot unexpectedly (as happens with servers) if it's coreos it's probably the auto update mechanism.
- corobo 10y ago
  Not in this case, Flynn runs on Ubuntu 14.04. I think the reboot was a DigitalOcean droplet's host being restarted at the time though I can't be sure.
- nathancahill 10y ago
  Unbelievable that auto update is the default. Got bit by that.
_lmars 10y ago
Really sorry to hear you have had issues with your Flynn cluster. Stability is our number one concern and we are pushing stability fixes all the time. Flynn does self-heal after node reboots, so if that is not working for you then it is a bug we need to fix. When you do have issues it would be really helpful to us and other users if you could open a GitHub issue or come and chat to us in IRC (#flynn on Freenode) so we can fix any bugs you are encountering. I fully understand you just want to be concentrating on building your product, this is why we are building Flynn!
- corobo 10y ago
  Aye like I say I'll grab the logs next time, unfortunately it's happened at the worst possible times so far (e.g. preparing to demo the site so can't really be trying to debug a broken cluster!) so I've not had the time to debug things as they've happened yet. If nothing else I can't fault the installer, it's damned easy to whip up a fresh Flynn! One of those things where I wish I could say it's something I'm doing that's breaking it but in each of the situations I've had I've not been interacting with it at all when it happens. I know my waffling here doesn't help anyone without logs of some sort, this post just caught me a couple of days after a cluster brick. I'll set it all up again and will grab the debug info and jump on IRC this time if it all goes to pot.
  - jpgvm 10y ago
    Please do, the node reboot logic has been through a lot of incremental fixes over time and we are always working hard to iron out the wrinkles that are left. If you don't have time to interactively debug with us on IRC just open a Github issue with what details you can and the output of the `flynn-host collect-debug-info` command and hopefully we will be able to discern what went wrong. Thanks for trying out Flynn!
    - corobo 10y ago
      I've scribbled the command down and will do so, thanks! Just to blow away some of my grump cobwebs I'm not just trying out Flynn, I'm still using Flynn! The hiccups were just going to be temporarily patched by hijacking my ServerPilot controlled machine (WordPress on Flynn.. Haven't managed that one yet!) until it felt a bit more stable in the dev/stage environment. :)
      - jpgvm 10y ago
        Awesome! Would be cool to learn how you are approaching Wordpress on Flynn. I have poked around at writing an asset backend that talks to Flynn's blobstore because a family member wanted me to host their Wordpress site but haven't quite gotten around to it. We merged the MariaDB database appliance a little while back which took care of the other major roadblock to getting Wordpress to "just work" on Flynn. I'm `jpg@freenode and josephglanville on Github, hope to see you around. :)

by corobo 10y ago

merb 10y ago

> Sometimes a node will reboot unexpectedly (as happens with servers) if it's coreos it's probably the auto update mechanism.

corobo 10y ago
Not in this case, Flynn runs on Ubuntu 14.04. I think the reboot was a DigitalOcean droplet's host being restarted at the time though I can't be sure.
nathancahill 10y ago
Unbelievable that auto update is the default. Got bit by that.

_lmars 10y ago

Really sorry to hear you have had issues with your Flynn cluster. Stability is our number one concern and we are pushing stability fixes all the time. Flynn does self-heal after node reboots, so if that is not working for you then it is a bug we need to fix. When you do have issues it would be really helpful to us and other users if you could open a GitHub issue or come and chat to us in IRC (#flynn on Freenode) so we can fix any bugs you are encountering. I fully understand you just want to be concentrating on building your product, this is why we are building Flynn!

corobo 10y ago
Aye like I say I'll grab the logs next time, unfortunately it's happened at the worst possible times so far (e.g. preparing to demo the site so can't really be trying to debug a broken cluster!) so I've not had the time to debug things as they've happened yet. If nothing else I can't fault the installer, it's damned easy to whip up a fresh Flynn! One of those things where I wish I could say it's something I'm doing that's breaking it but in each of the situations I've had I've not been interacting with it at all when it happens. I know my waffling here doesn't help anyone without logs of some sort, this post just caught me a couple of days after a cluster brick. I'll set it all up again and will grab the debug info and jump on IRC this time if it all goes to pot.
- jpgvm 10y ago
  Please do, the node reboot logic has been through a lot of incremental fixes over time and we are always working hard to iron out the wrinkles that are left. If you don't have time to interactively debug with us on IRC just open a Github issue with what details you can and the output of the `flynn-host collect-debug-info` command and hopefully we will be able to discern what went wrong. Thanks for trying out Flynn!
  - corobo 10y ago
    I've scribbled the command down and will do so, thanks! Just to blow away some of my grump cobwebs I'm not just trying out Flynn, I'm still using Flynn! The hiccups were just going to be temporarily patched by hijacking my ServerPilot controlled machine (WordPress on Flynn.. Haven't managed that one yet!) until it felt a bit more stable in the dev/stage environment. :)
    - jpgvm 10y ago
      Awesome! Would be cool to learn how you are approaching Wordpress on Flynn. I have poked around at writing an asset backend that talks to Flynn's blobstore because a family member wanted me to host their Wordpress site but haven't quite gotten around to it. We merged the MariaDB database appliance a little while back which took care of the other major roadblock to getting Wordpress to "just work" on Flynn. I'm `jpg@freenode and josephglanville on Github, hope to see you around. :)