25 ms·

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the. > More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover. > Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this. > The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated. > It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner. [1] https://community.fly.io/t/service-interruption-cant-destroy-machine-deploy-or-restart https://community.fly.io/t/service-interruption-cant-destroy...

gowthamgts12 3y ago
> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If it really got missed, then I don't understand how the thread was made private to only logged-in users?
- deleted 3y ago
  [deleted]
- p-e-w 3y ago
  Whoa, what? That's a much bigger red flag than the downtime itself.
  - throwawayfly 3y ago
    Ok as long as we’re getting conspiratorial, something similar I observed has bugged me. About a year ago fly awarded a few people in the forums, I think it was 3, the “aeronaut” badge. Basically just pointless bling for a “routinely very helpful” person or somesuch. Still, I can imagine it was cool to get it. No, it wasn’t me. One person I saw with it absolutely deserved it: this person is, to this day, always hopping in and helping people; linking to docs; raising their own issues with a big dose of “fellow builder” understanding and empathy; that sort of person. My own queries typically led me to a thread that this person has answered. In short - the kind of helpful, proactive, high knowledge volunteer early adopter that every community needs - and a handful are blessed to find. Then one day I saw this same person had offered — to one random newbie with build problems in one of the many HALP threads — a reply like, “maybe Fly isn’t the best option for you. here are some other places that can host an app”. The thread was left alone and faded, like many when a lost newbie is involved. But 1 day later, I noticed this tireless early adopter no longer had their “aeronaut” badge. I still refuse to believe my own eyes about something that petty.
    - michaeldwan 3y ago
      Get out of here with this nonsense. We tell people when we’re a bad option all the time. Do you really think we have a desire (or time) to punish somebody for doing the same? Also, here’s the long forgotten badge, still with 3 people… https://community.fly.io/badges/107/aeronaut https://community.fly.io/badges/107/aeronaut
      - sho 3y ago
        > Do you really think we have a desire (or time) to punish somebody for doing the same? idk man, there's these awfully convenient disappearing forum threads too. The benefit of the doubt is starting to expire. I see you're a co-founder, so presumably you have some sway on priorities and skin in the game. I think you should take the reputational damage you're accruing here much more seriously than you apparently are. A few more incidents like this and it won't just be you telling people you're a bad option. * edited to tone down the forum thread disappearance angle. FWIW I do believe that it likely wasn't deliberate. My main point was that these things add up and "of course we wouldn't do that!" starts to ring a little hollow the 10th time you hear it...
- teraflop 3y ago
  It looks like all 166 threads with the "App not working" tag are invisible when not logged in. So I'm guessing somebody applied that tag retroactively. https://community.fly.io/c/questions-and-help/app-not-working/28 https://community.fly.io/c/questions-and-help/app-not-workin... EDIT: it now appears that the "app-not-working" tag itself has been deleted, and no longer shows up even when logged in.
  - buro9 3y ago
    This is why companies should not run their own forums. It's cheap support and marketing, it's not really community.
    - vasco 3y ago
      I never thought to make friends with people who's only common thing with me is that they shop at the same place. Companies creating a "community" is exactly as you described.
      - toyg 3y ago
        I am an interested party in the process space, and I think that's ungenerous. When you work with a complex tool every day, and you have to find solutions for this or that issue, develop strategies for this or that business case, etc etc, you're not really shopping - it's more like you're in the trenches. At that point, finding people who have the same issues and talking shop with them, can be great for both knowledge exchange and camaraderie. Linux wouldn't be what it is today without the LUGs era, for example.
        vasco 3y ago
        We're talking about private companies running forum software instead of providing support. We're not talking about the power of IRC or mailing list communities for open source projects and the like. If I pay for something I want the person I pay money to help me fix problems I get.
  - 3y ago
mrcwinn 3y ago
For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.
- steve_adams_86 3y ago
  I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them. It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.
  - danielvaughn 3y ago
    I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.
    - eduction 3y ago
      I went to DO's site due to your comment and I don't see anywhere where I can just get a server. Do you mean a VPS/Droplet? (I'm looking under Products and Solutions.)
      - gregsadetsky 3y ago
bongobingo1 3y ago
Seems like the OP should have made a HN thread in the first place instead of posting to community.stri^H^H^H^Hfly.io
- revskill 3y ago
  But HN is not a customer service forum ?
  - noizejoy 3y ago
    > But HN is not a customer service forum ? you must be new here ;-)
  - gtirloni 3y ago
    It's often used as an escalation point when people can't get support from certain companies (most notably, Google). If an employee lurks in here and sees your post, they might contact the right people to fix your issue. Smaller companies also do a lot of PR damage control and constantly monitor HN for threads complaining about their services. You're not wrong but that's how it works.
    - tptacek 3y ago
      That's not what happened here. We're talking about an outage that was resolved days ago, long before this thread went up.
- 5e92cb50239222b 3y ago
  > ^H^H^H^H alt+backspace will wipe that substring in most shells in one go.
  - mewmew07 3y ago
    it would loose the comic appeal though
  - alias_neo 3y ago
emmelaich 3y ago
The irony or perhaps the tragedy of building a low friction service is that you have to have experts on the lower level high friction stuff. I would hope that after a couple of hours downtime, they'd bring up a fresh machine with Ansible or whatever. Hardware or AWS/GCP Vm.
- ps 3y ago
  > I would hope that after a couple of hours downtime, they'd bring up a fresh machine with Ansible or whatever. It is not just about a fresh machine which hopefully sits in each datacenter. I can imagine they needed the clone of the system due to the design of the fly.io service and that's where the "fun" begins.
yla92 3y ago
Is it me or the page is now gone? "Oops! That page doesn’t exist or is private." Edit: Ok, I can see after sign up / log in.
quickthrower2 3y ago
> We strongly recommend running multiple instances to mitigate the impact of single-host failures like this. Make it impossible not to do so, and make it frictionless then.
- remus 3y ago
  That would presumably cost more money which is not a trade off every user would want to make.
  - marcinzm 3y ago
    You cannot make every user happy, and its generally better to not have a user than to have an unhappy user.
benjaminwootton 3y ago
Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect resilience to host failure?
- DoubleFree 3y ago
  Fly postgres is not managed postgres, it's cli sugar over a normal fly app, which the [docs](https://fly.io/docs/postgres/ https://fly.io/docs/postgres/) make quite clear. Their docs also make clear that if you run postgres in a single-instance configuration, if the hardware it's running on has problems, you database will go down. I believe the underlying reason that precludes failing over to a different host machine, is that fly volumes are slices of host-attached nvme drives. If the host goes down, these can't be migrated. I _think_ instances without attached volumes will fail-over to a different host. Of course, that's not ideal, and maybe their CLI should also warn about this loudly when creating the cluster.
- smallerfish 3y ago
  If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.
  - marcinzm 3y ago
    Downtime but limited downtime since the data is stored with redundantly across multiple machines in the same AZ. So unless the AZ goes down (which is a different failure than what happened here) you can restart the DB on a different instance pretty quickly and I'm guessing AWS will do it automatically for you. edit: Remove triple as not certain about level of redundancy
oefrha 3y ago
I was confused why support for platform failure relies on a forum where employees may or may not check. After checking docs[1], apparently you have to be on a paid plan (at least $29/mo) to access email support, so you may not have it even you’re paying for resources. I won’t be using it for side projects where I’m okay with paying $5-10/mo but don’t want to have three day outages. [1] https://fly.io/docs/about/support/ https://fly.io/docs/about/support/
- MuffinFlavored 3y ago
  Forewarning: I am not being critical of fly.io nor their free support whatsoever when I say this. From a technical perspective, could they have "been better" from a technical perspective? I see their name a lot on HN so I know they are doing really cool + advanced things and this is probably some super small edge case that slipped through the cracks. Could they have added some message / do we as the HN community feel they needed to be like "we're gonna add some extra logging/monitoring going forward so it won't happen again"? By all means, they probably don't owe anybody in terms of stability + uptime guarantees when it comes to a free tier. Sh*t happens.
  - riwsky 3y ago
    They broke uptime for the paid tier, not just the free tier. The relevance of paid/free is that free (and cheap paid) plans don’t get fly support over email
  - azemetre 3y ago
    They may not owe anyone anything but over time these types of issues can cause a large reputation hit. If I was just searching online or trying to find out what various communities think about Fly.io and see several threads about major outages with poor communications, do you think I will use their services? It would be an immediate pass. It takes a long time to build a reputation, and you can lose it instantly.
  - elderlydoofus 3y ago
    FWIW: I am on the bottom tier of the paid plans ($29/mo) so I could get access to the email support, and even with that their response time is still not great. I have an ongoing issue with one of my PG clusters where one of the nodes was failing and all my attempts at fixing it are failing (mainly cloning one of the other machines to bring the cluster numbers back to normal). I emailed my account’s support email mid Friday morning last week and did not hear back until this past Monday night. Sucks, because like a lot of others in this thread I like what Fly is trying to do and am rooting for them, but IMO they should use a significant chunk of that funding they just received on hiring a ton of SREs and front line customer support. EDIT: I should add, the past times I have emailed them the response time was good. It's just this most recent time was so egregious (3 days!) to get even that initial response that I bring it up.
thefreeman 3y ago
total shot in the dark, but, was it a transaction id wrap around?
plagiarist 3y ago
Why is it my responsibility to move instances from machine to machine to mitigate a cloud host's outages? What is their utility if not performing the bare minimum of cloud host responsibilities keeping my container up?

gowthamgts12 3y ago

> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If it really got missed, then I don't understand how the thread was made private to only logged-in users?

deleted 3y ago
[deleted]
p-e-w 3y ago
Whoa, what? That's a much bigger red flag than the downtime itself.
- throwawayfly 3y ago
  Ok as long as we’re getting conspiratorial, something similar I observed has bugged me. About a year ago fly awarded a few people in the forums, I think it was 3, the “aeronaut” badge. Basically just pointless bling for a “routinely very helpful” person or somesuch. Still, I can imagine it was cool to get it. No, it wasn’t me. One person I saw with it absolutely deserved it: this person is, to this day, always hopping in and helping people; linking to docs; raising their own issues with a big dose of “fellow builder” understanding and empathy; that sort of person. My own queries typically led me to a thread that this person has answered. In short - the kind of helpful, proactive, high knowledge volunteer early adopter that every community needs - and a handful are blessed to find. Then one day I saw this same person had offered — to one random newbie with build problems in one of the many HALP threads — a reply like, “maybe Fly isn’t the best option for you. here are some other places that can host an app”. The thread was left alone and faded, like many when a lost newbie is involved. But 1 day later, I noticed this tireless early adopter no longer had their “aeronaut” badge. I still refuse to believe my own eyes about something that petty.
  - michaeldwan 3y ago
    Get out of here with this nonsense. We tell people when we’re a bad option all the time. Do you really think we have a desire (or time) to punish somebody for doing the same? Also, here’s the long forgotten badge, still with 3 people… https://community.fly.io/badges/107/aeronaut https://community.fly.io/badges/107/aeronaut
    - sho 3y ago
      > Do you really think we have a desire (or time) to punish somebody for doing the same? idk man, there's these awfully convenient disappearing forum threads too. The benefit of the doubt is starting to expire. I see you're a co-founder, so presumably you have some sway on priorities and skin in the game. I think you should take the reputational damage you're accruing here much more seriously than you apparently are. A few more incidents like this and it won't just be you telling people you're a bad option. * edited to tone down the forum thread disappearance angle. FWIW I do believe that it likely wasn't deliberate. My main point was that these things add up and "of course we wouldn't do that!" starts to ring a little hollow the 10th time you hear it...
teraflop 3y ago
It looks like all 166 threads with the "App not working" tag are invisible when not logged in. So I'm guessing somebody applied that tag retroactively. https://community.fly.io/c/questions-and-help/app-not-working/28 https://community.fly.io/c/questions-and-help/app-not-workin... EDIT: it now appears that the "app-not-working" tag itself has been deleted, and no longer shows up even when logged in.
- buro9 3y ago
  This is why companies should not run their own forums. It's cheap support and marketing, it's not really community.
  - vasco 3y ago
    I never thought to make friends with people who's only common thing with me is that they shop at the same place. Companies creating a "community" is exactly as you described.
    - toyg 3y ago
      I am an interested party in the process space, and I think that's ungenerous. When you work with a complex tool every day, and you have to find solutions for this or that issue, develop strategies for this or that business case, etc etc, you're not really shopping - it's more like you're in the trenches. At that point, finding people who have the same issues and talking shop with them, can be great for both knowledge exchange and camaraderie. Linux wouldn't be what it is today without the LUGs era, for example.
      - vasco 3y ago
        We're talking about private companies running forum software instead of providing support. We're not talking about the power of IRC or mailing list communities for open source projects and the like. If I pay for something I want the person I pay money to help me fix problems I get.
- 3y ago

mrcwinn 3y ago

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

steve_adams_86 3y ago
I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them. It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.
- danielvaughn 3y ago
  I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.
  - eduction 3y ago
    I went to DO's site due to your comment and I don't see anywhere where I can just get a server. Do you mean a VPS/Droplet? (I'm looking under Products and Solutions.)
    - gregsadetsky 3y ago

bongobingo1 3y ago

Seems like the OP should have made a HN thread in the first place instead of posting to community.stri^H^H^H^Hfly.io

revskill 3y ago
But HN is not a customer service forum ?
- noizejoy 3y ago
  > But HN is not a customer service forum ? you must be new here ;-)
- gtirloni 3y ago
  It's often used as an escalation point when people can't get support from certain companies (most notably, Google). If an employee lurks in here and sees your post, they might contact the right people to fix your issue. Smaller companies also do a lot of PR damage control and constantly monitor HN for threads complaining about their services. You're not wrong but that's how it works.
  - tptacek 3y ago
    That's not what happened here. We're talking about an outage that was resolved days ago, long before this thread went up.
5e92cb50239222b 3y ago
> ^H^H^H^H alt+backspace will wipe that substring in most shells in one go.
- mewmew07 3y ago
  it would loose the comic appeal though
- alias_neo 3y ago

emmelaich 3y ago

The irony or perhaps the tragedy of building a low friction service is that you have to have experts on the lower level high friction stuff. I would hope that after a couple of hours downtime, they'd bring up a fresh machine with Ansible or whatever. Hardware or AWS/GCP Vm.

ps 3y ago
> I would hope that after a couple of hours downtime, they'd bring up a fresh machine with Ansible or whatever. It is not just about a fresh machine which hopefully sits in each datacenter. I can imagine they needed the clone of the system due to the design of the fly.io service and that's where the "fun" begins.

yla92 3y ago

Is it me or the page is now gone? "Oops! That page doesn’t exist or is private." Edit: Ok, I can see after sign up / log in.

quickthrower2 3y ago

> We strongly recommend running multiple instances to mitigate the impact of single-host failures like this. Make it impossible not to do so, and make it frictionless then.

remus 3y ago
That would presumably cost more money which is not a trade off every user would want to make.
- marcinzm 3y ago
  You cannot make every user happy, and its generally better to not have a user than to have an unhappy user.

benjaminwootton 3y ago

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect resilience to host failure?

DoubleFree 3y ago
Fly postgres is not managed postgres, it's cli sugar over a normal fly app, which the [docs](https://fly.io/docs/postgres/ https://fly.io/docs/postgres/) make quite clear. Their docs also make clear that if you run postgres in a single-instance configuration, if the hardware it's running on has problems, you database will go down. I believe the underlying reason that precludes failing over to a different host machine, is that fly volumes are slices of host-attached nvme drives. If the host goes down, these can't be migrated. I _think_ instances without attached volumes will fail-over to a different host. Of course, that's not ideal, and maybe their CLI should also warn about this loudly when creating the cluster.
smallerfish 3y ago
If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.
- marcinzm 3y ago
  Downtime but limited downtime since the data is stored with redundantly across multiple machines in the same AZ. So unless the AZ goes down (which is a different failure than what happened here) you can restart the DB on a different instance pretty quickly and I'm guessing AWS will do it automatically for you. edit: Remove triple as not certain about level of redundancy

oefrha 3y ago

I was confused why support for platform failure relies on a forum where employees may or may not check. After checking docs[1], apparently you have to be on a paid plan (at least $29/mo) to access email support, so you may not have it even you’re paying for resources. I won’t be using it for side projects where I’m okay with paying $5-10/mo but don’t want to have three day outages. [1] https://fly.io/docs/about/support/ https://fly.io/docs/about/support/

MuffinFlavored 3y ago
Forewarning: I am not being critical of fly.io nor their free support whatsoever when I say this. From a technical perspective, could they have "been better" from a technical perspective? I see their name a lot on HN so I know they are doing really cool + advanced things and this is probably some super small edge case that slipped through the cracks. Could they have added some message / do we as the HN community feel they needed to be like "we're gonna add some extra logging/monitoring going forward so it won't happen again"? By all means, they probably don't owe anybody in terms of stability + uptime guarantees when it comes to a free tier. Sh*t happens.
- riwsky 3y ago
  They broke uptime for the paid tier, not just the free tier. The relevance of paid/free is that free (and cheap paid) plans don’t get fly support over email
- azemetre 3y ago
  They may not owe anyone anything but over time these types of issues can cause a large reputation hit. If I was just searching online or trying to find out what various communities think about Fly.io and see several threads about major outages with poor communications, do you think I will use their services? It would be an immediate pass. It takes a long time to build a reputation, and you can lose it instantly.
- elderlydoofus 3y ago
  FWIW: I am on the bottom tier of the paid plans ($29/mo) so I could get access to the email support, and even with that their response time is still not great. I have an ongoing issue with one of my PG clusters where one of the nodes was failing and all my attempts at fixing it are failing (mainly cloning one of the other machines to bring the cluster numbers back to normal). I emailed my account’s support email mid Friday morning last week and did not hear back until this past Monday night. Sucks, because like a lot of others in this thread I like what Fly is trying to do and am rooting for them, but IMO they should use a significant chunk of that funding they just received on hiring a ton of SREs and front line customer support. EDIT: I should add, the past times I have emailed them the response time was good. It's just this most recent time was so egregious (3 days!) to get even that initial response that I bring it up.

thefreeman 3y ago

total shot in the dark, but, was it a transaction id wrap around?

plagiarist 3y ago

Why is it my responsibility to move instances from machine to machine to mitigate a cloud host's outages? What is their utility if not performing the bare minimum of cloud host responsibilities keeping my container up?