6 ms·

Fly.io deleted all my apps without notifying me

111 points by lukas_b 2y ago

I noticed my app was down and went to check it out. Lo and Behold my account had been flagged by their automatic fraud detection algorithm. This is despite my account having close to $500 of credit and being active for around a year now. I was not informed about this at all. I followed the prompt to allow them to charge my credit card in order to verify that I'm not a bot and now I have control of my account again. However none of my apps were restored. Has anyone else experienced something like this in the past?

frafart 2y ago
I haven’t had my apps deleted but the fraud detection system flags me almost every time I interact with the platform. It’s very disruptive.
sergiotapia 2y ago
this sounds terrifying. so their "algorithm" can just nuke your entire business without any signaling?
- lukas_b 2y ago
  apparently I'm not the only one. https://community.fly.io/t/account-flagged-apps-gone/14570 https://community.fly.io/t/account-flagged-apps-gone/14570 https://web.archive.org/web/20240710225525/https://community.fly.io/t/account-flagged-apps-gone/14570 https://web.archive.org/web/20240710225525/https://community...
  - gatnoodle 2y ago
    This is bad, really bad!
add-sub-mul-div 2y ago
Dropbox deleted all my data because I didn't log in for a long time. Guess you're supposed to check in on your completely static data occasionally. Whatever, the lesson is not to let yourself seriously rely on any of these services.
- cpach 2y ago
  No matter what service provider one uses, one should always have backups and a plan B/disaster recovery. The 3-2-1 rule is as relevant as ever.
altairprime 2y ago
What were your apps?
- lukas_b 2y ago
  a personal vpn and a company website
  - yellowapple 2y ago
    I'd guess that the personal VPN had something to do with it. Did you do anything on it that put Fly.io at risk of getting in trouble?
yellowapple 2y ago
Well that just evaporated the interest I had in using Fly.io. Thanks for the heads-up.
- oliwarner 2y ago
  That's what makes me so deeply suspicious of posts like this. It costs nothing but a few minutes to spring up a fake account and dump on the competition. Who knows if Lukas is legit, but they don't seem to have asked the service provider where their data is yet. Or what happened. You're not going to get those answers here. Going straight to social media before you've found anything out stinks.
  - lukas_b 2y ago
    This is a reasonable criticism. I don't know how I can prove it as screenshots are not hard to fake, but here they are in any case. Also I emailed them and still haven't gotten a response, so I'm not inclined to be very understanding seeing as this appears to be happening quite frequently to people. https://imgur.com/a/xLj3tqd https://imgur.com/a/xLj3tqd my company site is infoproc dot co. it is currently down. but if you google it you can see it has been indexed. And you can search for my company on linkedin if you want to know who I am.
    - oliwarner 2y ago
      I really wasn't directly attacking you or demanding proof, just pointing out why I don't like the normalisation of complaining in public as an early step in a technical support.
- bananapub 2y ago
  you've made such a big change based on one post from a basically brand new account on HN? why?
  - yellowapple 2y ago
    It ain't really a big change; fly.io looked cool, but it ain't like I'm already a customer. And lukas_b doesn't seem to be all that brand-new of an account, either. As for "why", it's because I've been burned by this sort of thing before.
Arjuna144 2y ago
Thanks for telling. It may be a bug in their system, but the way this seems to have been handled by them is really ugly
- r0ks0n 2y ago
  that bug could even be... a fly...
bmitc 2y ago
It's my understanding that Fly.io heavily uses Elixir and has several Elixir bigshots on staff. But I only see mention of Fly.io for people are moving off of it or having major issues. I'm wondering of that's really the case and how given the tech stack.
- g15jv2dp 2y ago
  How is the programming language they use at all relevant?
- taspeotis 2y ago
  I forgot functional programming in Elixir meant no bugs! It's built on Erlang which halves bugs tenfold too! So 0 bugs because Elixir / 10 = 0 QED
- strzibny 2y ago
  The issue is political and has nothing to do with Elixir (which is pretty great btw). Please read the post again.
  - bmitc 2y ago
    Political? It was performed by their system. It doesn't sound like a human was in the loop at all, besides those who developed the system. And like I said, it isn't just this post alone. Read my comment again.
    - jtdowney 2y ago
      It was posted after your comment, but https://news.ycombinator.com/item?id=40938025 https://news.ycombinator.com/item?id=40938025 indicates that a human was in the loop.
nomilk 2y ago
> Has anyone else experienced something like this in the past My account was marked as high risk and was prevented from deploying for a few hours: https://stackoverflow.com/a/76400473/5783745 https://stackoverflow.com/a/76400473/5783745 But it didn't wipe out my apps. Took a couple hours to figure out what was going on and how to fix it (purchase some credit and run an esoteric and possibly unrelated command in the stack overflow answer). That (and a dozen other issues) was why I left fly. For now it's just not production-ready. I do hope they build a successful product (as defined by happy customers, not by fund raising). The more players there are in this space, the more options we have as developers, which is good even if the path there is bumpy.
- lukas_b 2y ago
  Thanks for this tip. I tried it and I keep getting the error: "Error: Your account got flagged by our internal fraud protection service" even after verifying my credit card.
yieldcrv 2y ago
well guess I’m keeping my current solution I use the free tier of vercel or netlify and host all the static assets on IPFS, I keep everything pinned using filecoin for free. lol yup I use filecoin to pin IPFS CIDs and because IPFS gateway CDNs get super unreliable, I created a js script that does a series of promises to every ipfs gateway and after one actually loads it cancels all the other promises, so that ensures availability to users and can handle spikes in traffic, without messing with the free tier
zacksiri 2y ago
The thing about fly.io / heroku / vercel is you are gaining convenience by giving up control over the underlying infrastructure. When you give up control of the infrastructure you also give up the ability to decide what's right and wrong. If you want to work with something that allows you retain that control but still have the convenience of a nice deployment workflow check out my profile.
- ec109685 2y ago
  Unless you build your own data center and run your own fiber, you’re at the mercy of someone else’s policies and could be shutdown at their whim.
  - miki123211 2y ago
    Even if you build your own datacenter and your own fiber, other ISPs might still either depeer from you or threaten to do so if you don't boot customer x off your service.
  - johnchristopher 2y ago
    Don't forget to bring a diesel generator if you forgot to build your own electric grid.
  - qhwudbebd 2y ago
    Whilst this is true in theory, the experience of being a data centre customer, taking transit from one or two providers and peering with a handful of others is very different to being a user of a bulk cloud infrastructure provider. I think the difference is automation, or rather lack of it. You build personal relationships with real people. Senior network engineers at the supplier don't consider it beneath them to ring up similarly competent people at the customer and chat directly. In the past I've hosted kit on a larger scale for my businesses, but it seems you get the same personal experience as a tiny individual customer. Even with just a couple of racks of kit of for hobby projects and friends' stuff, I know the network engineers at the DC and transit providers and can email or call direct to discuss things when I need to. More importantly, if (for example) one of my friends starts gobbling 10Gbps of $$$ transit because of a typo in a script, they ring my mobile to chuckle about it while we fix in realtime, rather than some automated process pulling the plug with the real engineers hiding behind support tickets and first-line support staff that can't string a sentence together.
strzibny 2y ago
This is where having Kamal config at hand is great. Even if I would lose my VM and access, I can be up and running again pretty quickly on a different host. I just change the IP in my config/deploy.yml
- ec109685 2y ago
  And I guess offsite backups too?
- alexedw 2y ago
  Would love to set up everything on Kamal but just feels like slightly too much management cost in maintaining a database yourself for mission critical stuff. If I'm going to use a managed DB, might as well do the same for the container? Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.
- tasuki 2y ago
  You mention Kamal as if it was some kind of a ubiquitous standard. It is the first time I hear about it. In what way is Kamal better than the hundreds of alternatives?
rizalp 2y ago
thanks for informing this, will avoid fly.io at all cost
deleted 2y ago
[deleted]
deely3 2y ago
Why not mention the name of the company? Then post will not become flagged, I suppose.
- lukas_b 2y ago
  the name is fly.io. it's in the title
  - deely3 2y ago
    Name of the posters company. Which site was erased by fly.io.
    - lukas_b 2y ago
      infoproc dot co. which is currently down.
mrkurt 2y ago
This account was not banned in error. Lukas, if you’d like me to explain publicly, I'm happy to do so.
- lukas_b 2y ago
  Maybe you could start by replying to my email? I would not have complained publicly if there had been any form of communication from you but there was none.
  - tptacek 2y ago
    I think here will do nicely. That OK with you?
    - bbg2401 2y ago
      How is that an acceptable proposal? I use Fly, but the policy to communicate with customers only via public forum is a disaster.
      - tptacek 2y ago
        That's not a decision we made. Anyways: I really don't think we have much more to say without this person's consent to proceed, beyond what was already said: we do not have a system that automatically deletes flagged accounts, and this was not an error. (What you're describing as our policy is certainly not our policy; these are extraordinarily distinctive circumstances.)
        deleted 2y ago
        [deleted]
    - tptacek 2y ago
      Things got real quiet here, didn't they?
      - 2y ago

111 points by lukas_b 2y ago

frafart 2y ago

I haven’t had my apps deleted but the fraud detection system flags me almost every time I interact with the platform. It’s very disruptive.

sergiotapia 2y ago

this sounds terrifying. so their "algorithm" can just nuke your entire business without any signaling?

lukas_b 2y ago
apparently I'm not the only one. https://community.fly.io/t/account-flagged-apps-gone/14570 https://community.fly.io/t/account-flagged-apps-gone/14570 https://web.archive.org/web/20240710225525/https://community.fly.io/t/account-flagged-apps-gone/14570 https://web.archive.org/web/20240710225525/https://community...
- gatnoodle 2y ago
  This is bad, really bad!

add-sub-mul-div 2y ago

Dropbox deleted all my data because I didn't log in for a long time. Guess you're supposed to check in on your completely static data occasionally. Whatever, the lesson is not to let yourself seriously rely on any of these services.

cpach 2y ago
No matter what service provider one uses, one should always have backups and a plan B/disaster recovery. The 3-2-1 rule is as relevant as ever.

altairprime 2y ago

What were your apps?

lukas_b 2y ago
a personal vpn and a company website
- yellowapple 2y ago
  I'd guess that the personal VPN had something to do with it. Did you do anything on it that put Fly.io at risk of getting in trouble?

yellowapple 2y ago

Well that just evaporated the interest I had in using Fly.io. Thanks for the heads-up.

oliwarner 2y ago
That's what makes me so deeply suspicious of posts like this. It costs nothing but a few minutes to spring up a fake account and dump on the competition. Who knows if Lukas is legit, but they don't seem to have asked the service provider where their data is yet. Or what happened. You're not going to get those answers here. Going straight to social media before you've found anything out stinks.
- lukas_b 2y ago
  This is a reasonable criticism. I don't know how I can prove it as screenshots are not hard to fake, but here they are in any case. Also I emailed them and still haven't gotten a response, so I'm not inclined to be very understanding seeing as this appears to be happening quite frequently to people. https://imgur.com/a/xLj3tqd https://imgur.com/a/xLj3tqd my company site is infoproc dot co. it is currently down. but if you google it you can see it has been indexed. And you can search for my company on linkedin if you want to know who I am.
  - oliwarner 2y ago
    I really wasn't directly attacking you or demanding proof, just pointing out why I don't like the normalisation of complaining in public as an early step in a technical support.
bananapub 2y ago
you've made such a big change based on one post from a basically brand new account on HN? why?
- yellowapple 2y ago
  It ain't really a big change; fly.io looked cool, but it ain't like I'm already a customer. And lukas_b doesn't seem to be all that brand-new of an account, either. As for "why", it's because I've been burned by this sort of thing before.

Arjuna144 2y ago

Thanks for telling. It may be a bug in their system, but the way this seems to have been handled by them is really ugly

r0ks0n 2y ago
that bug could even be... a fly...

bmitc 2y ago

It's my understanding that Fly.io heavily uses Elixir and has several Elixir bigshots on staff. But I only see mention of Fly.io for people are moving off of it or having major issues. I'm wondering of that's really the case and how given the tech stack.

g15jv2dp 2y ago
How is the programming language they use at all relevant?
taspeotis 2y ago
I forgot functional programming in Elixir meant no bugs! It's built on Erlang which halves bugs tenfold too! So 0 bugs because Elixir / 10 = 0 QED
strzibny 2y ago
The issue is political and has nothing to do with Elixir (which is pretty great btw). Please read the post again.
- bmitc 2y ago
  Political? It was performed by their system. It doesn't sound like a human was in the loop at all, besides those who developed the system. And like I said, it isn't just this post alone. Read my comment again.
  - jtdowney 2y ago
    It was posted after your comment, but https://news.ycombinator.com/item?id=40938025 https://news.ycombinator.com/item?id=40938025 indicates that a human was in the loop.

nomilk 2y ago

> Has anyone else experienced something like this in the past My account was marked as high risk and was prevented from deploying for a few hours: https://stackoverflow.com/a/76400473/5783745 https://stackoverflow.com/a/76400473/5783745 But it didn't wipe out my apps. Took a couple hours to figure out what was going on and how to fix it (purchase some credit and run an esoteric and possibly unrelated command in the stack overflow answer). That (and a dozen other issues) was why I left fly. For now it's just not production-ready. I do hope they build a successful product (as defined by happy customers, not by fund raising). The more players there are in this space, the more options we have as developers, which is good even if the path there is bumpy.

lukas_b 2y ago
Thanks for this tip. I tried it and I keep getting the error: "Error: Your account got flagged by our internal fraud protection service" even after verifying my credit card.

yieldcrv 2y ago

well guess I’m keeping my current solution I use the free tier of vercel or netlify and host all the static assets on IPFS, I keep everything pinned using filecoin for free. lol yup I use filecoin to pin IPFS CIDs and because IPFS gateway CDNs get super unreliable, I created a js script that does a series of promises to every ipfs gateway and after one actually loads it cancels all the other promises, so that ensures availability to users and can handle spikes in traffic, without messing with the free tier

zacksiri 2y ago

The thing about fly.io / heroku / vercel is you are gaining convenience by giving up control over the underlying infrastructure. When you give up control of the infrastructure you also give up the ability to decide what's right and wrong. If you want to work with something that allows you retain that control but still have the convenience of a nice deployment workflow check out my profile.

ec109685 2y ago
Unless you build your own data center and run your own fiber, you’re at the mercy of someone else’s policies and could be shutdown at their whim.
- miki123211 2y ago
  Even if you build your own datacenter and your own fiber, other ISPs might still either depeer from you or threaten to do so if you don't boot customer x off your service.
- johnchristopher 2y ago
  Don't forget to bring a diesel generator if you forgot to build your own electric grid.
- qhwudbebd 2y ago
  Whilst this is true in theory, the experience of being a data centre customer, taking transit from one or two providers and peering with a handful of others is very different to being a user of a bulk cloud infrastructure provider. I think the difference is automation, or rather lack of it. You build personal relationships with real people. Senior network engineers at the supplier don't consider it beneath them to ring up similarly competent people at the customer and chat directly. In the past I've hosted kit on a larger scale for my businesses, but it seems you get the same personal experience as a tiny individual customer. Even with just a couple of racks of kit of for hobby projects and friends' stuff, I know the network engineers at the DC and transit providers and can email or call direct to discuss things when I need to. More importantly, if (for example) one of my friends starts gobbling 10Gbps of $$$ transit because of a typo in a script, they ring my mobile to chuckle about it while we fix in realtime, rather than some automated process pulling the plug with the real engineers hiding behind support tickets and first-line support staff that can't string a sentence together.

strzibny 2y ago

This is where having Kamal config at hand is great. Even if I would lose my VM and access, I can be up and running again pretty quickly on a different host. I just change the IP in my config/deploy.yml

ec109685 2y ago
And I guess offsite backups too?
alexedw 2y ago
Would love to set up everything on Kamal but just feels like slightly too much management cost in maintaining a database yourself for mission critical stuff. If I'm going to use a managed DB, might as well do the same for the container? Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.
tasuki 2y ago
You mention Kamal as if it was some kind of a ubiquitous standard. It is the first time I hear about it. In what way is Kamal better than the hundreds of alternatives?

rizalp 2y ago

thanks for informing this, will avoid fly.io at all cost

deleted 2y ago

[deleted]

deely3 2y ago

Why not mention the name of the company? Then post will not become flagged, I suppose.

lukas_b 2y ago
the name is fly.io. it's in the title
- deely3 2y ago
  Name of the posters company. Which site was erased by fly.io.
  - lukas_b 2y ago
    infoproc dot co. which is currently down.

mrkurt 2y ago

This account was not banned in error. Lukas, if you’d like me to explain publicly, I'm happy to do so.

lukas_b 2y ago
Maybe you could start by replying to my email? I would not have complained publicly if there had been any form of communication from you but there was none.
- tptacek 2y ago
  I think here will do nicely. That OK with you?
  - bbg2401 2y ago
    How is that an acceptable proposal? I use Fly, but the policy to communicate with customers only via public forum is a disaster.
    - tptacek 2y ago
      That's not a decision we made. Anyways: I really don't think we have much more to say without this person's consent to proceed, beyond what was already said: we do not have a system that automatically deletes flagged accounts, and this was not an error. (What you're describing as our policy is certainly not our policy; these are extraordinarily distinctive circumstances.)
      - deleted 2y ago
        [deleted]
  - tptacek 2y ago
    Things got real quiet here, didn't they?
    - 2y ago