7 ms·

My main complaint about flyio is instability, which you won't notice in your first couple weeks. A couple of times per mo, redis (and occasionally postgres) ti

by itake 4y ago

My main complaint about flyio is instability, which you won't notice in your first couple weeks. A couple of times per mo, redis (and occasionally postgres) times out. My TLS cert did not renew, resulting in 8 hours of downtime. You get what you pay for.

CGamesPlay 4y ago
I got lucky and hit all of these problems in my first few hours of trying Fly (much to their detriment, unfortunately). I wanted to love it, but I encountered nothing but bugs when I played with it over a weekend a month or so ago. Internal DNS worked for one but not the other of my nodes; Dashboard accepted my credit card but didn't actually upgrade my plan; SSH console would hang, requiring killing the client process with SIGKILL; my builder hung with a Dockerfile that was literally just "FROM ubuntu" and "apt update"; this failure resulted in `flyctl machines list` hanging and `flyctl machines kill` hanging. Maybe it was just a bad weekend at Fly, but their status page was all green, too. So many problems between me and even giving fly a serious try. Unfortunate, but maybe in a year it will be better.
- mrkurt 4y ago
  It'll definitely be better in a year. It sounds like you hit some of our capacity issues. In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former. The SSH and Builder issues are interesting. Those sound like something related to our wireguard stack.
benhoyt 4y ago
Yeah, that would certainly have slowed me down too if I'd experienced that. I haven't had any of these issues. Fingers crossed! And hopefully they're working on stability.
selcuka 4y ago
My account got marked as "high possibility of fraud", even though I validated my email and added a credit card they even charged. The error message said that I had to contact support@fly.io but that address only accepts email from paid subscriptions. With no way to evaluate the service, I opted to delete my account.
- omeletdufromage 4y ago
  Exactly what happened to me as well. In my case, I was (fortunately?) messaged by someone from Fly.io after asking around the community forum how to remedy the issue. I think the person I talked to was inferring that the system got tripped because I was using my own domain for my email address. It's since been corrected, and I still have a handful of services running with them, but getting started with the platform definitely was a rougher experience than I was expecting it to be.
  - lloydatkinson 4y ago
    So they will regard any large company using their service as fraud? Insane
    - wasmitnetzen 4y ago
      Large company? Literally every company I ever worked for, the smallest one were 5 people, had their own domain for emails. And they were not all in IT.
      - lloydatkinson 4y ago
        So we agree its insane to cut off every company with a domain? Good.
        ceejayoz 4y ago
        That would be insane, yes. It isn’t the case, though. I signed up with my own domain just fine.
        wstuartcl 4y ago
        I don't know what merchant system they are using but I will say it is very common for them to have their own anti fraud detection and rejection on charges. Your specific case could be related to a huge list of possibilities (even many of them a rollup of many other interactions on other sites that happen to use the same merchant or foundational fraud data). Thinking this is just because of a domain name is silly. Also Fly.io -- You may want to clearly accept and manage trial issues (or at least a subset) via a support path. It seems silly to effectively bounce users with the impression that you have no support because they are not YET paying customers while in the trial phase.
ZephyrBlu 4y ago
My main gripe is the wild variation in response time. My server response time seemed to vary from <100ms to >300ms for the same operation.
- re-thc 4y ago
  Is this due to the network i.e. you get routed to a different server or due to CPU sharing?
  - achempion 4y ago
    isn't the whole selling point of fly is edge servers which supposed to reduce latency
    - re-thc 4y ago
      Sure, but what's edge? The world is big. Can't have servers everywhere (ok, maybe Cloudflare is close). It's possible due to routing decisions that you get sent to a "different edge" at different times.
      - marcinzm 4y ago
        So what's the point then? 250ms is enough to literally send traffic to the other side of the world. So why bother with edges if you can just have a single server where most of your customers are and get the same latencies?
  - ZephyrBlu 4y ago
    I thought it could have been because of a shared CPU at first as well, but I upgraded to dedicated CPUs and it didn't make a difference. I don't know whether this is a platform issue with Fly or something else, but it was kind of annoying to have such a high variation with seemingly no way to fix it.
- ignoramous 4y ago
  fwiw, for non-html workloads we serve (nodejs) with shared-cpu (several 100 requests per second), the latencies (p50, p75) remain consistent.
quickthrower2 4y ago
Venture funded stuff like this should be fast, good, cheap, pick 3! Until they need to make a profit or shut down.
- deleted 4y ago
  [deleted]
butterfly771 4y ago
After I have experienced several fly.io render railway northflank platforms, I think the problem with fly.io is that the web console is not fully functional, too dependent on CLI and the response time of the application is unstable. The problem with render is that it is too slow to build, north is similar to railway, and the experience is good. But although the north console is very comprehensive, it is too complicated and dazzling. By contrast, the interface of railway is very simple and comprehensive, and UX is great. (found by wappalyzer analysis that it is based on nextjs,very nice)
- eschatology 4y ago
  I tried railway and northflank. Really like the UI northflank has; I think anyone familiar with Kubernetes should feel right at home. I also agree with the sentiment that railway ui is simpler.
- fyzix 4y ago
  I get up to 20secs of downtime when redeploying on railway... I was using the sample golang repo from fly : https://github.com/fly-apps/go-example https://github.com/fly-apps/go-example That's just unacceptible imo.
  - fyzix 4y ago
    More context: The UI shows that the service is available but when I visit the live url, I get a 404. What I think is happening is that the UI shows the status of the underlying container and there is some fault with the reverse proxy they are using to expose the container to the internet. Fly.io doesn't have this issue but I am leery of all the complaints here.
  - ndneighbor 4y ago
    Hey there fyzix! Angelo here, Support Engineer from Railway. I was here procrastinating from work when I saw this comment. The downtime you see is our proxy taking time to cutover- however, we have https://docs.railway.app/deploy/healthchecks https://docs.railway.app/deploy/healthchecks that will only cutover once we have a 200 from your API. This way we can keep your old deploy live and serving requests if and only if it's live. Theres more we can do to make it magical, but this should help in the meantime.
jamil7 4y ago
There’s an edge case in which you can lose access to your account as well when switching from github to email login with the same email. It’s an edge case but quite a bad one.
callahad 4y ago
Yep. Fly isn't fully baked. Deployed a service earlier this month and got bitten in a few ways: - It took several hours to get a 1 GB disk provisioned (their API would time out and I'd get left with an allocated but unusable disk that couldn't be deleted without deleting the entire app it was attached to.) The Fly forums have numerous reports of the same issue, going back months, across multiple data centers. All resolved with the moral equivalent of "oops, try again later?" - The fly.toml reference is incomplete and internally inconsistent, documenting keys which are not valid, while omitting valid keys that appear elsewhere in the docs. - Volume snapshots are retained for either 5 or 7 days, depending on which of the multiple conflicting docs pages you believe. - I wanted to let them know about that last one, so I emailed the address listed at https://fly.io/docs/about/support/ https://fly.io/docs/about/support/, only to get an autoresponse saying that they don't read that mailbox and that I should pound sand unless I'm already paying them to read my emails. Reading between the lines of their recent blog posts, it seems like a lot of the infrastructure pains are pinned on Nomad, with the hope that moving to Fly Machines / Apps V2 will obviate a lot of those issues. I hope they're right, because what they're trying to deliver is exactly what I want.
- mrkurt 4y ago
  Well Nomad certainly isn't at fault for our docs. Volume provisioning issues have been a game of whackamole, largely as the result of scaling pains. We've grown like 4x in the last 3 months (thanks, Heroku) and it's put a strain on pretty much everything.
incangold 4y ago
Echo this, and I _did_ notice it in my first two weeks. 3 separate issues prevented me deploying. One was my fault, but hard to diagnose because error messages contain no useful information. The other two were the platform falling over, with the support forum having no resolution but plenty of other people in the same boat. Lots to love about fly. The latency is noticeably better than anything I’ve used except cloudflare. Prices are good. Hopefully they’ll get the growing pains sorted and we’ll have another contender.
- mrkurt 4y ago
  These deploy issues are all kinds of ulcer inducing. They're not strictly Nomad's fault (we're holding it wrong), but the complexity of flyctl -> nomad -> global hosts makes it difficult to bubble errors back up to you. If you end up trying again, give the machines based apps a shot. It's much simpler infrastructure: https://community.fly.io/t/fly-apps-on-machine-prerelease/10219 https://community.fly.io/t/fly-apps-on-machine-prerelease/10...
mrkurt 4y ago
The TLS cert renewal was a big failure on our part. I know that's, like, a one strike and you're out problem, but we've improved that part of our stack substantially. It should be much more resilient. Redis/Postgres timeouts sound like they might be a problem we can help with, though. Both those services work through a load balancer with a TCP idle timeout. If you get timeouts after a period of inactivity, it's usually a simple config tweak to have a driver handle that seamlessly. The timeout issues surprised me, more drivers than we expected can't handle a DB connection timing out when it's idle.
Existenceblinks 4y ago
I had (a couple years ago) some weird websocket throttle or debouncing characteristic. I tried to construct an evidence, but it's quite hassle so I just ignored it. I think they are too engineering culture. Target is devs, sure, but they could use some product guys.
itbeho 4y ago
I'd spend some time reading community.fly.io before committing to it. I recently helped a startup that has been on fly.io and there was a lot of frustration about things not working. it was very frustrating to see un-answered problems reported, fly.io down for what seemed to be many users, the same time seeing a fly.io co-founder arguing about politics all day long here on HN...
- butterfly771 4y ago
  In fact, when you enter the fly.io console, you will feel that this is an unfinished construction site, especially when you start to deploy a new app, and the frame text icons in the options are simply truncated, which feels rough.

by itake 4y ago

CGamesPlay 4y ago

I got lucky and hit all of these problems in my first few hours of trying Fly (much to their detriment, unfortunately). I wanted to love it, but I encountered nothing but bugs when I played with it over a weekend a month or so ago. Internal DNS worked for one but not the other of my nodes; Dashboard accepted my credit card but didn't actually upgrade my plan; SSH console would hang, requiring killing the client process with SIGKILL; my builder hung with a Dockerfile that was literally just "FROM ubuntu" and "apt update"; this failure resulted in `flyctl machines list` hanging and `flyctl machines kill` hanging. Maybe it was just a bad weekend at Fly, but their status page was all green, too. So many problems between me and even giving fly a serious try. Unfortunate, but maybe in a year it will be better.

mrkurt 4y ago
It'll definitely be better in a year. It sounds like you hit some of our capacity issues. In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former. The SSH and Builder issues are interesting. Those sound like something related to our wireguard stack.

benhoyt 4y ago

Yeah, that would certainly have slowed me down too if I'd experienced that. I haven't had any of these issues. Fingers crossed! And hopefully they're working on stability.

selcuka 4y ago

My account got marked as "high possibility of fraud", even though I validated my email and added a credit card they even charged. The error message said that I had to contact support@fly.io but that address only accepts email from paid subscriptions. With no way to evaluate the service, I opted to delete my account.

omeletdufromage 4y ago
Exactly what happened to me as well. In my case, I was (fortunately?) messaged by someone from Fly.io after asking around the community forum how to remedy the issue. I think the person I talked to was inferring that the system got tripped because I was using my own domain for my email address. It's since been corrected, and I still have a handful of services running with them, but getting started with the platform definitely was a rougher experience than I was expecting it to be.
- lloydatkinson 4y ago
  So they will regard any large company using their service as fraud? Insane
  - wasmitnetzen 4y ago
    Large company? Literally every company I ever worked for, the smallest one were 5 people, had their own domain for emails. And they were not all in IT.
    - lloydatkinson 4y ago
      So we agree its insane to cut off every company with a domain? Good.
      - ceejayoz 4y ago
        That would be insane, yes. It isn’t the case, though. I signed up with my own domain just fine.
      - wstuartcl 4y ago
        I don't know what merchant system they are using but I will say it is very common for them to have their own anti fraud detection and rejection on charges. Your specific case could be related to a huge list of possibilities (even many of them a rollup of many other interactions on other sites that happen to use the same merchant or foundational fraud data). Thinking this is just because of a domain name is silly. Also Fly.io -- You may want to clearly accept and manage trial issues (or at least a subset) via a support path. It seems silly to effectively bounce users with the impression that you have no support because they are not YET paying customers while in the trial phase.

ZephyrBlu 4y ago

My main gripe is the wild variation in response time. My server response time seemed to vary from <100ms to >300ms for the same operation.

re-thc 4y ago
Is this due to the network i.e. you get routed to a different server or due to CPU sharing?
- achempion 4y ago
  isn't the whole selling point of fly is edge servers which supposed to reduce latency
  - re-thc 4y ago
    Sure, but what's edge? The world is big. Can't have servers everywhere (ok, maybe Cloudflare is close). It's possible due to routing decisions that you get sent to a "different edge" at different times.
    - marcinzm 4y ago
      So what's the point then? 250ms is enough to literally send traffic to the other side of the world. So why bother with edges if you can just have a single server where most of your customers are and get the same latencies?
- ZephyrBlu 4y ago
  I thought it could have been because of a shared CPU at first as well, but I upgraded to dedicated CPUs and it didn't make a difference. I don't know whether this is a platform issue with Fly or something else, but it was kind of annoying to have such a high variation with seemingly no way to fix it.
ignoramous 4y ago
fwiw, for non-html workloads we serve (nodejs) with shared-cpu (several 100 requests per second), the latencies (p50, p75) remain consistent.

quickthrower2 4y ago

Venture funded stuff like this should be fast, good, cheap, pick 3! Until they need to make a profit or shut down.

deleted 4y ago
[deleted]

butterfly771 4y ago

After I have experienced several fly.io render railway northflank platforms, I think the problem with fly.io is that the web console is not fully functional, too dependent on CLI and the response time of the application is unstable. The problem with render is that it is too slow to build, north is similar to railway, and the experience is good. But although the north console is very comprehensive, it is too complicated and dazzling. By contrast, the interface of railway is very simple and comprehensive, and UX is great. (found by wappalyzer analysis that it is based on nextjs,very nice)

eschatology 4y ago
I tried railway and northflank. Really like the UI northflank has; I think anyone familiar with Kubernetes should feel right at home. I also agree with the sentiment that railway ui is simpler.
fyzix 4y ago
I get up to 20secs of downtime when redeploying on railway... I was using the sample golang repo from fly : https://github.com/fly-apps/go-example https://github.com/fly-apps/go-example That's just unacceptible imo.
- fyzix 4y ago
  More context: The UI shows that the service is available but when I visit the live url, I get a 404. What I think is happening is that the UI shows the status of the underlying container and there is some fault with the reverse proxy they are using to expose the container to the internet. Fly.io doesn't have this issue but I am leery of all the complaints here.
- ndneighbor 4y ago
  Hey there fyzix! Angelo here, Support Engineer from Railway. I was here procrastinating from work when I saw this comment. The downtime you see is our proxy taking time to cutover- however, we have https://docs.railway.app/deploy/healthchecks https://docs.railway.app/deploy/healthchecks that will only cutover once we have a 200 from your API. This way we can keep your old deploy live and serving requests if and only if it's live. Theres more we can do to make it magical, but this should help in the meantime.

jamil7 4y ago

There’s an edge case in which you can lose access to your account as well when switching from github to email login with the same email. It’s an edge case but quite a bad one.

callahad 4y ago

Yep. Fly isn't fully baked. Deployed a service earlier this month and got bitten in a few ways: - It took several hours to get a 1 GB disk provisioned (their API would time out and I'd get left with an allocated but unusable disk that couldn't be deleted without deleting the entire app it was attached to.) The Fly forums have numerous reports of the same issue, going back months, across multiple data centers. All resolved with the moral equivalent of "oops, try again later?" - The fly.toml reference is incomplete and internally inconsistent, documenting keys which are not valid, while omitting valid keys that appear elsewhere in the docs. - Volume snapshots are retained for either 5 or 7 days, depending on which of the multiple conflicting docs pages you believe. - I wanted to let them know about that last one, so I emailed the address listed at https://fly.io/docs/about/support/ https://fly.io/docs/about/support/, only to get an autoresponse saying that they don't read that mailbox and that I should pound sand unless I'm already paying them to read my emails. Reading between the lines of their recent blog posts, it seems like a lot of the infrastructure pains are pinned on Nomad, with the hope that moving to Fly Machines / Apps V2 will obviate a lot of those issues. I hope they're right, because what they're trying to deliver is exactly what I want.

mrkurt 4y ago
Well Nomad certainly isn't at fault for our docs. Volume provisioning issues have been a game of whackamole, largely as the result of scaling pains. We've grown like 4x in the last 3 months (thanks, Heroku) and it's put a strain on pretty much everything.

incangold 4y ago

Echo this, and I _did_ notice it in my first two weeks. 3 separate issues prevented me deploying. One was my fault, but hard to diagnose because error messages contain no useful information. The other two were the platform falling over, with the support forum having no resolution but plenty of other people in the same boat. Lots to love about fly. The latency is noticeably better than anything I’ve used except cloudflare. Prices are good. Hopefully they’ll get the growing pains sorted and we’ll have another contender.

mrkurt 4y ago
These deploy issues are all kinds of ulcer inducing. They're not strictly Nomad's fault (we're holding it wrong), but the complexity of flyctl -> nomad -> global hosts makes it difficult to bubble errors back up to you. If you end up trying again, give the machines based apps a shot. It's much simpler infrastructure: https://community.fly.io/t/fly-apps-on-machine-prerelease/10219 https://community.fly.io/t/fly-apps-on-machine-prerelease/10...

mrkurt 4y ago

The TLS cert renewal was a big failure on our part. I know that's, like, a one strike and you're out problem, but we've improved that part of our stack substantially. It should be much more resilient. Redis/Postgres timeouts sound like they might be a problem we can help with, though. Both those services work through a load balancer with a TCP idle timeout. If you get timeouts after a period of inactivity, it's usually a simple config tweak to have a driver handle that seamlessly. The timeout issues surprised me, more drivers than we expected can't handle a DB connection timing out when it's idle.

Existenceblinks 4y ago

I had (a couple years ago) some weird websocket throttle or debouncing characteristic. I tried to construct an evidence, but it's quite hassle so I just ignored it. I think they are too engineering culture. Target is devs, sure, but they could use some product guys.

itbeho 4y ago

I'd spend some time reading community.fly.io before committing to it. I recently helped a startup that has been on fly.io and there was a lot of frustration about things not working. it was very frustrating to see un-answered problems reported, fly.io down for what seemed to be many users, the same time seeing a fly.io co-founder arguing about politics all day long here on HN...

butterfly771 4y ago
In fact, when you enter the fly.io console, you will feel that this is an unfinished construction site, especially when you start to deploy a new app, and the frame text icons in the options are simply truncated, which feels rough.