15 ms·
Fly.io outage – resolved
- EGreg 2y agoWhat exactly does flyio.net do?
- michaelbuckbee 2y agoHosting service that has a lot of interesting distributed features.
- HellsMaddy 2y agoIf you mean specifically flyio.net and not just fly.io the company, I'm guessing they host their status page on a separate domain in case of DNS/registrar issues with their primary domain.
- stackghost 2y agoIIRC their value prop is that they let you rapidly spin up deployments/machines in regions that are closest to your users, the idea being that it will be lower latency and thus better UX.
- eek2121 2y agoWEB 2.0. SEE. TOLD YA! THEY SHOULDA UPGRADED TO THAT NEWFANGLED 3.0! ;)
- vachina 2y agoIt’s basically what Heroku used to be but with CDN-like presence.
- punkpeye 2y agoIt is not reflected in their status page, but fly.io itself is not even loading.
- arusahni 2y agoOof, hugops to the team.
- stevefan1999 2y agoYep...can confirm my self hosted Bitwarden there is completely FUBAR connection wise even if it is in EA, so it should be a worldwide outage...lemme guess, some internal tooling error, consensus split brain, or if it looks like someone leaked BGP routes again?
- jasonjayr 2y agoDNS. It's always DNS. /s
- jart 2y agohttps://github.com/jart/cosmopolitan/blob/master/third_party/musl/dns.png https://github.com/jart/cosmopolitan/blob/master/third_party...
- monkaiju 2y agoMight be! Shameless plug of a DNS tool i wrote years ago for anyone this pushes to learn more about DNS https://dug.unfrl.com/ https://dug.unfrl.com/
- satoru42 2y agoMine is in Asia and it's still accessible.
- odo1242 2y agoIt was a consensus split-brain (“database replication failure”) it seems
- redslazer 2y agofly.io just has the weirdest outages. It has issues so regularly we dont even need to run mock outages to make sure our system fail overs work.
- duxup 2y agoWhen I worked for a company who worked with big banks / financial institutions we used to run disaster recovery tests. Effectively a simulated outage where the company would try to run off their backup sites. They ran everything from those sites, it was impressive. Once in a while we'd have a real outage that matched the test we ran as recently as the weekend before. I was helping a bank switch over to the DR site(s) one day during such a real outage and I left my mic open when someone asked me what the commotion was on the upper floors of our HQ. I said "super happy fun surprise disaster recovery test for company X". VP of BIG bank was on the line monitoring and laughed "I'm using that one on the executive call in 15, thanks!" Supposedly it got picked up at the bank internally after the VP made the joke and was an unofficial code for such an outage for a long time.
- NetOpWibby 2y agoThankfully your comment was positive!
- latch 2y agoIn most BIG banks, "Vice President" is almost an entry-level title. Easily have 1000s of them. For example, this article points out that Goldman Sachs had ~12K VPs out of more than 30K employees: https://web.archive.org/web/20150311012855/https://www.wsj.com/articles/SB10001424052702304692804577281252012689294 https://web.archive.org/web/20150311012855/https://www.wsj.c...
- jart 2y agoVP at Goldman is equivalent to Senior SWE according to levels.fyi and their entry level is Analyst. I'm surprised by the compensation though. I would have thought people working at a place with gold in the name would be making more. Also apparently Morgan Stanley pays their VPs $67k/year.
- shubhamjain 2y agoThis is probably 5th or 6th major outage from Fly.io that I have personally seen. Pretty sure there were many others and some just went unnoticed. I recommended the service to a friend, and within two days he faced two outages. Fly.io seriously needs to get it together. Why it hasn’t happened yet is a mystery to me. They have a good product but stability needs to be an absolute top for a hosting service. Everything else is secondary.
- mcqueenjordan 2y agoReliability is hard when your volume is (presumably) scaling geometrically.
- paxys 2y agoCan't use the "reliability is hard" excuse when you are quite literally in the business of selling reliability.
- mcqueenjordan 2y agoIt’s just not that big of a mystery. It’s not an excuse; it’s just true. Also, they’re not especially selling reliability as much as they’re selling small geo-distributed deployments.
- ilrwbwrkhv 2y agoDoes anyone use them beyond the free tier? Same with Vercel for example.
- benhoyt 2y agoMy fly.io-hosted website went down for 5 minutes (6 hours ago), but then came right back up, and has been up ever since. I use a free monitoring service that checks it every 5 minutes, so it's possible it missed another short bit of downtime. But fly.io has been pretty reliable overall for me!
- dprotaso 2y agoWhat free monitoring tool do you use?
- deleted 2y ago[deleted]
- nomilk 2y agoWould be fascinated to see your data over a period of months. Application up time is flakey, but what was worse were fly deploys failing for no clear reason. Sometimes layers would just hang and eventually fail for no particular reason; I'd run the same command an hour or two later without any changes and it would just work as expected. I'd love to make a monitoring service to deploy a basic app (i.e. run the fly deploy command) every 5 minutes and see how often those deploys fail or hang. I'd guess ~5% inexplicably fail, which is frustrating unless you've got a lot of spare time.
- Joel_Mckay 2y ago[flagged]
- LorenzoGood 2y agoWhat does rust have to do with fly.io?
- Joel_Mckay 2y ago[flagged]
- 2y ago
- MaxfordAndSons 2y agoKinda funny that they've named their global state store "Corrosion"... not really a word I'd associate with stability and persistence.
- kermatt 2y agohttps://community.fly.io/t/reliability-its-not-great/11253 https://community.fly.io/t/reliability-its-not-great/11253 https://github.com/superfly/corrosion https://github.com/superfly/corrosion
- lordofgibbons 2y agoIt's an internal project based on Rust, not a product. So I don't think it matters too much what they name it. It's opens source which is great, but still not a product that they need to market.
- SOLAR_FIELDS 2y agoAnd to be fair, it’s a bit of a cute meme to name rust projects things that relate to it. Oxide, etc
- dumah 2y agoI take your point but corrosion-resistant metals such as Aluminum, Titanium, Weathering Steel and Stainless Steel don’t avoid corrosion entirely but form a thin and extremely stable corrosion layer (under the right conditions).
- littlestymaar 2y agoGold and platinum really are corrosion resistant though (but have questionable mechanical properties…)
- toast0 2y agoI stored important data in mnesia, so who would I be to talk. :p
- deleted 2y ago[deleted]
- veggieWHITES 2y agoI was considering these guys the other day until I saw their pricing page: https://fly.io/pricing/ https://fly.io/pricing/ (There's not a single price on there, why even create the page?)
- deleted 2y ago[deleted]
- schmichael 2y agoThe prices are just one click deeper. Hardly a nefarious dark pattern.
- rascul 2y agoThere's a link to what appears to be the actual pricing page https://fly.io/docs/about/pricing/ https://fly.io/docs/about/pricing/ There's also a link to the pricing calculator https://fly.io/calculator https://fly.io/calculator
- totetsu 2y agoIs that calculator hourly or monthly?
- radicalriddler 2y agoLiterally says "Monthly Costs" in the green panel on the right that calculates the total.
- eviks 2y agoIt's right there: "Monthly Cost"
- veggieWHITES 2y ago[flagged]
- tptacek 2y ago
- deleted 2y ago[deleted]
- HellsMaddy 2y agoSuspiciously, Turso started having issues around the same time. Their CEO confirmed on Discord it's due to the Fly outage: > Ok.I caught up with our oncall and This seems related to the Fly.io incident that is reported in our status page. Our login does call things in the Fly.io API > we are already in touch with Fly and will see if we can speed this up
- pier25 2y agoNot the first time Turso goes down because of Fly issues. It must suck to have built a db service and have this downtime. Apparently Turso are going to offer an AWS tier at some point.
- jonasdoesthings 2y agoLast month Turso released AWS-hosted databases to the public (still in Beta): https://turso.tech/blog/turso-aws-beta https://turso.tech/blog/turso-aws-beta
- pier25 2y agoThanks!
- DataOverload 2y agoWe switched from Fly to CF workers a while ago, and never looked back
- eek2121 2y agocongrats on not developing a playbook for the time you have to 'look back'. Providers will fail. good contingencies won't. ...hears faint sound...I SAID GOOD, QUIET YOU!
- punkpeye 2y agoThey are fundamentally different. If Cloudflare provided a way to host docker containers with volumes though, that would be game over for so many paas platforms.
- stoicjumbotron 2y agoCan't wait: https://blog.cloudflare.com/container-platform-preview/ https://blog.cloudflare.com/container-platform-preview/
- rstupek 2y agoHow are they equivalent?
- frakkingcylons 2y agoI switched from apples to oranges and never looked back.
- mrcwinn 2y agoI tried Fly early. I was very excited about this service, but I've never had a worse hosting experience. So I left. Coincidentally I tried it again a few days ago. Surely things must be better. Nope. Auth issues in the CLI, frustrations deploying a Docker app to a Fly machine. I wouldn't recommend it to anyone.
- steve_adams_86 2y agoI find their user experience to be exceptional. The only flake I’ve encountered is in uptime and general reliability of services I don’t interface with directly. They’ve done a stellar job on the stuff you actually deal with, but the glue holding your services together seems pretty wobbly.
- teaearlgraycold 2y agoI'm grateful to HN for keeping me well aware of Fly's issues. I'll never use them.
- kachapopopow 2y agoIt's still 99.99+% SLA? Would you really pay 100% more for <0.01% more uptime?
- mrcwinn 2y agoThis is not my experience at all, as a former paying customer.
- cj 2y agoI think what a lot of people fail to understand is that there are certain categories of apps that simply “can never go down” Examples include basically any PaaS, IaaS, or any company that provides a mission-critical service to another company (B2B SaaS). If you run a basic B2C CRUD app, maybe it’s not a big deal if you service goes down for 5 minutes. Unfortunately there are quite a few categories of companies where downtime simply isn’t tolerated by customers. (I operate a company with a “zero downtime” expectation from customers - it’s no joke, and I would never use any infrastructure abstraction layer other than AWS, GCP or Azure - preferably AWS us-east-1 because, well, if you know the joke…)
- macNchz 2y agoEvery PaaS and IaaS I’ve ever used has had some amount of downtime, often considerably more than 5 minutes, and I’ve run production services on many of them. Plenty of random issues on major cloud providers as well. Certainly plenty of situations with dozens of Twitter posts happening but never any acknowledgement on the AWS status page. Nothing’s perfect.
- cj 2y agoYea, when running services where 5 minutes of downtime results in lots of support tickets, you learn to accept that the incident will happen and learn to manage the incident rather than relying that it will never occur.
- akshayshah 2y agoThe series of outages early in 2023 also had some Corrosion-related pain: https://community.fly.io/t/reliability-its-not-great/11253 https://community.fly.io/t/reliability-its-not-great/11253
- __turbobrew__ 2y agoSeems like rolling their own datastore turned out to be a bad bet. Im not super familiar with their constraints but scylladb can do eventual consistency and is generally quite flexible. CouchDB is also an option for multi-leader replication.
- pier25 2y agoMy apps on Fly have not gone down this time.
- marvin-hansen 2y agoNo surprise. About a year ago, I looked at fly.io because of it's low pricing and I was wondering where they were cutting corners to still make some money. Ultimately, I found the answer in their tech docs where it was spelled out clearly that an fly instance is hardwired to one physical server and thus cannot fail over in case that server dies. Not sure if that part still is in the official documentation. In practice, that means if a server goes down, they have to load the last snapshot from that instance from the Backup and push it on a new server, update the network path, and pray to god that not more server fail than spare capacity is available. Otherwise you have to wait for a restore until the datacenter mounted a few more boxes in the rack. That explains quite a bit the randomness of those outage reports i.e. my app is down vs the other is fine and mine came back in 5 minutes vs the other took forever. As a business on a budget, I think anything else i.e. a small civo cluster serves you better.
- fulafel 2y agoThe status tells a story about a high-availability/clustering system failure so I think in this case the problem is rather the complexity of the HA machinery hurting the system's availability vs something like a simple VPS.
- ignoramous 2y agoFly.io can migrate vm+volume now: https://fly.io/docs/reference/machine-migration/ https://fly.io/docs/reference/machine-migration/ / https://archive.md/rAK0V https://archive.md/rAK0V > a fly instance is hardwired to one physical server and thus cannot fail over I'm having trouble understanding how else this is supposed to be? I understand that live migration is a thing, but even in those cases, a VM is "hardwired" to some physical server, no?
- mzi 2y ago> I'm having trouble understanding how else this is supposed to be? I understand that live migration is a thing, but even in those cases, a VM is "hardwired" to some physical server, no? You can run your workload (in this case a VM) on top of a scheduler, so if one node goes down the workload is just spun up on another available node. You will have downtime, but it will be limited.
- theideaofcoffee 2y agoColor me not surprised. My few interactions with people there just gave off the impression of them being in a bit over their heads. I don't know how well that translated to their actual ops, but it's difficult to not connect the two when they continue to have major outage after major outage for a product that 'should' be their customer's bedrock upon which they build everything else.
- deleted 2y ago[deleted]
- xyst 2y agoRecurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-squads that were unfortunate to draw the short stick.
- aaomidi 2y agoWhat do "Freezes" mean? Like, do you stop renewing your certificates? Do you stop taking in security updates for your software? Sure maybe "unnecessary" changes, but the line gets very gray very fast.
- vrosas 2y agoNo unnecessary code deployments.
- Spivak 2y agoIt's not very grey, prod becomes as if you told everyone but your ops team to go home and then sent your ops team on a cruise with pagers. If it's not important enough to merit interrupting their vacation you don't do it.
- fragmede 2y agoCerts shouldn't still be done by hand that this point; if another heartbleed comes out in the next 7 days then the risk can be examined, escalated, and the CISO can overrule the freeze. If it's a patch for remote root via Bluetooth drivers on a server that has no Bluetooth hardware, it's gonna wait. you're right that there's a grey line, but crossing that line involves waking up several people and the on call person makes a judgement call. if it's not important enough to wake up several people over, then things stay frozen.
- aaomidi 2y agoRight, that's basically what I mean. There are a lot of automated changes happening in the background for services. I guess the whole thing I'm saying is that not every breakage is happening because of a code change.
- jart 2y agofly.io publishes their post-mortems here: https://fly.io/infra-log/ https://fly.io/infra-log/ The last post-mortem they wrote is very interesting and full of details. Basically back in 2016 the heart or keystone component of fly.io production infrastructure was called consul, which is a highly secure TLS server that tracks shared state and it requires that both the server certificate and the client certificate be authenticated. Since it was centralized, it had scaling issues, so fly.io wrote a replacement for it in 2020 called corrosion, and quickly forgot about consul, but didn't have the heart to kill it. Then in October 2024 consul's root key signing key expires, which brought down all connectivity, and since it uses bidirectional authentication, they couldn't bring it back online until they deployed new SSL certificates to every machine in their fleet. Somehow they did this in half an hour, but the chain of dominoes had already been set in motion to reveal other weaknesses in their infrastructure that they could eliminate. There was this other internal service whose own independent set of TLS keys had also expired long ago, but they didn't notice until they tried rebooting it as part of the consul rekey, since doing so severed the TCP connections it had established way back when its certificate was valid. Plus the whole time this is happening, their logging tools are DDOSing their network provider. It took some real heroes to save the company and all their customers too when that many things explode at once.
- ignoramous 2y agoOn that Consul outage, Fly Infra concludes, "The moral of the story is, no more half-measures." On their careers page [1], the Fly team goes, "We're not big believers in tech debt." As an outsider, reads like a cacophony of contradictions? [1] https://fly.io/docs/hiring/working/#we-re-ruthless-about-doing-customer-visible-work https://fly.io/docs/hiring/working/#we-re-ruthless-about-doi...
- jart 2y agoNo one actually lives up to their principles, but it's still important that we have them. If you actually do live up to yours, then you need to adopt better principles.
- 2y ago
- gigapotential 2y agoHUGOPS Everything is going to be 200 OK!
- xyst 2y agoI can’t even login to my old account. Password reset is timing out yet still receive password reset e-mail. Password reset link broken, with 500 status code.
- neya 2y agoPersonal experience between Fly.io and Railway.com - Railway wins for me hands down. I have used both and the Railways support is stellar too, in comparison. Fly.io never responded to my query about data deletion till date. Despite emailing on their support email. I have had my Railway app online till date without any major downtimes too. I recommend anyone looking for a decent replacement to try them.
- punkpeye 2y agoHow does it compare in terms of price?
- justjake 2y agoWe actually only charge you for what you use. As a result people often see 30%+ savings when moving stuff over from other providers (especially Heroku) https://railway.com/pricing https://railway.com/pricing
- andai 2y agoI've used Railway control panel maybe a total of 10 times in my life and half the time it was having weird issues. Control panel UI not loading or not working, actions failing, deploys randomly failing... I love the idea but in practice it's not something I'd want to use for anything serious.
- justjake 2y agoWhile we've always aimed for great reliability on compute, the dashboard reliability wasn't very good at the start of the year. We ack'd this and then pretty heavily to making it stellar, so if you're still having issues please let us know (that should not be the case) Best, Jake from Railway
- andai 2y agoI used Railway as a "set it and forget it" for a client project, and I hadn't heard from him in over a year until some Railway update caused some issues with the deploy (something about group permissions). But support was very helpful in getting that fixed very quickly, so credit there! (And to be fair it did apparently work without any problems for like a year and a half, so credit there too!)
- punkpeye 2y agoContrary to the title of the post, Fly.io API remains inaccessible. Meaning, users still cannot access deploys/databases, etc. For accurate updates, follow https://community.fly.io/t/fly-io-site-is-currently-inaccessible/22791/40 https://community.fly.io/t/fly-io-site-is-currently-inaccess...
- cryptos 2y agoFly.io seems to be a bit of a mixed bag: https://news.ycombinator.com/item?id=41917436 https://news.ycombinator.com/item?id=41917436 https://news.ycombinator.com/item?id=35044516 https://news.ycombinator.com/item?id=35044516 https://news.ycombinator.com/item?id=34742946 https://news.ycombinator.com/item?id=34742946 https://news.ycombinator.com/item?id=34229751 https://news.ycombinator.com/item?id=34229751 If a cloud platform doesn't really provide reliability, I'd say it's probably not worth it. You could better just rent a (virtual) server and save the cloud tax.
- qeternity 2y agoI don't really understand the value prop of fly.io. They seem to have an impressive engineering team despite the outages, but is edge compute really something that 99.9% of devs need? There are tons of large companies that operate out of a single AWS region and those services are used by millions around the globe. It just strikes me as something that enables premature optimization right out of the box.
- jrockway 2y agoI would take edge compute if it's free and easy. That's fly.io's value prop. In a world where much web browsing starts with ACK SYN ACK, it is nice if the server is close to you.
- k__ 2y agoIt's basically the new Heroku with less lock-in, because it works with Docker. You get edge computing, autoscaling, and load balancing without additional configuration. Not as flexible as AWS, but also much easier to setup and maintain. But the reliability issues suck now and then.
- nikodotio 2y agoThis is precisely it. The ease of deploy, https domain configuration, scaling. Additionally, having machines that turn off when not in use is easy to configure, which I never managed on AWS.
- tamrix 2y ago[flagged]
- viraptor 2y agoThey actually hired some serious people. If they keep failing, it's not due to lack of total experience. That experience may not be well utilised, but the potential exists and they're far from kids.
- mattbee 2y agoIt feels like fly is trying to repeat a growth model that worked 20 years ago: throw interesting toys at engineers, then wait for engineers to recommend their services as they move on in their careers. Part of that playbook is the old Move Fast & Break Things. That can still be the right call for young projects, but it has two big problems: 1) AWS successfully moved themselves into the position of "safe" hosting choice, so it's much rarer for engineers to have influence on something that's seen by money men as a humdrum, solved problem; 2) engineers are not the internal influencers they used to be, being laid off left and right the last few years, and without time for hobby projects. (maybe also 3) it's much harder to build a useful free tier on a hosting service, which used to be a necessary marketing expense to reach those engineers). So idk, I feel like the bar is just higher for hosting stability than it used to be, and novelty is a much harder sell, even here. Or rather: if you're going to brag about reinventing so many wheels, they need to not to come off the cart as often.
- travisgriggs 2y agoDon’t a bunch of Elixir/Erlang guys work at fly.io? It’s weird to me that that hallmark of reliability is associated with something that the public sees as unreliable. What gives with that association?
- Huppie 2y agoIt's interesting to see this discussion about fly.io's reliability on a day that (after over three days of downtime) Microsoft Azure finally decided the update of Azure Static Web Apps they deployed last Friday is indeed broken for customers using specific authentication settings... ...with not a single status update from Microsoft in sight.