26 ms·

Reliability: It’s not great

1226 points by bishopsmother 4y ago

emschwartz 4y ago
One of my colleagues keeps repeating “reliability is our number one feature”. I’m not sure it is for 100% of early stage startups, but I guess it is once you exceed some minimum usage threshold. That said, definitely appreciate the detailed explanation.
- jpdb 4y ago
  > One of my colleagues keeps repeating “reliability is our number one feature”. I think reliability is the #1 feature at any stage because if you're unavailable, you're at best useless and more than likely you are actively harmful because your users have an expectation. However, if you're unavailable outside of times customers don't expect you to be there then you're not actually unavailable. This is more likely for an early stage start-up, but you don't typically choose or know when you're expected to be available nor do you always get to choose when you're unavailable.
  - sa46 4y ago
    In terms of confidentiality, availability, and integrity: I'll bet LastPass would gladly trade availability right now to regain confidentiality.
  - ignoramous 4y ago
    Our team at AWS had a poster up on the wall that more or less went: 1. Security 2. Durability 3. Availability 4. Speed Similar: https://twitter.com/colmmacc/status/1071088017190711296 https://twitter.com/colmmacc/status/1071088017190711296
- jrochkind1 4y ago
  > One of my colleagues keeps repeating “reliability is our number one feature”. > I’m not sure it is for 100% of early stage startups, I mean, it probably depends on the nature of the startup? Platform-as-service seems particularly sensitive to reliability (whether or not it's "#1 feature"), in a way that might not be true of startups in other spaces.
- cschep 4y ago
  TL;DR -- It's very domain specific if reliability is your number one feature. For a startup that is hosting other people's production application/data then this is absolutely true. Less than 100% always needs to be addressed. For a startup that is selling bingo cards then reliability probably isn't nearly as important. I'm guessing there were certain holidays that were more important than others as far as reliability goes though? Maybe patio11 can chime in :)
- nprateem 4y ago
  That's what crossing the chasm is about. Before the chasm are tinkerers who don't mind things failing as much, but after it, people want reliable things that Just Work.
soperj 4y ago
For django, they should really contribute to 2 scoops django cookie cutter program, so that you can get an out of the box django instance that can just deploy to Fly.io.
- mixmastamyk 4y ago
  May be wrong, but they seem to be very focused on interesting stacks like Elixir and RoR while building on Go/Rust. The corollary being neglect of the bread/butter stacks with high market share, like Python and Java/JavaScript. Don't think I've seen a blog post discussing those three beyond a passing mention? Not the end of the world, but mildly disappointing. At least they are all in with Postgres and Linux, a great foundation.
  - jjtheblunt 4y ago
    I thought the only thing their tech stack was all in on was docker images being rewritten to run on firecracker as a substrate. is it not agnostic about things like Elixir etc, at the tech level, though they've got super nice documentation for those tools you mentioned?
    - ilaksh 4y ago
      What's necessary to change for them to run on Firecracker?
      - ign0ramus 4y ago
        Here's a post explaining how they do it https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
        ilaksh 4y ago
        Yeah, saw that before. I thought he was saying you have to change your Dockerfiles to use them with fly.io. Just misinterpreted that sentence.
deleted 4y ago
[deleted]
iamdbtoo 4y ago
I'm a big fan of fly.io. From their hiring process to the product itself it's all carried out in a thoughtful manner. I hope they can weather this rough time.
yamrzou 4y ago
I'm not a user of Fly.io. I can't help but notice how remarkable the effect of open communication on potential end users like me. I remember reading about their reliability problems on HN some time ago. That biased my view of the company. After reading this, the open communication and transparency restored my trust in them, and would make them again a potential candidate for future projects. Because now I know that they acknowledge the problem and that they are trying to improve things.
- leetrout 4y ago
  mrkurt[1] is also active here and has been very transparent in his comments about scaling issues. Similar to this post he commented a week ago: > In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former. I am rooting for Fly! Great team. The company reminds me of early HashiCorp. [1] https://news.ycombinator.com/user?id=mrkurt https://news.ycombinator.com/user?id=mrkurt
- mbesto 4y ago
  Me too. However, this is a double edge sword. Their key value proposition is scale / speed which makes it concerning that they haven't "solved" that yet.
- willio58 4y ago
  Agreed, this is how company communication should be. I don't use Fly but would consider them in the future even given their recent issues. I look at this in contrast to Twitter who had/has? an outage today. Their leadership is opaque and doesn't take responsibility for the issues they are causing.
  - alfalfasprout 4y ago
    In fairness, a CEO who has basically been Kanye-ing himself and his company into irrelevance is a low bar.
- gizmo 4y ago
  This post is carefully worded corporate messaging, but because they write for their developer audience it has an informal "oh shucks we messed up bad y'all" vibe to it. But make no mistake, this is 100% corporate messaging. I get that growing is super hard. And maybe fly will grow up to be a good platform some day. But that's the future. Today, they're flying by the seat of their pants and I mostly feel sorry for people who were tricked into thinking this platform is ready for production use.
deleted 4y ago
[deleted]
tiffanyh 4y ago
Would the simple solve be that Fly.io just mark any new service of theirs “beta” for x-months post launch?
- samwillis 4y ago
  None of the services that they have had issues with in the recent past are new, they have been running for at least a couple of years. They would need to put a "beta" sticker on the whole platform for that suggestion to work. But the post makes it clear that the issue isn't that they had problems with new services. It was rapid customer growth before they had time to scale up the infrastructure as they had planned to do.
sergiotapia 4y ago
It's been almost a year since I gave Fly a review (https://news.ycombinator.com/item?id=31391116 https://news.ycombinator.com/item?id=31391116) and it's a bummer that they're still struggling to get things right. Double bummer because I love Phoenix and Elixir and they employ Chris McCord there. Maybe they were _too_ ambitious at the start? They have a hard road ahead of them, and competition like Render.com and Northflank have provided me with solutions to all of my problems. Great dev ux, great prices and predictable solutions. They also keep pushing out very useful features. A third competitor also sprung up Railway! There's certainly blood in the water. Will they catch up to others before the competition solves the "global mesh" unique value proposition Fly.io currently has? That's the $1MM question.
- zomglings 4y ago
  I read your review, and had a question so I thought I'd follow up here. You mentioned render.com as a competitor - does render host its own infrastructure or do they act as a go-between between their users and AWS/GCP/whatever?
  - cldellow 4y ago
    They act as a go-between in that they ultimately host on AWS/GCP. They host their own infrastructure in that they appear to run Kubernetes and have built out their own deployment and service fabric, so they're just using the underlying machines as dumb compute, they're not, eg, building on RDS. In March 2021, someone asked a question about carbon emissions of their data centres. They said they hosted on both GCP and AWS, but mentioned they were interested in moving to their own bare metal [1]. In April 2021, I asked a question about egress fees to Google, and they walked back a bit the comment about moving to bare metal [2]. As of March 2022, they're still in AWS/GCP [3]. As of September 2022, workloads for new users deploy into AWS, even in regions that were previously served by GCP [4]. [1]: https://community.render.com/t/does-render-use-green-energy/912/6 https://community.render.com/t/does-render-use-green-energy/... [2]: https://community.render.com/t/is-render-com-hosted-in-google-cloud/1187 https://community.render.com/t/is-render-com-hosted-in-googl... [3]: https://community.render.com/t/are-your-servers-owned-by-you/3719 https://community.render.com/t/are-your-servers-owned-by-you... [4]: https://community.render.com/t/which-render-regions-map-to-which-cloud-providers/5987/3 https://community.render.com/t/which-render-regions-map-to-w...
claytonjy 4y ago
Very interesting to see Kurt assert theyre going to "solve managed Postgres", and I'm super curious to know what that means. Does it mean something like RDS, or more like CrunchyData? I could see them building something RDS-like on their own, but if they're trying to go further than that I wonder if they'll buy or partner with other companies rather than doing it themselves. Neon strikes me as a Postgres-as-a-service that could pair well with Fly.
- mmcclure 4y ago
  That comment jumped out to me too, my recollection was that they've been pretty vocal about that not being something they wanted to solve themselves as a core competency. I'm not quite parsing if these two blurbs should still be taken together or if the second sentence is refuting the first. > The second problem we have with Postgres was a poor choice on my part. We decided to ship “unmanaged Postgres” to buy ourselves time to wait for the right managed Postgres provider to show up. > We’re going to solve managed Postgres. It’s going to take a while to get there, but it’s a core component of the infrastructure stack and we can’t afford to pretend otherwise. +1 to Neon seeming like a good fit, but it's also very much a beta (alpha?) both as a product and company (at least from my impression). I'm not sure that's a bet they'd want to make right now given the context of this post.
  - nikita 4y ago
    (Neon CEO) We are launching our paid tier March 15th and will be production ready shortly after. We are running 20K+ databases and measuring reliability and uptime. Generally reliability is a function of architecture (we are solid there), good SRE practices, and a long tail of event you live through, fix, and make sure they never happen again. The bigger the fleet the faster the learning.
    - mmcclure 4y ago
      That's great, love to hear it! Really excited about what you all are working on.
    - 4y ago
chucky_z 4y ago
mrkurt have you considered some of the lower tiers of vault enterprise that allow for performance replicas that just outright solve that problem? might be cheaper than an engineer at this point.
nu11ptr 4y ago
Not a client of fly.io, but dang impressive for the company to be this open and honest. Definite respect - wish more companies were like this. It puts them on my short list almost immediately for future needs.
- nathell 4y ago
  Came here to say pretty much this. Technical issues come and go; open communication is a core part of the company culture, and builds up trust.
- deleted 4y ago
  [deleted]
- nprateem 4y ago
  Why would you shortlist a company that admits to reliability issues?
  - macintux 4y ago
    All companies have reliability issues, especially in this problem space. I’d rather go with someone who acknowledges their failings in public.
  - jen20 4y ago
    Because if they admit to them they’re probably in the process of fixing them. When compared to a certain B tier cloud who gaslight users about flaws in the product and API, the amount of effort expended to get to an admission of fault is much lower for companies not inclined to lie by omission.
samwillis 4y ago
Fundamentally I think some of the problems come down to the difference between what Fly set out to build and what the market currently want. Fly (to my understanding) at its core is about edge compute. That is where they started and what the team are most excited about developing. It's a brilliant idea, they have the skills and expertise. They are going to be successful at it. However, at the same time the market is looking for a successor to Heroku. A zero dev ops PAAS with instant deployment, dirt simple managed Postgres, generous free level of service, lower cost as you scale, and a few regions around the world. That isn't what Fly set out to do... exactly, but is sort of the market they find themselves in when Heroku then basically told its low value customers to go away. It's that slight miss alignment of strategy and market fit that results in maybe decisions being made that benefit the original vision, but not necessarily the immediate influx of customers. I don't envy the stress the Fly team are under, but what an exciting set of problems they are trying to solve, I do envy that!
- vineyardmike 4y ago
  I agree - fly is so easy to use (when it works) that it’s hard not to be impressed. BUT what I’ve found is that we don’t need edge compute, since our customers aren’t that latency sensitive, so it’s lost on us. It’s only a few more milliseconds to us-east-1. I’ve heard (on HN) of a dozen different companies vying for the heroku replacement spots and yet Fly seemed to capture the attention. I couldn’t name another one off hand. What I truly want and probably lots of other people too is Flyctl (and workflow) for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles.
  - latchkey 4y ago
    > What I truly want and probably lots of other people too is Flyctl for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles. Google Cloud. It is painfully easy to spin up managed postgres, super easy to deploy gcp cloud functions or gcp cloud run. It isn't expensive either and just works.
    - 4y ago
outworlder 4y ago
> This is a theme. Existing open source is not designed for global deployment Eh? Unless you are consuming something as a service and it actually advertises it as a feature, nothing is ready for 'global deployment'. If you have a 'centralized' secret storage, then you have made it tied to a region. Want to have redundancies and lower latency? You'll have to distribute it. Vault has docs about this: https://developer.hashicorp.com/vault/tutorials/day-one-raft/multi-cluster-architecture https://developer.hashicorp.com/vault/tutorials/day-one-raft...
- josegonzalez 4y ago
  This one is interesting because Vault has an enterprise product which I assume (hope?) Fly is paying for. That enterprise version includes performance replicas, which allows for cross-region replication of secrets with region-local reads (and slightly lower writes). The OP almost makes it sound like they are using the non-enterprise versions (or at the very least, not taking advantage of this particular functionality). That said, I'd imagine with large enough scale, these sorts of features break anyways.
theloco 4y ago
I love reading stuff like this. I don't use fly, don't plan to, not totally sure everything it does and will check it out. But this is some great raw data on how stressful it is after you launch.
tebbers 4y ago
I really feel for Fly, as a potential customer. They are trying really hard. I would still love to use them one day and this post is definitely a step in the right direction. Growing is painful but they have smart people working there so fingers crossed that they sort this ASAP and it doesn't become existential.
jrochkind1 4y ago
I remain kind of amazed about how heroku managed to pull off what they pulled off, in the first case. Also: > The Heroku exodus broke our assumptions. Pre-Heroku, most of the apps we were running were spread across regions. And: we were growing about 15% per month. But post-Heroku, we got a huge influx of apps in just a few hot spots — and at 30% per month. I hadn't before seen anyone with a big picture view confirm a heroku exodus was happening, although a lot of people suspected it or had anecdotes. But if fly is seeing a pretty enormous number of customers moving from heroku to fly... oh wait, now I'm wondering, is this mainly a result of heroku ending free services, and those are free customers coming to fly for free services? If so... that's a pretty big burden to take on without revenue to match, it does seem kind of dangerous for fly.
- mike_d 4y ago
  > I remain kind of amazed about how heroku managed to pull off what they pulled off Heroku was built on top of AWS. They didn't have to handle many of the hard problems Fly does.
- 8organicbits 4y ago
  Do they unpack what the 30% includes? Is that revenue, VMs, accounts?
lopatin 4y ago
It sounds like they need more money to scale the shared stack
ec109685 4y ago
I wonder what types of RPS they are seeing that required a gossip based protocol to broadcast state around versus a more traditional data store. I take it that it’s far more important that the local region know about changes than a remote region, which makes a mastered store in one location as the source of truth problematic. I also wonder why these companies don’t backstop themselves on the public cloud? Failing into an AWS seems better than running out of capacity and some its services could be used in circumstances where an open source technology isn’t ready.
- likecarter 4y ago
  Yeah, reading the post made it seem like they followed “best practices” without really thinking things through. KISS.
e1g 4y ago
This reads like a mea culpa from an indie hacker, but Fly.io had 5+ years and raised $40M to get these basic fundamentals right. And we get promises of a new status page.
- ignoramous 4y ago
  Well, that's one way to look at it. Fly's been many things over the course of its lifetime [0], but I believe their latest pivot (on what they call "Machines") is pretty darn good. I've been using Machines since Oct last year, and things have gotten better week-over-week. Like with any platform, Fly has its own idiosyncrasies, which don't take much to get hang of. That said, I am the only person in my tech shop that deals with Fly. Some orgs with larger teams and heavier apps that deploy frequently or run DBs / disks on Fly (I don't) have had a rough few months; so that's there too. [0] Ex A: https://news.ycombinator.com/item?id=13985940 https://news.ycombinator.com/item?id=13985940
  - mbStavola 4y ago
    Fly Machines, if I understand them correctly, feels like a step backwards. Sure they might work better than "standard" Fly apps, but one of the motivating cases for Fly is being able to effortlessly scale across the world without having a Ph.D. in CS and a fistful of certs for Cloud engineering. That vision for Fly is awesome, game changing even, ignoring their current stability issues. Machines isn't that. From the documentation, it appears as though it's "just" a VM pinned to a single region and none of the "magic" of Fly really applies. If the server your VM is hosted on goes down, Fly won't redeploy your container. It's just downtime. Spinning up in other regions is something you have to think about and actually do. It seems closer to Heroku than it does Fly. Maybe I am totally misunderstanding Fly Machines and their use-case, maybe they're aiming to close the gap between Machines and Fly apps. It's just a bit of a bummer to see something that looks like walking back the original "promise" of Fly and makes me question whether or not Fly is going to just become like every other PaaS (even if it's a really good one).
    - ignoramous 4y ago
pier25 4y ago
I've been using Fly for over two years or so. The sentiment of this post doesn't align with my personal (anecdotal) experience. The PG issues hit me two times in the previous weeks but other than that it's been working great for me. With the move to v2 apps (using their new machines infra) things are actually faster and smoother than ever. About a year or so ago their CLI was quite buggy but I haven't really hit any bugs in months. I will remain with Fly for the time being. Hopefully they don't close shop!
- tptacek 4y ago
  We're nowhere even within the line of sight of closing up shop. We just haven't been doing a good job of aggressively communicating (a) when things go wrong and (b) what we're doing to account for it. The Fly.io of 2023 looks almost nothing like that of 2021 (all for the better), and it's not obvious to our users what's changed. We've been doing a shitty job of communicating, and we're taking our licks for it now.
  - chris_st 4y ago
    A lot can happen in 11 years :-) And thanks a lot for fly.io -- it's working great for my (rather small) use cases.
    - mrkurt 4y ago
      Oh my god that's a great callback.
- russellthehippo 4y ago
  Agree - V2 apps on machines are incredibly slick to launch (create/start/stop), get info on with graphql, and scale up and out. Magic. When the PG administration experience is that good I'll move it all over.
victorbjorklund 4y ago
Great post. Love how transparent Fly is. Im a customer but without any life and death important apps. And yea they had some issues lately so good they are addressing it.
deleted 4y ago
[deleted]
lll-o-lll 4y ago
At first I was all like “Ha ha, losers can’t scale” And then I was “Huh, these technical challenges are actually pretty difficult” And then I was all “crap, these are a bunch of technologies I was about to add to our stack” Thanks heaps fly.io people; having the humility to honestly talk about the challenges and failures massively helps people such as myself as we navigate new unfamiliar technologies. If more companies were willing to do this, it’d be a lot easier to avoid common pitfalls.
- chucky_z 4y ago
  The tech in their stack is still pretty good. Unless you’re supporting tens of thousands of customers and trying to make the promises that fly makes today. Look at the fly engineer replies in this thread. Also they basically only use OSS versions, they could go give Hashicorp some money to solve their Vault problems. They could probably partner with SecondQuadrant for PG as two examples. That might not make sense for their business though. Hard problems are hard no matter the choice.
  - lll-o-lll 4y ago
    Sure, I was going for a little humour there. A little riff on the whole “we always judge others until we walk in their shoes”. The take away I was hoping for is “providing insights into how we struggle helps others”
deleted 4y ago
[deleted]
ChrisMarshallNY 4y ago
Well, I feel for them. Scaling up is a bitch. I've been lucky, in the past, but a lot of that, is because I have "overengineered," and the tools/frameworks have advanced to meet the new demand. I am in the middle of a complete, bottom-to-top rewrite of the app we've been developing for the last couple of years. It's going great, but making this leap was a fraught decision. It's mainly, so I wouldn't have to write a post like that, in a year or two. We spent all the time refining it, until we had what we wanted, and it worked great on our small test team. Then, I loaded up a test server with 10,000 fake users, and tossed the app at that. To be fair, we don't think we'll have even that many users for quite a while. It's a very specialized demographic. * SOB * It no do so well. At that point, I had to decide whether to fix the issues (they were quite fixable), or revisit the architecture. The main issue with the architecture, was that it was an "accreted" app, with changes gradually being factored in, as we progressed. The main reason for this, is because no one really knew what they wanted, until we ran it up the flagpole (sound familiar?). The business logic was distributed throughout the app. That was ... ugly. I envisioned myself, a year or two down the road, sucking on a magnum, because the app had turned into a Cruftosaurus, and was coming for me in my nightmares. So I decided to rewrite, as we hadn't done any kind of MVP or public beta, so we actually had the runway to do this. I refined the entire business logic of the app into a single, platform-agnostic SPM module, which took just over a month, and have started to develop the app around that. It's pretty much a rewrite, but I am recycling a lot of the app code. We also brought in our designer, and he's looking at every screen I make. It's working well for him. Like I said, it's going great. Better than I expected. I know that I have a huge luxury, and I'm grateful. I can credit a lot of that, to doing some stress-testing before we got to a point where we had a bunch of users to support. I was able to go in, and go all Victor Frankenstein on the model. The result, so far, is that this thing screams, and you don't really even notice that there's that many users on it. The model has already been proven (that SPM module), and all we're doing, is chrome (which is a ton of work).
- bbkane 4y ago
  Hope this isn't a dumb question, but what's an "SPM module"?
  - 4y ago
skywhopper 4y ago
Interesting issues. Nothing surprising for anyone who’s run a global SaaS before, especially if growth has been incredibly fast. I find the gripes about Consul, Nomad, and Vault interesting since it sounds like the problems are mainly due to poor architectural decisions. Fly is rewriting those tools rather than invest in deploying them properly and in the process are running into new issues that those tools have already solved, which doesn’t give me confidence that the path forward will be any less bumpy.
ashiban 4y ago
One of the key challenges we observe is that if you're small enough, a Heroku like experience works well - and most of your needs would be covered by virtually any combination of techstacks. It gets significantly more challenging when you grow, either in feature complexity or scale complexity - and then very few services can offer what AWS/GCP/Azure offer - albeit at the increased engineering/monetary cost of using them. We're building a different kind of approach[0] that aims to absorb the mechanical cost of using public cloud capabilities (that are proven to scale) without hiding it altogether. [0] https://github.com/KlothoPlatform/klotho https://github.com/KlothoPlatform/klotho
a3w 4y ago
[flagged]
pyentropy 4y ago
Almost half of the issues are caused by their use of HashiCorp products. As someone that has started tons of Consul clusters, analyzed tons of Terraform states, developed providers and wrote a HCL parser, I must say this: HashiCorp built a brand of consistent design & docs, security, strict configuration, distributed-algos-made-approachable... but at its core, it's a very fragile ecosystem. The only benefit of HashiCorp headaches is that you will quickly learn Golang while reading some obscure github.com/hashicorp/blah/blah/file.go :)
- tptacek 4y ago
  We are asking to HashiCorp products to do things they were not designed to do, in configurations that they don't expect to be deployed in. Take a step back, and the idea of a single global namespace bound up with Raft consistency for a fleet deployed in dozens of regions, providing near-real-time state propagation, is just not at all reasonable. Our state propagation needs are much closer to those of a routing protocol than a distributed key-value database. I have only positive things to say about every HashiCorp product I've worked with since I got here.
  - pyentropy 4y ago
    I respect that. Can you elaborate a bit on the routing protocol thing? I assume you used WAN gossip? I love the simplicity of fly.io & wish you all the best improving Fly's reliability!
    - tptacek 4y ago
      If you've ever implemented IS-IS or OSPF before, like 80% of the work is "LSP flooding", which is just the process that gets updates about available links from one end of the network to another as fast as possible without drowning the links themselves in update messages. Flooding algorithms don't build consensus, unlike Raft quorums, which intrinsically have a centralized set of authorities that keep a single source of truth for all the valid updates. An OSPF router uses those updates to do build a forwarding table with a single-point shortest path first routine, but there's nothing to say that you couldn't instead use the same notion of publishing weighted advertisements of connectivity to, for instance, build a table to map incoming HTTP requests to backends that can field them. The point is, if you're going to do distributed consensus, you've got a dilemma: either you're going to have the Ents moot in a single forest, close together, and round trip updates from across the globe in and out of that forest (painfully slow to get things in and out of the cluster), or you're going to try to have them moot long distance (painfully slow to have the cluster converge). The other thing you can do, though, is just sidestep this: we really don't have the Raft problem at all, in that different hosts on our network do not disagree with each other about whether they're running particular apps; if worker-sfu-ord-1934 says it's running an instance of app-4839, I pretty much don't give a shit if worker-sfu-maa-382a says otherwise; I can just take ORD's word for it. That's the intuition behind why you'd want to do something like SWIM update propagation rather than Raft for a global state propagation scheme. But if you're just doing service discovery for a well-bounded set of applications (like you would be if you were running engineering for a single large company and their internal apps), Raft gives you some handy tools you might reasonably take advantage of --- a key-value store, for instance. You're mostly in a single data center anyways, so you don't have the long-distance-Entmoot problem. And HashiCorp's tools will federate out across multiple data centers; the constraints you inherit by doing that federation mostly don't matter for a single company's engineering, but they're extremely painful if you're servicing an unbounded set of customer applications and providing each of them a single global picture of their deployments. Or we're just holding it wrong. Also a possibility.
bmorton 4y ago
[flagged]
- sph 4y ago
  It's a post on their forum, hardly an ad.
deivid 4y ago
I'm a bit sour reading this. I've always liked fly and particularly the engineering blog, so much so that a couple of months ago I decided to apply for an infra position, to work on some of these very topics. Sadly after 4~5 rounds of interviews (including a workday) they just ghosted me.
- deleted 4y ago
  [deleted]
- tptacek 4y ago
  If that happened, it absolutely was not on purpose. Shoot me an email at thomas@fly.io.
- ignoramous 4y ago
  > Don't feel too bad nor take it personal. They probably have a lot of applicants, and are looking to grow their team by hiring someone with very specific skills. > I also applied a few months ago while I was in the middle of my job search. For one, I couldn’t really answer their "favorite syscall question" because I’ve never dealt with syscalls :) so maybe I just wasn't a good fit. Surely, everyone's favourite syscall is exit()
  - jlokier 4y ago
    For what it's worth, exit() is not a syscall, but _exit() often is.
revskill 4y ago
Fly.io seems like Vercel 1.0 (where you can just deploy docker image and done), but it's more than that, with configurable volumes, secrets,... I'm bullish on fly.io.
1023bytes 4y ago
I get it, I like fly.io, but the last outage made me switch to Railway.app
nathants 4y ago
i also wanted a good cli for aws, and built one: https://github.com/nathants/libaws https://github.com/nathants/libaws companies like fly are fantastic. they provide a good service, and they put market pressure on aws. a free tier isn’t important anymore. with usage based pricing for lambda/dynamo/s3, an app with usage approaching zero has no cost.
plasma 4y ago
Thanks for sharing! Would it help to replace Corrosion with a simpler "Here's my local known state" blob that is POST'd to blob storage (for example) on a major cloud provider, and have another service read that at intervals? Just to make it really simple. There will be a better way than that, but my thought is if you can make it simpler (known state is always just pushed, so missing updates auto-recovers and avoids corruption) then you can be building on top of a more stable service discovery system. Centralized secret storage, can you keep the US instance read/write, but replicate read-only copies (a side-car tool that copies the database to other regions at various intervals?) so each region can fetch secrets locally? Or perhaps both can be solved with a general "Copy local state to other regions" service that is pretty simple but gives each region its own copy of other region's information (secrets, provisioning states, ...). I've needed to do similar things for some of the apps I've built, where a service needed another (simpler) service in front of it to bear the traffic load but was operationally simple (deferred the smarts to the system it was using as the source of truth) and automatically recovered from failure due to its simplicity.
- preseinger 4y ago
  +1 — service discovery _feels_ like something that gossip can solve, but that's only true at global/open/untrusted scale, in a well-scoped and authoritative domain like a company it's really not, it's really just another typical, expected-to-be-consistent state problem
djha-skin 4y ago
> In response, we’ve shipped a project called Corrosion. Corrosion is a gossip > based service discovery system. I wonder why they didn't try to use Serf[1] for this, since they were so into HashiCorp tools. It also uses the gossip protocol. 1: https://www.serf.io/docs/index.html https://www.serf.io/docs/index.html
swamp40 4y ago
"CEO finds novel way to fix Capacity problems..." They just lost about 40% of their paying users with that blog post.
nomilk 4y ago
What an earnest post, and how damn refreshing it is to see such concern for users, accountability, honesty and openness (quite a contrast to another PaaS)! I moved one app successfully from heroku to fly and attempted to move a few others. These are my experiences (both good and bad): Great: - The load time on the pages is insanely faster on fly than heroku. Sometimes I thought I was on the localhost version of the app, it was that snappy. - Love that it uses a Dockerfile - Love paying for what I use (compared to Heroku's rigid minimum of $16/month for hobby dyno w/ postgres for baby apps, or $34/month just to get a second web dyno for toddler apps). The same apps are <$5/month each on fly. Not great: - I find the fly.toml file hard to understand and use, and the cycle time slow to fix or tinker with it. It's partly (entirely?) a 'me' problem because I haven't spent a huge amount of time reading the documentation. - I found scheduling a rake task in a rails app time consuming (~days) the first time, but very easy (15 minutes) the second and subsequent times, once I knew a way that worked (cron didn't work; had to use a tool I hadn't used before 'supercronic'). - Deploys sometimes time out with `Error failed to fetch an image or build from source: error rendering push status stream: EOF`. Most layers copied, but randomly, some layers wouldn't. All I could do is keep trying until it worked, which it did, 2 hours later. Not the end of the world, but an annoying complication when you're already trying to solve complex problems. - I followed a youtube video on how to move a rails app from heroku to fly, and it worked on a modern app, but I couldn't quite get fly happy when moving the older app - something to do with postgres versions, and I didn't want to spend all day figuring it out. I'm not hugely experienced with docker, it could have been an easy fix for someone more experienced. On reflection, 3 of the 4 negatives above are solvable by me reading the docs more thoroughly and getting more proficient with docker. I look forward to continuing using and exploring fly, and can't be happier with the directness, transparency and care from fly staff. A platform with huge potential.
- bradgessler 4y ago
  Did you try migrating with this guide? https://fly.io/docs/rails/getting-started/migrate-from-heroku/ https://fly.io/docs/rails/getting-started/migrate-from-herok... The issues you ran into with older versions of Rails was probably because the Dockerfile that `fly launch` generated was for new versions of Rails. We switched to https://github.com/rubys/dockerfile-rails https://github.com/rubys/dockerfile-rails to streamline Dockerfile generation and support older versions of Rails. If you try it again and run into issues you can open an issue at https://github.com/rubys/dockerfile-rails/issues https://github.com/rubys/dockerfile-rails/issues or post in https://community.fly.io https://community.fly.io and somebody will help get that sorted out. The more versions of Rails we can deploy the better!
andy_ppp 4y ago
How easy is it to deploy things with GitHub actions, AWS and terraform instead? I think I’d like full infrastructure control after having accidentally got a job doing DevOps which I’m starting to get quite into. I should probably write up a blogpost about converting everything over…
none_to_remain 4y ago
Three little pigs, but they all live in a house of straw built on top of several houses of sticks. (The foundation is an old house of brick, but the wolf isn't bothered by that.)
Karupan 4y ago
Thanks for the honest technical write up - not easy to air one's dirty laundry to users. Given the scalability and stability issues, I am curious to understand the percent of apps deployed to fly are actually used in production/critical to business. Sounds like they have quite a few hobby/free tier users (myself included) who probably won't notice certain issues unlike paid customers.
hinkley 4y ago
> We’ve put a lot of work and resources into growing the platform and maturing our engineering organization. But that work has lagged growth. I fundamentally don't understand why people are in such a big hurry to get 'famous'. I've worked a couple of places where the marketing side was working as hard as they could to make sure that our heads were on fire at all possible moments. At one job I had a (very, very junior) manager come up to me and say great news we landed <big customer> and my immediate reply was, "fuck me". We were already running to stay upright and now we're about to have twice as much scrutiny. Wonderful. If you push hard enough, eventually everyone looks like an idiot. The number of humans for whom that is not true could fit into a book. Both alive and deceased. They most definitely do not work for the companies I've described, at least not enough of them so you'd notice.
computomatic 4y ago
Where does fly.io document their per-account services limits? For example, max apps, databases, etc. I took a quick look and couldn't find them. Do they have any documented service limits? A google search turned up [0] which does not inspire optimism. > ...there isn’t a limit to number of apps from a billing standpoint... [0] https://community.fly.io/t/free-tier-limits-and-quota-needs-clarification/8755 https://community.fly.io/t/free-tier-limits-and-quota-needs-...
rtpg 4y ago
I have two extremely tiny sites (like, "handful of users/1 user" sites) on Fly. I have had multiple incidents despite me not even touching them. The thing that worries me about these incidents is they haven't been, like, full service outages. A small subset of users talking about issues in forums. This makes me just feel like Fly has an immense amount of issues. At least if like 50% of fly goes down then it feels like a config fat finger. When it's a bunch of tiny issues now all my ops debugging has to start with going to the fly forums (and it's _always been issues on fly's side_). The price is "right" (though like with all PaaS the gaslighting about running multiple processes in one container makes me feel bad about the state of cloud computing). And I really like the CLI stuff mostly! But I extremely don't care about edge computing so for me fly is just heroku and I would love to feel more confident on that end. (EDIT: the nice thing is I get email support with a bit of cash. This is a thing that will go away when they get bigger but it's here while things are still breaking often)
daguava 4y ago
[dead]
debarshri 4y ago
This is a great blog and have so many insights for an SRE or devops person. But this goes to show you how difficult it is self host stuff at scale. I used to work for a company that built deployment platforms for law firms. All our deployments where on prem and we had the same complexity with kubernetes. We had similar setup with vault and stolon for HA PG. More moving parts you have in infra, more permutations and combinations of failure modes you have. What these guys are building is something I have seen in many orgs trying to do it internally and fail. PaaS is a hard problem if you want to solve it "reliabily"
anacrolix 4y ago
- Machines seem like a waste of time - Access directly to VMs is being removed (and doesn't support TCP over IPv4, or UDP over IPv6) - The CDN is nice but should support private networking too. - Volume management is deficient: It should be possible to access and fix volumes outside the context of an its app instance. - Egress traffic should be free between apps over private networking, at least in the same DC.
KETpXDDzR 4y ago
That sounds like a typical problem of unnecessary complexity to me. I wonder how many over engineered (web) applications could run as a single, efficient process on a single machine.
- TheDong 4y ago
  Do you know what fly.io does? Like, it cannot possibly run on a single machine by definition because the product they're building is running customer code on edge compute nodes distributed across the globe. The main selling point of their service is that it's _not_ a single machine.
throwawaaarrgh 4y ago
I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability. Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatened by competing things and literal bugs and weather and the soil it's grown in and more. It needs constant care. Over time it grows stronger, and can eventually fend for itself pretty well. Sometimes you get lucky and it just grows fine by itself. And sometimes 50 different things conspire to kill it. But you have to be there monitoring it, finding the problems, learning how to prevent them. Every garden is a little different. It doesn't matter what a company like Fly does technology wise. It takes time and care and churning. Eventually they will be reliable. But the initial process takes a while. And every new piece of tech they throw in is another plant in the garden. So the good news is, they can become really reliable. But the bad news is, it doesn't come fast, and the more new plants they put in the ground, the more concerns there are to address before the garden is self sustaining.
- mrkurt 4y ago
  This is an excellent description.
- IshKebab 4y ago
  Maybe so but there are definitely technology choices that have vastly different "initial reliability". What would you expect to be more reliable - a bash script or a Rust program? I'm not that familiar with setting up global network infrastructure but I imagine there are similar choices that can vastly affect initial reliability.
  - capableweb 4y ago
    I'd say it depends more on the person rather than the technology. A master in bash will build more reliable API (in bash no less!) than a beginner in Rust, simply because of experience and knowing their way around the tools they're using. Newer/different technologies won't simply solve a problem unless the person has some sort of domain knowledge of said problem.
benatkin 4y ago
It's too late. They've already misled people.
siliconc0w 4y ago
Respect the post, building your own infrastructure provider is playing on hard, the big players have had armies of engineers iterating on their stacks for a decade+
birracerveza 4y ago
What is this? A company being open and honest about problems their customers are facing? What is happening? Has the world gone mad??
nprateem 4y ago
Interesting take. As a non customer I now won't consider them for any projects as they've confirmed their unreliability.
vinay_ys 4y ago
Corrosion seems over-engineered. Instead of doing a simpler federation of multiple databases (one per datacenter) across the globe, they decided to do gossip amongst every single VM across the globe! You don't really gain much but you do get all the noisy complexity for sure.
clement_b 4y ago
I've been with fly.io at small scale and I have always loved their approach to content (docs, blogs, forums), and needless to say, their product. They are very talented and are building something truly great. Their openness, shown in this post, is an example to follow. It's very hard to be that honest and direct when you're meant to be an infallible entity, but it's not a surprise at all to see fly authoring such a post. That's how they operate and that's why I trust them over anyone else.
gpjanik 4y ago
I like Fly's response to the problems - honesty and openness - so I'm going to add to their problems and try to use it ;)
davedx 4y ago
@kurt happy (sometimes paying) Fly user here. Keep up the great work!
ecmascript 4y ago
Wow, posts like these kind of want me to sign up and become a customer. The honesty and openness is something I highly value and it's weird that this is so uncommon that you get surprised by reading stuff like this. Seems like they have a good understanding what the problems are so they will most likely be solved sooner or later. Good work and keep honesty as open as you've done so far :)
sidcool 4y ago
I have a debilitating impostor syndrome around resiliency and reliability of my systems. I always feel I am doing something wrong compared to the Googles, the Microsofts and the Fly.ios of the world. This does help feel better.
al_be_back 4y ago
in my view, a new edge computing entrant needs a niche market e.g low latency gaming or privacy-heavy computing, and to stay away from MAAG cloud territory.
Dave3of5 4y ago
I always thought it would be really interesting to work for a company like fly.io. Solving hard problems like this seems interesting. On the other hand it could be a giant shit show of micromanagement and toxicity, who knows really. At the moment they aren't hiring though so that's that.
tonnydourado 4y ago
Damn. I don't know, and I guess almost no one can know, how much of this is genuine honesty and how much is calculated messaging, but I barely care. It's refreshing enough that I kinda wanna give the thing a try. Which, ironically, might make their problems just a tiny little bit worse, since I likely won't be a paying customer XD
crabbone 4y ago
I'm not in the Web business, and have no idea what fly.io is offering, but whenever I hear anyone trashing Consul, I give them a standing ovation. An application which decided to use Base64-encoded JSON for its communication protocol deserves every bit of mockery it can get.
pwelch 4y ago
Growing pains are never fun. It doesn't mention (at least that I read) if they're using HashiCorp Open Source or Enterprise. Open Source is great and I owe my career to it but they might be hitting the scale when the Enterprise features and support start to be worth the price. I've only used Fly.io for a personal app but I think it's a great option so I hope they keep growing.
- pwelch 4y ago
  Ah my question was answered more or less. They're an edge case which makes sense: https://news.ycombinator.com/item?id=35048318 https://news.ycombinator.com/item?id=35048318

1226 points by bishopsmother 4y ago

One of my colleagues keeps repeating “reliability is our number one feature”. I’m not sure it is for 100% of early stage startups, but I guess it is once you exceed some minimum usage threshold. That said, definitely appreciate the detailed explanation.

jpdb 4y ago
> One of my colleagues keeps repeating “reliability is our number one feature”. I think reliability is the #1 feature at any stage because if you're unavailable, you're at best useless and more than likely you are actively harmful because your users have an expectation. However, if you're unavailable outside of times customers don't expect you to be there then you're not actually unavailable. This is more likely for an early stage start-up, but you don't typically choose or know when you're expected to be available nor do you always get to choose when you're unavailable.
- sa46 4y ago
  In terms of confidentiality, availability, and integrity: I'll bet LastPass would gladly trade availability right now to regain confidentiality.
- ignoramous 4y ago
  Our team at AWS had a poster up on the wall that more or less went: 1. Security 2. Durability 3. Availability 4. Speed Similar: https://twitter.com/colmmacc/status/1071088017190711296 https://twitter.com/colmmacc/status/1071088017190711296
jrochkind1 4y ago
> One of my colleagues keeps repeating “reliability is our number one feature”. > I’m not sure it is for 100% of early stage startups, I mean, it probably depends on the nature of the startup? Platform-as-service seems particularly sensitive to reliability (whether or not it's "#1 feature"), in a way that might not be true of startups in other spaces.
cschep 4y ago
TL;DR -- It's very domain specific if reliability is your number one feature. For a startup that is hosting other people's production application/data then this is absolutely true. Less than 100% always needs to be addressed. For a startup that is selling bingo cards then reliability probably isn't nearly as important. I'm guessing there were certain holidays that were more important than others as far as reliability goes though? Maybe patio11 can chime in :)
nprateem 4y ago
That's what crossing the chasm is about. Before the chasm are tinkerers who don't mind things failing as much, but after it, people want reliable things that Just Work.

soperj 4y ago

For django, they should really contribute to 2 scoops django cookie cutter program, so that you can get an out of the box django instance that can just deploy to Fly.io.

mixmastamyk 4y ago
May be wrong, but they seem to be very focused on interesting stacks like Elixir and RoR while building on Go/Rust. The corollary being neglect of the bread/butter stacks with high market share, like Python and Java/JavaScript. Don't think I've seen a blog post discussing those three beyond a passing mention? Not the end of the world, but mildly disappointing. At least they are all in with Postgres and Linux, a great foundation.
- jjtheblunt 4y ago
  I thought the only thing their tech stack was all in on was docker images being rewritten to run on firecracker as a substrate. is it not agnostic about things like Elixir etc, at the tech level, though they've got super nice documentation for those tools you mentioned?
  - ilaksh 4y ago
    What's necessary to change for them to run on Firecracker?
    - ign0ramus 4y ago
      Here's a post explaining how they do it https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
      - ilaksh 4y ago
        Yeah, saw that before. I thought he was saying you have to change your Dockerfiles to use them with fly.io. Just misinterpreted that sentence.

deleted 4y ago

[deleted]

iamdbtoo 4y ago

I'm a big fan of fly.io. From their hiring process to the product itself it's all carried out in a thoughtful manner. I hope they can weather this rough time.

yamrzou 4y ago

I'm not a user of Fly.io. I can't help but notice how remarkable the effect of open communication on potential end users like me. I remember reading about their reliability problems on HN some time ago. That biased my view of the company. After reading this, the open communication and transparency restored my trust in them, and would make them again a potential candidate for future projects. Because now I know that they acknowledge the problem and that they are trying to improve things.

leetrout 4y ago
mrkurt[1] is also active here and has been very transparent in his comments about scaling issues. Similar to this post he commented a week ago: > In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former. I am rooting for Fly! Great team. The company reminds me of early HashiCorp. [1] https://news.ycombinator.com/user?id=mrkurt https://news.ycombinator.com/user?id=mrkurt
mbesto 4y ago
Me too. However, this is a double edge sword. Their key value proposition is scale / speed which makes it concerning that they haven't "solved" that yet.
willio58 4y ago
Agreed, this is how company communication should be. I don't use Fly but would consider them in the future even given their recent issues. I look at this in contrast to Twitter who had/has? an outage today. Their leadership is opaque and doesn't take responsibility for the issues they are causing.
- alfalfasprout 4y ago
  In fairness, a CEO who has basically been Kanye-ing himself and his company into irrelevance is a low bar.
gizmo 4y ago
This post is carefully worded corporate messaging, but because they write for their developer audience it has an informal "oh shucks we messed up bad y'all" vibe to it. But make no mistake, this is 100% corporate messaging. I get that growing is super hard. And maybe fly will grow up to be a good platform some day. But that's the future. Today, they're flying by the seat of their pants and I mostly feel sorry for people who were tricked into thinking this platform is ready for production use.

deleted 4y ago

[deleted]

tiffanyh 4y ago

Would the simple solve be that Fly.io just mark any new service of theirs “beta” for x-months post launch?

samwillis 4y ago
None of the services that they have had issues with in the recent past are new, they have been running for at least a couple of years. They would need to put a "beta" sticker on the whole platform for that suggestion to work. But the post makes it clear that the issue isn't that they had problems with new services. It was rapid customer growth before they had time to scale up the infrastructure as they had planned to do.

sergiotapia 4y ago

It's been almost a year since I gave Fly a review (https://news.ycombinator.com/item?id=31391116 https://news.ycombinator.com/item?id=31391116) and it's a bummer that they're still struggling to get things right. Double bummer because I love Phoenix and Elixir and they employ Chris McCord there. Maybe they were _too_ ambitious at the start? They have a hard road ahead of them, and competition like Render.com and Northflank have provided me with solutions to all of my problems. Great dev ux, great prices and predictable solutions. They also keep pushing out very useful features. A third competitor also sprung up Railway! There's certainly blood in the water. Will they catch up to others before the competition solves the "global mesh" unique value proposition Fly.io currently has? That's the $1MM question.

zomglings 4y ago
I read your review, and had a question so I thought I'd follow up here. You mentioned render.com as a competitor - does render host its own infrastructure or do they act as a go-between between their users and AWS/GCP/whatever?
- cldellow 4y ago
  They act as a go-between in that they ultimately host on AWS/GCP. They host their own infrastructure in that they appear to run Kubernetes and have built out their own deployment and service fabric, so they're just using the underlying machines as dumb compute, they're not, eg, building on RDS. In March 2021, someone asked a question about carbon emissions of their data centres. They said they hosted on both GCP and AWS, but mentioned they were interested in moving to their own bare metal [1]. In April 2021, I asked a question about egress fees to Google, and they walked back a bit the comment about moving to bare metal [2]. As of March 2022, they're still in AWS/GCP [3]. As of September 2022, workloads for new users deploy into AWS, even in regions that were previously served by GCP [4]. [1]: https://community.render.com/t/does-render-use-green-energy/912/6 https://community.render.com/t/does-render-use-green-energy/... [2]: https://community.render.com/t/is-render-com-hosted-in-google-cloud/1187 https://community.render.com/t/is-render-com-hosted-in-googl... [3]: https://community.render.com/t/are-your-servers-owned-by-you/3719 https://community.render.com/t/are-your-servers-owned-by-you... [4]: https://community.render.com/t/which-render-regions-map-to-which-cloud-providers/5987/3 https://community.render.com/t/which-render-regions-map-to-w...

claytonjy 4y ago

Very interesting to see Kurt assert theyre going to "solve managed Postgres", and I'm super curious to know what that means. Does it mean something like RDS, or more like CrunchyData? I could see them building something RDS-like on their own, but if they're trying to go further than that I wonder if they'll buy or partner with other companies rather than doing it themselves. Neon strikes me as a Postgres-as-a-service that could pair well with Fly.

mmcclure 4y ago
That comment jumped out to me too, my recollection was that they've been pretty vocal about that not being something they wanted to solve themselves as a core competency. I'm not quite parsing if these two blurbs should still be taken together or if the second sentence is refuting the first. > The second problem we have with Postgres was a poor choice on my part. We decided to ship “unmanaged Postgres” to buy ourselves time to wait for the right managed Postgres provider to show up. > We’re going to solve managed Postgres. It’s going to take a while to get there, but it’s a core component of the infrastructure stack and we can’t afford to pretend otherwise. +1 to Neon seeming like a good fit, but it's also very much a beta (alpha?) both as a product and company (at least from my impression). I'm not sure that's a bet they'd want to make right now given the context of this post.
- nikita 4y ago
  (Neon CEO) We are launching our paid tier March 15th and will be production ready shortly after. We are running 20K+ databases and measuring reliability and uptime. Generally reliability is a function of architecture (we are solid there), good SRE practices, and a long tail of event you live through, fix, and make sure they never happen again. The bigger the fleet the faster the learning.
  - mmcclure 4y ago
    That's great, love to hear it! Really excited about what you all are working on.
  - 4y ago

chucky_z 4y ago

mrkurt have you considered some of the lower tiers of vault enterprise that allow for performance replicas that just outright solve that problem? might be cheaper than an engineer at this point.

nu11ptr 4y ago

Not a client of fly.io, but dang impressive for the company to be this open and honest. Definite respect - wish more companies were like this. It puts them on my short list almost immediately for future needs.

nathell 4y ago
Came here to say pretty much this. Technical issues come and go; open communication is a core part of the company culture, and builds up trust.
deleted 4y ago
[deleted]
nprateem 4y ago
Why would you shortlist a company that admits to reliability issues?
- macintux 4y ago
  All companies have reliability issues, especially in this problem space. I’d rather go with someone who acknowledges their failings in public.
- jen20 4y ago
  Because if they admit to them they’re probably in the process of fixing them. When compared to a certain B tier cloud who gaslight users about flaws in the product and API, the amount of effort expended to get to an admission of fault is much lower for companies not inclined to lie by omission.

samwillis 4y ago

Fundamentally I think some of the problems come down to the difference between what Fly set out to build and what the market currently want. Fly (to my understanding) at its core is about edge compute. That is where they started and what the team are most excited about developing. It's a brilliant idea, they have the skills and expertise. They are going to be successful at it. However, at the same time the market is looking for a successor to Heroku. A zero dev ops PAAS with instant deployment, dirt simple managed Postgres, generous free level of service, lower cost as you scale, and a few regions around the world. That isn't what Fly set out to do... exactly, but is sort of the market they find themselves in when Heroku then basically told its low value customers to go away. It's that slight miss alignment of strategy and market fit that results in maybe decisions being made that benefit the original vision, but not necessarily the immediate influx of customers. I don't envy the stress the Fly team are under, but what an exciting set of problems they are trying to solve, I do envy that!

vineyardmike 4y ago
I agree - fly is so easy to use (when it works) that it’s hard not to be impressed. BUT what I’ve found is that we don’t need edge compute, since our customers aren’t that latency sensitive, so it’s lost on us. It’s only a few more milliseconds to us-east-1. I’ve heard (on HN) of a dozen different companies vying for the heroku replacement spots and yet Fly seemed to capture the attention. I couldn’t name another one off hand. What I truly want and probably lots of other people too is Flyctl (and workflow) for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles.
- latchkey 4y ago
  > What I truly want and probably lots of other people too is Flyctl for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles. Google Cloud. It is painfully easy to spin up managed postgres, super easy to deploy gcp cloud functions or gcp cloud run. It isn't expensive either and just works.
  - 4y ago

outworlder 4y ago

> This is a theme. Existing open source is not designed for global deployment Eh? Unless you are consuming something as a service and it actually advertises it as a feature, nothing is ready for 'global deployment'. If you have a 'centralized' secret storage, then you have made it tied to a region. Want to have redundancies and lower latency? You'll have to distribute it. Vault has docs about this: https://developer.hashicorp.com/vault/tutorials/day-one-raft/multi-cluster-architecture https://developer.hashicorp.com/vault/tutorials/day-one-raft...

josegonzalez 4y ago
This one is interesting because Vault has an enterprise product which I assume (hope?) Fly is paying for. That enterprise version includes performance replicas, which allows for cross-region replication of secrets with region-local reads (and slightly lower writes). The OP almost makes it sound like they are using the non-enterprise versions (or at the very least, not taking advantage of this particular functionality). That said, I'd imagine with large enough scale, these sorts of features break anyways.

theloco 4y ago

I love reading stuff like this. I don't use fly, don't plan to, not totally sure everything it does and will check it out. But this is some great raw data on how stressful it is after you launch.

tebbers 4y ago

I really feel for Fly, as a potential customer. They are trying really hard. I would still love to use them one day and this post is definitely a step in the right direction. Growing is painful but they have smart people working there so fingers crossed that they sort this ASAP and it doesn't become existential.

jrochkind1 4y ago

I remain kind of amazed about how heroku managed to pull off what they pulled off, in the first case. Also: > The Heroku exodus broke our assumptions. Pre-Heroku, most of the apps we were running were spread across regions. And: we were growing about 15% per month. But post-Heroku, we got a huge influx of apps in just a few hot spots — and at 30% per month. I hadn't before seen anyone with a big picture view confirm a heroku exodus was happening, although a lot of people suspected it or had anecdotes. But if fly is seeing a pretty enormous number of customers moving from heroku to fly... oh wait, now I'm wondering, is this mainly a result of heroku ending free services, and those are free customers coming to fly for free services? If so... that's a pretty big burden to take on without revenue to match, it does seem kind of dangerous for fly.

mike_d 4y ago
> I remain kind of amazed about how heroku managed to pull off what they pulled off Heroku was built on top of AWS. They didn't have to handle many of the hard problems Fly does.
8organicbits 4y ago
Do they unpack what the 30% includes? Is that revenue, VMs, accounts?

lopatin 4y ago

It sounds like they need more money to scale the shared stack

ec109685 4y ago

I wonder what types of RPS they are seeing that required a gossip based protocol to broadcast state around versus a more traditional data store. I take it that it’s far more important that the local region know about changes than a remote region, which makes a mastered store in one location as the source of truth problematic. I also wonder why these companies don’t backstop themselves on the public cloud? Failing into an AWS seems better than running out of capacity and some its services could be used in circumstances where an open source technology isn’t ready.

likecarter 4y ago
Yeah, reading the post made it seem like they followed “best practices” without really thinking things through. KISS.

e1g 4y ago

This reads like a mea culpa from an indie hacker, but Fly.io had 5+ years and raised $40M to get these basic fundamentals right. And we get promises of a new status page.

ignoramous 4y ago
Well, that's one way to look at it. Fly's been many things over the course of its lifetime [0], but I believe their latest pivot (on what they call "Machines") is pretty darn good. I've been using Machines since Oct last year, and things have gotten better week-over-week. Like with any platform, Fly has its own idiosyncrasies, which don't take much to get hang of. That said, I am the only person in my tech shop that deals with Fly. Some orgs with larger teams and heavier apps that deploy frequently or run DBs / disks on Fly (I don't) have had a rough few months; so that's there too. [0] Ex A: https://news.ycombinator.com/item?id=13985940 https://news.ycombinator.com/item?id=13985940
- mbStavola 4y ago
  Fly Machines, if I understand them correctly, feels like a step backwards. Sure they might work better than "standard" Fly apps, but one of the motivating cases for Fly is being able to effortlessly scale across the world without having a Ph.D. in CS and a fistful of certs for Cloud engineering. That vision for Fly is awesome, game changing even, ignoring their current stability issues. Machines isn't that. From the documentation, it appears as though it's "just" a VM pinned to a single region and none of the "magic" of Fly really applies. If the server your VM is hosted on goes down, Fly won't redeploy your container. It's just downtime. Spinning up in other regions is something you have to think about and actually do. It seems closer to Heroku than it does Fly. Maybe I am totally misunderstanding Fly Machines and their use-case, maybe they're aiming to close the gap between Machines and Fly apps. It's just a bit of a bummer to see something that looks like walking back the original "promise" of Fly and makes me question whether or not Fly is going to just become like every other PaaS (even if it's a really good one).
  - ignoramous 4y ago

pier25 4y ago

I've been using Fly for over two years or so. The sentiment of this post doesn't align with my personal (anecdotal) experience. The PG issues hit me two times in the previous weeks but other than that it's been working great for me. With the move to v2 apps (using their new machines infra) things are actually faster and smoother than ever. About a year or so ago their CLI was quite buggy but I haven't really hit any bugs in months. I will remain with Fly for the time being. Hopefully they don't close shop!

tptacek 4y ago
We're nowhere even within the line of sight of closing up shop. We just haven't been doing a good job of aggressively communicating (a) when things go wrong and (b) what we're doing to account for it. The Fly.io of 2023 looks almost nothing like that of 2021 (all for the better), and it's not obvious to our users what's changed. We've been doing a shitty job of communicating, and we're taking our licks for it now.
- chris_st 4y ago
  A lot can happen in 11 years :-) And thanks a lot for fly.io -- it's working great for my (rather small) use cases.
  - mrkurt 4y ago
    Oh my god that's a great callback.
russellthehippo 4y ago
Agree - V2 apps on machines are incredibly slick to launch (create/start/stop), get info on with graphql, and scale up and out. Magic. When the PG administration experience is that good I'll move it all over.

victorbjorklund 4y ago

Great post. Love how transparent Fly is. Im a customer but without any life and death important apps. And yea they had some issues lately so good they are addressing it.

deleted 4y ago

[deleted]

lll-o-lll 4y ago

At first I was all like “Ha ha, losers can’t scale” And then I was “Huh, these technical challenges are actually pretty difficult” And then I was all “crap, these are a bunch of technologies I was about to add to our stack” Thanks heaps fly.io people; having the humility to honestly talk about the challenges and failures massively helps people such as myself as we navigate new unfamiliar technologies. If more companies were willing to do this, it’d be a lot easier to avoid common pitfalls.

chucky_z 4y ago
The tech in their stack is still pretty good. Unless you’re supporting tens of thousands of customers and trying to make the promises that fly makes today. Look at the fly engineer replies in this thread. Also they basically only use OSS versions, they could go give Hashicorp some money to solve their Vault problems. They could probably partner with SecondQuadrant for PG as two examples. That might not make sense for their business though. Hard problems are hard no matter the choice.
- lll-o-lll 4y ago
  Sure, I was going for a little humour there. A little riff on the whole “we always judge others until we walk in their shoes”. The take away I was hoping for is “providing insights into how we struggle helps others”

deleted 4y ago

[deleted]

ChrisMarshallNY 4y ago

Well, I feel for them. Scaling up is a bitch. I've been lucky, in the past, but a lot of that, is because I have "overengineered," and the tools/frameworks have advanced to meet the new demand. I am in the middle of a complete, bottom-to-top rewrite of the app we've been developing for the last couple of years. It's going great, but making this leap was a fraught decision. It's mainly, so I wouldn't have to write a post like that, in a year or two. We spent all the time refining it, until we had what we wanted, and it worked great on our small test team. Then, I loaded up a test server with 10,000 fake users, and tossed the app at that. To be fair, we don't think we'll have even that many users for quite a while. It's a very specialized demographic. * SOB * It no do so well. At that point, I had to decide whether to fix the issues (they were quite fixable), or revisit the architecture. The main issue with the architecture, was that it was an "accreted" app, with changes gradually being factored in, as we progressed. The main reason for this, is because no one really knew what they wanted, until we ran it up the flagpole (sound familiar?). The business logic was distributed throughout the app. That was ... ugly. I envisioned myself, a year or two down the road, sucking on a magnum, because the app had turned into a Cruftosaurus, and was coming for me in my nightmares. So I decided to rewrite, as we hadn't done any kind of MVP or public beta, so we actually had the runway to do this. I refined the entire business logic of the app into a single, platform-agnostic SPM module, which took just over a month, and have started to develop the app around that. It's pretty much a rewrite, but I am recycling a lot of the app code. We also brought in our designer, and he's looking at every screen I make. It's working well for him. Like I said, it's going great. Better than I expected. I know that I have a huge luxury, and I'm grateful. I can credit a lot of that, to doing some stress-testing before we got to a point where we had a bunch of users to support. I was able to go in, and go all Victor Frankenstein on the model. The result, so far, is that this thing screams, and you don't really even notice that there's that many users on it. The model has already been proven (that SPM module), and all we're doing, is chrome (which is a ton of work).

bbkane 4y ago
Hope this isn't a dumb question, but what's an "SPM module"?
- 4y ago

skywhopper 4y ago

Interesting issues. Nothing surprising for anyone who’s run a global SaaS before, especially if growth has been incredibly fast. I find the gripes about Consul, Nomad, and Vault interesting since it sounds like the problems are mainly due to poor architectural decisions. Fly is rewriting those tools rather than invest in deploying them properly and in the process are running into new issues that those tools have already solved, which doesn’t give me confidence that the path forward will be any less bumpy.

ashiban 4y ago

One of the key challenges we observe is that if you're small enough, a Heroku like experience works well - and most of your needs would be covered by virtually any combination of techstacks. It gets significantly more challenging when you grow, either in feature complexity or scale complexity - and then very few services can offer what AWS/GCP/Azure offer - albeit at the increased engineering/monetary cost of using them. We're building a different kind of approach[0] that aims to absorb the mechanical cost of using public cloud capabilities (that are proven to scale) without hiding it altogether. [0] https://github.com/KlothoPlatform/klotho https://github.com/KlothoPlatform/klotho

a3w 4y ago

[flagged]

pyentropy 4y ago

Almost half of the issues are caused by their use of HashiCorp products. As someone that has started tons of Consul clusters, analyzed tons of Terraform states, developed providers and wrote a HCL parser, I must say this: HashiCorp built a brand of consistent design & docs, security, strict configuration, distributed-algos-made-approachable... but at its core, it's a very fragile ecosystem. The only benefit of HashiCorp headaches is that you will quickly learn Golang while reading some obscure github.com/hashicorp/blah/blah/file.go :)

tptacek 4y ago
We are asking to HashiCorp products to do things they were not designed to do, in configurations that they don't expect to be deployed in. Take a step back, and the idea of a single global namespace bound up with Raft consistency for a fleet deployed in dozens of regions, providing near-real-time state propagation, is just not at all reasonable. Our state propagation needs are much closer to those of a routing protocol than a distributed key-value database. I have only positive things to say about every HashiCorp product I've worked with since I got here.
- pyentropy 4y ago
  I respect that. Can you elaborate a bit on the routing protocol thing? I assume you used WAN gossip? I love the simplicity of fly.io & wish you all the best improving Fly's reliability!
  - tptacek 4y ago
    If you've ever implemented IS-IS or OSPF before, like 80% of the work is "LSP flooding", which is just the process that gets updates about available links from one end of the network to another as fast as possible without drowning the links themselves in update messages. Flooding algorithms don't build consensus, unlike Raft quorums, which intrinsically have a centralized set of authorities that keep a single source of truth for all the valid updates. An OSPF router uses those updates to do build a forwarding table with a single-point shortest path first routine, but there's nothing to say that you couldn't instead use the same notion of publishing weighted advertisements of connectivity to, for instance, build a table to map incoming HTTP requests to backends that can field them. The point is, if you're going to do distributed consensus, you've got a dilemma: either you're going to have the Ents moot in a single forest, close together, and round trip updates from across the globe in and out of that forest (painfully slow to get things in and out of the cluster), or you're going to try to have them moot long distance (painfully slow to have the cluster converge). The other thing you can do, though, is just sidestep this: we really don't have the Raft problem at all, in that different hosts on our network do not disagree with each other about whether they're running particular apps; if worker-sfu-ord-1934 says it's running an instance of app-4839, I pretty much don't give a shit if worker-sfu-maa-382a says otherwise; I can just take ORD's word for it. That's the intuition behind why you'd want to do something like SWIM update propagation rather than Raft for a global state propagation scheme. But if you're just doing service discovery for a well-bounded set of applications (like you would be if you were running engineering for a single large company and their internal apps), Raft gives you some handy tools you might reasonably take advantage of --- a key-value store, for instance. You're mostly in a single data center anyways, so you don't have the long-distance-Entmoot problem. And HashiCorp's tools will federate out across multiple data centers; the constraints you inherit by doing that federation mostly don't matter for a single company's engineering, but they're extremely painful if you're servicing an unbounded set of customer applications and providing each of them a single global picture of their deployments. Or we're just holding it wrong. Also a possibility.

bmorton 4y ago

[flagged]

sph 4y ago
It's a post on their forum, hardly an ad.

deivid 4y ago

I'm a bit sour reading this. I've always liked fly and particularly the engineering blog, so much so that a couple of months ago I decided to apply for an infra position, to work on some of these very topics. Sadly after 4~5 rounds of interviews (including a workday) they just ghosted me.

deleted 4y ago
[deleted]
tptacek 4y ago
If that happened, it absolutely was not on purpose. Shoot me an email at thomas@fly.io.
ignoramous 4y ago
> Don't feel too bad nor take it personal. They probably have a lot of applicants, and are looking to grow their team by hiring someone with very specific skills. > I also applied a few months ago while I was in the middle of my job search. For one, I couldn’t really answer their "favorite syscall question" because I’ve never dealt with syscalls :) so maybe I just wasn't a good fit. Surely, everyone's favourite syscall is exit()
- jlokier 4y ago
  For what it's worth, exit() is not a syscall, but _exit() often is.

revskill 4y ago

Fly.io seems like Vercel 1.0 (where you can just deploy docker image and done), but it's more than that, with configurable volumes, secrets,... I'm bullish on fly.io.

1023bytes 4y ago

I get it, I like fly.io, but the last outage made me switch to Railway.app

nathants 4y ago

i also wanted a good cli for aws, and built one: https://github.com/nathants/libaws https://github.com/nathants/libaws companies like fly are fantastic. they provide a good service, and they put market pressure on aws. a free tier isn’t important anymore. with usage based pricing for lambda/dynamo/s3, an app with usage approaching zero has no cost.

plasma 4y ago

Thanks for sharing! Would it help to replace Corrosion with a simpler "Here's my local known state" blob that is POST'd to blob storage (for example) on a major cloud provider, and have another service read that at intervals? Just to make it really simple. There will be a better way than that, but my thought is if you can make it simpler (known state is always just pushed, so missing updates auto-recovers and avoids corruption) then you can be building on top of a more stable service discovery system. Centralized secret storage, can you keep the US instance read/write, but replicate read-only copies (a side-car tool that copies the database to other regions at various intervals?) so each region can fetch secrets locally? Or perhaps both can be solved with a general "Copy local state to other regions" service that is pretty simple but gives each region its own copy of other region's information (secrets, provisioning states, ...). I've needed to do similar things for some of the apps I've built, where a service needed another (simpler) service in front of it to bear the traffic load but was operationally simple (deferred the smarts to the system it was using as the source of truth) and automatically recovered from failure due to its simplicity.

preseinger 4y ago
+1 — service discovery _feels_ like something that gossip can solve, but that's only true at global/open/untrusted scale, in a well-scoped and authoritative domain like a company it's really not, it's really just another typical, expected-to-be-consistent state problem

djha-skin 4y ago

> In response, we’ve shipped a project called Corrosion. Corrosion is a gossip > based service discovery system. I wonder why they didn't try to use Serf[1] for this, since they were so into HashiCorp tools. It also uses the gossip protocol. 1: https://www.serf.io/docs/index.html https://www.serf.io/docs/index.html

swamp40 4y ago

"CEO finds novel way to fix Capacity problems..." They just lost about 40% of their paying users with that blog post.

nomilk 4y ago

What an earnest post, and how damn refreshing it is to see such concern for users, accountability, honesty and openness (quite a contrast to another PaaS)! I moved one app successfully from heroku to fly and attempted to move a few others. These are my experiences (both good and bad): Great: - The load time on the pages is insanely faster on fly than heroku. Sometimes I thought I was on the localhost version of the app, it was that snappy. - Love that it uses a Dockerfile - Love paying for what I use (compared to Heroku's rigid minimum of $16/month for hobby dyno w/ postgres for baby apps, or $34/month just to get a second web dyno for toddler apps). The same apps are <$5/month each on fly. Not great: - I find the fly.toml file hard to understand and use, and the cycle time slow to fix or tinker with it. It's partly (entirely?) a 'me' problem because I haven't spent a huge amount of time reading the documentation. - I found scheduling a rake task in a rails app time consuming (~days) the first time, but very easy (15 minutes) the second and subsequent times, once I knew a way that worked (cron didn't work; had to use a tool I hadn't used before 'supercronic'). - Deploys sometimes time out with `Error failed to fetch an image or build from source: error rendering push status stream: EOF`. Most layers copied, but randomly, some layers wouldn't. All I could do is keep trying until it worked, which it did, 2 hours later. Not the end of the world, but an annoying complication when you're already trying to solve complex problems. - I followed a youtube video on how to move a rails app from heroku to fly, and it worked on a modern app, but I couldn't quite get fly happy when moving the older app - something to do with postgres versions, and I didn't want to spend all day figuring it out. I'm not hugely experienced with docker, it could have been an easy fix for someone more experienced. On reflection, 3 of the 4 negatives above are solvable by me reading the docs more thoroughly and getting more proficient with docker. I look forward to continuing using and exploring fly, and can't be happier with the directness, transparency and care from fly staff. A platform with huge potential.

bradgessler 4y ago
Did you try migrating with this guide? https://fly.io/docs/rails/getting-started/migrate-from-heroku/ https://fly.io/docs/rails/getting-started/migrate-from-herok... The issues you ran into with older versions of Rails was probably because the Dockerfile that `fly launch` generated was for new versions of Rails. We switched to https://github.com/rubys/dockerfile-rails https://github.com/rubys/dockerfile-rails to streamline Dockerfile generation and support older versions of Rails. If you try it again and run into issues you can open an issue at https://github.com/rubys/dockerfile-rails/issues https://github.com/rubys/dockerfile-rails/issues or post in https://community.fly.io https://community.fly.io and somebody will help get that sorted out. The more versions of Rails we can deploy the better!

andy_ppp 4y ago

How easy is it to deploy things with GitHub actions, AWS and terraform instead? I think I’d like full infrastructure control after having accidentally got a job doing DevOps which I’m starting to get quite into. I should probably write up a blogpost about converting everything over…

none_to_remain 4y ago

Three little pigs, but they all live in a house of straw built on top of several houses of sticks. (The foundation is an old house of brick, but the wolf isn't bothered by that.)

Karupan 4y ago

Thanks for the honest technical write up - not easy to air one's dirty laundry to users. Given the scalability and stability issues, I am curious to understand the percent of apps deployed to fly are actually used in production/critical to business. Sounds like they have quite a few hobby/free tier users (myself included) who probably won't notice certain issues unlike paid customers.

hinkley 4y ago

> We’ve put a lot of work and resources into growing the platform and maturing our engineering organization. But that work has lagged growth. I fundamentally don't understand why people are in such a big hurry to get 'famous'. I've worked a couple of places where the marketing side was working as hard as they could to make sure that our heads were on fire at all possible moments. At one job I had a (very, very junior) manager come up to me and say great news we landed <big customer> and my immediate reply was, "fuck me". We were already running to stay upright and now we're about to have twice as much scrutiny. Wonderful. If you push hard enough, eventually everyone looks like an idiot. The number of humans for whom that is not true could fit into a book. Both alive and deceased. They most definitely do not work for the companies I've described, at least not enough of them so you'd notice.

computomatic 4y ago

Where does fly.io document their per-account services limits? For example, max apps, databases, etc. I took a quick look and couldn't find them. Do they have any documented service limits? A google search turned up [0] which does not inspire optimism. > ...there isn’t a limit to number of apps from a billing standpoint... [0] https://community.fly.io/t/free-tier-limits-and-quota-needs-clarification/8755 https://community.fly.io/t/free-tier-limits-and-quota-needs-...

rtpg 4y ago

I have two extremely tiny sites (like, "handful of users/1 user" sites) on Fly. I have had multiple incidents despite me not even touching them. The thing that worries me about these incidents is they haven't been, like, full service outages. A small subset of users talking about issues in forums. This makes me just feel like Fly has an immense amount of issues. At least if like 50% of fly goes down then it feels like a config fat finger. When it's a bunch of tiny issues now all my ops debugging has to start with going to the fly forums (and it's _always been issues on fly's side_). The price is "right" (though like with all PaaS the gaslighting about running multiple processes in one container makes me feel bad about the state of cloud computing). And I really like the CLI stuff mostly! But I extremely don't care about edge computing so for me fly is just heroku and I would love to feel more confident on that end. (EDIT: the nice thing is I get email support with a bit of cash. This is a thing that will go away when they get bigger but it's here while things are still breaking often)

daguava 4y ago

[dead]

debarshri 4y ago

This is a great blog and have so many insights for an SRE or devops person. But this goes to show you how difficult it is self host stuff at scale. I used to work for a company that built deployment platforms for law firms. All our deployments where on prem and we had the same complexity with kubernetes. We had similar setup with vault and stolon for HA PG. More moving parts you have in infra, more permutations and combinations of failure modes you have. What these guys are building is something I have seen in many orgs trying to do it internally and fail. PaaS is a hard problem if you want to solve it "reliabily"

anacrolix 4y ago

- Machines seem like a waste of time - Access directly to VMs is being removed (and doesn't support TCP over IPv4, or UDP over IPv6) - The CDN is nice but should support private networking too. - Volume management is deficient: It should be possible to access and fix volumes outside the context of an its app instance. - Egress traffic should be free between apps over private networking, at least in the same DC.

KETpXDDzR 4y ago

That sounds like a typical problem of unnecessary complexity to me. I wonder how many over engineered (web) applications could run as a single, efficient process on a single machine.

TheDong 4y ago
Do you know what fly.io does? Like, it cannot possibly run on a single machine by definition because the product they're building is running customer code on edge compute nodes distributed across the globe. The main selling point of their service is that it's _not_ a single machine.

throwawaaarrgh 4y ago

I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability. Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatened by competing things and literal bugs and weather and the soil it's grown in and more. It needs constant care. Over time it grows stronger, and can eventually fend for itself pretty well. Sometimes you get lucky and it just grows fine by itself. And sometimes 50 different things conspire to kill it. But you have to be there monitoring it, finding the problems, learning how to prevent them. Every garden is a little different. It doesn't matter what a company like Fly does technology wise. It takes time and care and churning. Eventually they will be reliable. But the initial process takes a while. And every new piece of tech they throw in is another plant in the garden. So the good news is, they can become really reliable. But the bad news is, it doesn't come fast, and the more new plants they put in the ground, the more concerns there are to address before the garden is self sustaining.

mrkurt 4y ago
This is an excellent description.
IshKebab 4y ago
Maybe so but there are definitely technology choices that have vastly different "initial reliability". What would you expect to be more reliable - a bash script or a Rust program? I'm not that familiar with setting up global network infrastructure but I imagine there are similar choices that can vastly affect initial reliability.
- capableweb 4y ago
  I'd say it depends more on the person rather than the technology. A master in bash will build more reliable API (in bash no less!) than a beginner in Rust, simply because of experience and knowing their way around the tools they're using. Newer/different technologies won't simply solve a problem unless the person has some sort of domain knowledge of said problem.

benatkin 4y ago

It's too late. They've already misled people.

siliconc0w 4y ago

Respect the post, building your own infrastructure provider is playing on hard, the big players have had armies of engineers iterating on their stacks for a decade+

birracerveza 4y ago

What is this? A company being open and honest about problems their customers are facing? What is happening? Has the world gone mad??

nprateem 4y ago

Interesting take. As a non customer I now won't consider them for any projects as they've confirmed their unreliability.

vinay_ys 4y ago

Corrosion seems over-engineered. Instead of doing a simpler federation of multiple databases (one per datacenter) across the globe, they decided to do gossip amongst every single VM across the globe! You don't really gain much but you do get all the noisy complexity for sure.

clement_b 4y ago

I've been with fly.io at small scale and I have always loved their approach to content (docs, blogs, forums), and needless to say, their product. They are very talented and are building something truly great. Their openness, shown in this post, is an example to follow. It's very hard to be that honest and direct when you're meant to be an infallible entity, but it's not a surprise at all to see fly authoring such a post. That's how they operate and that's why I trust them over anyone else.

gpjanik 4y ago

I like Fly's response to the problems - honesty and openness - so I'm going to add to their problems and try to use it ;)

davedx 4y ago

@kurt happy (sometimes paying) Fly user here. Keep up the great work!

ecmascript 4y ago

Wow, posts like these kind of want me to sign up and become a customer. The honesty and openness is something I highly value and it's weird that this is so uncommon that you get surprised by reading stuff like this. Seems like they have a good understanding what the problems are so they will most likely be solved sooner or later. Good work and keep honesty as open as you've done so far :)

sidcool 4y ago

I have a debilitating impostor syndrome around resiliency and reliability of my systems. I always feel I am doing something wrong compared to the Googles, the Microsofts and the Fly.ios of the world. This does help feel better.

al_be_back 4y ago

in my view, a new edge computing entrant needs a niche market e.g low latency gaming or privacy-heavy computing, and to stay away from MAAG cloud territory.

Dave3of5 4y ago

I always thought it would be really interesting to work for a company like fly.io. Solving hard problems like this seems interesting. On the other hand it could be a giant shit show of micromanagement and toxicity, who knows really. At the moment they aren't hiring though so that's that.

tonnydourado 4y ago

Damn. I don't know, and I guess almost no one can know, how much of this is genuine honesty and how much is calculated messaging, but I barely care. It's refreshing enough that I kinda wanna give the thing a try. Which, ironically, might make their problems just a tiny little bit worse, since I likely won't be a paying customer XD

crabbone 4y ago

I'm not in the Web business, and have no idea what fly.io is offering, but whenever I hear anyone trashing Consul, I give them a standing ovation. An application which decided to use Base64-encoded JSON for its communication protocol deserves every bit of mockery it can get.

pwelch 4y ago

Growing pains are never fun. It doesn't mention (at least that I read) if they're using HashiCorp Open Source or Enterprise. Open Source is great and I owe my career to it but they might be hitting the scale when the Enterprise features and support start to be worth the price. I've only used Fly.io for a personal app but I think it's a great option so I hope they keep growing.

pwelch 4y ago
Ah my question was answered more or less. They're an edge case which makes sense: https://news.ycombinator.com/item?id=35048318 https://news.ycombinator.com/item?id=35048318