8 ms·

Migrating our backend from Vercel to Fly.io

147 points by Hooopo 3y ago

schneems 3y ago
I’m curious if you looked at Heroku (I work there). You mention functions (which we don’t support), also servers (which we definitely support). I’m curious if that’s it or there was more to the decision.
- drwl 3y ago
  I’m a bit out of the loop but I thought heroku died or is languishing under Salesforce. That’s my current perception of everything and no longer see it recommended in HN threads. Hopefully this does not come off as an attack (it’s not).
  - case 3y ago
    We’ve run Domainr on Heroku for over a decade, and it’s been rock solid all along.
  - animal_spirits 3y ago
    I'm currently using Heroku for a small business app, and it is working wonderfully for me
  - brundolf 3y ago
    It never stopped working, it just... stopped. More of an omen than a practical issue (so far)
    - antod 3y ago
      That's a good way of describing it. Another issue is they've locked a whole lot of useful (practically required these days) features behind requiring an enterprise account - the trouble is that "enterprise" isn't just paying a whole lot more (if only). Enterprise involves getting involved in Salesforce style opaque fixed priced annual paid upfront contracts etc. It's just not a cloud provider any more at that point - you know the whole elastic thing the cloud was supposed to do.
- preciousoo 3y ago
  I discovered fly because I made an heroku account, connected the wrong card (I’m a broke college student), and heroku told me I couldn’t change the card for the next 30 days. This was all within 5 mins of making my account. I couldn’t find a support avenues. I tried many clever workarounds but their alt-account defector is top notch (props to that team). Asked around in dev circles and they recommended me fly io. Idk who hurt heroku for them to put such measures in place but I’ve never encountered such strict policies before, and I’ll forever avoid places like that.
deleted 3y ago
[deleted]
rjh29 3y ago
> Edge functions are cost-effective as you only pay for the actual CPU execution. > We have over 1000 monitors, and the monthly cost to run them would be $150. > While on fly we only have 6 servers with 2vcpu/512Mb It cost us $23.34 monthly ($3.89*6). So edge functions are in no way cost-effective right? People using lambda functions are getting ripped off, they could just buy a couple of VPS.
- 616c 3y ago
  If you're bursty and only run 1000 invocations every few days or weeks and otherwise you run it 0 or 1 times per increment then you can end up spending a lot less than that estimated server cost with fly.io no?
  - rjh29 3y ago
    Definitely there will be cases where it makes sense. But intuition would suggest if your servers are say 80% idle then serverless functions would be cheaper, but that isn't actually the case. Cloud companies don't incur much of a cost from a VPS either if it's idle. My team noticed the same with AWS Aurora Serverless (a database), it was so expensive that it was easier to just run a normal instance of RDS.
- calvinmorrison 3y ago
  Both are free for business,
- NicoJuicy 3y ago
  Edge functions are cost effective, the problem is that they are comparing from Vercel. Vercel is basically a dev friendly wrapper for tier 1 services: https://news.ycombinator.com/item?id=35774730 https://news.ycombinator.com/item?id=35774730 Eg. Vercel is 25x more expensive than eg. Cloudflare Workers. Raw guess would be that their 150$ bill would have become 6$. https://news.ycombinator.com/item?id=37891412 https://news.ycombinator.com/item?id=37891412 Eg. Image resizing > Vercel : 5$ / 1000 requests > Cloudflare : 9$ / 50.000 requests Edit for comment below: That's a blog post. I got my info from here: https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/ See: image resizing > 50,000 monthly resizing requests included with Pro, Business. $9 per additional 50,000 resizing requests
kavaruka 3y ago
I would be curious to know the performance using node.js as runtime, given that at the moment there is no evidence that bun on a real application offers better performance
xmonkee 3y ago
Every day I need to add a new feature to my app, I am grateful I picked fly (serverful) rather than Vercel. The fact that as far as I'm concerned, it's just a computer, is incredibly useful. We've added long-running tasks, background jobs, scheduled tasks, side-car processes, custom-code execution, etc etc. Then, the fact that I can run something like Redis or Metabase within the same VPN with just a dockerfile is incredibly empowering. And just giving up basic things like SSH access to your server seems like an incredibly short-sighted thing to do. Maybe I'm too old, I just don't get it.
- lmm 3y ago
  It's not "just" a computer, a computer is a whole bunch of complicated stuff that I don't want to have to care about. I want to write some code and have it run and I don't want or need to care about the details of how that happens as long as it works reliably. Being able to ssh into your server is giving you more tools to fix problems, sure, but mostly problems that you created for yourself by having a server in the first place.
  - apavlinovic 3y ago
    [flagged]
    - lmm 3y ago
      > If you are a software engineer, and you think that a server is "complicated", then why are you in this business at all? I'm in this business to solve big problems with a minimum of effort. The three great virtues of a programmer are laziness, impatience, and hubris. (I do in fact know how to build and maintain a server. But unless you're getting some unique value-add from doing that, it's a waste of time that you could be spending more productively)
  - xmonkee 3y ago
    > I want to write some code and have it run and I don't want or need to care about the details of how that happens as long as it works reliably I'm sorry, this is an incredibly stupid take. You always "need" to care about the abstraction that your infrastructure is providing to you. Vercel also provides a abstraction in terms of serverless functions. >I want to write some code and have it run and I don't want or need to care about the details of how that happens as long as it works reliably. Yeah, same. As long as it works, I have no problem. Now add background tasks or streaming responses or a cron job. Oh, guess what, you have to suddenly care about the options your provider is giving you, or go out and buy some stupid cron-as-service or ssh-as-service because you don't have any control over your infrastructure. And now suddenly your infra is way more complicated than mine. I am still one that single dockerfile. >Being able to ssh into your server is giving you more tools to fix problems, sure, but mostly problems that you created for yourself by having a server in the first place. How is running a clean-up script anything to do with having a server? That is the most common use-case for ssh-ing into your server. In fact I am wracking my brains right now to come up with anytime I had a problem because of having a server and coming up short. Fly.io (or AWS, or GCP) has problems, for sure, but none of them are because I am running a server.
steve_adams_86 3y ago
Unreliable deployments are my experience as well. I also encountered unexpected and unannounced downtimes surprisingly often. I was excited about fly, but ended up sticking with digitalocean. I have only had one issue with deployment reliability there (when they changed their build tooling for Python applications on the apps platform), but they responded quickly with a fix and shortly after announced the change and potential issues to all customers. Fly is not like this, and as a hobbyist I don’t have the time or energy to deal with their platform’s issues. I’d rather pay for something I can depend on. DO has been amazing in that regard, and their tooling is excellent. I’ve used vercel in a professional context and wouldn’t use it for personal work. The markup is crazy and the tooling isn’t appealing enough to justify the cost. This is definitely a subjective matter as opposed to reliability and communication which are objectively necessary. Vercel just “rubs me the wrong way”, and I’m sure many people here love it.
- danpalmer 3y ago
  At my last place we ran into a number of issues with using DO in production. It was fine for dev machines, testing, etc, but we had production downtime due to DO's networking setup, and support were unable to understand the problem, let alone fix it. Quick summary: we backed up our other prod hosting to DO over SSH. One day our backups went offline, DO claimed this was because of a DDoS attack, but our backups were working fine and there were no noticeable effects. Only one port was open, SSH, and we had great security on it. Support re-enabled networking for the host, backups resumed, then the next day the same thing happened again. We told them not to do this, and they said they could not, and that we should "put Cloudflare in front of it", completely missing how that was not possible or useful for our case, and missing the fact that we were not having any problems other than DO disabling networking.
  - solarkraft 3y ago
    That level of uselessness is impressive. They must've trained on Microsoft's forum.
  - steve_adams_86 3y ago
    Wow, that’s exceptionally unhelpful. It hasn’t been my experience, but this mirrors my experience with fly. I guess we’re never safe, haha. I’ve had several projects of varying complexity running with excellent uptime, both on the apps platform and on plain old droplets, for longer than I can say with certainty. Close to 8 years I guess. I might just be lucky, but in that time I really can only remember the one unexpected outage.
NicoJuicy 3y ago
TLDR: They could have done it cheaper, quicker and without adding DevOps to their workload with just migrating to Cloudflare. - Vercel: 150$/m. - Fly: 23$/m ( + managing servers and devops) - Cloudflare: 11 $/m. --- (original comment) They could have gone from Vercel to Cloudflare to reduce their costs. But that would have been almost no work to create a blog post about :p https://developers.cloudflare.com/pages/framework-guides/deploy-a-hono-site/ https://developers.cloudflare.com/pages/framework-guides/dep... Did some raw math. Cloudflare is 0,15$ per million requests and Vercel is 2$ per million requests. Their calculation for Vercel was: 77,600 * (2/1,000,000) = 0.15c per monitor monthly So that's ~0,011c per monitor monthly on Cloudflare. That would be a bill of 11€ per month ( vs 150 € per month on Vercel ). Probably less, since Cloudflare doesn't count idle CPU time, which is very relevant in this use-case ( outbound http calls) ... - https://blog.cloudflare.com/workers-pricing-scale-to-zero/ https://blog.cloudflare.com/workers-pricing-scale-to-zero/ Which is cheaper than their VPS of 23.34$ / month. And would have avoided managing servers + security to their workload...
- grrowl 3y ago
  Can you deploy 700MB Dockerfiles to Cloudflare, as they mention as a minimum requirement in the article?
  - NicoJuicy 3y ago
    Why would they need that on Cloudflare? Since they didn't had to change much of their original functions to docker, if they would have switched to Cloudflare from Vercel directly. That would have been a lot quicker for them to do... Alternatively, Cloudflare supports hono which they moved too. https://developers.cloudflare.com/pages/framework-guides/deploy-a-hono-site/ https://developers.cloudflare.com/pages/framework-guides/dep...
  - radicalriddler 3y ago
    Didn't they only need the 700MB dockerfile due to Fly.io requiring it?
    - 3y ago
Lucasoato 3y ago
I really don’t understand how people can trust platforms like Vercel, Fly.io over robust could providers like Cloudflare, AWS or Azure. I mean, Vercel has its usefulness, it’s so well integrated with the NextJS stack, it totally makes sense for small amateurish projects since it saves you time and money… but once you want to push to production, have real customers and satisfy them reliably, these platforms can’t compete with the big ones.
- leerob 3y ago
  Edit: Nevermind, wrong thread. Vercel does honor DCMA, of course, though.
  - jiayo 3y ago
    You work at Vercel. Are you saying Vercel does not honour DMCA takedown requests and that is a selling point of using Vercel? This seems like a strange thing to brag about.
    - deleted 3y ago
      [deleted]
    - DaiPlusPlus 3y ago
      > This seems like a strange thing to brag about. Not to me, it isn't. There are plenty of areas-of-interest that attract both hobbyists and serious academics alike - which also tends to attract unwanted attention from callous legal departments who are keep to adopt a shoot-first-ask-questions-later policy - things like (lawful) research into DRM techniques, retro video-games (and not ROM hosting), infosec disclosures, and so on - so if you're really into those areas and want a safe place for your lawful content but without worrying about your site/content/services being taken-down without good cause then it makes sense to side with a provider who is able to resist DMCA requests.
  - deleted 3y ago
    [deleted]
- elliotec 3y ago
pech0rin 3y ago
I find it interesting that people seem to be trading short term gains with long term reliability and maintenance costs. This glut of 0-friction deploy services lull people into a nice false sense of security. But in actuality you are wasting hours, days, weeks of time when they become unreliable, support is unresponsive, or something unexpected pops up. There is a huge advantage (outside of amateur, low importance projects) for putting in place - at the beginning - an infrastructure that is dead simple and reliable. AWS, GCP may have some upfront complexity but provide advantages in terms of reliability, knowledgeable support, and proven track records. I would never recommend these current platforms to be used for building a long term business on top of. I have been tempted by the siren song of one click deploys but in the long run so much extra time is wasted.
- deleted 3y ago
  [deleted]
- heraldev 3y ago
  This! Can't agree more, I think we share the same idea, that's what the tool we're making is about: https://github.com/mify-io/mify/ https://github.com/mify-io/mify/. It generates backend service code in a scalable way from the beginning, so that you wouldn't have to rewrite and move services to some other platform. It's better to have good architecture from the beginning, but I understand why people choose these platforms - they are saving a lot of time in the initial development, that helps them iterate quickly. What will happen next is that people spending time and resources to perform costly migrations, and some do this more that once.
- yowlingcat 3y ago
  > I find it interesting that people seem to be trading short term gains with long term reliability and maintenance costs. This glut of 0-friction deploy services lull people into a nice false sense of security. I find it interesting as well. I agree that it's a false sense of security, and there is no real long-term gain from avoiding the one time paydown of deploying to a big 3 cloud services provider. Still, I think the impulse reflects something a very real pain, and something I find my team continuing to face as we try to manage a the operationally minimalistic stack we can get away with on AWS -- poor DX. It does still boggle my mind that AWS still doesn't have a Heroku-esque happy path DX that lets you get started easily and then add in complexity on an as needed basis rather than forcing it to get the most basic thing running. It seems like every minor customization requires in AWS parlance spinning up a Lambda to do something that should be a first class feature in the platform by default. Will I migrate off the platform? No. Would I use a simpler, opinionated interface that let me focus on my application and not arcanae, if AWS made it avaiable? Absolutely.
notnmeyer 3y ago
these issues aren’t particularly severe and strike me as the kind of thing you’d generally run into switching hosts.
rozenmd 3y ago
My uptime monitoring business made a similar migration (AWS Lambda to fly.io), and I ended up rolling it back a few months later. I wrote more about the move to fly.io here: https://onlineornot.com/on-moving-million-uptime-checks-onto-fly-io https://onlineornot.com/on-moving-million-uptime-checks-onto... and (part of) the move back to AWS here: https://onlineornot.com/scaling-aws-lambda-postgres-to-thousands-of-uptime-checks https://onlineornot.com/scaling-aws-lambda-postgres-to-thous... Edit: forgot that second link doesn't actually explain that I moved off fly.io, will write a follow-up.
- deleted 3y ago
  [deleted]
- factormeta 3y ago
  [flagged]
robertlagrant 3y ago
Why is Fly apparently so unstable? I like many love the idea, but get a little scared by the many many anecdotes of issues. What are they doing that makes it unstable? Lots of new locations spinning up that shake bugs loose? Cost-reducing refactorings that reduce stability?
- urschrei 3y ago
  (Fly customer for the past 12 months: small web app (three machines across two regions plus replicated Postgres across two regions, on a paid plan)). Fly has been extremely stable for us, with the sole exception of deploys: once a month or so, deploys from CI start failing for a couple of hours. That doesn’t result in any downtime (I have never experienced any downtime due to a failing machine on Fly), just that new code doesn’t end up on prod until it’s fixed. If it’s urgent I email support (highly competent), or wait it out. I would describe myself as “extremely happy with the service, yet also annoyed by this aspect”. Fly allows me to manage my resources in a way that isn’t really possible elsewhere (from standard Python web apps in multi-hundred-mb containers to specialised Rust apps in < 10mb containers), and in a way that is (now) extremely simple to reason about, and the support has been excellent when I’ve needed it (they were very patient and understanding when I screwed up a region move and managed to somehow break my db leader beyond repair), but I’d like them to address this, because it’s a widespread issue. Given the evolution of their architecture, I suspect they will. But I’d also like them to talk about it more.
  - robertlagrant 3y ago
    Thanks for the insight!
- elxx 3y ago
  (Background: I'm currently using Fly for some hobby apps. I like it.) It is still wildly unstable right now because they're basically still building the platform and figuring out how to run a business. Earlier this year there was a migration to their "Apps V2" platform [0] which was supposed to be simple but it was extremely poorly communicated which led to a lot of users hitting issues along the way and being forced to make forum posts to try and desperately figure out how to keep their production apps up. None of the migrations worked for me either, I didn't complain as a freeloader - but seeing the support requests from paying customers painted a really bad picture. [0] https://community.fly.io/t/get-in-losers-were-getting-off-nomad/12914 https://community.fly.io/t/get-in-losers-were-getting-off-no...
mvdtnz 3y ago
From one toy to another.
- reducesuffering 3y ago
  Are Adobe, Splunk, Washington Post, Netflix, Zapier, Notion, and Uber toys? Because they're running on Vercel infra.
  - ies7 3y ago
    Its not about vercel or fly.io. Its about openstatus dev Their migration timeline from their blog: 1. August 2, 48 hours after public launch 400+ users 2. August 20, migrate from planetscale to turso (sqlite) 3. Oct 29, migrate from vercel to fly.io, migrate from nextjs to hono, also mentioned change to bunjs. This is seems like they tend to (sorry I'm judging here): 1. move fast break things or 2. don't have a plan before launch day or 3. only chasing the latest tech buzz.
    - tibozaurus 3y ago
      We are trying, breaking and learning. And you were right we did not have plan before launch, we wanted to build something that bring us excitement. and we are planing more about the future, since the project took off FYI we still both have full time job
      - ies7 3y ago
        Sorry for the harsh words. Obviously I can't speak about excitement but if you're worried about cost, you might want to research aws grant or things like that. In 2019 my company invest in a startup, while doing IT due diligence I found out they get $100K AWS grant to be used for 2 years. Fyi this company business is writing articles about baby and almost no revenue at that time.
recroad 3y ago
I have been using Vercel for production NextJS apps and have been very satisfied.
- tibozaurus 3y ago
  We are too but for a simple REST API it might not be the best
  - recroad 3y ago
    No, probably not. I use it for NextJS hosting and I absolutely love the page invalidation. It probably has saved me thousands in server costs.
shoo 3y ago
I love a good migration post-mortem, thank you to the author for publishing it! There's a bit of extra detail I'd be curious to know, as someone completely unfamiliar with both Vercel & Fly.io: Re: "we required a lightweight server" as one of the drivers to migrate -- how did deploying to Vercel impede this? What specific business/operational issues was this causing? Re: migration issue of large container image -- what business or operational issues did the large container image size cause? Why was it necessary to shrink the image size, when it could be previously ignored? edit: it appears that fly.io previously had a 2GB container image size limit, relaxed on 2023/08/11 to "roughly 8GB" -- https://community.fly.io/t/docker-image-size-limit-raised-from-2gb-to-8gb/14749 https://community.fly.io/t/docker-image-size-limit-raised-fr...
factormeta 3y ago
Hmm if they hole point is save memory and smaller sizes, and since they are willing go with Bun (very experimental type of tech), then they should also just tried Deno.
bambam24 3y ago
[dead]
haney 3y ago
I joined a project that was fully deployed on Vercel. We routinely ran into issues with limitations, outages and sharp edges. Our junior devs had also taken advantage of Vercel specific features (I remember a Vercel specific request object in the code specifically). Given all the problems and the vendor lock in from tight coupling I advise everyone I discuss Vercel with to avoid them like the plague.
- sonofssam 3y ago
  Can you elaborate more on this? Or point me to a discussion? My org is planning to move to Vercel it'd be nice to know its pitfalls
tehlike 3y ago
Next up: Migrating to Hetzner.
konaraddi 3y ago
> Additionally, we have not discovered a quick method to rollback to the previous version I feel like this should be a high priority. Deployments should be quickly reversible so that a livesite caused by a bad deployment can be mitigated quickly.
aurareturn 3y ago
I've tried Fly.io probably 3 times. I've never gotten a simple Node.js project to deploy correctly. Meanwhile, I deployed the same projects to DigitalOcean and Render without a single change successfully every time.

147 points by Hooopo 3y ago

I’m curious if you looked at Heroku (I work there). You mention functions (which we don’t support), also servers (which we definitely support). I’m curious if that’s it or there was more to the decision.

drwl 3y ago
I’m a bit out of the loop but I thought heroku died or is languishing under Salesforce. That’s my current perception of everything and no longer see it recommended in HN threads. Hopefully this does not come off as an attack (it’s not).
- case 3y ago
  We’ve run Domainr on Heroku for over a decade, and it’s been rock solid all along.
- animal_spirits 3y ago
  I'm currently using Heroku for a small business app, and it is working wonderfully for me
- brundolf 3y ago
  It never stopped working, it just... stopped. More of an omen than a practical issue (so far)
  - antod 3y ago
    That's a good way of describing it. Another issue is they've locked a whole lot of useful (practically required these days) features behind requiring an enterprise account - the trouble is that "enterprise" isn't just paying a whole lot more (if only). Enterprise involves getting involved in Salesforce style opaque fixed priced annual paid upfront contracts etc. It's just not a cloud provider any more at that point - you know the whole elastic thing the cloud was supposed to do.
preciousoo 3y ago
I discovered fly because I made an heroku account, connected the wrong card (I’m a broke college student), and heroku told me I couldn’t change the card for the next 30 days. This was all within 5 mins of making my account. I couldn’t find a support avenues. I tried many clever workarounds but their alt-account defector is top notch (props to that team). Asked around in dev circles and they recommended me fly io. Idk who hurt heroku for them to put such measures in place but I’ve never encountered such strict policies before, and I’ll forever avoid places like that.

deleted 3y ago

[deleted]

rjh29 3y ago

> Edge functions are cost-effective as you only pay for the actual CPU execution. > We have over 1000 monitors, and the monthly cost to run them would be $150. > While on fly we only have 6 servers with 2vcpu/512Mb It cost us $23.34 monthly ($3.89*6). So edge functions are in no way cost-effective right? People using lambda functions are getting ripped off, they could just buy a couple of VPS.

616c 3y ago
If you're bursty and only run 1000 invocations every few days or weeks and otherwise you run it 0 or 1 times per increment then you can end up spending a lot less than that estimated server cost with fly.io no?
- rjh29 3y ago
  Definitely there will be cases where it makes sense. But intuition would suggest if your servers are say 80% idle then serverless functions would be cheaper, but that isn't actually the case. Cloud companies don't incur much of a cost from a VPS either if it's idle. My team noticed the same with AWS Aurora Serverless (a database), it was so expensive that it was easier to just run a normal instance of RDS.
calvinmorrison 3y ago
Both are free for business,
NicoJuicy 3y ago
Edge functions are cost effective, the problem is that they are comparing from Vercel. Vercel is basically a dev friendly wrapper for tier 1 services: https://news.ycombinator.com/item?id=35774730 https://news.ycombinator.com/item?id=35774730 Eg. Vercel is 25x more expensive than eg. Cloudflare Workers. Raw guess would be that their 150$ bill would have become 6$. https://news.ycombinator.com/item?id=37891412 https://news.ycombinator.com/item?id=37891412 Eg. Image resizing > Vercel : 5$ / 1000 requests > Cloudflare : 9$ / 50.000 requests Edit for comment below: That's a blog post. I got my info from here: https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/ See: image resizing > 50,000 monthly resizing requests included with Pro, Business. $9 per additional 50,000 resizing requests

kavaruka 3y ago

I would be curious to know the performance using node.js as runtime, given that at the moment there is no evidence that bun on a real application offers better performance

xmonkee 3y ago

Every day I need to add a new feature to my app, I am grateful I picked fly (serverful) rather than Vercel. The fact that as far as I'm concerned, it's just a computer, is incredibly useful. We've added long-running tasks, background jobs, scheduled tasks, side-car processes, custom-code execution, etc etc. Then, the fact that I can run something like Redis or Metabase within the same VPN with just a dockerfile is incredibly empowering. And just giving up basic things like SSH access to your server seems like an incredibly short-sighted thing to do. Maybe I'm too old, I just don't get it.

lmm 3y ago
It's not "just" a computer, a computer is a whole bunch of complicated stuff that I don't want to have to care about. I want to write some code and have it run and I don't want or need to care about the details of how that happens as long as it works reliably. Being able to ssh into your server is giving you more tools to fix problems, sure, but mostly problems that you created for yourself by having a server in the first place.
- apavlinovic 3y ago
  [flagged]
  - lmm 3y ago
    > If you are a software engineer, and you think that a server is "complicated", then why are you in this business at all? I'm in this business to solve big problems with a minimum of effort. The three great virtues of a programmer are laziness, impatience, and hubris. (I do in fact know how to build and maintain a server. But unless you're getting some unique value-add from doing that, it's a waste of time that you could be spending more productively)
- xmonkee 3y ago
  > I want to write some code and have it run and I don't want or need to care about the details of how that happens as long as it works reliably I'm sorry, this is an incredibly stupid take. You always "need" to care about the abstraction that your infrastructure is providing to you. Vercel also provides a abstraction in terms of serverless functions. >I want to write some code and have it run and I don't want or need to care about the details of how that happens as long as it works reliably. Yeah, same. As long as it works, I have no problem. Now add background tasks or streaming responses or a cron job. Oh, guess what, you have to suddenly care about the options your provider is giving you, or go out and buy some stupid cron-as-service or ssh-as-service because you don't have any control over your infrastructure. And now suddenly your infra is way more complicated than mine. I am still one that single dockerfile. >Being able to ssh into your server is giving you more tools to fix problems, sure, but mostly problems that you created for yourself by having a server in the first place. How is running a clean-up script anything to do with having a server? That is the most common use-case for ssh-ing into your server. In fact I am wracking my brains right now to come up with anytime I had a problem because of having a server and coming up short. Fly.io (or AWS, or GCP) has problems, for sure, but none of them are because I am running a server.

steve_adams_86 3y ago

Unreliable deployments are my experience as well. I also encountered unexpected and unannounced downtimes surprisingly often. I was excited about fly, but ended up sticking with digitalocean. I have only had one issue with deployment reliability there (when they changed their build tooling for Python applications on the apps platform), but they responded quickly with a fix and shortly after announced the change and potential issues to all customers. Fly is not like this, and as a hobbyist I don’t have the time or energy to deal with their platform’s issues. I’d rather pay for something I can depend on. DO has been amazing in that regard, and their tooling is excellent. I’ve used vercel in a professional context and wouldn’t use it for personal work. The markup is crazy and the tooling isn’t appealing enough to justify the cost. This is definitely a subjective matter as opposed to reliability and communication which are objectively necessary. Vercel just “rubs me the wrong way”, and I’m sure many people here love it.

danpalmer 3y ago
At my last place we ran into a number of issues with using DO in production. It was fine for dev machines, testing, etc, but we had production downtime due to DO's networking setup, and support were unable to understand the problem, let alone fix it. Quick summary: we backed up our other prod hosting to DO over SSH. One day our backups went offline, DO claimed this was because of a DDoS attack, but our backups were working fine and there were no noticeable effects. Only one port was open, SSH, and we had great security on it. Support re-enabled networking for the host, backups resumed, then the next day the same thing happened again. We told them not to do this, and they said they could not, and that we should "put Cloudflare in front of it", completely missing how that was not possible or useful for our case, and missing the fact that we were not having any problems other than DO disabling networking.
- solarkraft 3y ago
  That level of uselessness is impressive. They must've trained on Microsoft's forum.
- steve_adams_86 3y ago
  Wow, that’s exceptionally unhelpful. It hasn’t been my experience, but this mirrors my experience with fly. I guess we’re never safe, haha. I’ve had several projects of varying complexity running with excellent uptime, both on the apps platform and on plain old droplets, for longer than I can say with certainty. Close to 8 years I guess. I might just be lucky, but in that time I really can only remember the one unexpected outage.

NicoJuicy 3y ago

TLDR: They could have done it cheaper, quicker and without adding DevOps to their workload with just migrating to Cloudflare. - Vercel: 150$/m. - Fly: 23$/m ( + managing servers and devops) - Cloudflare: 11 $/m. --- (original comment) They could have gone from Vercel to Cloudflare to reduce their costs. But that would have been almost no work to create a blog post about :p https://developers.cloudflare.com/pages/framework-guides/deploy-a-hono-site/ https://developers.cloudflare.com/pages/framework-guides/dep... Did some raw math. Cloudflare is 0,15$ per million requests and Vercel is 2$ per million requests. Their calculation for Vercel was: 77,600 * (2/1,000,000) = 0.15c per monitor monthly So that's ~0,011c per monitor monthly on Cloudflare. That would be a bill of 11€ per month ( vs 150 € per month on Vercel ). Probably less, since Cloudflare doesn't count idle CPU time, which is very relevant in this use-case ( outbound http calls) ... - https://blog.cloudflare.com/workers-pricing-scale-to-zero/ https://blog.cloudflare.com/workers-pricing-scale-to-zero/ Which is cheaper than their VPS of 23.34$ / month. And would have avoided managing servers + security to their workload...

grrowl 3y ago
Can you deploy 700MB Dockerfiles to Cloudflare, as they mention as a minimum requirement in the article?
- NicoJuicy 3y ago
  Why would they need that on Cloudflare? Since they didn't had to change much of their original functions to docker, if they would have switched to Cloudflare from Vercel directly. That would have been a lot quicker for them to do... Alternatively, Cloudflare supports hono which they moved too. https://developers.cloudflare.com/pages/framework-guides/deploy-a-hono-site/ https://developers.cloudflare.com/pages/framework-guides/dep...
- radicalriddler 3y ago
  Didn't they only need the 700MB dockerfile due to Fly.io requiring it?
  - 3y ago

Lucasoato 3y ago

I really don’t understand how people can trust platforms like Vercel, Fly.io over robust could providers like Cloudflare, AWS or Azure. I mean, Vercel has its usefulness, it’s so well integrated with the NextJS stack, it totally makes sense for small amateurish projects since it saves you time and money… but once you want to push to production, have real customers and satisfy them reliably, these platforms can’t compete with the big ones.

leerob 3y ago
Edit: Nevermind, wrong thread. Vercel does honor DCMA, of course, though.
- jiayo 3y ago
  You work at Vercel. Are you saying Vercel does not honour DMCA takedown requests and that is a selling point of using Vercel? This seems like a strange thing to brag about.
  - deleted 3y ago
    [deleted]
  - DaiPlusPlus 3y ago
    > This seems like a strange thing to brag about. Not to me, it isn't. There are plenty of areas-of-interest that attract both hobbyists and serious academics alike - which also tends to attract unwanted attention from callous legal departments who are keep to adopt a shoot-first-ask-questions-later policy - things like (lawful) research into DRM techniques, retro video-games (and not ROM hosting), infosec disclosures, and so on - so if you're really into those areas and want a safe place for your lawful content but without worrying about your site/content/services being taken-down without good cause then it makes sense to side with a provider who is able to resist DMCA requests.
- deleted 3y ago
  [deleted]
elliotec 3y ago

pech0rin 3y ago

I find it interesting that people seem to be trading short term gains with long term reliability and maintenance costs. This glut of 0-friction deploy services lull people into a nice false sense of security. But in actuality you are wasting hours, days, weeks of time when they become unreliable, support is unresponsive, or something unexpected pops up. There is a huge advantage (outside of amateur, low importance projects) for putting in place - at the beginning - an infrastructure that is dead simple and reliable. AWS, GCP may have some upfront complexity but provide advantages in terms of reliability, knowledgeable support, and proven track records. I would never recommend these current platforms to be used for building a long term business on top of. I have been tempted by the siren song of one click deploys but in the long run so much extra time is wasted.

deleted 3y ago
[deleted]
heraldev 3y ago
This! Can't agree more, I think we share the same idea, that's what the tool we're making is about: https://github.com/mify-io/mify/ https://github.com/mify-io/mify/. It generates backend service code in a scalable way from the beginning, so that you wouldn't have to rewrite and move services to some other platform. It's better to have good architecture from the beginning, but I understand why people choose these platforms - they are saving a lot of time in the initial development, that helps them iterate quickly. What will happen next is that people spending time and resources to perform costly migrations, and some do this more that once.
yowlingcat 3y ago
> I find it interesting that people seem to be trading short term gains with long term reliability and maintenance costs. This glut of 0-friction deploy services lull people into a nice false sense of security. I find it interesting as well. I agree that it's a false sense of security, and there is no real long-term gain from avoiding the one time paydown of deploying to a big 3 cloud services provider. Still, I think the impulse reflects something a very real pain, and something I find my team continuing to face as we try to manage a the operationally minimalistic stack we can get away with on AWS -- poor DX. It does still boggle my mind that AWS still doesn't have a Heroku-esque happy path DX that lets you get started easily and then add in complexity on an as needed basis rather than forcing it to get the most basic thing running. It seems like every minor customization requires in AWS parlance spinning up a Lambda to do something that should be a first class feature in the platform by default. Will I migrate off the platform? No. Would I use a simpler, opinionated interface that let me focus on my application and not arcanae, if AWS made it avaiable? Absolutely.

notnmeyer 3y ago

these issues aren’t particularly severe and strike me as the kind of thing you’d generally run into switching hosts.

rozenmd 3y ago

My uptime monitoring business made a similar migration (AWS Lambda to fly.io), and I ended up rolling it back a few months later. I wrote more about the move to fly.io here: https://onlineornot.com/on-moving-million-uptime-checks-onto-fly-io https://onlineornot.com/on-moving-million-uptime-checks-onto... and (part of) the move back to AWS here: https://onlineornot.com/scaling-aws-lambda-postgres-to-thousands-of-uptime-checks https://onlineornot.com/scaling-aws-lambda-postgres-to-thous... Edit: forgot that second link doesn't actually explain that I moved off fly.io, will write a follow-up.

deleted 3y ago
[deleted]
factormeta 3y ago
[flagged]

robertlagrant 3y ago

Why is Fly apparently so unstable? I like many love the idea, but get a little scared by the many many anecdotes of issues. What are they doing that makes it unstable? Lots of new locations spinning up that shake bugs loose? Cost-reducing refactorings that reduce stability?

urschrei 3y ago
(Fly customer for the past 12 months: small web app (three machines across two regions plus replicated Postgres across two regions, on a paid plan)). Fly has been extremely stable for us, with the sole exception of deploys: once a month or so, deploys from CI start failing for a couple of hours. That doesn’t result in any downtime (I have never experienced any downtime due to a failing machine on Fly), just that new code doesn’t end up on prod until it’s fixed. If it’s urgent I email support (highly competent), or wait it out. I would describe myself as “extremely happy with the service, yet also annoyed by this aspect”. Fly allows me to manage my resources in a way that isn’t really possible elsewhere (from standard Python web apps in multi-hundred-mb containers to specialised Rust apps in < 10mb containers), and in a way that is (now) extremely simple to reason about, and the support has been excellent when I’ve needed it (they were very patient and understanding when I screwed up a region move and managed to somehow break my db leader beyond repair), but I’d like them to address this, because it’s a widespread issue. Given the evolution of their architecture, I suspect they will. But I’d also like them to talk about it more.
- robertlagrant 3y ago
  Thanks for the insight!
elxx 3y ago
(Background: I'm currently using Fly for some hobby apps. I like it.) It is still wildly unstable right now because they're basically still building the platform and figuring out how to run a business. Earlier this year there was a migration to their "Apps V2" platform [0] which was supposed to be simple but it was extremely poorly communicated which led to a lot of users hitting issues along the way and being forced to make forum posts to try and desperately figure out how to keep their production apps up. None of the migrations worked for me either, I didn't complain as a freeloader - but seeing the support requests from paying customers painted a really bad picture. [0] https://community.fly.io/t/get-in-losers-were-getting-off-nomad/12914 https://community.fly.io/t/get-in-losers-were-getting-off-no...

mvdtnz 3y ago

From one toy to another.

reducesuffering 3y ago
Are Adobe, Splunk, Washington Post, Netflix, Zapier, Notion, and Uber toys? Because they're running on Vercel infra.
- ies7 3y ago
  Its not about vercel or fly.io. Its about openstatus dev Their migration timeline from their blog: 1. August 2, 48 hours after public launch 400+ users 2. August 20, migrate from planetscale to turso (sqlite) 3. Oct 29, migrate from vercel to fly.io, migrate from nextjs to hono, also mentioned change to bunjs. This is seems like they tend to (sorry I'm judging here): 1. move fast break things or 2. don't have a plan before launch day or 3. only chasing the latest tech buzz.
  - tibozaurus 3y ago
    We are trying, breaking and learning. And you were right we did not have plan before launch, we wanted to build something that bring us excitement. and we are planing more about the future, since the project took off FYI we still both have full time job
    - ies7 3y ago
      Sorry for the harsh words. Obviously I can't speak about excitement but if you're worried about cost, you might want to research aws grant or things like that. In 2019 my company invest in a startup, while doing IT due diligence I found out they get $100K AWS grant to be used for 2 years. Fyi this company business is writing articles about baby and almost no revenue at that time.

recroad 3y ago

I have been using Vercel for production NextJS apps and have been very satisfied.

tibozaurus 3y ago
We are too but for a simple REST API it might not be the best
- recroad 3y ago
  No, probably not. I use it for NextJS hosting and I absolutely love the page invalidation. It probably has saved me thousands in server costs.

shoo 3y ago

I love a good migration post-mortem, thank you to the author for publishing it! There's a bit of extra detail I'd be curious to know, as someone completely unfamiliar with both Vercel & Fly.io: Re: "we required a lightweight server" as one of the drivers to migrate -- how did deploying to Vercel impede this? What specific business/operational issues was this causing? Re: migration issue of large container image -- what business or operational issues did the large container image size cause? Why was it necessary to shrink the image size, when it could be previously ignored? edit: it appears that fly.io previously had a 2GB container image size limit, relaxed on 2023/08/11 to "roughly 8GB" -- https://community.fly.io/t/docker-image-size-limit-raised-from-2gb-to-8gb/14749 https://community.fly.io/t/docker-image-size-limit-raised-fr...

factormeta 3y ago

Hmm if they hole point is save memory and smaller sizes, and since they are willing go with Bun (very experimental type of tech), then they should also just tried Deno.

bambam24 3y ago

[dead]

haney 3y ago

I joined a project that was fully deployed on Vercel. We routinely ran into issues with limitations, outages and sharp edges. Our junior devs had also taken advantage of Vercel specific features (I remember a Vercel specific request object in the code specifically). Given all the problems and the vendor lock in from tight coupling I advise everyone I discuss Vercel with to avoid them like the plague.

sonofssam 3y ago
Can you elaborate more on this? Or point me to a discussion? My org is planning to move to Vercel it'd be nice to know its pitfalls

tehlike 3y ago

Next up: Migrating to Hetzner.

konaraddi 3y ago

> Additionally, we have not discovered a quick method to rollback to the previous version I feel like this should be a high priority. Deployments should be quickly reversible so that a livesite caused by a bad deployment can be mitigated quickly.

aurareturn 3y ago

I've tried Fly.io probably 3 times. I've never gotten a simple Node.js project to deploy correctly. Meanwhile, I deployed the same projects to DigitalOcean and Render without a single change successfully every time.