18 ms·

Migrating from AWS to Fly.io

252 points by sausagefeet 4y ago

itsmemattchung 4y ago
No where do I see mentioned how much was actually saved in dollar figures. What are we talking here—thousands, hundreds? Similarly, I'd be interested in knowing engineering effort involved in the migration. As a software consultant focusing on designing, building, and deploying software on AWS, more often than not, the infrastructure cost issues I've seen have less to do with the underlying infrastructure provider — AWS in this case — and more to do with the application itself that's being deployed. Most recently, we were able to shave off 50% of a client's bill that was using Lambda for compute. The issue? Several, including sleep timers (on a service where you pay by the millisecond!) as well as pathological code (i.e. consume message from queue, enqueue same message, rinse and repeat). Yes. You can save money by transitioning to another provider. But I'd start with reviewing the underlying code and architecture first
- mrdoops 4y ago
  The biggest $ cost I'd have on my radar WRT AWS vs Fly is complexity and need-to-know of employee time to do any given thing. Fly is way easier to navigate and use than AWS and its labyrinth of Cloud Scale™ horrors. The trade-off being you can do more given things with AWS.
  - biorach 4y ago
    Yeah, this. AWS enables you to do anything you can think of, at any scale and complexity, given enough time spent reading their documentation and crafting terraform configurations. Fly makes it really easy to do the 80% of things that most small/medium operations need to just get done now.
    - deleted 4y ago
      [deleted]
  - yodon 4y ago
    >Fly is way easier to navigate and use than AWS and its labyrinth of Cloud Scale™ horrors Give fly.io a couple years and it will end up forced to take on all the complexity and edge cases AWS and Azure have to deal with today. A shiny new codebase is always nicer than an old one, because the old one has had to accommodate all those pesky customer needs.
    - Existenceblinks 4y ago
      You think it's inevitable? I think it's unlikely, they probably run into a different kind of targets. Those "edge cases" horror seem unnecessary like permission hell, pricing model hell (and too many size)
    - rendaw 4y ago
      Did that ever happen to Heroku? Ignoring 10% of use cases for their target customers and making the 90% doesn't sound crazy to me.
  - intelVISA 4y ago
    I can't wait for Fly to disrupt the market of Cloud Native Eldritch Horrors™ Their SLA's a bit too iffy for my use case atm but I wish them well in this noble endeavor.
throwthere 4y ago
The article mentions this, but that’s strongly enough in my opinion. With fly you have no managed PostgreSQL. In my opinion, it’s not really comparable on cost to aws when Postgres is in the stack. You do read about interesting hacks where someone will set up rds in a region that may be single digit milliseconds away from a fly region. then, presumably, you could put PG bouncer on a sort of bastion host that connects to the fly wire guard VPN. But obviously, there’s no guarantee that the latency will always be that good.
- TheDong 4y ago
  > But obviously, there’s no guarantee that the latency will always be that good That guarantee already doesn't exist with RDS. The RDS master is in one AZ, your server may be in another (and if it isn't, well, RDS will fail over to another AZ eventually). The network latency between AWS AZs is usually good, but it can be arbitrarily high, up to full outages between AZs. This is just a measure of degrees. Your application and/or business already has to handle some degree of network issues (aws network outages), so it's a tradeoff of whether the extra hops increase the chance enough to matter.
- mike_d 4y ago
  https://fly.io/docs/postgres/ https://fly.io/docs/postgres/ It isn't "Managed Postgres," but the differences are minimal. RDS is ultimately a solution for people who look in the mirror and confidently say "you don't know how to run a database."
  - Octabrain 4y ago
    > RDS is ultimately a solution for people who look in the mirror and confidently say "you don't know how to run a database." I think this is a terrible oversimplification and something tells me that you haven't had to deal with a complex database setup from an operations perspective. RDS reduces a huge overhead in terms of operations (ha, backups, upgrades and clustering being the first ones that come to my mind). Between RDS and running a database on a virtual machine(s) and manage it with, let's say, Ansible for providing the four aforementioned features I would chose RDS any day of the week.
viraptor 4y ago
I missed when this was added: > if you configure your application to expose a Prometheus endpoint, those metrics will automatically show up on your Grafana dashboard But that's such an amazing and simple idea to integrate observability. Love the approach.
- deleted 4y ago
  [deleted]
- fastball 4y ago
  Isn't this just how Prometheus works alongside Grafana? I've never used fly.io but I've used that exact setup with these two services.
  - alexeldeib 4y ago
    I believe GP is commenting on fly’s integration. Fly comes with built in Prometheus scraping and prepopulated grafana dashboards for VM resources some connection/request metrics. I believe the value is that if you simply expose a prom endpoint, they’ll scrape it and you can use their grafana (or hook your own into their Prometheus). Nothing special vs vanilla prom/grafana, but seamless “no-ops” integration vs DIY.
    - viraptor 4y ago
      Correct. It's common for companies to have their own custom systems or specific agents, then build a very custom and limited dashboard on top. Fly.io went with "you know grafana and prometheus? yeah, we'll do just that". And I think that's perfect.
- kitd 4y ago
  Openshift does something similar (though you need to annotate your deployments). But yes, the simplicity is refreshing.
jokethrowaway 4y ago
I found debugging setup errors on fly to be incredibly frustrating You can't even ssh until the container started and they have some requirements on hostnames (0.0.0.0) At least a few of their examples are broken. The documentation could be better but it's pretty good Is it better than Aws? Yes of course, you need to spend way more to get a dx as bad as Aws. Setting up a VPS is a smoother experience than fly, though. Once it's running, it's lovely and reminds me of heroku.
- biorach 4y ago
  > You can't even ssh until the container started In fairness, I'm struggling to think of a way they could get you to ssh to a container that hasn't started yet > and they have some requirements on hostnames (0.0.0.0) Can you clarify?
- tptacek 4y ago
  Help me understand what you'd be SSH'ing into before your VM boots up? (I'm responsible for most of our SSH goop, so if there's an improvement to be made, I'm happy to make it). A reminder: we don't run "containers". We take containers, unpack them, and transform them into VMs. There's no host OS for you to access.
  - indigodaddy 4y ago
    Maybe he means/wants some kind of “console” access for his “instance” ? Seems crazy as it takes I’m sure less than a second to boot, and if there is a problem booting up it’s probably not going to be a problem that a user/customer would be able to solve at console.
  - mostly_lurks 4y ago
    Caveats: 1) this may not be what grandparent was trying to do and 2) I have limited experience debugging "regular" Docker, let alone firecracker or whatever Fly.io does. With that said, I succesfully deployed a simple Rust/Actix/Sqlite app with a Dockerfile to Fly.io. I then thought I'd try out litestream. For reasons I'm still not sure about, having `ENTRYPOINT ["litestream replicate … -exec myapp"]` resulted in immediate kernel panics.[1] As I was debugging, my instinct was to want to `fly ssh console` into a running container to see if running the same command from the shell produced any clues for further debugging. Though I understand this is the wrong mental model, the thought was something like "well, I know everything else works, I just need Fly to ignore this failing command so I can have a minute to poke around." To do this, I ended up just removing litestream from the ENTRYPOINT so the deploy would succeed, then I could SSH in and play around at the shell to see what was going on. Again, I have no idea whether this is the same sort of problem the other person was having, but for my case, what would be helpful is probably not changing how `fly ssh console` behaves, but perhaps some documentation of suggested debugging techniques in case your app is failing to start. [1] Putting the exact same command into a "start.sh" and making that the ENTRYPOINT worked fine so that's what I ended up doing.
sosodev 4y ago
I recently migrated some services from DigitalOcean to Fly and it was quite pleasant. My only issue is that routing to Fly from Southern California is awful. My friend and I have different ISPs and yet we’re both routed to the east coast instead of the much closer Los Angeles edge server. So our requests end up going to the east coast(edge), then back to the west coast (app), back to the east coast(edge), and then back to west coast (origin). The latency adds up and makes requests to a nearby app almost as slow as to one hosted across an ocean. I’m worried that other users will hit this and make hosting on Fly terrible for apps that require lower latency :/
- whalesalad 4y ago
  What ISP's are involved here?
  - sosodev 4y ago
    Cox and Spectrum. They route slightly different but both to the east coast. I linked the forum thread above.
- mrkurt 4y ago
  This is uncommon. When did it happen? We had some cross country routing issues a few weeks ago. We can generally fix these if you don't mind sending us traveroutes.
  - ryanisnan 4y ago
    This is cool, I'd love to know more about how you generally fix issues like these?
    - mrkurt 4y ago
      I usually just go complain to network folks and they figure it out. I'm good at complaining! Routing issues like this are typically the result of some weird network provider politics. The fix is typically ti change how we announce IPs to force a route to do something we want. It's a dark art.
biorach 4y ago
I'm not sure this is well-known enough yet - fly now supports scale-to-zero services using fly machines - https://fly.io/docs/machines/ https://fly.io/docs/machines/
- hamandcheese 4y ago
  Does scale to zero work with ordinary fly apps yet?
  - biorach 4y ago
    Probably. Depends. For a simple app should be straighforward. I set up a machine - a few weeks ago this involved a few direct api calls using curl - maybe `flyctl` can do it all now. Then I deployed on that machine a version of my app that gracefully shut down if no requests were received for 60s. Works great. Shuts down after a minute, boots back up in under a second when a request comes in. See the docs for details
    - TheHappyOddish 4y ago
      I found the docs difficult and inconsistent when moving between flyctl and cURL when I tried playing with machines a few months ago. I daresay they'll be wonderful once the team get the machine stuff working under flyctl fully. Their generous free tier does make it easy to play and assess though.
  - mrkurt 4y ago
    No, ordinary Fly apps (what we call v1 apps) are managed by nomad. They don't scale to zero. You can create a machine based app and it'll scale to zero just fine, but you lose most of the magic of `fly deploy` right now.
- rendaw 4y ago
  I've been trying fly machines with terraform and running into one bug after another: https://github.com/fly-apps/terraform-provider-fly/issues/125 https://github.com/fly-apps/terraform-provider-fly/issues/12... https://github.com/fly-apps/terraform-provider-fly/issues/126 https://github.com/fly-apps/terraform-provider-fly/issues/12... https://github.com/fly-apps/terraform-provider-fly/issues/127 https://github.com/fly-apps/terraform-provider-fly/issues/12... I was really excited since fly seems dead simple, but I haven't gotten it running yet...
alberth 4y ago
Isn’t Fly.io hosted on AWS (they don’t have their own hardware). Just like Heroku.
- hamandcheese 4y ago
  No, I don't think they are. The things they do with Anycast IPs I don't think is possible on AWS.
- mrkurt 4y ago
  We have our own hardware.
  - alberth 4y ago
    That’s awesome to hear! Thank you btw for all your funding/support of the Erlang/Elixir ecosystem.
  - garganzol 4y ago
    Stay cool, Fly. You guys will rule the world!
nomilk 4y ago
I moved a rails MVP from heroku to fly.io. Biggest negative so far: no easy way to reset the database [1]. I'd typically like to blow it away and re-create it frequently as I tinker with it e.g. adding/removing/renaming a column or changing a data type. Those small changes now require migrations, which are a fair bit more work (but not a huge problem tbh). Biggest positive: ability to select a zone meant response time for Australian users was very fast compared to heroku; making a big difference to UX. [1] https://stackoverflow.com/q/74664822/5783745 https://stackoverflow.com/q/74664822/5783745
- rtpg 4y ago
  This is stuff where I highly recommend writing up helper scripts. I have one to pull in stuff from a fly Postgres DB, spin up my local docker compose setup, and then pump the data into there. I think that's the real niceness here, a CLI that is pretty straightforward (not perfect by any means! but I can still wrap my head around it)
  - nomilk 4y ago
    Sounds awesome/slick. If you know of any examples that can be shared would love to see them. My use of the fly CLI is rudimentary at best, and I don't use scripts at all, just (mostly basic) commands.
mtlynch 4y ago
I've migrated several services from GCP to Fly.io, and I've been very happy with Fly. All of my apps are low-volume hobbyist web apps that run on a single machine, so I don't do any Terraform, k8s, or Postgres, so my use case is a little simpler than OP. My biggest complaint has been with outages,[0, 1, 2] but that's gotten better. The original architecture meant that larger servers could evict other users' running servers, which they admitted was a bad idea.[2] I'm not sure if they've fixed that since, but I haven't seen it happen in a few months. >The container logging solution provided by Fly.io is basic. It's easy to view logs with the Fly.io CLI and via the Fly.io dashboard. However, there's only a small window of logs that are kept, forcing you to create a remote logging solution either internal to your application or via a separate Fly.io application that ships logs to an external service. This is a piece of operational overhead that I'd like to see removed. I very much agree. This is a big feature gap I feel when using Fly as well. >Fly.io is not a support-first company. If this bothers you, then Fly.io is probably not for you. >When emailing support, and email is the only option, it can take from hours to days to get a response. Sometimes they don't follow up. It does not feel like their level of support is on par with other cloud providers. This hasn't been my experience. I go through the forum rather than email, but I typically see responses within hours, even on the weekends. By contrast, when I reported bugs to GCP through their designated support channels, there was multi-day latency, and they'd just keep dismissing my report.[3] [0] https://community.fly.io/t/cant-deploy-my-app-in-iad/7746 https://community.fly.io/t/cant-deploy-my-app-in-iad/7746 [1] https://community.fly.io/t/application-vms-down-without-any-change-cant-deploy/7618/11?u=mtlynch https://community.fly.io/t/application-vms-down-without-any-... [2] https://community.fly.io/t/app-stuck-in-pending-state-after-restarts-rescaling/5972/6?u=mtlynch https://community.fly.io/t/app-stuck-in-pending-state-after-... [3] https://stackoverflow.com/q/53410165/90388 https://stackoverflow.com/q/53410165/90388
- mrkurt 4y ago
  You probably had a different community experience than recent folks. We've grown like 4x in the last 3 months and haven't kept up with community as well as I'd like (because hiring). And over holidays, we've had a difficult time with email support as well. It'll get better, but it's not great right now!
philliphaydon 4y ago
So it seems the 'machine' is the equiv of AWSs EC2? I think I would only save $5~ by not having to have HAProxy (cos AWS LB is exensive) but I would need to manage PG myself which seems like a hassle for a hobby project.
tommoor 4y ago
Fly is great. I know the Fly team has an aversion to it but I really just wish they would hire some database folks and take on managed Postgres. I'm already running 8+ apps on there and it would be the peace of mind needed to move the rest across… one day.
- mrkurt 4y ago
  I'm not completely averse (we've done this before!), it's just not how I think the world should work. If we hold out, and the future Fly cloud has a better-than-RDS Postgres option, it will be worth it. If we build our own managed Postgres, no one better is going to bring their stuff to our platform. Same with MySQL.
  - subleq 4y ago
    I'd trust a first-party managed postgres built by fly more than a third-party one.
    - belmont_sup 4y ago
      I like their strategy of outsourcing what they’re not best at to the best. They partnered with upstash, and now they’ve got a new feature almost immediately. Sure there’s a lot of vetting, but wrapping products is common. I’d be happy if they can wrap Neon’s serverless postgres service into their networks somehow.
      - nikita 4y ago
        Neon CEO here. We are working on becoming first party service on several major problems now. We will be happy to provide this for Fly. Kurt and I have been chatting. Fly wants cross region replication which is coming soon. Once it's there we can integrate. With that Neon is still on AWS and can be close to Fly, but not run on Fly servers. It's relatively straightforward for us to run on Fly, but the S3 part will still be Amazon.
        4y ago
habibur 4y ago
Would have loved to read how much money they saved from this migration. The number's not there, unfortunately.
- yashap 4y ago
  Author mentioned here that they went from $200/month on AWS to $90/month of Fly: https://news.ycombinator.com/item?id=34242923 https://news.ycombinator.com/item?id=34242923 Pretty tiny cloud costs either way, probably safe to assume they’re a small, early stage startup. But even still, it was likely a bad call to spend significant engineering effort on saving $110/month, which is a rounding error even for small startups. Hard to imagine that time couldn’t have been better spent on things that grow the business, like new features and bug fixes.
  - lawnchair 4y ago
    Definitely tiny but we're just overall happier with Fly. The migration in total probably cost us a weekend of work all said and done. We get a lot for free with Fly (observability, multi-region, etc.) so all in all a win.
    - yashap 4y ago
      Ah fair enough, that’s a pretty small amount of work!
  - habibur 4y ago
    I would read it as a 60% saving. That's a very good one. Considering that this is a recurring cost and not a one time and it's proportional to their sales. Aka, they now will be able to provide their clients twice the amount of contents for the same price. At least in theory.
    - yashap 4y ago
      My thought was more timing - there’s so much that’s URGENT for a young startup, is cost cutting really urgent, something you have to do NOW, when the current savings are just $110/month? Part of what allows startups to survive is ruthless focus on the work that that’s most important today, and saving $110/month seems unlikely to be that. However, the blog post author responded, noting that this migration took only 1 weekend, and they like other aspects of Fly than just the cost. So fair enough, 1 weekend is a small investment, less than I expected, I would have guessed this took 2+ weeks.
garganzol 4y ago
For me, Fly.io is like a seminal step forward to an imaginary OS of the future: standardized, auto-clustered, just works, dead simple to grasp. Kind of Windows 95 of its time.
- deleted 4y ago
  [deleted]
memset 4y ago
Since the fly folks are here: I'd love the ability to trigger jobs as I can with aws batch. The ability to do `fly run -a journalize-batch --command="python3 upload.py"` is something I need right now and am in the process of setting up aws batch to do it.
- mrkurt 4y ago
  You can probably hack together with machines, though we're not a great fit for big bursts of compute. "fly machine run" is fun.
boundlessdreamz 4y ago
Since the fly folks are here, do you plan to support monorepo apps in a single fly.toml. i.e define all services in a single fly.toml like docker compose or render's blueprint? Also a way to reuse the definition but run a different command. For example for a rails app, web worker (puma) and job worker (sidekiq) differs only in the command run. In fly, I would have to duplicate those. If there is an easier way, do let us know.
- mrkurt 4y ago
  The second case has experimental support already: https://fly.io/docs/reference/configuration/#the-processes-section https://fly.io/docs/reference/configuration/#the-processes-s... The first, we're not completely sure about yet. Probably someday, though.
- deleted 4y ago
  [deleted]
jesuspiece 4y ago
> Fly.io is not a support-first company. If this bothers you, then Fly.io is probably not for you. > When emailing support, and email is the only option, it can take from hours to days to get a response. Sometimes they don't follow up. It does not feel like their level of support is on par with other cloud providers. This enough to keep me on AWS. You can say a lot about AWS but their support is great
- Aeolun 4y ago
  Responsive at least. They’re very quick to inform you that x is a known issue with their service, and the request has been dispatched to the dev team.
- deleted 4y ago
  [deleted]
- wpietri 4y ago
  Yeah, I'm ok with "hours to days" of mysterious downtime for some personal projects. But for my day job? Deal killer for sure.
chaychoong 4y ago
I would love to try fly.io but unfortunately it doesn't support GPU workloads yet. Waiting eagerly!
- deleted 4y ago
  [deleted]
lawnchair 4y ago
Author here. Some folks wanted the details on pricing. Pre migration, the Terrateam AWS monthly bill was about $200/mo. With Fly.io, we're paying around $90/mo.
- moltar 4y ago
  Could you break down the costs of AWS? From your post you say you needed a simple service plus a Postgres instance. I think you can have that for around $50 on AWS using a small EC2 and Aurora Serverless v2 (min $40/mo)
  - lawnchair 4y ago
    I don't have access to the bills anymore. Two AZs, RDS HA, bandwidth, and EC2 all added up. It's possible I could have cut down the bill. The initial motivation to migrate was cost, but at the end of the day we're just happy with the Fly.io developer experience. It's a better fit for us.
- hknmtt 4y ago
  61$/m for 1vCPU and 8GB ram? Try Hetzner next time, on bare metal, for cheaper.
  - dist1ll 4y ago
    You can get 1vCPU + 2GB RAM for 3.92 Euro/month at Hetzner. Those prices are hard to beat.
    - dxxvi 4y ago
      You can get 4 virtual ARM CPU + 24GB RAM + 200GB hard disk for $0/month at Oracle cloud. Nothing can beat that.
      - dist1ll 4y ago
        Any caveats? This sounds really great.
tuananh 4y ago
if the author says it doesn't take more than few hours + DX is really better, who are we to judge :)
calltrak 4y ago
[dead]
wink 4y ago
I've been running a hobby project on their free tier for a while and it's mostly cool. Wrote up some stuff back then: https://f5n.org/blog/2022/trying-out-some-hosting-options/ https://f5n.org/blog/2022/trying-out-some-hosting-options/ but unfortunately none of my pain points/caveats seem to have been fixed * deploying (re)builds your container but doesn't even tag it locally, so if you just built one, it will build the exact same one again and on the other hand you are left guessing which hash is the container that the tool just built. * if you start with your own Dockerfile the mandatory options in fly.toml are not described overly well, so a bit of fiddling * recently I redeployed and without any real change (just a new code build) it wouldn't deploy, I had to remove some of the settings in fly.toml that I had guessed at to make it work in the summer to make it work now - no changelog anywhere * if your app is coming close to some assumed limit on the services.concurrency (which is not documented well), again you need to guess and redeploy until it works Maybe I've run into an edge case with a JVM app where 256MB of RAM is tight, but my problem is more that all the feedback I get from the tooling is a little meh and doesn't make me confident to ever try production workloads here. Not complaining about the free service though and the dashboard/metrics stuff seems nice.
- sieabahlpark 4y ago
  [dead]
- afandian 4y ago
  I never got as far as running my Spring Boot application on fly.io. Killed after a few seconds. I’d love to know why. https://community.fly.io/t/java-app-is-killed-on-startup/8379/6 https://community.fly.io/t/java-app-is-killed-on-startup/837...
  - wink 4y ago
    Mine's a Clojure app and I'm running it via `java -Xmx180m -Xms180m` and my fly.toml has a kill_timeout of 15 and in services.tcp_checks I now have interval 25000 and timeout 9000 - but as I wrote, it's a very basic app, anything JVM on 256MB has always (even years before fly.io) been hit or miss, sadly. Good luck!
H12 4y ago
With all the talk of PostgreSQL, I'm curious if anyone has tried the offerings Fly.io has been pushing with SQLite (Litestream & LiteFS) in any serious capacity, and can comment on the experience. Wondering whether they would be a viable alternative to Postgres in the typical "Database + Docker Container" architecture.
- benbjohnson 4y ago
  Litestream & LiteFS author here. Tailscale has been using Litestream in production for a while and blogged about it in April 2022[1]. I've had a number of folks contact me out-of-band and tell me they're using in production at a surprising scale. As for LiteFS, it's still early beta stage so I don't know of many people using it in a production capacity. I would expect it to be another six months or year before confidence builds enough to see more regular usage. [1]: https://tailscale.com/blog/database-for-2022/ https://tailscale.com/blog/database-for-2022/
strzibny 4y ago
Nice post. I also documented steps how to run Rails application if someone needs it: https://businessclasskit.com/docs/how-to-deploy-rails-sidekiq-fly-io/ https://businessclasskit.com/docs/how-to-deploy-rails-sideki...
l5870uoo9y 4y ago
> The main motivation to migrate was cost. Is Fly.io really than much cheaper for hosting CDN/Postgres/Docker Container?
- itake 4y ago
  I saved about 25% from Heroku, but fly is much less reliable. I've had TLS certs expire without warning and redis has timeouts
  - biorach 4y ago
    > redis has timeouts I see closed connections on low traffic redis connections - for now a dumb catch-and-reconnect is doing ok
    - itake 4y ago
      This was the problem I was experiencing: https://community.fly.io/t/redis-socketclosedunexpectedlyerror-socket-closed-unexpectedly/5897 https://community.fly.io/t/redis-socketclosedunexpectedlyerr... Taking this thread's advice, I started self-hosting on flyio and have only seen 1 timeout in ~2 months.
  - mrkurt 4y ago
    Did we diagnose the TLS certs expiring? This is most often something like a CAA record that fails invisibly. We're not very good at notifying you of these kinds of issues, but certs should always renew when the config remains good. People who proxy through cloudflare hit this a lot, unfortunately.
    - itake 4y ago
      CF is my DNS. Not sure if I had the proxy enabled. I was able to re-configure everything and it started working. I made a reminder to check around the 65 day mark to see if it successfully renews. I suspect what may have caused the issue is I have multiple fly hosts on subdomains, but each subdomain had a wildcard certificate against the root (*.example.com).

252 points by sausagefeet 4y ago

No where do I see mentioned how much was actually saved in dollar figures. What are we talking here—thousands, hundreds? Similarly, I'd be interested in knowing engineering effort involved in the migration. As a software consultant focusing on designing, building, and deploying software on AWS, more often than not, the infrastructure cost issues I've seen have less to do with the underlying infrastructure provider — AWS in this case — and more to do with the application itself that's being deployed. Most recently, we were able to shave off 50% of a client's bill that was using Lambda for compute. The issue? Several, including sleep timers (on a service where you pay by the millisecond!) as well as pathological code (i.e. consume message from queue, enqueue same message, rinse and repeat). Yes. You can save money by transitioning to another provider. But I'd start with reviewing the underlying code and architecture first

mrdoops 4y ago
The biggest $ cost I'd have on my radar WRT AWS vs Fly is complexity and need-to-know of employee time to do any given thing. Fly is way easier to navigate and use than AWS and its labyrinth of Cloud Scale™ horrors. The trade-off being you can do more given things with AWS.
- biorach 4y ago
  Yeah, this. AWS enables you to do anything you can think of, at any scale and complexity, given enough time spent reading their documentation and crafting terraform configurations. Fly makes it really easy to do the 80% of things that most small/medium operations need to just get done now.
  - deleted 4y ago
    [deleted]
- yodon 4y ago
  >Fly is way easier to navigate and use than AWS and its labyrinth of Cloud Scale™ horrors Give fly.io a couple years and it will end up forced to take on all the complexity and edge cases AWS and Azure have to deal with today. A shiny new codebase is always nicer than an old one, because the old one has had to accommodate all those pesky customer needs.
  - Existenceblinks 4y ago
    You think it's inevitable? I think it's unlikely, they probably run into a different kind of targets. Those "edge cases" horror seem unnecessary like permission hell, pricing model hell (and too many size)
  - rendaw 4y ago
    Did that ever happen to Heroku? Ignoring 10% of use cases for their target customers and making the 90% doesn't sound crazy to me.
- intelVISA 4y ago
  I can't wait for Fly to disrupt the market of Cloud Native Eldritch Horrors™ Their SLA's a bit too iffy for my use case atm but I wish them well in this noble endeavor.

throwthere 4y ago

The article mentions this, but that’s strongly enough in my opinion. With fly you have no managed PostgreSQL. In my opinion, it’s not really comparable on cost to aws when Postgres is in the stack. You do read about interesting hacks where someone will set up rds in a region that may be single digit milliseconds away from a fly region. then, presumably, you could put PG bouncer on a sort of bastion host that connects to the fly wire guard VPN. But obviously, there’s no guarantee that the latency will always be that good.

TheDong 4y ago
> But obviously, there’s no guarantee that the latency will always be that good That guarantee already doesn't exist with RDS. The RDS master is in one AZ, your server may be in another (and if it isn't, well, RDS will fail over to another AZ eventually). The network latency between AWS AZs is usually good, but it can be arbitrarily high, up to full outages between AZs. This is just a measure of degrees. Your application and/or business already has to handle some degree of network issues (aws network outages), so it's a tradeoff of whether the extra hops increase the chance enough to matter.
mike_d 4y ago
https://fly.io/docs/postgres/ https://fly.io/docs/postgres/ It isn't "Managed Postgres," but the differences are minimal. RDS is ultimately a solution for people who look in the mirror and confidently say "you don't know how to run a database."
- Octabrain 4y ago
  > RDS is ultimately a solution for people who look in the mirror and confidently say "you don't know how to run a database." I think this is a terrible oversimplification and something tells me that you haven't had to deal with a complex database setup from an operations perspective. RDS reduces a huge overhead in terms of operations (ha, backups, upgrades and clustering being the first ones that come to my mind). Between RDS and running a database on a virtual machine(s) and manage it with, let's say, Ansible for providing the four aforementioned features I would chose RDS any day of the week.

viraptor 4y ago

I missed when this was added: > if you configure your application to expose a Prometheus endpoint, those metrics will automatically show up on your Grafana dashboard But that's such an amazing and simple idea to integrate observability. Love the approach.

deleted 4y ago
[deleted]
fastball 4y ago
Isn't this just how Prometheus works alongside Grafana? I've never used fly.io but I've used that exact setup with these two services.
- alexeldeib 4y ago
  I believe GP is commenting on fly’s integration. Fly comes with built in Prometheus scraping and prepopulated grafana dashboards for VM resources some connection/request metrics. I believe the value is that if you simply expose a prom endpoint, they’ll scrape it and you can use their grafana (or hook your own into their Prometheus). Nothing special vs vanilla prom/grafana, but seamless “no-ops” integration vs DIY.
  - viraptor 4y ago
    Correct. It's common for companies to have their own custom systems or specific agents, then build a very custom and limited dashboard on top. Fly.io went with "you know grafana and prometheus? yeah, we'll do just that". And I think that's perfect.
kitd 4y ago
Openshift does something similar (though you need to annotate your deployments). But yes, the simplicity is refreshing.

jokethrowaway 4y ago

I found debugging setup errors on fly to be incredibly frustrating You can't even ssh until the container started and they have some requirements on hostnames (0.0.0.0) At least a few of their examples are broken. The documentation could be better but it's pretty good Is it better than Aws? Yes of course, you need to spend way more to get a dx as bad as Aws. Setting up a VPS is a smoother experience than fly, though. Once it's running, it's lovely and reminds me of heroku.

biorach 4y ago
> You can't even ssh until the container started In fairness, I'm struggling to think of a way they could get you to ssh to a container that hasn't started yet > and they have some requirements on hostnames (0.0.0.0) Can you clarify?
tptacek 4y ago
Help me understand what you'd be SSH'ing into before your VM boots up? (I'm responsible for most of our SSH goop, so if there's an improvement to be made, I'm happy to make it). A reminder: we don't run "containers". We take containers, unpack them, and transform them into VMs. There's no host OS for you to access.
- indigodaddy 4y ago
  Maybe he means/wants some kind of “console” access for his “instance” ? Seems crazy as it takes I’m sure less than a second to boot, and if there is a problem booting up it’s probably not going to be a problem that a user/customer would be able to solve at console.
- mostly_lurks 4y ago
  Caveats: 1) this may not be what grandparent was trying to do and 2) I have limited experience debugging "regular" Docker, let alone firecracker or whatever Fly.io does. With that said, I succesfully deployed a simple Rust/Actix/Sqlite app with a Dockerfile to Fly.io. I then thought I'd try out litestream. For reasons I'm still not sure about, having `ENTRYPOINT ["litestream replicate … -exec myapp"]` resulted in immediate kernel panics.[1] As I was debugging, my instinct was to want to `fly ssh console` into a running container to see if running the same command from the shell produced any clues for further debugging. Though I understand this is the wrong mental model, the thought was something like "well, I know everything else works, I just need Fly to ignore this failing command so I can have a minute to poke around." To do this, I ended up just removing litestream from the ENTRYPOINT so the deploy would succeed, then I could SSH in and play around at the shell to see what was going on. Again, I have no idea whether this is the same sort of problem the other person was having, but for my case, what would be helpful is probably not changing how `fly ssh console` behaves, but perhaps some documentation of suggested debugging techniques in case your app is failing to start. [1] Putting the exact same command into a "start.sh" and making that the ENTRYPOINT worked fine so that's what I ended up doing.

sosodev 4y ago

I recently migrated some services from DigitalOcean to Fly and it was quite pleasant. My only issue is that routing to Fly from Southern California is awful. My friend and I have different ISPs and yet we’re both routed to the east coast instead of the much closer Los Angeles edge server. So our requests end up going to the east coast(edge), then back to the west coast (app), back to the east coast(edge), and then back to west coast (origin). The latency adds up and makes requests to a nearby app almost as slow as to one hosted across an ocean. I’m worried that other users will hit this and make hosting on Fly terrible for apps that require lower latency :/

whalesalad 4y ago
What ISP's are involved here?
- sosodev 4y ago
  Cox and Spectrum. They route slightly different but both to the east coast. I linked the forum thread above.
mrkurt 4y ago
This is uncommon. When did it happen? We had some cross country routing issues a few weeks ago. We can generally fix these if you don't mind sending us traveroutes.
- ryanisnan 4y ago
  This is cool, I'd love to know more about how you generally fix issues like these?
  - mrkurt 4y ago
    I usually just go complain to network folks and they figure it out. I'm good at complaining! Routing issues like this are typically the result of some weird network provider politics. The fix is typically ti change how we announce IPs to force a route to do something we want. It's a dark art.

biorach 4y ago

I'm not sure this is well-known enough yet - fly now supports scale-to-zero services using fly machines - https://fly.io/docs/machines/ https://fly.io/docs/machines/

hamandcheese 4y ago
Does scale to zero work with ordinary fly apps yet?
- biorach 4y ago
  Probably. Depends. For a simple app should be straighforward. I set up a machine - a few weeks ago this involved a few direct api calls using curl - maybe `flyctl` can do it all now. Then I deployed on that machine a version of my app that gracefully shut down if no requests were received for 60s. Works great. Shuts down after a minute, boots back up in under a second when a request comes in. See the docs for details
  - TheHappyOddish 4y ago
    I found the docs difficult and inconsistent when moving between flyctl and cURL when I tried playing with machines a few months ago. I daresay they'll be wonderful once the team get the machine stuff working under flyctl fully. Their generous free tier does make it easy to play and assess though.
- mrkurt 4y ago
  No, ordinary Fly apps (what we call v1 apps) are managed by nomad. They don't scale to zero. You can create a machine based app and it'll scale to zero just fine, but you lose most of the magic of `fly deploy` right now.
rendaw 4y ago
I've been trying fly machines with terraform and running into one bug after another: https://github.com/fly-apps/terraform-provider-fly/issues/125 https://github.com/fly-apps/terraform-provider-fly/issues/12... https://github.com/fly-apps/terraform-provider-fly/issues/126 https://github.com/fly-apps/terraform-provider-fly/issues/12... https://github.com/fly-apps/terraform-provider-fly/issues/127 https://github.com/fly-apps/terraform-provider-fly/issues/12... I was really excited since fly seems dead simple, but I haven't gotten it running yet...

alberth 4y ago

Isn’t Fly.io hosted on AWS (they don’t have their own hardware). Just like Heroku.

hamandcheese 4y ago
No, I don't think they are. The things they do with Anycast IPs I don't think is possible on AWS.
mrkurt 4y ago
We have our own hardware.
- alberth 4y ago
  That’s awesome to hear! Thank you btw for all your funding/support of the Erlang/Elixir ecosystem.
- garganzol 4y ago
  Stay cool, Fly. You guys will rule the world!

nomilk 4y ago

I moved a rails MVP from heroku to fly.io. Biggest negative so far: no easy way to reset the database [1]. I'd typically like to blow it away and re-create it frequently as I tinker with it e.g. adding/removing/renaming a column or changing a data type. Those small changes now require migrations, which are a fair bit more work (but not a huge problem tbh). Biggest positive: ability to select a zone meant response time for Australian users was very fast compared to heroku; making a big difference to UX. [1] https://stackoverflow.com/q/74664822/5783745 https://stackoverflow.com/q/74664822/5783745

rtpg 4y ago
This is stuff where I highly recommend writing up helper scripts. I have one to pull in stuff from a fly Postgres DB, spin up my local docker compose setup, and then pump the data into there. I think that's the real niceness here, a CLI that is pretty straightforward (not perfect by any means! but I can still wrap my head around it)
- nomilk 4y ago
  Sounds awesome/slick. If you know of any examples that can be shared would love to see them. My use of the fly CLI is rudimentary at best, and I don't use scripts at all, just (mostly basic) commands.

mtlynch 4y ago

I've migrated several services from GCP to Fly.io, and I've been very happy with Fly. All of my apps are low-volume hobbyist web apps that run on a single machine, so I don't do any Terraform, k8s, or Postgres, so my use case is a little simpler than OP. My biggest complaint has been with outages,[0, 1, 2] but that's gotten better. The original architecture meant that larger servers could evict other users' running servers, which they admitted was a bad idea.[2] I'm not sure if they've fixed that since, but I haven't seen it happen in a few months. >The container logging solution provided by Fly.io is basic. It's easy to view logs with the Fly.io CLI and via the Fly.io dashboard. However, there's only a small window of logs that are kept, forcing you to create a remote logging solution either internal to your application or via a separate Fly.io application that ships logs to an external service. This is a piece of operational overhead that I'd like to see removed. I very much agree. This is a big feature gap I feel when using Fly as well. >Fly.io is not a support-first company. If this bothers you, then Fly.io is probably not for you. >When emailing support, and email is the only option, it can take from hours to days to get a response. Sometimes they don't follow up. It does not feel like their level of support is on par with other cloud providers. This hasn't been my experience. I go through the forum rather than email, but I typically see responses within hours, even on the weekends. By contrast, when I reported bugs to GCP through their designated support channels, there was multi-day latency, and they'd just keep dismissing my report.[3] [0] https://community.fly.io/t/cant-deploy-my-app-in-iad/7746 https://community.fly.io/t/cant-deploy-my-app-in-iad/7746 [1] https://community.fly.io/t/application-vms-down-without-any-change-cant-deploy/7618/11?u=mtlynch https://community.fly.io/t/application-vms-down-without-any-... [2] https://community.fly.io/t/app-stuck-in-pending-state-after-restarts-rescaling/5972/6?u=mtlynch https://community.fly.io/t/app-stuck-in-pending-state-after-... [3] https://stackoverflow.com/q/53410165/90388 https://stackoverflow.com/q/53410165/90388

mrkurt 4y ago
You probably had a different community experience than recent folks. We've grown like 4x in the last 3 months and haven't kept up with community as well as I'd like (because hiring). And over holidays, we've had a difficult time with email support as well. It'll get better, but it's not great right now!

philliphaydon 4y ago

So it seems the 'machine' is the equiv of AWSs EC2? I think I would only save $5~ by not having to have HAProxy (cos AWS LB is exensive) but I would need to manage PG myself which seems like a hassle for a hobby project.

tommoor 4y ago

Fly is great. I know the Fly team has an aversion to it but I really just wish they would hire some database folks and take on managed Postgres. I'm already running 8+ apps on there and it would be the peace of mind needed to move the rest across… one day.

mrkurt 4y ago
I'm not completely averse (we've done this before!), it's just not how I think the world should work. If we hold out, and the future Fly cloud has a better-than-RDS Postgres option, it will be worth it. If we build our own managed Postgres, no one better is going to bring their stuff to our platform. Same with MySQL.
- subleq 4y ago
  I'd trust a first-party managed postgres built by fly more than a third-party one.
  - belmont_sup 4y ago
    I like their strategy of outsourcing what they’re not best at to the best. They partnered with upstash, and now they’ve got a new feature almost immediately. Sure there’s a lot of vetting, but wrapping products is common. I’d be happy if they can wrap Neon’s serverless postgres service into their networks somehow.
    - nikita 4y ago
      Neon CEO here. We are working on becoming first party service on several major problems now. We will be happy to provide this for Fly. Kurt and I have been chatting. Fly wants cross region replication which is coming soon. Once it's there we can integrate. With that Neon is still on AWS and can be close to Fly, but not run on Fly servers. It's relatively straightforward for us to run on Fly, but the S3 part will still be Amazon.
      - 4y ago

habibur 4y ago

Would have loved to read how much money they saved from this migration. The number's not there, unfortunately.

yashap 4y ago
Author mentioned here that they went from $200/month on AWS to $90/month of Fly: https://news.ycombinator.com/item?id=34242923 https://news.ycombinator.com/item?id=34242923 Pretty tiny cloud costs either way, probably safe to assume they’re a small, early stage startup. But even still, it was likely a bad call to spend significant engineering effort on saving $110/month, which is a rounding error even for small startups. Hard to imagine that time couldn’t have been better spent on things that grow the business, like new features and bug fixes.
- lawnchair 4y ago
  Definitely tiny but we're just overall happier with Fly. The migration in total probably cost us a weekend of work all said and done. We get a lot for free with Fly (observability, multi-region, etc.) so all in all a win.
  - yashap 4y ago
    Ah fair enough, that’s a pretty small amount of work!
- habibur 4y ago
  I would read it as a 60% saving. That's a very good one. Considering that this is a recurring cost and not a one time and it's proportional to their sales. Aka, they now will be able to provide their clients twice the amount of contents for the same price. At least in theory.
  - yashap 4y ago
    My thought was more timing - there’s so much that’s URGENT for a young startup, is cost cutting really urgent, something you have to do NOW, when the current savings are just $110/month? Part of what allows startups to survive is ruthless focus on the work that that’s most important today, and saving $110/month seems unlikely to be that. However, the blog post author responded, noting that this migration took only 1 weekend, and they like other aspects of Fly than just the cost. So fair enough, 1 weekend is a small investment, less than I expected, I would have guessed this took 2+ weeks.

garganzol 4y ago

For me, Fly.io is like a seminal step forward to an imaginary OS of the future: standardized, auto-clustered, just works, dead simple to grasp. Kind of Windows 95 of its time.

deleted 4y ago
[deleted]

memset 4y ago

Since the fly folks are here: I'd love the ability to trigger jobs as I can with aws batch. The ability to do `fly run -a journalize-batch --command="python3 upload.py"` is something I need right now and am in the process of setting up aws batch to do it.

mrkurt 4y ago
You can probably hack together with machines, though we're not a great fit for big bursts of compute. "fly machine run" is fun.

boundlessdreamz 4y ago

Since the fly folks are here, do you plan to support monorepo apps in a single fly.toml. i.e define all services in a single fly.toml like docker compose or render's blueprint? Also a way to reuse the definition but run a different command. For example for a rails app, web worker (puma) and job worker (sidekiq) differs only in the command run. In fly, I would have to duplicate those. If there is an easier way, do let us know.

mrkurt 4y ago
The second case has experimental support already: https://fly.io/docs/reference/configuration/#the-processes-section https://fly.io/docs/reference/configuration/#the-processes-s... The first, we're not completely sure about yet. Probably someday, though.
deleted 4y ago
[deleted]

jesuspiece 4y ago

> Fly.io is not a support-first company. If this bothers you, then Fly.io is probably not for you. > When emailing support, and email is the only option, it can take from hours to days to get a response. Sometimes they don't follow up. It does not feel like their level of support is on par with other cloud providers. This enough to keep me on AWS. You can say a lot about AWS but their support is great

Aeolun 4y ago
Responsive at least. They’re very quick to inform you that x is a known issue with their service, and the request has been dispatched to the dev team.
deleted 4y ago
[deleted]
wpietri 4y ago
Yeah, I'm ok with "hours to days" of mysterious downtime for some personal projects. But for my day job? Deal killer for sure.

chaychoong 4y ago

I would love to try fly.io but unfortunately it doesn't support GPU workloads yet. Waiting eagerly!

deleted 4y ago
[deleted]

lawnchair 4y ago

Author here. Some folks wanted the details on pricing. Pre migration, the Terrateam AWS monthly bill was about $200/mo. With Fly.io, we're paying around $90/mo.

moltar 4y ago
Could you break down the costs of AWS? From your post you say you needed a simple service plus a Postgres instance. I think you can have that for around $50 on AWS using a small EC2 and Aurora Serverless v2 (min $40/mo)
- lawnchair 4y ago
  I don't have access to the bills anymore. Two AZs, RDS HA, bandwidth, and EC2 all added up. It's possible I could have cut down the bill. The initial motivation to migrate was cost, but at the end of the day we're just happy with the Fly.io developer experience. It's a better fit for us.
hknmtt 4y ago
61$/m for 1vCPU and 8GB ram? Try Hetzner next time, on bare metal, for cheaper.
- dist1ll 4y ago
  You can get 1vCPU + 2GB RAM for 3.92 Euro/month at Hetzner. Those prices are hard to beat.
  - dxxvi 4y ago
    You can get 4 virtual ARM CPU + 24GB RAM + 200GB hard disk for $0/month at Oracle cloud. Nothing can beat that.
    - dist1ll 4y ago
      Any caveats? This sounds really great.

tuananh 4y ago

if the author says it doesn't take more than few hours + DX is really better, who are we to judge :)

calltrak 4y ago

[dead]

wink 4y ago

I've been running a hobby project on their free tier for a while and it's mostly cool. Wrote up some stuff back then: https://f5n.org/blog/2022/trying-out-some-hosting-options/ https://f5n.org/blog/2022/trying-out-some-hosting-options/ but unfortunately none of my pain points/caveats seem to have been fixed * deploying (re)builds your container but doesn't even tag it locally, so if you just built one, it will build the exact same one again and on the other hand you are left guessing which hash is the container that the tool just built. * if you start with your own Dockerfile the mandatory options in fly.toml are not described overly well, so a bit of fiddling * recently I redeployed and without any real change (just a new code build) it wouldn't deploy, I had to remove some of the settings in fly.toml that I had guessed at to make it work in the summer to make it work now - no changelog anywhere * if your app is coming close to some assumed limit on the services.concurrency (which is not documented well), again you need to guess and redeploy until it works Maybe I've run into an edge case with a JVM app where 256MB of RAM is tight, but my problem is more that all the feedback I get from the tooling is a little meh and doesn't make me confident to ever try production workloads here. Not complaining about the free service though and the dashboard/metrics stuff seems nice.

sieabahlpark 4y ago
[dead]
afandian 4y ago
I never got as far as running my Spring Boot application on fly.io. Killed after a few seconds. I’d love to know why. https://community.fly.io/t/java-app-is-killed-on-startup/8379/6 https://community.fly.io/t/java-app-is-killed-on-startup/837...
- wink 4y ago
  Mine's a Clojure app and I'm running it via `java -Xmx180m -Xms180m` and my fly.toml has a kill_timeout of 15 and in services.tcp_checks I now have interval 25000 and timeout 9000 - but as I wrote, it's a very basic app, anything JVM on 256MB has always (even years before fly.io) been hit or miss, sadly. Good luck!

H12 4y ago

With all the talk of PostgreSQL, I'm curious if anyone has tried the offerings Fly.io has been pushing with SQLite (Litestream & LiteFS) in any serious capacity, and can comment on the experience. Wondering whether they would be a viable alternative to Postgres in the typical "Database + Docker Container" architecture.

benbjohnson 4y ago
Litestream & LiteFS author here. Tailscale has been using Litestream in production for a while and blogged about it in April 2022[1]. I've had a number of folks contact me out-of-band and tell me they're using in production at a surprising scale. As for LiteFS, it's still early beta stage so I don't know of many people using it in a production capacity. I would expect it to be another six months or year before confidence builds enough to see more regular usage. [1]: https://tailscale.com/blog/database-for-2022/ https://tailscale.com/blog/database-for-2022/

strzibny 4y ago

Nice post. I also documented steps how to run Rails application if someone needs it: https://businessclasskit.com/docs/how-to-deploy-rails-sidekiq-fly-io/ https://businessclasskit.com/docs/how-to-deploy-rails-sideki...

l5870uoo9y 4y ago

> The main motivation to migrate was cost. Is Fly.io really than much cheaper for hosting CDN/Postgres/Docker Container?

itake 4y ago
I saved about 25% from Heroku, but fly is much less reliable. I've had TLS certs expire without warning and redis has timeouts
- biorach 4y ago
  > redis has timeouts I see closed connections on low traffic redis connections - for now a dumb catch-and-reconnect is doing ok
  - itake 4y ago
    This was the problem I was experiencing: https://community.fly.io/t/redis-socketclosedunexpectedlyerror-socket-closed-unexpectedly/5897 https://community.fly.io/t/redis-socketclosedunexpectedlyerr... Taking this thread's advice, I started self-hosting on flyio and have only seen 1 timeout in ~2 months.
- mrkurt 4y ago
  Did we diagnose the TLS certs expiring? This is most often something like a CAA record that fails invisibly. We're not very good at notifying you of these kinds of issues, but certs should always renew when the config remains good. People who proxy through cloudflare hit this a lot, unfortunately.
  - itake 4y ago
    CF is my DNS. Not sure if I had the proxy enabled. I was able to re-configure everything and it started working. I made a reminder to check around the 65 day mark to see if it successfully renews. I suspect what may have caused the issue is I have multiple fly hosts on subdomains, but each subdomain had a wildcard certificate against the root (*.example.com).