10 ms·

Show HN: Fly – A global load balancer with middleware

112 points by mrkurt 10y ago

thomcrowe 10y ago
I've been running Fly in beta for one of my apps and it's pretty damn cool. Adding a couple more today.
- romanovcode 10y ago
  I don't get it. What's the difference between this and cloudflare?
  - mrkurt 10y ago
    Fly is way more "aware" of underlying applications than Cloudflare. We offer most of the same quick wins (easy SSL, global termination). We also replace your local load balancers, handle service discovery (just boot an app process with our agent), and let you run middleware at the edge. Our site is a good example: the static docs are hosted on Github Pages, the landing page is a middleware + Rack app, and our dashboard is a Rails app hosted on Kubernetes. All of these are mounted under https://fly.io https://fly.io. The edges are aware of the session cookies Rails uses and we have routing rules to send people to the right backends if they're logged in. It's early, but the middleware is already super powerful.
    - thomcrowe 10y ago
      What /u/mrkurt said.
    - bpicolo 10y ago
      I don't see docs related to service discovery. If that's part of the package, sounds like it's more front-page worthy, even.
      - mrkurt 10y ago
        We have some docs about it here: https://fly.io/docs/agents/ https://fly.io/docs/agents/ It doesn't say service discovery (it should!). This will mostly let us build some neat features, though. We can let people build rules to send traffic to a particular branch.
        bpicolo 10y ago
        Phrase that in terms of A/B testing and it's probably a pretty easy selling point.
garry 10y ago
This lets you do smart obvious things like route /blog on your home domain to your blog provider. There's a lot of stuff you can do that makes your website better and faster using a smart load balancer and so this is kind of an obvious missing link in web stacks. Glad that Kurt and Jerome are working on this!
secstate 10y ago
I'm sure the tech is really neat, but after the last CloudFlare debacle, piping all your HTTP requests through a for profit company just seems like it's ripe for abuse. This is not meant to slander Fly, and I'm sure the devs have all the best intentions in the world, but the internet will only become more brittle the more services like this or CloudFlare become the arbiters of vast swaths of TCP packets.
- mrkurt 10y ago
  This is a real concern. We think about it a lot. In fact, we have a fairly nice way of isolating customers who can afford global server installs from other customers on shared infrastructure. Unfortunately, setting servers up across the world is really complicated and expensive. There's a definite trade off — we want to make this kind of power accessible to every developer, and running shared infrastructure is the only way we've found to do it. I do think apps are more secure by default running through fly than most load balancer services. Traffic between visitors and our edges is SSL (unless a customer opts out, at which point we nag them), application processes establish a secure tunnel back to our edge nodes to handle traffic. There are no network hops between fly and customer apps that happen in the clear. Replicating this with a normal load balancer setup means handling SSL termination for visitors and terminating SSL with client cert verification between load balancers and app servers. It's a pain, and most people don't get it right because it's too finicky. But I don't want to disregard your concern either. It's definitely something we think about a lot. We would like to figure out a way to offer all of this power easily, cheaply (for developers), and in a way that is entirely opaque to us. As soon as we figure out how, we will.
  - secstate 10y ago
    Thanks for the thoughtful reply :) At the end of the day, I suppose there's a certain inevitability to the sort of service you're providing, and I'd rather there were 20 smaller companies, than just CloudFlare. So carry on, and I'll check out your tech when I have a chance!
- markbnj 10y ago
  Aren't virtually all of us doing this anyway? Who doesn't pipe all their http requests through a for profit company? Given that such a large percentage of web-accessible applications are on the cloud these days, most people are probably using some sort of managed ingress solution from their provider.
moondev 10y ago
Looks like a very interesting product. I can see how off-loading middleware can really make teams more productive. Can you go into more detail on how you are running this on k8s? A bunch of federated clusters around the world? SSH tunnels in sidecars?
- mrkurt 10y ago
  We're actually only running some of it in k8s, the edge nodes are ansible managed bare metal. We do however, work really really well with k8s apps. Our apps are all k8s deployments. We have a user facing dashboard, a marketing app, and a backend administrative dashboard. Sidecar pods would work fine for our agents, but we actually bundle them with the apps ... this lets our edges detect release version, branch info, etc (that stuff is easy to read from git, hard to detect from a sidecar). Those agents connect to the nearest edge to setup a tunnel, then we use very fast backhaul between edges to connect users back to apps. We actually started by running the edges as k8s nodes, and even toyed around with federated clusters. It was a bit unwieldy and had a few too many layers of indirection between visitors and our proxies so we simplified.
  - moondev 10y ago
    Very cool! Thanks for the info.
mrkurt 10y ago
(founder here) We've been shipping apps for many years now, and every time around we're frustrated by a rudimentary routing layer. We think a proxy/CDN can do a lot for devs, and Fly is an early look at the power a smarter routing layer can give to developers. Fly connects visitors directly to apps, handles all the chores, and provides edge-level middleware to help devs ship features faster. It's designed to run in place of local load balancers like nginx and haproxy, and works as both a load balancer and CDN for dynamic apps (we have servers in 6 datacenters, more soon). For me, the most interesting part is the middleware, and this is what we expect to really matter long term. You're probably familiar with local app middleware to provide auth, modify content, do analytics, etc, etc. Most of these can actually run at the edge, and Fly currently has middleware for: * Server side Google analytics * Identity Aware Proxy (require auth to even load an app) * Geo data / connection speed information as headers * Session cookie knowledge * Routing rules You can mix and match these. If your app is getting DDoS'ed from a botnet in China, for example, you can use the geo data middleware + a routing rule to send all traffic from China to a higher capacity. backend. Or, you can extract a user_id from session cookie and send it to Google analytics. Or, send authenticated users to one backend (like your app) and anonymous users to another (like a static site hosted in GH pages). This is something we've wanted for a really long time, we've worked on Ars Technica, Compose, and even a bunch of low volume apps that would have been better with a service like this. Traffic routing is really powerful if applied properly, and we're hoping to give any developer the same flexibility as they'd have at one of the big 3 tech companies (who tend to build all this themselves).
- Dinius 10y ago
  The first heading overflows on mobile, hiding the "a" and "y" in "application delivery"and causing horizontal scrolling.
  - mrkurt 10y ago
    Thanks, it'll be fixed soon.
- FooBarWidget 10y ago
  Thanks for releasing this. How does this differ from just combining a CDN like Cloudflare, with an API gateway like Tyk or Kong?
Xorlev 10y ago
Wow, this is _really_ expensive! I'm sorry, but $0.05/1000 requests is really really premium. 1000 request/s is $0.05/s. 86400s/day * 30 days * $0.05 -> $129.6K/30d. 1k req/s is not that uncommon either. And that's before 'paid middleware'! APIs and sites that'd truly benefit from this are immediately priced out. I see why you priced it the way you did (gotta start somewhere with such a large upfront investment if you're not using cloud capacity), but you aren't exactly going to get any large customers with numbers like that. Project it to when you have ~500k req/s going through your service (778M ARR!): you cannot expect to still be making $0.05/1000 requests. To put it into perspective, costs to call S3 are 10x-100x cheaper (GETs are 0.004/10000, PUT/POST/LIST/COPY is 0.005/1000) and S3 arguably does a lot more per request. I really want to like it. The service sounds _awesome_, don't get me wrong. I'm a huge nerd for smart loadbalancing. I've written more loadbalancing code than I care to admit, but the economics just aren't there for me to even consider adopting your service. It's a lot cheaper for me to keep a pool of nginx servers in AWS regions around the globe with pre-established TLS connections back to the home DCs for much of the benefit fly.io brings.
- AtheistOfFail 10y ago
  Digital Ocean's new Load Balancers are $20/month, WTF is this thing?
- bpicolo 10y ago
  That makes 1k r/s about as much as 6-12 full time engineers to configure your load balancers.
- mrkurt 10y ago
  This math isn't right, we bill based on total requests (not req/s capacity). It's $0.05 per thousand requests. So 86k/day * 30 days / 1000 * $0.05 means $129/mo. Can we tweak the wording to clarify what we mean? We'll add examples, too. I really hate opaque pricing and wanted to make this as simple to understand as possible.
  - Xorlev 10y ago
    This math _is_ right. 1000 requests/s is 86.4M requests per day. So for 1000 requests in a given second, that's $0.05 per second if you substitute the term. But to show you otherwise: 1000 req/s * 86400s * 30d / 1000 * 0.05 = 129600 It's only $129/mo if it's 0.05c and not $0.05, which is not what the website says.
marcosscriven 10y ago
I just tried this, and the IP for my test site seems to resolve to a Chicago based ISP called Secure Central Network - is that the closest one to UK? Do you have a list of edge locations somewhere? The simple SSL cert setup seems novel, though the routing/middle stuff reminds me of Fastly: https://www.fastly.com https://www.fastly.com
- mrkurt 10y ago
  That's an anycasted IP. We use one pool of IPs globally and let BGP select the shortest route. If you run a traceroute you'll probably find that you hit a server in Amsterdam.
  - subway 10y ago
    You're running TCP over Anycast? Bold move.
willejs 10y ago
If anyone is considering this, consider fastly and custom VCLs too.
- jedisct1 10y ago
  +1 for Fastly.
- mrkurt 10y ago
  You can do some of what we do if you're a VCL wizard, but not much. I'm a recovering VCL wizard, it's part of the reason I wanted to build this. :)
tyingq 10y ago
This is very cool. I suspect though, you'll eventually be competing with CDNs that are adding API management features. You're starting off with middleware that's smarter than a typical CDN, but less smart than a typical API gateway. See https://tyk.io/tyk-documentation/get-started/ https://tyk.io/tyk-documentation/get-started/ for a good overview of some of the features of an API gateway. At a high level, an API gateway already has most of what your middleware does (tweaking requests and responses, injecting headers, modifying body, orchestrating multiple endpoints, etc). But, they add some things your middleware doesn't do, like smart conditional caching of api responses, rate limits, quotas, access control + tokens, web UI for management, and more. You have a pretty unique value proposition now, but if the various CDNs start to integrate API gateways, it may not be so unique. Might be worth looking at some existing API gateway software to quickly add to your feature list.
andrewbarba 10y ago
Looking at the response headers from your own site, everything is being served up from Fastly edge servers. Why wouldn't you serve your own site from your own infrastructure? The reason I even bothered to look at the response headers is because I'm a huge fan of Fastly and wanted to see what you guys were sending back to compare. Turns out they look identical because they are Fastly headers and Fastly edge servers. And maybe have a look at Fastly pricing: https://www.fastly.com/pricing https://www.fastly.com/pricing $0.0075 / 10,000 requests seems much more reasonable than your $0.05 / 1,000 requests...
- matahwoosh 10y ago
  We do serve our own site! We're running multiple apps on the same fly.io domain. Our landing page is currently hosted on Github Pages (and that's probably how you're seeing Fastly headers), but you're hitting our load balancer first :)
- goodroot 10y ago
  Fellow from Fly here, nice to meet you. We do serve our own site. Right now we run multiple applications on our domain. The landing page that you see on http://fly.io http://fly.io is hosted on GitHub pages, which explains Fastly. Our /docs/, too. You are hitting our Load Balancer first, though.
otterley 10y ago
> When your visitors explore your application, their HTTPS requests arrive at the closest regional edge node, buff up with Middleware, then intelligently route to your application processes over a hardened SSH tunnel. ssh is a terrible way to transfer bulk traffic. The send/receive buffer sizes are fixed to small sizes, so you can never get the benefits of TCP window scaling. So there will always be an artificial transfer rate ceiling that's directly proportional to the latency between the edge node and the origin server. You'll quickly find you'll have to use an ordinary TLS-over-TCP connection in order to attain adequate performance over high-latency connections.

112 points by mrkurt 10y ago

thomcrowe 10y ago

I've been running Fly in beta for one of my apps and it's pretty damn cool. Adding a couple more today.

romanovcode 10y ago
I don't get it. What's the difference between this and cloudflare?
- mrkurt 10y ago
  Fly is way more "aware" of underlying applications than Cloudflare. We offer most of the same quick wins (easy SSL, global termination). We also replace your local load balancers, handle service discovery (just boot an app process with our agent), and let you run middleware at the edge. Our site is a good example: the static docs are hosted on Github Pages, the landing page is a middleware + Rack app, and our dashboard is a Rails app hosted on Kubernetes. All of these are mounted under https://fly.io https://fly.io. The edges are aware of the session cookies Rails uses and we have routing rules to send people to the right backends if they're logged in. It's early, but the middleware is already super powerful.
  - thomcrowe 10y ago
    What /u/mrkurt said.
  - bpicolo 10y ago
    I don't see docs related to service discovery. If that's part of the package, sounds like it's more front-page worthy, even.
    - mrkurt 10y ago
      We have some docs about it here: https://fly.io/docs/agents/ https://fly.io/docs/agents/ It doesn't say service discovery (it should!). This will mostly let us build some neat features, though. We can let people build rules to send traffic to a particular branch.
      - bpicolo 10y ago
        Phrase that in terms of A/B testing and it's probably a pretty easy selling point.

garry 10y ago

This lets you do smart obvious things like route /blog on your home domain to your blog provider. There's a lot of stuff you can do that makes your website better and faster using a smart load balancer and so this is kind of an obvious missing link in web stacks. Glad that Kurt and Jerome are working on this!

secstate 10y ago

I'm sure the tech is really neat, but after the last CloudFlare debacle, piping all your HTTP requests through a for profit company just seems like it's ripe for abuse. This is not meant to slander Fly, and I'm sure the devs have all the best intentions in the world, but the internet will only become more brittle the more services like this or CloudFlare become the arbiters of vast swaths of TCP packets.

mrkurt 10y ago
This is a real concern. We think about it a lot. In fact, we have a fairly nice way of isolating customers who can afford global server installs from other customers on shared infrastructure. Unfortunately, setting servers up across the world is really complicated and expensive. There's a definite trade off — we want to make this kind of power accessible to every developer, and running shared infrastructure is the only way we've found to do it. I do think apps are more secure by default running through fly than most load balancer services. Traffic between visitors and our edges is SSL (unless a customer opts out, at which point we nag them), application processes establish a secure tunnel back to our edge nodes to handle traffic. There are no network hops between fly and customer apps that happen in the clear. Replicating this with a normal load balancer setup means handling SSL termination for visitors and terminating SSL with client cert verification between load balancers and app servers. It's a pain, and most people don't get it right because it's too finicky. But I don't want to disregard your concern either. It's definitely something we think about a lot. We would like to figure out a way to offer all of this power easily, cheaply (for developers), and in a way that is entirely opaque to us. As soon as we figure out how, we will.
- secstate 10y ago
  Thanks for the thoughtful reply :) At the end of the day, I suppose there's a certain inevitability to the sort of service you're providing, and I'd rather there were 20 smaller companies, than just CloudFlare. So carry on, and I'll check out your tech when I have a chance!
markbnj 10y ago
Aren't virtually all of us doing this anyway? Who doesn't pipe all their http requests through a for profit company? Given that such a large percentage of web-accessible applications are on the cloud these days, most people are probably using some sort of managed ingress solution from their provider.

moondev 10y ago

Looks like a very interesting product. I can see how off-loading middleware can really make teams more productive. Can you go into more detail on how you are running this on k8s? A bunch of federated clusters around the world? SSH tunnels in sidecars?

mrkurt 10y ago
We're actually only running some of it in k8s, the edge nodes are ansible managed bare metal. We do however, work really really well with k8s apps. Our apps are all k8s deployments. We have a user facing dashboard, a marketing app, and a backend administrative dashboard. Sidecar pods would work fine for our agents, but we actually bundle them with the apps ... this lets our edges detect release version, branch info, etc (that stuff is easy to read from git, hard to detect from a sidecar). Those agents connect to the nearest edge to setup a tunnel, then we use very fast backhaul between edges to connect users back to apps. We actually started by running the edges as k8s nodes, and even toyed around with federated clusters. It was a bit unwieldy and had a few too many layers of indirection between visitors and our proxies so we simplified.
- moondev 10y ago
  Very cool! Thanks for the info.

mrkurt 10y ago

(founder here) We've been shipping apps for many years now, and every time around we're frustrated by a rudimentary routing layer. We think a proxy/CDN can do a lot for devs, and Fly is an early look at the power a smarter routing layer can give to developers. Fly connects visitors directly to apps, handles all the chores, and provides edge-level middleware to help devs ship features faster. It's designed to run in place of local load balancers like nginx and haproxy, and works as both a load balancer and CDN for dynamic apps (we have servers in 6 datacenters, more soon). For me, the most interesting part is the middleware, and this is what we expect to really matter long term. You're probably familiar with local app middleware to provide auth, modify content, do analytics, etc, etc. Most of these can actually run at the edge, and Fly currently has middleware for: * Server side Google analytics * Identity Aware Proxy (require auth to even load an app) * Geo data / connection speed information as headers * Session cookie knowledge * Routing rules You can mix and match these. If your app is getting DDoS'ed from a botnet in China, for example, you can use the geo data middleware + a routing rule to send all traffic from China to a higher capacity. backend. Or, you can extract a user_id from session cookie and send it to Google analytics. Or, send authenticated users to one backend (like your app) and anonymous users to another (like a static site hosted in GH pages). This is something we've wanted for a really long time, we've worked on Ars Technica, Compose, and even a bunch of low volume apps that would have been better with a service like this. Traffic routing is really powerful if applied properly, and we're hoping to give any developer the same flexibility as they'd have at one of the big 3 tech companies (who tend to build all this themselves).

Dinius 10y ago
The first heading overflows on mobile, hiding the "a" and "y" in "application delivery"and causing horizontal scrolling.
- mrkurt 10y ago
  Thanks, it'll be fixed soon.
FooBarWidget 10y ago
Thanks for releasing this. How does this differ from just combining a CDN like Cloudflare, with an API gateway like Tyk or Kong?

Xorlev 10y ago

Wow, this is _really_ expensive! I'm sorry, but $0.05/1000 requests is really really premium. 1000 request/s is $0.05/s. 86400s/day * 30 days * $0.05 -> $129.6K/30d. 1k req/s is not that uncommon either. And that's before 'paid middleware'! APIs and sites that'd truly benefit from this are immediately priced out. I see why you priced it the way you did (gotta start somewhere with such a large upfront investment if you're not using cloud capacity), but you aren't exactly going to get any large customers with numbers like that. Project it to when you have ~500k req/s going through your service (778M ARR!): you cannot expect to still be making $0.05/1000 requests. To put it into perspective, costs to call S3 are 10x-100x cheaper (GETs are 0.004/10000, PUT/POST/LIST/COPY is 0.005/1000) and S3 arguably does a lot more per request. I really want to like it. The service sounds _awesome_, don't get me wrong. I'm a huge nerd for smart loadbalancing. I've written more loadbalancing code than I care to admit, but the economics just aren't there for me to even consider adopting your service. It's a lot cheaper for me to keep a pool of nginx servers in AWS regions around the globe with pre-established TLS connections back to the home DCs for much of the benefit fly.io brings.

AtheistOfFail 10y ago
Digital Ocean's new Load Balancers are $20/month, WTF is this thing?
bpicolo 10y ago
That makes 1k r/s about as much as 6-12 full time engineers to configure your load balancers.
mrkurt 10y ago
This math isn't right, we bill based on total requests (not req/s capacity). It's $0.05 per thousand requests. So 86k/day * 30 days / 1000 * $0.05 means $129/mo. Can we tweak the wording to clarify what we mean? We'll add examples, too. I really hate opaque pricing and wanted to make this as simple to understand as possible.
- Xorlev 10y ago
  This math _is_ right. 1000 requests/s is 86.4M requests per day. So for 1000 requests in a given second, that's $0.05 per second if you substitute the term. But to show you otherwise: 1000 req/s * 86400s * 30d / 1000 * 0.05 = 129600 It's only $129/mo if it's 0.05c and not $0.05, which is not what the website says.

marcosscriven 10y ago

I just tried this, and the IP for my test site seems to resolve to a Chicago based ISP called Secure Central Network - is that the closest one to UK? Do you have a list of edge locations somewhere? The simple SSL cert setup seems novel, though the routing/middle stuff reminds me of Fastly: https://www.fastly.com https://www.fastly.com

mrkurt 10y ago
That's an anycasted IP. We use one pool of IPs globally and let BGP select the shortest route. If you run a traceroute you'll probably find that you hit a server in Amsterdam.
- subway 10y ago
  You're running TCP over Anycast? Bold move.

willejs 10y ago

If anyone is considering this, consider fastly and custom VCLs too.

jedisct1 10y ago
+1 for Fastly.
mrkurt 10y ago
You can do some of what we do if you're a VCL wizard, but not much. I'm a recovering VCL wizard, it's part of the reason I wanted to build this. :)

tyingq 10y ago

This is very cool. I suspect though, you'll eventually be competing with CDNs that are adding API management features. You're starting off with middleware that's smarter than a typical CDN, but less smart than a typical API gateway. See https://tyk.io/tyk-documentation/get-started/ https://tyk.io/tyk-documentation/get-started/ for a good overview of some of the features of an API gateway. At a high level, an API gateway already has most of what your middleware does (tweaking requests and responses, injecting headers, modifying body, orchestrating multiple endpoints, etc). But, they add some things your middleware doesn't do, like smart conditional caching of api responses, rate limits, quotas, access control + tokens, web UI for management, and more. You have a pretty unique value proposition now, but if the various CDNs start to integrate API gateways, it may not be so unique. Might be worth looking at some existing API gateway software to quickly add to your feature list.

andrewbarba 10y ago

Looking at the response headers from your own site, everything is being served up from Fastly edge servers. Why wouldn't you serve your own site from your own infrastructure? The reason I even bothered to look at the response headers is because I'm a huge fan of Fastly and wanted to see what you guys were sending back to compare. Turns out they look identical because they are Fastly headers and Fastly edge servers. And maybe have a look at Fastly pricing: https://www.fastly.com/pricing https://www.fastly.com/pricing $0.0075 / 10,000 requests seems much more reasonable than your $0.05 / 1,000 requests...

matahwoosh 10y ago
We do serve our own site! We're running multiple apps on the same fly.io domain. Our landing page is currently hosted on Github Pages (and that's probably how you're seeing Fastly headers), but you're hitting our load balancer first :)
goodroot 10y ago
Fellow from Fly here, nice to meet you. We do serve our own site. Right now we run multiple applications on our domain. The landing page that you see on http://fly.io http://fly.io is hosted on GitHub pages, which explains Fastly. Our /docs/, too. You are hitting our Load Balancer first, though.

otterley 10y ago

> When your visitors explore your application, their HTTPS requests arrive at the closest regional edge node, buff up with Middleware, then intelligently route to your application processes over a hardened SSH tunnel. ssh is a terrible way to transfer bulk traffic. The send/receive buffer sizes are fixed to small sizes, so you can never get the benefits of TCP window scaling. So there will always be an artificial transfer rate ceiling that's directly proportional to the latency between the edge node and the origin server. You'll quickly find you'll have to use an ordinary TLS-over-TCP connection in order to attain adequate performance over high-latency connections.