19 ms·

Migrating Dropbox from Nginx to Envoy

427 points by SaveTheRbtz 6y ago

dmicher 6y ago
I know some people might find it a little controversial, but I’m super excited about our load balancing future and that we probably have the biggest Envoy deployment in the world now. When we moved most of Dropbox traffic to Envoy, we had to seamlessly migrate a system that already handles tens of millions of open connections, millions of requests per second, and terabits of bandwidth. This effectively made us into one of the biggest Envoy users.
- user5994461 6y ago
  Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling. What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have? I was running a comparable setup in a large company, all based on HAProxy, there was a significant amount of complexity in routing requests to applications that might ultimately be in any of 30 datacenters.
  - SaveTheRbtz 6y ago
    We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS, now Katran[4]) that encapsulate traffic and DSR it to L7 balancers (previously nginx, now mostly Envoy.) Overall, we have ~25 PoPs and 4 datacenters. [1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastructure-edge-network https://dropbox.tech/infrastructure/dropbox-traffic-infrastr... [2] https://dropbox.tech/infrastructure/intelligent-dns-based-load-balancing-at-dropbox https://dropbox.tech/infrastructure/intelligent-dns-based-lo... [3] https://www.dropbox.com/peering https://www.dropbox.com/peering [4] https://github.com/facebookincubator/katran https://github.com/facebookincubator/katran
    - user5994461 6y ago
      Great. Exactly what I was looking for =)
    - alexeldeib 6y ago
      Cool to see someone using Katran in production. Really interesting stack you have there.
      - SaveTheRbtz 6y ago
        Actually, all the props for that go to Katran's author himeself. When we hired Nikita V. Shirokov (tehnerd), the first thing he did was replacing IPVS with XDP/eBPF-based Katran, which improved our Edge servers throughput by 10x, from ~2MPps to ~20Mpps. He also contributed a lot to Envoy migration migrating our desktop client to it and adding perf-related thing like TLS session tickets' lifetime to SDS.
    - dilyevsky 6y ago
      Katran - nice! Any issues with it at all? Do you use it with xdp capable hardware or just normal driver offload?
      - SaveTheRbtz 6y ago
        It works beautifully. We use driver offload (i40e on the Edge.)
    - traceroute66 6y ago
      @SaveTheRbtz "we have an open peering policy" That's a bit of a lie given you have a minimum 50Mbps requirement before you even consider a peering request. I would call that Selective, not Open !
Havoc 6y ago
Sensing a bit of a trend here. Didn't another major player recently make the same switch?
- SaveTheRbtz 6y ago
  I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]: * Lyft (of course) * Spotify * Stripe * Square * eBay * Yelp * Pinterest Plus the support from major cloud providers: Google, Microsoft, and Amazon. [1] https://envoyconna18.sched.com/ https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/ https://envoycon2019.sched.com/
  - stock_toaster 6y ago
    So, seems like nginx is fine until your company reaches the "we are worth billions now" scale?
    - user5994461 6y ago
      nginx is never fine for load balancing, they put basic features like metrics behind the paid edition. It's not sane to operate in production. https://thehftguy.com/2016/10/03/haproxy-vs-nginx-why-you-should-never-use-nginx-for-load-balancing/ https://thehftguy.com/2016/10/03/haproxy-vs-nginx-why-you-sh...
    - donavanm 6y ago
      I work for a billions company. Nginx is still fine. Youll need to be prepared to pay for better operations, management, visibility, and protocol support. You can either pay them or build it in house, but you will want to pay.
  - user5994461 6y ago
    They must all be GRPC users. Developers are pushing GRPC and protobuf pretty hard in companies. The next step down the road is to move to envoy as the load balancer. Otherwise these protocols don't work well over traditional HTTP infrastructure.
rdli 6y ago
Really great post. I'm glad the post in particular mentioned community, because I think in the end this is the huge advantage Envoy has over NGINX. NGINX, could, in theory, resolve all technical issues raised in the post. But the fundamental tension between the open source and commercial versions cannot be resolved. (Disclosure: We use Envoy as part of Ambassador, and so of course we're big fans!)
caiobegotti 6y ago
I'm positively surprised that Dropbox (at least from what I understood from the post) didn't require lots of changes or patches on top of the upstream codebase of Envoy to migrate their traffic!
- SaveTheRbtz 6y ago
  We did require some of them[1]. Esp. painful were Transfer-Encoding quirks, and some dances around old HTTP/1.0 backends and request buffering. Compared to NGINX though, it was relatively easy to push these fixes upstream. Community is very welcoming to outside contributions. [1] https://dropbox.tech/infrastructure/how-we-migrated-dropbox-from-nginx-to-envoy#-issues-we-encountered- https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...
  - veshij 6y ago
    We do have some local patches as well (mostly for integration with out own infrastructure - stats collection, some RPC specific stuff). As SaveTheRbtz mentioned we encountered some issues with non-RFC clients, corner cases which were not exposed when envoy is used in "trusted" environment, etc., but all our fixes are now in upstream, so next migrations will be way easier both for us and for other envoy users.
deleted 6y ago
[deleted]
techntoke 6y ago
Who does Dropbox compete with these days? They have pretty much the highest prices for the least amount of value. The only reason I see them mentioned here frequently is their connection with Y Combinator.
- rmoriz 6y ago
  I've admired Drew and the early Dropbox team for getting things done and shipped even when compiled python GUIs was edgy as the initial Rails version of Twitter was. But they shipped and validated the market. Now adding all those fancy and cool tech mentioned in the blog post will increase the complexity by a lot but it's not clear what are the real benefits. Does a decreased number of machines running really justify the migration and addition of complexity? Maybe they have some new products in the pipeline that built upon the new stack. Or they waste their time. We will see.
  - sanxiyn 6y ago
    > Does a decreased number of machines running really justify the migration and addition of complexity? Of course it does. I am not sure why you think it doesn't.
    - rmoriz 6y ago
      servers are a commodity and cheap compared to runtime complexity
      - sanxiyn 6y ago
        Yes, but it's a lot of servers.
        rmoriz 6y ago
        How many? 5000 servers? As a public company it should not really matter at that size. Developers are way more expensive.
      - 6y ago
mperham 6y ago
Did you consider using commercial nginx? If so, what made you decide against it?
- xmichael0 6y ago
  The price is really insane for Nginx commercial.
  - user5994461 6y ago
    About $2000 per host.
  - mperham 6y ago
    As an Enterprise software vendor myself, I can assure you: everything is negotiable at Dropbox’s scale including very deep discounts.
- dmicher 6y ago
  Sadly, it would probably be as hard to maintain as an opensource version. We really want to have access to the code to make sure we can fix, troubleshoot it, understand it fast... Things that may've help: -- Configuration definition (e.g. protobufs.) -- More focus on observability: error metrics (instead of logs), tracing, etc. -- gRPC control plane. -- C++ module development SDK. -- (ideally) bazel. Some dataplane features like gRPC JSON transcoding, gRPC-Web, and http/2 to backends.
  - dmarble 6y ago
    Don't any of the major commercial open source vendors offer custom terms to give access to the commercial source? I'd imagine they'd contemplate it for big deals. Seems like one of the only ways to keep some of these sophisticated customers onboard.
    - SaveTheRbtz 6y ago
      open source argument is valid -- most software enterprise vendors do provide source code access (under NDA.) The rest of the arguments stand though: as it is right now, it way more developer/operator friendly to use Envoy in our production.
sroussey 6y ago
One thing nice about OpenResty (nginx) and their Lua support is that it plugs in at TLS negotiation. Does Envoy?
- SaveTheRbtz 6y ago
  Can you describe your use-case? If you are talking about the ability to select a certificate on the fly via `ssl_certificate_by_lua_block`[1] we are not aware of such functionality. If you are missing something, I would highly encourage you discuss it with the community on a github! From Oleg Guba, Traffic Team TL, co-author, and person driving the deployment: * ListenerFilters + NetworkFilters are flexible enough, that some of the custom logic could be just moved to the config. From Ruslan Nigmatullin, our head Envoy developer: If you are talking more about a custom verification code there is already couple of ways to do that: * Client TLS auth Network Filter: https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/client_ssl_auth_filter#config-network-filters-client-ssl-auth https://www.envoyproxy.io/docs/envoy/latest/configuration/li... * Alternatively, if you are writing C++ extension you can use Network::ReadFilter, Network::ConnectionCallbacks. [1] https://github.com/openresty/lua-nginx-module#ssl_certificate_by_lua_block https://github.com/openresty/lua-nginx-module#ssl_certificat... [2] https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md https://github.com/openresty/lua-resty-core/blob/master/lib/...
  - sroussey 6y ago
    Wordpress and others use this to load certain on the fly. When you are a multidomain host this matters a lot. You don’t just load up a million cents as files and restart the server (though I do know a company that does something like this, but man, quite brittle).
user5994461 6y ago
A shame they picked nginx in the first place, it has all the stats and critical features behind the paid edition. HAProxy is always a better choice for load balancing. Besides that, it looks like the move was significantly driven by GRPC and profobuf. No surprise here, GRPC really doesn't work well over HTTP. Once a company start using the google stack, they have to move to more of the google stack to make it usable.
- SaveTheRbtz 6y ago
  Our technology stack is very gRPC friendly, so developer experience is actually better with it, than without (though this is very subjective.) As for the middleboxes, using gRPC-WEB[1] allowed us to switch Desktop Client App to gRPC even behind firewalls/IDSes that do not speak HTTP/2 yet. As for the HAProxy, Dropbox used to use (circa 2013) it specifically for loadbalancing, but we eventually replaced it with our Golang proxy. That said, recent HAProxy improvements (v2.0+) make it quite an awesome dataplane and an excellent loadbalancer! [1] https://github.com/grpc/grpc-web https://github.com/grpc/grpc-web
weitzj 6y ago
I did not quite get how they configure envoy? Did they write their own control plane? Use ambassador/Istio/Gloo?
- euroelessar 6y ago
  We have built our own control plane in golang tightly integrated with an existing infrastructure (service discovery, secrets/certificates management, configs delivery, feature gating, and so on).
- veshij 6y ago
  We have a mix of static and dynamic configuration. We started with almost everything defined in the configuration and implemented our control plane only for endpoint discovery service. Over the time we implemented more and more features there (certificates, tls tickets, route and vhost configuration, etc). We decided to write own implementation on control plane - actually the core part is pretty simple and easily expandable.
e40 6y ago
Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anything but OSS. What they make? Non-OSS software. Just blows my mind.
- SaveTheRbtz 6y ago
  The subject of monetizing opensource software is a tricky one. Some companies pursue the Open-Core principle, others monetize through the consulting services or cloud infrastructures. As for investing into opensource, Dropbox is trying to do that when possible, for example we (along with Automattic) did sponsor HTTP/2 development in Nginx.
  - gitgud 6y ago
    Personally I think that monetisation of open-source goes against the consumer of the OSS in practically all cases. - Open-Core::: Features are not added to core, as they want people to upgrade. - Consulting::: Ease of use is ignored, as if it's too easy people won't need consultants. - Sponsoring Goals::: Software is almost held at ransom, until goals are reached. The best way to help open-source software is to donate or contribute code... if you're trying to maximise profits, then just make it propitiatory
    - JoshTriplett 6y ago
      > - Consulting::: Ease of use is ignored, as if it's too easy people won't need consultants. Some problems can only be made so easy. Some problems require custom work. Sometimes you need paid support not because the product is low-quality but because you need to know that you can call someone at 3am because your service is down. There are lots of reasons to have consulting. > - Sponsoring Goals::: Software is almost held at ransom, until goals are reached. You're assuming the work would get done one way or another. Sometimes people have many other things they could be doing, and they need to justify spending more time on a project than they already do. Or sometimes, people have a fixed amount of time but they're happy to prioritize things people want and will pay for. (No argument about open-core; that definitely has problems.) Other great approaches include hosting the software as a service. Depending on the nature of a project, many people may want a service whose primary value proposition is "we'll host this for you so you don't have to maintain and administrate it".
pmlnr 6y ago
If anyone was wondering, this is solely for proxying, not for oldschool web server functionalities, eg. static file serving.
- SaveTheRbtz 6y ago
  We've actually started experimenting with converting our static file serving to "proxying to S3 + caching." This is simpler from deployment and development perspectives (for companies that do not have a distributed filesystem, like Google with its GFS): * for deployment we do not need to maintain a pool of stateful boxes with files on them and keep these files in sync. * for development, engineers now have a programatic interface for managing your static assets.
MuffinFlavored 6y ago
Can somebody speak to why dynamic upstreams included in a file paired with `sudo service nginx reload` for prod deploys stopped scaling?
- SaveTheRbtz 6y ago
  It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.) On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.
  - DarkWiiPlayer 6y ago
    nginx is not well suited for constantly reconfiguring your infrastructure on very hot servers. This is a problem when you expose such infrastructure configurations to users (think cloudflare), but otherwise you can just mitigate this problem by having a sane deployment strategy.
- nielsole 6y ago
  Nginx configuration is bound to its workers. When you reload nginx new workers are created(and start responding to new connections) and the old ones are drained. The draining finishes when the last connection is finished or a timeout is reached. In OSS nginx every upstream change requires a configuration reload. If you have lots of upstream changes and don't want to terminate connections prematurely, this can quickly require lots of RAM as you have many workers. Stock nginx worker is around 150Mb, but issues with openresty integration(they mention lua usage) can bloat this to > 1GB.
varbhat 6y ago
https://h2o.examp1e.net/ https://h2o.examp1e.net/ This is also good web server.Configuration is done in yaml. Also,it claims to be very fast.
eric4smith 6y ago
It's interesting almost no web server provides an easy way to deal with multi-tenant multi-domain architectures in a good way that includes automatic SSL. Caddy is the closest, but still not near enough. There is this small segment of the market that we operate in that requires thousands of TLS connected domains to be hosted behind a dynamic backend. It's services like Tumblr, Wordpress.com, or any other hosting service where you can get a "custom domain" to point to your own blog or site. NGINX - No. Apache - Nope. Caddy - Can do (but need lots of workarounds) Envoy - Nope. Everyone focuses on a few hand-coded domains and no automatic TLS. Maybe this part of the market is too small anyway. Sigh.
- simplyinfinity 6y ago
  We are using OpenResty with lua-auto-ssl for exactly this purpose, and it works like a charm.
- mholt 6y ago
  Several companies use Caddy for exactly this purpose. Fathom Analytics for example uses it for their custom domains feature. Caddy can even reactively provision certs during TLS handshakes. It's a native feature. Why does it require lots of workarounds?
  - bschwindHN 6y ago
    Yeah I'm not sure what they're getting at, I've used Caddy as well for similar "custom domain" features, it was super easy. Thanks for creating it!
  - eric4smith 6y ago
    Yes. Caddy is what we use, since not much else can do it as easily as Caddy can. And it's our go-to tool for several projects that require custom domains. And we really, really, appreciate it! I'm just saying that it's not something that is documented well or purpose built for that scenario.
  - sladey 6y ago
    Is there any mature integration to achieve this with Kubernetes?
deleted 6y ago
[deleted]
ram_rar 6y ago
I feel so old now. There was a time, when I used to discuss with senior engineers @ Yahoo! to use NginX over Apache. Nginx was the hot thing, popularizing C10k [1]. Now in my current team, I have junior devs in my team pushing for Envoy over HAProxy/Nginx setup. Is this trend happening primarily because devs are pushing for GRPC over REST? What benefits does Envoy offer over Nginx, if you're still a REST based service. I am not fully convinced of operational overhead that NGINX brings. [1] https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem
- Matthias247 6y ago
  The sibling comments point towards the difference in configuration if you take the "out of the box" product. But there is also a vast difference in how code is organized, in case you ever have to touch it. From my point of view Nginx feels "old". It's a C codebase without a great amount of abstractions and interfaces, and instead having a bunch of #ifdefs here and there. Unit-tests and comments are not to be found. Build via autotools. Envoy looks as modern as it gets for a C++ codebase - apart from maybe the lack of using coroutines which could be interesting for that use-case. It uses C++14, seems to be extremely structured and documented, has unit-tests, uses Bazel for builds, etc. So I think the experience for anyone being interested in working on the code will be very different, and people that prefer the style of project A will have a very hard time with the style of project B and the other way around.
  - MrBuddyCasino 6y ago
    > uses Bazel for builds Is this unanimously good? I've heard both praise and horror, never used it myself.
    - st1ck 6y ago
      It's good for Dropbox, since they use Bazel.
    - SaveTheRbtz 6y ago
      One of the senior engineers once said to me that "Bazel is like a sewer: you get back what you put in." Bazel requires a lot of upfront effort but the power of (a programmatically accessible/modifiable) dependency graph and a common build/test system across all the languages is very hard to underestimate.
Snelius 6y ago
And finally we got nginx as legacy now, lul.
xet7 6y ago
How does Envoy compare to Caddy 2 ? https://caddyserver.com https://caddyserver.com
- xet7 6y ago
  Found some related discussion here: https://caddy.community/t/caddy-to-replace-envoy-for-ha-setup/9078 https://caddy.community/t/caddy-to-replace-envoy-for-ha-setu...
- SaveTheRbtz 6y ago
  To tell you the truth, we didn't consider it. From what I can get from the architecture docs[1], it can be a decent platform for apps, but might not be the best choice for a general purpose ingress/egress proxy (at least for now.) [1] https://caddyserver.com/docs/architecture https://caddyserver.com/docs/architecture
  - mholt 6y ago
    It is a great choice for a general purpose proxy. (That's kind of the point.)
    - atonse 6y ago
      But they mentioned that they wanted to use C++ instead of go to get even that extra performance out. I use Caddy a lot and it's perfectly fine for my scale, but at dropbox's scale, maybe go wouldn't be enough for the ingress part?
      - mholt 6y ago
        I just lament the increasing deployments of programs written in memory-unsafe languages to the edge, in general. I am more curious what makes the author think Caddy "might not be the best choice for a general purpose ingress/egress proxy" (there were no other qualifications to that statement, but no evidence to support it either).
polskibus 6y ago
Thank you for the thorough comparison. Could anyone chip in whether a recent haprozy version would be a better choice than nginx and /or envoy in a similar case?
DarkWiiPlayer 6y ago
Seems like they could have switched to openresty instead and saved quite a lot of effort in their migration, but oh well, they probably just couldn't handle the 1-indexing /s
- anonymoushn 6y ago
  Seems like they were already using openresty. Having used openresty professionally, I appreciate that it provides ways to write code to solve a lot of the problems outlined in TFA, but solving the problems out of the box is significantly better.
spacewander 6y ago
One of my friends brought me up this post in the morning. The post is awesome and inspirational (caused a discussion in our chant group), though I can't agree with some trivial points. > Nginx performance without stats collections is on part with Envoy, but our Lua stats collection slowed Nginx on the high-RPS test by a factor of 3. This was expected given our reliance on lua_shared_dict, which is synchronized across workers with a mutex. The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds. It look like that the compared Nginx is configured with a system which has been survived for years and not up-to-date. The company I worked with used a single virtual server to hold all traffic and routed them with Lua code dynamically. And the upstream is chosen by Lua code too. There is no need to reload Nginx when a new route/upstream is added. We even implemented 'Access Log Service' like feature so that each user can have her favorite access log (by modifying the Nginx core, of course). However, I don't think this post is incorrect. What Envoy surpasses Nginx is that it has a more thriving developer community. There are more features added into Envoy than Nginx in the recent years. Not only that, opening discussion of Nginx development is rare. Nginx is an old, slow giant.
- rolls-reus 6y ago
  > The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds. Any pointers on how to achieve this for someone just starting out with lua and openresty? I have the exact same thing (lua_shared_dict) for stats collection, would love to learn a better way.
  - spacewander 6y ago
    You can look at https://github.com/knyar/nginx-lua-prometheus/pull/75 https://github.com/knyar/nginx-lua-prometheus/pull/75 for inspiration.
- 6y ago
dandare 6y ago
Is Condoleezza Rice still working for Dropbox?
- teichmann 6y ago
  She’s on the board of directors: https://www.dropbox.com/about https://www.dropbox.com/about.
- Shorel 6y ago
  She has never worked for Dropbox. Dropbox works for her :D
shay_ker 6y ago
> C++14 is not much different from using Golang or, with a stretch, one may even say Python. That's... definitely a stretch.
deleted 6y ago
[deleted]

427 points by SaveTheRbtz 6y ago

I know some people might find it a little controversial, but I’m super excited about our load balancing future and that we probably have the biggest Envoy deployment in the world now. When we moved most of Dropbox traffic to Envoy, we had to seamlessly migrate a system that already handles tens of millions of open connections, millions of requests per second, and terabits of bandwidth. This effectively made us into one of the biggest Envoy users.

user5994461 6y ago
Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling. What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have? I was running a comparable setup in a large company, all based on HAProxy, there was a significant amount of complexity in routing requests to applications that might ultimately be in any of 30 datacenters.
- SaveTheRbtz 6y ago
  We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS, now Katran[4]) that encapsulate traffic and DSR it to L7 balancers (previously nginx, now mostly Envoy.) Overall, we have ~25 PoPs and 4 datacenters. [1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastructure-edge-network https://dropbox.tech/infrastructure/dropbox-traffic-infrastr... [2] https://dropbox.tech/infrastructure/intelligent-dns-based-load-balancing-at-dropbox https://dropbox.tech/infrastructure/intelligent-dns-based-lo... [3] https://www.dropbox.com/peering https://www.dropbox.com/peering [4] https://github.com/facebookincubator/katran https://github.com/facebookincubator/katran
  - user5994461 6y ago
    Great. Exactly what I was looking for =)
  - alexeldeib 6y ago
    Cool to see someone using Katran in production. Really interesting stack you have there.
    - SaveTheRbtz 6y ago
      Actually, all the props for that go to Katran's author himeself. When we hired Nikita V. Shirokov (tehnerd), the first thing he did was replacing IPVS with XDP/eBPF-based Katran, which improved our Edge servers throughput by 10x, from ~2MPps to ~20Mpps. He also contributed a lot to Envoy migration migrating our desktop client to it and adding perf-related thing like TLS session tickets' lifetime to SDS.
  - dilyevsky 6y ago
    Katran - nice! Any issues with it at all? Do you use it with xdp capable hardware or just normal driver offload?
    - SaveTheRbtz 6y ago
      It works beautifully. We use driver offload (i40e on the Edge.)
  - traceroute66 6y ago
    @SaveTheRbtz "we have an open peering policy" That's a bit of a lie given you have a minimum 50Mbps requirement before you even consider a peering request. I would call that Selective, not Open !

Havoc 6y ago

Sensing a bit of a trend here. Didn't another major player recently make the same switch?

SaveTheRbtz 6y ago
I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]: * Lyft (of course) * Spotify * Stripe * Square * eBay * Yelp * Pinterest Plus the support from major cloud providers: Google, Microsoft, and Amazon. [1] https://envoyconna18.sched.com/ https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/ https://envoycon2019.sched.com/
- stock_toaster 6y ago
  So, seems like nginx is fine until your company reaches the "we are worth billions now" scale?
  - user5994461 6y ago
    nginx is never fine for load balancing, they put basic features like metrics behind the paid edition. It's not sane to operate in production. https://thehftguy.com/2016/10/03/haproxy-vs-nginx-why-you-should-never-use-nginx-for-load-balancing/ https://thehftguy.com/2016/10/03/haproxy-vs-nginx-why-you-sh...
  - donavanm 6y ago
    I work for a billions company. Nginx is still fine. Youll need to be prepared to pay for better operations, management, visibility, and protocol support. You can either pay them or build it in house, but you will want to pay.
- user5994461 6y ago
  They must all be GRPC users. Developers are pushing GRPC and protobuf pretty hard in companies. The next step down the road is to move to envoy as the load balancer. Otherwise these protocols don't work well over traditional HTTP infrastructure.

rdli 6y ago

Really great post. I'm glad the post in particular mentioned community, because I think in the end this is the huge advantage Envoy has over NGINX. NGINX, could, in theory, resolve all technical issues raised in the post. But the fundamental tension between the open source and commercial versions cannot be resolved. (Disclosure: We use Envoy as part of Ambassador, and so of course we're big fans!)

caiobegotti 6y ago

I'm positively surprised that Dropbox (at least from what I understood from the post) didn't require lots of changes or patches on top of the upstream codebase of Envoy to migrate their traffic!

SaveTheRbtz 6y ago
We did require some of them[1]. Esp. painful were Transfer-Encoding quirks, and some dances around old HTTP/1.0 backends and request buffering. Compared to NGINX though, it was relatively easy to push these fixes upstream. Community is very welcoming to outside contributions. [1] https://dropbox.tech/infrastructure/how-we-migrated-dropbox-from-nginx-to-envoy#-issues-we-encountered- https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...
- veshij 6y ago
  We do have some local patches as well (mostly for integration with out own infrastructure - stats collection, some RPC specific stuff). As SaveTheRbtz mentioned we encountered some issues with non-RFC clients, corner cases which were not exposed when envoy is used in "trusted" environment, etc., but all our fixes are now in upstream, so next migrations will be way easier both for us and for other envoy users.

deleted 6y ago

[deleted]

techntoke 6y ago

Who does Dropbox compete with these days? They have pretty much the highest prices for the least amount of value. The only reason I see them mentioned here frequently is their connection with Y Combinator.

rmoriz 6y ago
I've admired Drew and the early Dropbox team for getting things done and shipped even when compiled python GUIs was edgy as the initial Rails version of Twitter was. But they shipped and validated the market. Now adding all those fancy and cool tech mentioned in the blog post will increase the complexity by a lot but it's not clear what are the real benefits. Does a decreased number of machines running really justify the migration and addition of complexity? Maybe they have some new products in the pipeline that built upon the new stack. Or they waste their time. We will see.
- sanxiyn 6y ago
  > Does a decreased number of machines running really justify the migration and addition of complexity? Of course it does. I am not sure why you think it doesn't.
  - rmoriz 6y ago
    servers are a commodity and cheap compared to runtime complexity
    - sanxiyn 6y ago
      Yes, but it's a lot of servers.
      - rmoriz 6y ago
        How many? 5000 servers? As a public company it should not really matter at that size. Developers are way more expensive.
    - 6y ago

mperham 6y ago

Did you consider using commercial nginx? If so, what made you decide against it?

xmichael0 6y ago
The price is really insane for Nginx commercial.
- user5994461 6y ago
  About $2000 per host.
- mperham 6y ago
  As an Enterprise software vendor myself, I can assure you: everything is negotiable at Dropbox’s scale including very deep discounts.
dmicher 6y ago
Sadly, it would probably be as hard to maintain as an opensource version. We really want to have access to the code to make sure we can fix, troubleshoot it, understand it fast... Things that may've help: -- Configuration definition (e.g. protobufs.) -- More focus on observability: error metrics (instead of logs), tracing, etc. -- gRPC control plane. -- C++ module development SDK. -- (ideally) bazel. Some dataplane features like gRPC JSON transcoding, gRPC-Web, and http/2 to backends.
- dmarble 6y ago
  Don't any of the major commercial open source vendors offer custom terms to give access to the commercial source? I'd imagine they'd contemplate it for big deals. Seems like one of the only ways to keep some of these sophisticated customers onboard.
  - SaveTheRbtz 6y ago
    open source argument is valid -- most software enterprise vendors do provide source code access (under NDA.) The rest of the arguments stand though: as it is right now, it way more developer/operator friendly to use Envoy in our production.

sroussey 6y ago

One thing nice about OpenResty (nginx) and their Lua support is that it plugs in at TLS negotiation. Does Envoy?

SaveTheRbtz 6y ago
Can you describe your use-case? If you are talking about the ability to select a certificate on the fly via `ssl_certificate_by_lua_block`[1] we are not aware of such functionality. If you are missing something, I would highly encourage you discuss it with the community on a github! From Oleg Guba, Traffic Team TL, co-author, and person driving the deployment: * ListenerFilters + NetworkFilters are flexible enough, that some of the custom logic could be just moved to the config. From Ruslan Nigmatullin, our head Envoy developer: If you are talking more about a custom verification code there is already couple of ways to do that: * Client TLS auth Network Filter: https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/client_ssl_auth_filter#config-network-filters-client-ssl-auth https://www.envoyproxy.io/docs/envoy/latest/configuration/li... * Alternatively, if you are writing C++ extension you can use Network::ReadFilter, Network::ConnectionCallbacks. [1] https://github.com/openresty/lua-nginx-module#ssl_certificate_by_lua_block https://github.com/openresty/lua-nginx-module#ssl_certificat... [2] https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md https://github.com/openresty/lua-resty-core/blob/master/lib/...
- sroussey 6y ago
  Wordpress and others use this to load certain on the fly. When you are a multidomain host this matters a lot. You don’t just load up a million cents as files and restart the server (though I do know a company that does something like this, but man, quite brittle).

user5994461 6y ago

A shame they picked nginx in the first place, it has all the stats and critical features behind the paid edition. HAProxy is always a better choice for load balancing. Besides that, it looks like the move was significantly driven by GRPC and profobuf. No surprise here, GRPC really doesn't work well over HTTP. Once a company start using the google stack, they have to move to more of the google stack to make it usable.

SaveTheRbtz 6y ago
Our technology stack is very gRPC friendly, so developer experience is actually better with it, than without (though this is very subjective.) As for the middleboxes, using gRPC-WEB[1] allowed us to switch Desktop Client App to gRPC even behind firewalls/IDSes that do not speak HTTP/2 yet. As for the HAProxy, Dropbox used to use (circa 2013) it specifically for loadbalancing, but we eventually replaced it with our Golang proxy. That said, recent HAProxy improvements (v2.0+) make it quite an awesome dataplane and an excellent loadbalancer! [1] https://github.com/grpc/grpc-web https://github.com/grpc/grpc-web

weitzj 6y ago

I did not quite get how they configure envoy? Did they write their own control plane? Use ambassador/Istio/Gloo?

euroelessar 6y ago
We have built our own control plane in golang tightly integrated with an existing infrastructure (service discovery, secrets/certificates management, configs delivery, feature gating, and so on).
veshij 6y ago
We have a mix of static and dynamic configuration. We started with almost everything defined in the configuration and implemented our control plane only for endpoint discovery service. Over the time we implemented more and more features there (certificates, tls tickets, route and vhost configuration, etc). We decided to write own implementation on control plane - actually the core part is pretty simple and easily expandable.

e40 6y ago

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anything but OSS. What they make? Non-OSS software. Just blows my mind.

SaveTheRbtz 6y ago
The subject of monetizing opensource software is a tricky one. Some companies pursue the Open-Core principle, others monetize through the consulting services or cloud infrastructures. As for investing into opensource, Dropbox is trying to do that when possible, for example we (along with Automattic) did sponsor HTTP/2 development in Nginx.
- gitgud 6y ago
  Personally I think that monetisation of open-source goes against the consumer of the OSS in practically all cases. - Open-Core::: Features are not added to core, as they want people to upgrade. - Consulting::: Ease of use is ignored, as if it's too easy people won't need consultants. - Sponsoring Goals::: Software is almost held at ransom, until goals are reached. The best way to help open-source software is to donate or contribute code... if you're trying to maximise profits, then just make it propitiatory
  - JoshTriplett 6y ago
    > - Consulting::: Ease of use is ignored, as if it's too easy people won't need consultants. Some problems can only be made so easy. Some problems require custom work. Sometimes you need paid support not because the product is low-quality but because you need to know that you can call someone at 3am because your service is down. There are lots of reasons to have consulting. > - Sponsoring Goals::: Software is almost held at ransom, until goals are reached. You're assuming the work would get done one way or another. Sometimes people have many other things they could be doing, and they need to justify spending more time on a project than they already do. Or sometimes, people have a fixed amount of time but they're happy to prioritize things people want and will pay for. (No argument about open-core; that definitely has problems.) Other great approaches include hosting the software as a service. Depending on the nature of a project, many people may want a service whose primary value proposition is "we'll host this for you so you don't have to maintain and administrate it".

pmlnr 6y ago

If anyone was wondering, this is solely for proxying, not for oldschool web server functionalities, eg. static file serving.

SaveTheRbtz 6y ago
We've actually started experimenting with converting our static file serving to "proxying to S3 + caching." This is simpler from deployment and development perspectives (for companies that do not have a distributed filesystem, like Google with its GFS): * for deployment we do not need to maintain a pool of stateful boxes with files on them and keep these files in sync. * for development, engineers now have a programatic interface for managing your static assets.

MuffinFlavored 6y ago

Can somebody speak to why dynamic upstreams included in a file paired with `sudo service nginx reload` for prod deploys stopped scaling?

SaveTheRbtz 6y ago
It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.) On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.
- DarkWiiPlayer 6y ago
  nginx is not well suited for constantly reconfiguring your infrastructure on very hot servers. This is a problem when you expose such infrastructure configurations to users (think cloudflare), but otherwise you can just mitigate this problem by having a sane deployment strategy.
nielsole 6y ago
Nginx configuration is bound to its workers. When you reload nginx new workers are created(and start responding to new connections) and the old ones are drained. The draining finishes when the last connection is finished or a timeout is reached. In OSS nginx every upstream change requires a configuration reload. If you have lots of upstream changes and don't want to terminate connections prematurely, this can quickly require lots of RAM as you have many workers. Stock nginx worker is around 150Mb, but issues with openresty integration(they mention lua usage) can bloat this to > 1GB.

varbhat 6y ago

https://h2o.examp1e.net/ https://h2o.examp1e.net/ This is also good web server.Configuration is done in yaml. Also,it claims to be very fast.

eric4smith 6y ago

It's interesting almost no web server provides an easy way to deal with multi-tenant multi-domain architectures in a good way that includes automatic SSL. Caddy is the closest, but still not near enough. There is this small segment of the market that we operate in that requires thousands of TLS connected domains to be hosted behind a dynamic backend. It's services like Tumblr, Wordpress.com, or any other hosting service where you can get a "custom domain" to point to your own blog or site. NGINX - No. Apache - Nope. Caddy - Can do (but need lots of workarounds) Envoy - Nope. Everyone focuses on a few hand-coded domains and no automatic TLS. Maybe this part of the market is too small anyway. Sigh.

simplyinfinity 6y ago
We are using OpenResty with lua-auto-ssl for exactly this purpose, and it works like a charm.
mholt 6y ago
Several companies use Caddy for exactly this purpose. Fathom Analytics for example uses it for their custom domains feature. Caddy can even reactively provision certs during TLS handshakes. It's a native feature. Why does it require lots of workarounds?
- bschwindHN 6y ago
  Yeah I'm not sure what they're getting at, I've used Caddy as well for similar "custom domain" features, it was super easy. Thanks for creating it!
- eric4smith 6y ago
  Yes. Caddy is what we use, since not much else can do it as easily as Caddy can. And it's our go-to tool for several projects that require custom domains. And we really, really, appreciate it! I'm just saying that it's not something that is documented well or purpose built for that scenario.
- sladey 6y ago
  Is there any mature integration to achieve this with Kubernetes?

deleted 6y ago

[deleted]

ram_rar 6y ago

I feel so old now. There was a time, when I used to discuss with senior engineers @ Yahoo! to use NginX over Apache. Nginx was the hot thing, popularizing C10k [1]. Now in my current team, I have junior devs in my team pushing for Envoy over HAProxy/Nginx setup. Is this trend happening primarily because devs are pushing for GRPC over REST? What benefits does Envoy offer over Nginx, if you're still a REST based service. I am not fully convinced of operational overhead that NGINX brings. [1] https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem

Matthias247 6y ago
The sibling comments point towards the difference in configuration if you take the "out of the box" product. But there is also a vast difference in how code is organized, in case you ever have to touch it. From my point of view Nginx feels "old". It's a C codebase without a great amount of abstractions and interfaces, and instead having a bunch of #ifdefs here and there. Unit-tests and comments are not to be found. Build via autotools. Envoy looks as modern as it gets for a C++ codebase - apart from maybe the lack of using coroutines which could be interesting for that use-case. It uses C++14, seems to be extremely structured and documented, has unit-tests, uses Bazel for builds, etc. So I think the experience for anyone being interested in working on the code will be very different, and people that prefer the style of project A will have a very hard time with the style of project B and the other way around.
- MrBuddyCasino 6y ago
  > uses Bazel for builds Is this unanimously good? I've heard both praise and horror, never used it myself.
  - st1ck 6y ago
    It's good for Dropbox, since they use Bazel.
  - SaveTheRbtz 6y ago
    One of the senior engineers once said to me that "Bazel is like a sewer: you get back what you put in." Bazel requires a lot of upfront effort but the power of (a programmatically accessible/modifiable) dependency graph and a common build/test system across all the languages is very hard to underestimate.

Snelius 6y ago

And finally we got nginx as legacy now, lul.

xet7 6y ago

How does Envoy compare to Caddy 2 ? https://caddyserver.com https://caddyserver.com

xet7 6y ago
Found some related discussion here: https://caddy.community/t/caddy-to-replace-envoy-for-ha-setup/9078 https://caddy.community/t/caddy-to-replace-envoy-for-ha-setu...
SaveTheRbtz 6y ago
To tell you the truth, we didn't consider it. From what I can get from the architecture docs[1], it can be a decent platform for apps, but might not be the best choice for a general purpose ingress/egress proxy (at least for now.) [1] https://caddyserver.com/docs/architecture https://caddyserver.com/docs/architecture
- mholt 6y ago
  It is a great choice for a general purpose proxy. (That's kind of the point.)
  - atonse 6y ago
    But they mentioned that they wanted to use C++ instead of go to get even that extra performance out. I use Caddy a lot and it's perfectly fine for my scale, but at dropbox's scale, maybe go wouldn't be enough for the ingress part?
    - mholt 6y ago
      I just lament the increasing deployments of programs written in memory-unsafe languages to the edge, in general. I am more curious what makes the author think Caddy "might not be the best choice for a general purpose ingress/egress proxy" (there were no other qualifications to that statement, but no evidence to support it either).

polskibus 6y ago

Thank you for the thorough comparison. Could anyone chip in whether a recent haprozy version would be a better choice than nginx and /or envoy in a similar case?

DarkWiiPlayer 6y ago

Seems like they could have switched to openresty instead and saved quite a lot of effort in their migration, but oh well, they probably just couldn't handle the 1-indexing /s

anonymoushn 6y ago
Seems like they were already using openresty. Having used openresty professionally, I appreciate that it provides ways to write code to solve a lot of the problems outlined in TFA, but solving the problems out of the box is significantly better.

spacewander 6y ago

One of my friends brought me up this post in the morning. The post is awesome and inspirational (caused a discussion in our chant group), though I can't agree with some trivial points. > Nginx performance without stats collections is on part with Envoy, but our Lua stats collection slowed Nginx on the high-RPS test by a factor of 3. This was expected given our reliance on lua_shared_dict, which is synchronized across workers with a mutex. The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds. It look like that the compared Nginx is configured with a system which has been survived for years and not up-to-date. The company I worked with used a single virtual server to hold all traffic and routed them with Lua code dynamically. And the upstream is chosen by Lua code too. There is no need to reload Nginx when a new route/upstream is added. We even implemented 'Access Log Service' like feature so that each user can have her favorite access log (by modifying the Nginx core, of course). However, I don't think this post is incorrect. What Envoy surpasses Nginx is that it has a more thriving developer community. There are more features added into Envoy than Nginx in the recent years. Not only that, opening discussion of Nginx development is rare. Nginx is an old, slow giant.

rolls-reus 6y ago
> The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds. Any pointers on how to achieve this for someone just starting out with lua and openresty? I have the exact same thing (lua_shared_dict) for stats collection, would love to learn a better way.
- spacewander 6y ago
  You can look at https://github.com/knyar/nginx-lua-prometheus/pull/75 https://github.com/knyar/nginx-lua-prometheus/pull/75 for inspiration.
6y ago

dandare 6y ago

Is Condoleezza Rice still working for Dropbox?

teichmann 6y ago
She’s on the board of directors: https://www.dropbox.com/about https://www.dropbox.com/about.
Shorel 6y ago
She has never worked for Dropbox. Dropbox works for her :D

shay_ker 6y ago

> C++14 is not much different from using Golang or, with a stretch, one may even say Python. That's... definitely a stretch.

deleted 6y ago

[deleted]