4 ms·

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS, now Katran[4]) that encapsulate traffic and DSR it to L7 balancers (previously nginx, now mostly Envoy.) Overall, we have ~25 PoPs and 4 datacenters. [1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastructure-edge-network https://dropbox.tech/infrastructure/dropbox-traffic-infrastr... [2] https://dropbox.tech/infrastructure/intelligent-dns-based-load-balancing-at-dropbox https://dropbox.tech/infrastructure/intelligent-dns-based-lo... [3] https://www.dropbox.com/peering https://www.dropbox.com/peering [4] https://github.com/facebookincubator/katran https://github.com/facebookincubator/katran

user5994461 6y ago
Great. Exactly what I was looking for =)
alexeldeib 6y ago
Cool to see someone using Katran in production. Really interesting stack you have there.
- SaveTheRbtz 6y ago
  Actually, all the props for that go to Katran's author himeself. When we hired Nikita V. Shirokov (tehnerd), the first thing he did was replacing IPVS with XDP/eBPF-based Katran, which improved our Edge servers throughput by 10x, from ~2MPps to ~20Mpps. He also contributed a lot to Envoy migration migrating our desktop client to it and adding perf-related thing like TLS session tickets' lifetime to SDS.
dilyevsky 6y ago
Katran - nice! Any issues with it at all? Do you use it with xdp capable hardware or just normal driver offload?
- SaveTheRbtz 6y ago
  It works beautifully. We use driver offload (i40e on the Edge.)
traceroute66 6y ago
@SaveTheRbtz "we have an open peering policy" That's a bit of a lie given you have a minimum 50Mbps requirement before you even consider a peering request. I would call that Selective, not Open !

user5994461 6y ago

Great. Exactly what I was looking for =)

alexeldeib 6y ago

Cool to see someone using Katran in production. Really interesting stack you have there.

SaveTheRbtz 6y ago
Actually, all the props for that go to Katran's author himeself. When we hired Nikita V. Shirokov (tehnerd), the first thing he did was replacing IPVS with XDP/eBPF-based Katran, which improved our Edge servers throughput by 10x, from ~2MPps to ~20Mpps. He also contributed a lot to Envoy migration migrating our desktop client to it and adding perf-related thing like TLS session tickets' lifetime to SDS.

dilyevsky 6y ago

Katran - nice! Any issues with it at all? Do you use it with xdp capable hardware or just normal driver offload?

SaveTheRbtz 6y ago
It works beautifully. We use driver offload (i40e on the Edge.)

traceroute66 6y ago

@SaveTheRbtz "we have an open peering policy" That's a bit of a lie given you have a minimum 50Mbps requirement before you even consider a peering request. I would call that Selective, not Open !