5 ms·

What’s new with our in-house load balancing service

49 points by ABS 2y ago

leetrout 2y ago
I am surprised to see zookeeper being used with etcd. Why use both? Why even consider ZK in 2024? Is it just popular in java shops? Maybe I am misunderstanding?
- bittermandel 2y ago
  Since it's written ZooKeeper/etcd-based, I assume they use ZK and are going over to Etcd.
- pgwhalen 2y ago
  > Why even consider ZK in 2024? I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.
  - zaphirplane 2y ago
    Great question
- aftbit 2y ago
  Dropbox has been using Zookeeper since the beginning. I assume they are just familiar with running it and understand its guarantees better.
nepthar 2y ago
We used a deterministic aperture load balancing to solve this at twitter - https://blog.x.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer https://blog.x.com/engineering/en_us/topics/infrastructure/2.... Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.
- renewiltord 2y ago
  Very cool. Low-state approaches are always more interesting.
- deleted 2y ago
  [deleted]
- jauntywundrkind 2y ago
  Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.... I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.
renewiltord 2y ago
I see: load-aware routing. Why skip the weight update phase on low weight reports? Isn’t that precisely the period when those nodes are likely overloaded?
dang 2y ago
[stub for offtopicness]
- terramauthe 2y ago
  [flagged]
  - yawnxyz 2y ago
    "load balancing" is a cold way to describe a layoff...
    - Simon_ORourke 2y ago
      "fully automated market-driven load balancing" more like.
    - blitzar 2y ago
      "load balancing" is apt description of the role of HR
      - toomuchtodo 2y ago
        More like container orchestration and bin packing, amirite? Faulty container? Terminate and launch a new container. Cattle, not pets.
  - guessmyname 2y ago
    He does → https://www.linkedin.com/in/yi-shu-tai-13160154/ https://www.linkedin.com/in/yi-shu-tai-13160154/ According to his LinkedIn profile: 7 years 6 months into the job. Although, it looks like he transfered to a new team a year ago to do ML Ops.
    - apocalyptic0n3 2y ago
      Dropbox just laid off 20% of the company yesterday. In many cases, people don't rush to update their LinkedIn so he may or may not still be there.

49 points by ABS 2y ago

I am surprised to see zookeeper being used with etcd. Why use both? Why even consider ZK in 2024? Is it just popular in java shops? Maybe I am misunderstanding?

bittermandel 2y ago
Since it's written ZooKeeper/etcd-based, I assume they use ZK and are going over to Etcd.
pgwhalen 2y ago
> Why even consider ZK in 2024? I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.
- zaphirplane 2y ago
  Great question
aftbit 2y ago
Dropbox has been using Zookeeper since the beginning. I assume they are just familiar with running it and understand its guarantees better.

nepthar 2y ago

We used a deterministic aperture load balancing to solve this at twitter - https://blog.x.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer https://blog.x.com/engineering/en_us/topics/infrastructure/2.... Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.

renewiltord 2y ago
Very cool. Low-state approaches are always more interesting.
deleted 2y ago
[deleted]
jauntywundrkind 2y ago
Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.... I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.

renewiltord 2y ago

I see: load-aware routing. Why skip the weight update phase on low weight reports? Isn’t that precisely the period when those nodes are likely overloaded?

dang 2y ago

[stub for offtopicness]

terramauthe 2y ago
[flagged]
- yawnxyz 2y ago
  "load balancing" is a cold way to describe a layoff...
  - Simon_ORourke 2y ago
    "fully automated market-driven load balancing" more like.
  - blitzar 2y ago
    "load balancing" is apt description of the role of HR
    - toomuchtodo 2y ago
      More like container orchestration and bin packing, amirite? Faulty container? Terminate and launch a new container. Cattle, not pets.
- guessmyname 2y ago
  He does → https://www.linkedin.com/in/yi-shu-tai-13160154/ https://www.linkedin.com/in/yi-shu-tai-13160154/ According to his LinkedIn profile: 7 years 6 months into the job. Although, it looks like he transfered to a new team a year ago to do ML Ops.
  - apocalyptic0n3 2y ago
    Dropbox just laid off 20% of the company yesterday. In many cases, people don't rush to update their LinkedIn so he may or may not still be there.