4 ms·

I don't need to tell you how much of an ecosystem exploded after the BigTable whitepapers. It might not be realistic to expect that kind of reaction for load ba

by gtaylor 11y ago

I don't need to tell you how much of an ecosystem exploded after the BigTable whitepapers. It might not be realistic to expect that kind of reaction for load balancing tech, but it's great to see them continue to do this sort of thing. Also looking forward to seeing what this release in particular may inspire.

jauer 11y ago
ECMP is pretty old-hat. Example from 2007: https://www.nanog.org/meetings/nanog41/presentations/Kapela-lightning.pdf https://www.nanog.org/meetings/nanog41/presentations/Kapela-... Exactly how the routes get steered to servers can vary (static routes with track objects, OSPF or BGP on the server), but the basic idea of using network gear to ECMP traffic between servers is, if not ancient, pretty common. I suspect there's something novel in control plane or hashing used to steer ECMP but I didn't see anything about that in the article.
- teraflop 11y ago
  Yeah, the interesting details are in the paper. If I understand it correctly, ECMP alone doesn't give you a very even distribution because of the small number of hash buckets. And every time the routing table changes (e.g. when taking backends in or out of rotation) it will drop connections because packets are suddenly being hashed to different destinations. Maglev adds an additional layer behind ECMP that solves those problems.
- transitorykris 11y ago
  ECMP is old, we've being pulling this stunt for a long time, for the right applications (e.g., UDP, short lived TCP). Adding/removing routes in ECMP can cause flows to hash to a new destination. The connection tracking and consistent hashing in Maglev is important to deal with faults and avoid disrupting unrelated flows. Still nothing new.
  - asuffield 11y ago
    (Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google, and I'm oncall for this service.) Well, it's "not new" in the sense that this system's been running google.com for about 8 years now ;) This is just the first time that we've published how it works.
  - kijiki 11y ago
    We (Cumulus Networks) support a feature called "resilient hashing" that ensures that if a software LB fails, only the flows going to that LB are redistributed to the remaining LBs. You still lose some connections when an LB fails, but only the ones going through the failed LB. Unrelated flows to other LBs are not impacted. We've got multiple customers doing variants of the LB architecture Google talks about here.
    - newman314 11y ago
      I didn't know Cumulus had a LB product. Any more details that you can share?
      - wmf 11y ago
        You can turn any switches into ghetto load balancers by running BGP on some hosts and advertising a /32 into your switches.
        bogomipz 11y ago
        You want BGP on your edge router, where your transit connections, these are what do the ECMP towards your LBs which speak iBGP. I'm not sure where you would use a cheap switch in this setup, certainly not at the edge.
        wmf 11y ago
        Many recent data centers run eBGP between all switches. I can't explain the rest without diagrams.
        bogomipz 11y ago
        Huh? Then you must not understand what you are talking about very well. Are you talking about running BGP to ToR switches? What does that have to do with ECMP or load balancers? Its just an L3 design and its not new.
      - 11y ago
  - bogomipz 11y ago
    Usually ECMP on your LBs, you run a routing daemon on the LB box - such Quagga or BIRD and the speak BGP to the edge, if an LB fails or goes away then the route is withdrawn from BGP peer and this withdrawl is how faults are dealt with. Can you or someone else elaborate on how Malev adds or differs from that? Unfortunately I can not get to google at the moment.
jayhuang 11y ago
Surprised there are so few comments after this many upvotes. It's always great seeing companies share their work, such as MapReduce, BigTable, TensorFlow, etc from Google (Netflix's ops tools also are very interesting, not to discount the contributions of Facebook, Microsoft et al). Would love to see the HN community's opinion on this, as someone who is not an expert in ops/infrastructure.
- deleted 11y ago
  [deleted]

by gtaylor 11y ago

jauer 11y ago

ECMP is pretty old-hat. Example from 2007: https://www.nanog.org/meetings/nanog41/presentations/Kapela-lightning.pdf https://www.nanog.org/meetings/nanog41/presentations/Kapela-... Exactly how the routes get steered to servers can vary (static routes with track objects, OSPF or BGP on the server), but the basic idea of using network gear to ECMP traffic between servers is, if not ancient, pretty common. I suspect there's something novel in control plane or hashing used to steer ECMP but I didn't see anything about that in the article.

teraflop 11y ago
Yeah, the interesting details are in the paper. If I understand it correctly, ECMP alone doesn't give you a very even distribution because of the small number of hash buckets. And every time the routing table changes (e.g. when taking backends in or out of rotation) it will drop connections because packets are suddenly being hashed to different destinations. Maglev adds an additional layer behind ECMP that solves those problems.
transitorykris 11y ago
ECMP is old, we've being pulling this stunt for a long time, for the right applications (e.g., UDP, short lived TCP). Adding/removing routes in ECMP can cause flows to hash to a new destination. The connection tracking and consistent hashing in Maglev is important to deal with faults and avoid disrupting unrelated flows. Still nothing new.
- asuffield 11y ago
  (Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google, and I'm oncall for this service.) Well, it's "not new" in the sense that this system's been running google.com for about 8 years now ;) This is just the first time that we've published how it works.
- kijiki 11y ago
  We (Cumulus Networks) support a feature called "resilient hashing" that ensures that if a software LB fails, only the flows going to that LB are redistributed to the remaining LBs. You still lose some connections when an LB fails, but only the ones going through the failed LB. Unrelated flows to other LBs are not impacted. We've got multiple customers doing variants of the LB architecture Google talks about here.
  - newman314 11y ago
    I didn't know Cumulus had a LB product. Any more details that you can share?
    - wmf 11y ago
      You can turn any switches into ghetto load balancers by running BGP on some hosts and advertising a /32 into your switches.
      - bogomipz 11y ago
        You want BGP on your edge router, where your transit connections, these are what do the ECMP towards your LBs which speak iBGP. I'm not sure where you would use a cheap switch in this setup, certainly not at the edge.
        wmf 11y ago
        Many recent data centers run eBGP between all switches. I can't explain the rest without diagrams.
        bogomipz 11y ago
        Huh? Then you must not understand what you are talking about very well. Are you talking about running BGP to ToR switches? What does that have to do with ECMP or load balancers? Its just an L3 design and its not new.
    - 11y ago
- bogomipz 11y ago
  Usually ECMP on your LBs, you run a routing daemon on the LB box - such Quagga or BIRD and the speak BGP to the edge, if an LB fails or goes away then the route is withdrawn from BGP peer and this withdrawl is how faults are dealt with. Can you or someone else elaborate on how Malev adds or differs from that? Unfortunately I can not get to google at the moment.

jayhuang 11y ago

Surprised there are so few comments after this many upvotes. It's always great seeing companies share their work, such as MapReduce, BigTable, TensorFlow, etc from Google (Netflix's ops tools also are very interesting, not to discount the contributions of Facebook, Microsoft et al). Would love to see the HN community's opinion on this, as someone who is not an expert in ops/infrastructure.

deleted 11y ago
[deleted]