14 ms·

I've never heard of routers using the ttl field for hashing, why would that ever be useful? Also it seems an incorrect use of anycast to terminate the same flo

I've never heard of routers using the ttl field for hashing, why would that ever be useful? Also it seems an incorrect use of anycast to terminate the same flow at different machines. At Google, anycast traffic goes through clusters of maglevs that directs individual flows to the same endpoint consistently. https://ai.google/research/pubs/pub44824 https://ai.google/research/pubs/pub44824 Also I'd like to echo that Fastly's NOC is great. Super responsive and smart. Disclaimer: I work at Google

mcpherrinm 8y ago
If you're doing multi-tier ECMP with the same hashing algorithm at each tier, using TTL can ensure you don't get polarization issues. Though there's plenty of other ways to avoid polarization, this works out of the box. I have heard fastly competitors tut-tut their use of anycast as potentially unreliable in the face of issues like this.
- greglindahl 8y ago
  "this works out of the box" seems a bit of an odd thing to say when it's not working for significant edge cases. Maybe "it worked when we first tested it" is better? :-)
  - p1necone 8y ago
    I dunno, in my experience anything that "works right out of the box" usually misses a bunch of edge cases (possibly necessarily/by definition). The description seems apt to me.
- bogomipz 8y ago
  What is "polarization" here? I am not familiar with this term applied to routing? Oscillating between different available paths?
  - zkms 8y ago
    > Oscillating between different available paths? The term for unwanted oscillation between different paths is, IIRC, "flapping". However, when working with ECMP (equal-cost multi path), you actually do want to use all your paths (hence "equal cost") simultaneously (for load-balancing purposes, say). "Polarisation" refers to the unwanted condition when the hash algorithm that decides which path a flow/packet takes is not properly distributing flows/packets across links, leading to underutilisation and/or overutilisation of links/routes.
    - bogomipz 8y ago
      Sure, I meant when the same 5 tuple is oscillating between different paths which really should not happen. You mentioned there were plenty other ways to avoid polarization. I would be curious to hear what those are.
      - alecbenzer 8y ago
        New to most of this but couldn't you just salt the hashes being used per-router?
        bogomipz 8y ago
        Yeah I think you could seed each router's hashing with something unique like a mac address hardware ID but you could also just use different hashing at each tier in your network like using source IP + dest IP on one tier and source IP + dest IP + IP protocol on another tier. This should achieve the same result.
        alecbenzer 8y ago
        Yeah but then you need to keep coming up with new ways of hashing things as you add tiers. It's easier to generate a new salt than it is to figure out a new but still-valid way of hashing the packets.
cft 8y ago
Incapsula response is different however: at some point we urgently needed to implement a DDoS protection using GRE tunnels. An Incapsula eng spoke with us as if he was challenging us to a programming competition puzzle: "I can tell you that you are big enough to implement GRE only if you can answer "yes" to these four networking questions". We went with Verisign as the result, they made things look simpler

mcpherrinm 8y ago

If you're doing multi-tier ECMP with the same hashing algorithm at each tier, using TTL can ensure you don't get polarization issues. Though there's plenty of other ways to avoid polarization, this works out of the box. I have heard fastly competitors tut-tut their use of anycast as potentially unreliable in the face of issues like this.

greglindahl 8y ago
"this works out of the box" seems a bit of an odd thing to say when it's not working for significant edge cases. Maybe "it worked when we first tested it" is better? :-)
- p1necone 8y ago
  I dunno, in my experience anything that "works right out of the box" usually misses a bunch of edge cases (possibly necessarily/by definition). The description seems apt to me.
bogomipz 8y ago
What is "polarization" here? I am not familiar with this term applied to routing? Oscillating between different available paths?
- zkms 8y ago
  > Oscillating between different available paths? The term for unwanted oscillation between different paths is, IIRC, "flapping". However, when working with ECMP (equal-cost multi path), you actually do want to use all your paths (hence "equal cost") simultaneously (for load-balancing purposes, say). "Polarisation" refers to the unwanted condition when the hash algorithm that decides which path a flow/packet takes is not properly distributing flows/packets across links, leading to underutilisation and/or overutilisation of links/routes.
  - bogomipz 8y ago
    Sure, I meant when the same 5 tuple is oscillating between different paths which really should not happen. You mentioned there were plenty other ways to avoid polarization. I would be curious to hear what those are.
    - alecbenzer 8y ago
      New to most of this but couldn't you just salt the hashes being used per-router?
      - bogomipz 8y ago
        Yeah I think you could seed each router's hashing with something unique like a mac address hardware ID but you could also just use different hashing at each tier in your network like using source IP + dest IP on one tier and source IP + dest IP + IP protocol on another tier. This should achieve the same result.
        alecbenzer 8y ago
        Yeah but then you need to keep coming up with new ways of hashing things as you add tiers. It's easier to generate a new salt than it is to figure out a new but still-valid way of hashing the packets.

cft 8y ago

Incapsula response is different however: at some point we urgently needed to implement a DDoS protection using GRE tunnels. An Incapsula eng spoke with us as if he was challenging us to a programming competition puzzle: "I can tell you that you are big enough to implement GRE only if you can answer "yes" to these four networking questions". We went with Verisign as the result, they made things look simpler