5 ms·

If we could use more than 1 IP, then we could treat 1 400Gb box as 4 100Gb boxes. That could lead to the "perfect case" every time, since connections would alw

If we could use more than 1 IP, then we could treat 1 400Gb box as 4 100Gb boxes. That could lead to the "perfect case" every time, since connections would always stay local to the numa node where content is present.

betaby 5y ago
There is no kTLS for IPv6? IPv6 space is abundant and most mobiles is USA/Canada have IPv6. Won't that solve the problem?
- drewg123 5y ago
  IPv4 and IPv6 can both use kTLS. We offer service via V6, but most clients connect via IPv4. It differs by region, and even time of day, but IPv4 is still the vast majority of traffic.
  - chungy 5y ago
    I've had to blackhole Netflix IPv6 ranges on my router because Netflix would identify my IPv6 connection as being a "VPN" even though it's not.
    - drewg123 5y ago
      If you could email me your ranges, I can try to look into it internally. Use my last name at gmail.com (or at freebsd.org)
      - genewitch 5y ago
        I'm not GP; Hurricane Electric ipv6 always had Netflix think I was on a VPN, but now I have real ipv6 through the same ISP, I just pay more money so Netflix doesn't complain anymore.
        0x0000000 5y ago
        Are you referring to HE's tunnel broker service? If so, then yeah, that's a VPN.
toast0 5y ago
I wonder if you could steer clients away from connections where the NIC and storage nodes are mismatched. Something like close the connection after N requests/N minutes if the nodes are mismatched, but leave it open indefinitely if they match. There's of course a lot of ways for that to not be very helpful. You'd still have only a 25% of getting a port number that hashes to the right node the next time (assuming tcp port number is involved at all, if it's just src and dest ips then client connections from the same IP would always hash the same, and that's probably a big portion of your clients), and if establishing connections is expensive enough (or clients aren't good at it) then that's a negative. Also if a stream's files don't tend to stay on the same node, then churning connections to get to the right node doesn't help if the next segment is on a different node. I'm sure there are other scenarios too. I know some other CDN appliance setups do use multiple IPs, so you probably could get more, but it would add administrative stress.
topranks 5y ago
Is there a reason you can’t? In a past life we broke LAGs up to use different subnets per port to prevent traffic crossing the NUMA bridge. I’m sure there are good reasons you didn’t take this approach, be interesting to hear them.

betaby 5y ago

There is no kTLS for IPv6? IPv6 space is abundant and most mobiles is USA/Canada have IPv6. Won't that solve the problem?

drewg123 5y ago
IPv4 and IPv6 can both use kTLS. We offer service via V6, but most clients connect via IPv4. It differs by region, and even time of day, but IPv4 is still the vast majority of traffic.
- chungy 5y ago
  I've had to blackhole Netflix IPv6 ranges on my router because Netflix would identify my IPv6 connection as being a "VPN" even though it's not.
  - drewg123 5y ago
    If you could email me your ranges, I can try to look into it internally. Use my last name at gmail.com (or at freebsd.org)
    - genewitch 5y ago
      I'm not GP; Hurricane Electric ipv6 always had Netflix think I was on a VPN, but now I have real ipv6 through the same ISP, I just pay more money so Netflix doesn't complain anymore.
      - 0x0000000 5y ago
        Are you referring to HE's tunnel broker service? If so, then yeah, that's a VPN.

toast0 5y ago

I wonder if you could steer clients away from connections where the NIC and storage nodes are mismatched. Something like close the connection after N requests/N minutes if the nodes are mismatched, but leave it open indefinitely if they match. There's of course a lot of ways for that to not be very helpful. You'd still have only a 25% of getting a port number that hashes to the right node the next time (assuming tcp port number is involved at all, if it's just src and dest ips then client connections from the same IP would always hash the same, and that's probably a big portion of your clients), and if establishing connections is expensive enough (or clients aren't good at it) then that's a negative. Also if a stream's files don't tend to stay on the same node, then churning connections to get to the right node doesn't help if the next segment is on a different node. I'm sure there are other scenarios too. I know some other CDN appliance setups do use multiple IPs, so you probably could get more, but it would add administrative stress.

topranks 5y ago

Is there a reason you can’t? In a past life we broke LAGs up to use different subnets per port to prevent traffic crossing the NUMA bridge. I’m sure there are good reasons you didn’t take this approach, be interesting to hear them.