10 ms·

In my defense, I was assuming SNI (aka the modern internet), and that the IP was reachable by those you care for it to be reachable by. Ignoring these issues, i

In my defense, I was assuming SNI (aka the modern internet), and that the IP was reachable by those you care for it to be reachable by. Ignoring these issues, is there an "engineering" reason why a single IP won't work, in terms of, for example, hardware can't demultiplex the aggregate ingress volume of CloudFlare and handle DOS mitigation? I guess I'm asking this because of how woeful looking the "load-balancing" solutions are from the major cloud providers. I feel they way they're externally documented, and how their APIs are specified, hitting them with more than a 40Gbps fat-server's load of traffic will cause issues, regardless of how many hosts you have serving that load. I'd appreciate some insight from those who handle such crazy amounts of traffic.

jgrahamc 12y ago
If you think of a connection is defined by the tuple (source_ip, source_port, destination_ip, destination_port) then you might run into problems if destination_ip was a single value, just because whatever hashing you are using/table lookups for connection management, DoS protection etc. etc. might have problems with the sheer size. We are doing a huge amount of traffic and I can imagine having to engineer around some things related to that. But the real issues are the ones that outline above.
cortesoft 12y ago
I work for a major CDN that uses anycast, and there are a number of reasons. I won't go into too many of them, but quickly: 1) Anycast doesn't give you fine grain control. Once we announce our anycast routes, what traffic actually gets sent where is out of our control - it is based on the peering arrangements of our transit providers. If we need to balance traffic between our pops, we need finer grained control than a single anycast IP. 2) IP addresses get blocked for all sorts of reasons (looking at you China!) If all customers were on one IP address, as soon as China decides to block one customer, they are all blocked. 3) Anycast sometimes has weird behavior. For example, traffic might be sent to a datacenter that might be close in terms of peer links, but far in terms of physical distance and latency. Using DNS, we can route around these issues. I am not sure what you mean about the "40gbps fat-server's load of traffic" causing issues. We handle many customers that push more than that.
- 15155 12y ago
  > 2) IP addresses get blocked for all sorts of reasons (looking at you China!) If all customers were on one IP address, as soon as China decides to block one customer, they are all blocked. Feature, not a bug.
- throwaway000002 11y ago
  Just disregard my "fat-server" comment. It's more from being disillusioned with all load-balancing solutions being tied to the service provider. I'd like something that was cloud agnostic, that was peered at multiple points with the major providers. I guess this is step 1 in the same effort from CloudFlare, before they add AWS and Azure. But their interface is over-simple, understandable considering the technical proficiency of their average customer. CloudFlare is too one-size fits all, but from a business perspective it's totally understandable. I know it's a pipe dream, but I wish we could defragment the IP space and clean up the BGP tables. It would at least make anycast more reliable without resorting to DNS tricks like edns-client-subnet. As for IP blocking, if undesirable sites are behind the same IP as publically demanded ones, it could make blocking actions harder to get the populace to support. But worrying about authoritative regimes is not my concern. After all, why make a service accessible if you cannot monetize the user base sufficiently. Yes, I'm a little jaded.

jgrahamc 12y ago

If you think of a connection is defined by the tuple (source_ip, source_port, destination_ip, destination_port) then you might run into problems if destination_ip was a single value, just because whatever hashing you are using/table lookups for connection management, DoS protection etc. etc. might have problems with the sheer size. We are doing a huge amount of traffic and I can imagine having to engineer around some things related to that. But the real issues are the ones that outline above.

cortesoft 12y ago

I work for a major CDN that uses anycast, and there are a number of reasons. I won't go into too many of them, but quickly: 1) Anycast doesn't give you fine grain control. Once we announce our anycast routes, what traffic actually gets sent where is out of our control - it is based on the peering arrangements of our transit providers. If we need to balance traffic between our pops, we need finer grained control than a single anycast IP. 2) IP addresses get blocked for all sorts of reasons (looking at you China!) If all customers were on one IP address, as soon as China decides to block one customer, they are all blocked. 3) Anycast sometimes has weird behavior. For example, traffic might be sent to a datacenter that might be close in terms of peer links, but far in terms of physical distance and latency. Using DNS, we can route around these issues. I am not sure what you mean about the "40gbps fat-server's load of traffic" causing issues. We handle many customers that push more than that.

15155 12y ago
> 2) IP addresses get blocked for all sorts of reasons (looking at you China!) If all customers were on one IP address, as soon as China decides to block one customer, they are all blocked. Feature, not a bug.
throwaway000002 11y ago
Just disregard my "fat-server" comment. It's more from being disillusioned with all load-balancing solutions being tied to the service provider. I'd like something that was cloud agnostic, that was peered at multiple points with the major providers. I guess this is step 1 in the same effort from CloudFlare, before they add AWS and Azure. But their interface is over-simple, understandable considering the technical proficiency of their average customer. CloudFlare is too one-size fits all, but from a business perspective it's totally understandable. I know it's a pipe dream, but I wish we could defragment the IP space and clean up the BGP tables. It would at least make anycast more reliable without resorting to DNS tricks like edns-client-subnet. As for IP blocking, if undesirable sites are behind the same IP as publically demanded ones, it could make blocking actions harder to get the populace to support. But worrying about authoritative regimes is not my concern. After all, why make a service accessible if you cannot monetize the user base sufficiently. Yes, I'm a little jaded.