5 ms·

"many clients, especially game consoles and other consumer electronics, only use one IP address returned from the DNS query." Is this referring to Netflix?

by Maxious 14y ago

EwanToo 14y ago
Netflix definitely had the ELB issues, I think it was described by them as "ELB routing issues" - the service was up, but some people were being sent to the wrong AZ. That certainly sounds like this doesn't it?
- rdl 14y ago
  It's especially interesting to look at Amazon.com vs. AWS. IIRC, Amazon.com doesn't use ELB, they use some proper load balancers (with DNS above) in front of several Regions of ELBs. I wonder if there's a market for global load balancing + advanced DNS tricks as a service, even if you're using Amazon entirely. Generally extending Amazon's IaaS with features Amazon should implement hasn't been a great strategy, as Amazon eventually implements an inferior but cheaper or easier (or at least default) solution for non-core parts. Kind of surprised to see www.amazon.com goes to DC for me (from SF), though. It's possible it's all in US-East, across all the AZ's, and just has localized sites like co.uk in other places.
  - zhoutong 14y ago
    The load balancer itself can always be a single point of failure, no matter how many layers of packaging you have. By default, DNS system doesn't support failover because multiple A records will result in round robin. So there are two possible solutions: 1. Use a DNS with health checking so that records can be changed dynamically for failover. 2. Use Anycast for load balances to broadcast only healthy nodes with the same IP prefix. Both ways are imperfect and not widely used in production systems.
    - rdl 14y ago
      Normally you Anycast DNS only, and then use DNS load balancing (of IPs) to load balance physical load balancers (which are themselves sharing IPs on subnets and doing high availability stuff) in front of banks of servers. You probably implement Geo, health, and load based optimization in the DNS layer and also in the physical load balancers. A single host out of ~30 dying at Site A doesn't affect how the DNS load balancing responds, but a critical number does. Due to broken DNS implementations in the wild, you ideally keep your per-site load balancers up in as many situations as possible (even if load must be shed, you keep core network and load balancers up) and redirect traffic from there to other sites. For larger sites, the big choice is running a single AS global network where IPs can move between datacenters vs. running each datacenter as a separate network. Advantages and disadvantages to each. The big disadvantage of all of this is you're now spending a few million a month on salaries, infrastructure (various PoPs, network gear, etc.) before you've bought a single server or served a single page. Not so lean.
      - zhoutong 14y ago
        > The big disadvantage of all of this is you're now spending a few million a month on salaries, infrastructure (various PoPs, network gear, etc.) before you've bought a single server or served a single page. Not so lean. So a business opportunity kicks in. That's how the cloud revolutionizes computing.
        rdl 14y ago
        The problem I think is that "really hard parts of the front end of your app, as a service" is only really valuable once someone has also provided "really hard parts of the back end of your app (database distribution/replication across multiple sites, ideally with resilience against various failures, and your choice on CAP), as a service", too. (on the front end, you should probably also be doing CDN+ (caching, WAN TCP acceleration, DoS prevention) too.) [also, wow, I didn't realize you were the "famous" Zhou Tong.]
        zhoutong 14y ago
  - micro_cam 14y ago
    I think this area will actually prove to be one of the key differences between AWS and Google's New Compute Engine. I'm not an expert in this area but the technical details of compute engine talk from google io (http://www.youtube.com/watch?v=ws2VRHq5ars http://www.youtube.com/watch?v=ws2VRHq5ars) makes it sound like they are doing something special so that global traffic travels on their backbone as much as possible and you can move ip from server to server.
  - xxpor 14y ago
    >Kind of surprised to see www.amazon.com goes to DC for me (from SF), though. It's possible it's all in US-East, across all the AZ's, and just has localized sites like co.uk in other places. Check where amazon.co.jp goes ;)
zhoutong 14y ago
AWS officially recommends CNAME records for ELBs, but the IP addresses don't change regularly and also CNAME for root host name won't work if other records are present, so many sysadmins straightaway use A records with the ELB IPs.
- rdl 14y ago
  I have never understood the design decision to require zone apex to be an A Record. (I mean, I know that is what the RFC says, but I don't know what the RFC authors were thinking back in ancient times.)
  - zhoutong 14y ago
    Well, disregarding the design decision, or the RFC, it's basically wrong to use CNAME alongside of other records, especially MX records. Considering most active domains have MX records under the root host name (like example.com.), CNAMEs won't work for these hostnames. If there's native implementation for an "ALIAS" record which only points to the corresponding A record of the target, then it will work anywhere.
- ceejayoz 14y ago
  > the IP addresses don't change regularly This is only true if you have a stable level of traffic, IIRC. If they have to scale the ELB up or down the IPs will change.

"many clients, especially game consoles and other consumer electronics, only use one IP address returned from the DNS query." Is this referring to Netflix?

by Maxious 14y ago

"many clients, especially game consoles and other consumer electronics, only use one IP address returned from the DNS query." Is this referring to Netflix?

EwanToo 14y ago

Netflix definitely had the ELB issues, I think it was described by them as "ELB routing issues" - the service was up, but some people were being sent to the wrong AZ. That certainly sounds like this doesn't it?

rdl 14y ago
It's especially interesting to look at Amazon.com vs. AWS. IIRC, Amazon.com doesn't use ELB, they use some proper load balancers (with DNS above) in front of several Regions of ELBs. I wonder if there's a market for global load balancing + advanced DNS tricks as a service, even if you're using Amazon entirely. Generally extending Amazon's IaaS with features Amazon should implement hasn't been a great strategy, as Amazon eventually implements an inferior but cheaper or easier (or at least default) solution for non-core parts. Kind of surprised to see www.amazon.com goes to DC for me (from SF), though. It's possible it's all in US-East, across all the AZ's, and just has localized sites like co.uk in other places.
- zhoutong 14y ago
  The load balancer itself can always be a single point of failure, no matter how many layers of packaging you have. By default, DNS system doesn't support failover because multiple A records will result in round robin. So there are two possible solutions: 1. Use a DNS with health checking so that records can be changed dynamically for failover. 2. Use Anycast for load balances to broadcast only healthy nodes with the same IP prefix. Both ways are imperfect and not widely used in production systems.
  - rdl 14y ago
    Normally you Anycast DNS only, and then use DNS load balancing (of IPs) to load balance physical load balancers (which are themselves sharing IPs on subnets and doing high availability stuff) in front of banks of servers. You probably implement Geo, health, and load based optimization in the DNS layer and also in the physical load balancers. A single host out of ~30 dying at Site A doesn't affect how the DNS load balancing responds, but a critical number does. Due to broken DNS implementations in the wild, you ideally keep your per-site load balancers up in as many situations as possible (even if load must be shed, you keep core network and load balancers up) and redirect traffic from there to other sites. For larger sites, the big choice is running a single AS global network where IPs can move between datacenters vs. running each datacenter as a separate network. Advantages and disadvantages to each. The big disadvantage of all of this is you're now spending a few million a month on salaries, infrastructure (various PoPs, network gear, etc.) before you've bought a single server or served a single page. Not so lean.
    - zhoutong 14y ago
      > The big disadvantage of all of this is you're now spending a few million a month on salaries, infrastructure (various PoPs, network gear, etc.) before you've bought a single server or served a single page. Not so lean. So a business opportunity kicks in. That's how the cloud revolutionizes computing.
      - rdl 14y ago
        The problem I think is that "really hard parts of the front end of your app, as a service" is only really valuable once someone has also provided "really hard parts of the back end of your app (database distribution/replication across multiple sites, ideally with resilience against various failures, and your choice on CAP), as a service", too. (on the front end, you should probably also be doing CDN+ (caching, WAN TCP acceleration, DoS prevention) too.) [also, wow, I didn't realize you were the "famous" Zhou Tong.]
        zhoutong 14y ago
- micro_cam 14y ago
  I think this area will actually prove to be one of the key differences between AWS and Google's New Compute Engine. I'm not an expert in this area but the technical details of compute engine talk from google io (http://www.youtube.com/watch?v=ws2VRHq5ars http://www.youtube.com/watch?v=ws2VRHq5ars) makes it sound like they are doing something special so that global traffic travels on their backbone as much as possible and you can move ip from server to server.
- xxpor 14y ago
  >Kind of surprised to see www.amazon.com goes to DC for me (from SF), though. It's possible it's all in US-East, across all the AZ's, and just has localized sites like co.uk in other places. Check where amazon.co.jp goes ;)

zhoutong 14y ago

AWS officially recommends CNAME records for ELBs, but the IP addresses don't change regularly and also CNAME for root host name won't work if other records are present, so many sysadmins straightaway use A records with the ELB IPs.

rdl 14y ago
I have never understood the design decision to require zone apex to be an A Record. (I mean, I know that is what the RFC says, but I don't know what the RFC authors were thinking back in ancient times.)
- zhoutong 14y ago
  Well, disregarding the design decision, or the RFC, it's basically wrong to use CNAME alongside of other records, especially MX records. Considering most active domains have MX records under the root host name (like example.com.), CNAMEs won't work for these hostnames. If there's native implementation for an "ALIAS" record which only points to the corresponding A record of the target, then it will work anywhere.
ceejayoz 14y ago
> the IP addresses don't change regularly This is only true if you have a stable level of traffic, IIRC. If they have to scale the ELB up or down the IPs will change.