7 ms·

Slow to scale/drain will make your utilization go into toilet since you’ll have to overprovision by a lot. There are many other issues with dns loadbalancing

fulafel 6y ago
Scaling up won't be a problem (new DNS names will just work). As for draining, it does slow down the draining some, but how many scenarios really need to retire endpoints at a quicker cadence than DNS can do, you can set the TTL to 10 minutes?
- cestith 6y ago
  We pull unhealthy VMs and containers from rotation in real time. Serving a bunch of 500 series errors to clients because your application server took a nosedive isn't great even with a 300 second TTL. We don't use NAT. When people say RFC1918 addresses aren't routable they mean you can't advertise them on the public Internet. You can totally have 10/8 switched and routed internally. Even if you don't want to mess with that, it's entirely possible to set a proxy to proxy from its public IP to another public IP, but why waste the IP space, especially in v4? I can server 80 applications each with half a dozen or more backend systems from about six proxy servers. The proxy isn't running the application and connecting to the RDBMS so each one can handle way, way more traffic than even an efficient application. Using a proxy that's layer 7 aware also lets you do things like scale different parts of a URI tree individually. If you're not proxying, every copy of app.example.com serves everything under https://app.example.com/ https://app.example.com/ even if it's one of 23 (about the most A records that will fit in UDP DNS) servers doing it. With a proxy, I can decide that https://app.example.com/freestuff/ https://app.example.com/freestuff/ is too busy and spikes the backends, and that the load of that and the rest of the stuff under / is too hard to scale for properly when taken together. So I just tell my proxies that everything for https://app.example.com/freestuff https://app.example.com/freestuff goes to server set A and the rest of stuff in / goes to server set B. Then the different performance and different demand for the two can be studied, improved, monitored, and reported separately at the VM or container (or even bare metal backend) level. I can also throw memcached or Redis into the mix and do site-wide limiting on an IP that's scanning or attacking my entire laundry list of applications. Even without those I can rate-limit what each proxy will accept from a single visitor globally or per backend type.
  - fulafel 6y ago
    How often does it happen that working containers suddenly fail like that though and it's critical to shave the minutes? I appreciate proxying does give you slightly better reaction time but this has to be weighed against the costs and outages caused by the complexity in the proxying. Re you app.example.com scenario: I was not thinking of serving the static assets and frontend code from the individual services, but rather having eg your SPA know about the different endpoints and serving the resources from their own service which would be stable. Re the memcached angle - the static serving can have caching by whatever mechanism and the API endpoints can also use memcached all they want, with at least as good control granularity as in the proxy case. But if you were thinking about a caching reverse proxy, I think that would be more prone to poisoning the cache with error replies than having the endpoints do their caching individually. Again, think of all the failure prone machinery you can elide in this scenario.
    - cestith 6y ago
      Reasonable rules for something like HAProxy are not at all prone to errors. Ours aren't even applied by hand. It only has to be made right when the automation module is updated, and that gets tested well before it goes to production. Varnish rules are also not that difficult, but I said nothing of caching reverse proxy. I mentioned Memcached specifically in the context of federating traffic counts across multiple non-caching proxies. Whether an application uses Memcached or not is irrelevant to the proxies.
  - fulafel 6y ago
    Re the v4 space question, for v4 from the public internet was covered in my original (fallback proxy switched on host header). Internally, using v4 10.x networks for internal systems is fraught with problems as you'll inevitably end up trying to connect other 10.x ambiguously addressed networks, and get bit by the ambiguity in security, monitoring, configuration etc. End-to-end addrssing with globally unique addresses is just a good idea, it's a big reason the internet model won over competing networking technologies.
    - 6y ago
- dilyevsky 6y ago
  It will not, in fact, just work bc many clients cache old entries for a long time so you won’t be able to solve your load spreading problem quickly at least in general case
  - fulafel 6y ago
    In browsers at least this should be a thing of the past - Chrome and Firefox both are documented to internally cache only for 1 minute. You might of course have non browser clients that don't respect DNS TTL which might be a problem if you're trying to introduce this into existing old systems.
    - dilyevsky 6y ago
      And then chrome and ff will query your isp resolver than caches for a day just because - oops
      - fulafel 6y ago
        Those ISP resolver caches do respect the DNS TTL per the spec, otherwise all kinds of stuff breaks. If you're concerned that this would still be an issue you could easily do some measurements. There's no incentive for ISPs to break the this given the "my internet is broken" UX and how tiny DNS traffic is.

fulafel 6y ago

Scaling up won't be a problem (new DNS names will just work). As for draining, it does slow down the draining some, but how many scenarios really need to retire endpoints at a quicker cadence than DNS can do, you can set the TTL to 10 minutes?

cestith 6y ago
We pull unhealthy VMs and containers from rotation in real time. Serving a bunch of 500 series errors to clients because your application server took a nosedive isn't great even with a 300 second TTL. We don't use NAT. When people say RFC1918 addresses aren't routable they mean you can't advertise them on the public Internet. You can totally have 10/8 switched and routed internally. Even if you don't want to mess with that, it's entirely possible to set a proxy to proxy from its public IP to another public IP, but why waste the IP space, especially in v4? I can server 80 applications each with half a dozen or more backend systems from about six proxy servers. The proxy isn't running the application and connecting to the RDBMS so each one can handle way, way more traffic than even an efficient application. Using a proxy that's layer 7 aware also lets you do things like scale different parts of a URI tree individually. If you're not proxying, every copy of app.example.com serves everything under https://app.example.com/ https://app.example.com/ even if it's one of 23 (about the most A records that will fit in UDP DNS) servers doing it. With a proxy, I can decide that https://app.example.com/freestuff/ https://app.example.com/freestuff/ is too busy and spikes the backends, and that the load of that and the rest of the stuff under / is too hard to scale for properly when taken together. So I just tell my proxies that everything for https://app.example.com/freestuff https://app.example.com/freestuff goes to server set A and the rest of stuff in / goes to server set B. Then the different performance and different demand for the two can be studied, improved, monitored, and reported separately at the VM or container (or even bare metal backend) level. I can also throw memcached or Redis into the mix and do site-wide limiting on an IP that's scanning or attacking my entire laundry list of applications. Even without those I can rate-limit what each proxy will accept from a single visitor globally or per backend type.
- fulafel 6y ago
  How often does it happen that working containers suddenly fail like that though and it's critical to shave the minutes? I appreciate proxying does give you slightly better reaction time but this has to be weighed against the costs and outages caused by the complexity in the proxying. Re you app.example.com scenario: I was not thinking of serving the static assets and frontend code from the individual services, but rather having eg your SPA know about the different endpoints and serving the resources from their own service which would be stable. Re the memcached angle - the static serving can have caching by whatever mechanism and the API endpoints can also use memcached all they want, with at least as good control granularity as in the proxy case. But if you were thinking about a caching reverse proxy, I think that would be more prone to poisoning the cache with error replies than having the endpoints do their caching individually. Again, think of all the failure prone machinery you can elide in this scenario.
  - cestith 6y ago
    Reasonable rules for something like HAProxy are not at all prone to errors. Ours aren't even applied by hand. It only has to be made right when the automation module is updated, and that gets tested well before it goes to production. Varnish rules are also not that difficult, but I said nothing of caching reverse proxy. I mentioned Memcached specifically in the context of federating traffic counts across multiple non-caching proxies. Whether an application uses Memcached or not is irrelevant to the proxies.
- fulafel 6y ago
  Re the v4 space question, for v4 from the public internet was covered in my original (fallback proxy switched on host header). Internally, using v4 10.x networks for internal systems is fraught with problems as you'll inevitably end up trying to connect other 10.x ambiguously addressed networks, and get bit by the ambiguity in security, monitoring, configuration etc. End-to-end addrssing with globally unique addresses is just a good idea, it's a big reason the internet model won over competing networking technologies.
  - 6y ago
dilyevsky 6y ago
It will not, in fact, just work bc many clients cache old entries for a long time so you won’t be able to solve your load spreading problem quickly at least in general case
- fulafel 6y ago
  In browsers at least this should be a thing of the past - Chrome and Firefox both are documented to internally cache only for 1 minute. You might of course have non browser clients that don't respect DNS TTL which might be a problem if you're trying to introduce this into existing old systems.
  - dilyevsky 6y ago
    And then chrome and ff will query your isp resolver than caches for a day just because - oops
    - fulafel 6y ago
      Those ISP resolver caches do respect the DNS TTL per the spec, otherwise all kinds of stuff breaks. If you're concerned that this would still be an issue you could easily do some measurements. There's no incentive for ISPs to break the this given the "my internet is broken" UX and how tiny DNS traffic is.