3 ms·

Skipping an unnecessary intermediary is worth considering. Load balancing isn't without cost, and load balancers subtly (or unsubtly) messing up connections is

by toast0 2y ago

Skipping an unnecessary intermediary is worth considering. Load balancing isn't without cost, and load balancers subtly (or unsubtly) messing up connections is an issue. I've also used providers where their load balancers had worse availability than our hosts. If you control the clients, it's reasonable to call the platform dns api to get a list of ips and shuffle and iterate through in an appropriate way. Even better if you have a few stablely allocated IPs you can distribute in client binaries for when DNS is broken; but DNS is often not broken and it's nice to use for operational changes without having to push new configuration/binaries everytime you update the cluster. If your clients are browsers, default behavior is ok; they usually use IPs in order, which can be problematic [1], but otherwise, they have good retry behavior: on connection refused they try another IP right away, in case of timeout, they try at least a few different IPs. It's not ideal, and I'd use a load balancer for browsers, at least to serve the initial page load if feasible, and maybe DNS RR and semi-smart client logic in JS for websockets/etc; but DNS RR is workable for a whole site too. If your clients are not browsers and not controlled by you, best of luck? I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. And that clients behind those resolvers will have trouble when you update DNS, but if your loadbalancer is behind a DNS name, when it needs to change addresses, you'll deal with that then, and you won't have experience. [1] one of the RFCs suggests that OS apis should sort responses by prefix match, which might make sense if IP prefixes were heirarchical as a proxy to get to a least network distance server. But in the real world, numerically adjacent /24s are often not network adjacent, but if your servers have widely disparate addresses, you may see traffic from some client ips gravitate towards numerically similar server ips.

ectospheno 2y ago
> I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. I’ve run a min ttl of 3600 on my home network for over a year. No one has complained yet.
- toast0 2y ago
  That's only because there's no way for service operators to effectively complain when your clients continue to hit service ips for 55 minutes after you should. And if there was, we'd first yell at all the people who continue to hit service ips for weeks and months after a change... by the time we get to complaining about one home using an hour ttl, it's not a big deal.
  - ectospheno 2y ago
    I take the point of view that if me not honoring your 60 second ttl breaks your site for me then I want to know so I stop going there.
ignoramous 2y ago
> you control the clients, it's reasonable to call the platform dns api to get a list of ips and shuffle and iterate through in an appropriate way. Even better if you have a few stable allocated IPs you can distribute in client binaries for when DNS is broken You know, not many apps do this but in particular WhatsApp does! Was it you?
- toast0 2y ago
  Not my idea, but I supported it. Originally, client build scripts resolved the service names at build time, and that worked ok because our hosts tended to have a lot of longevity, and DNS tends to work, but things got a little better when we were more intentional about selecting the servers to be in the list, and keep track of which ones were in the list, so retirements could be managed a bit better. And I pushed until we got agreement on a set of FB load balancer IPs to include as well.
  - ignoramous 2y ago
    Nice. Thanks! Another peculiar thing I observed (way back when) was... in the most losiest of lossy EDGE/2G environments in rural India, only WhatsApp managed to work (email clients, browsers, other chat apps didn't). Not only was WhatsApp able to send/recieve messages but also upload/download ~100KB PDFs (over what seemed like a 20m to 30m slow process, but it did complete alright). If it is okay to disclose, did WhatsApp build its own protocol/impl atop TCP/UDP for such scenarios?
    - toast0 2y ago
      The EU marketplace disclosures on protocol seem to be pretty close to what I remember of the non-public protocols. Chat is basically binary encoded XMPP, with essentially a compression dictionary, so per iq overhead is minimal. Especially for the start of connection stuff (login, offline message delivery), we counted bytes and made accomidations for typical network issues we would see. Not acking a big chunk of offline messages after a few tries? Let's send one at a time and see if that works, etc. Our socket timeouts were rather long as well. Before the move into Facebook infra, servers were in the US only, and rural India is a long ways from the US; and last mile contention on 2G gets real rough out there too... I want to say timeouts were on the order of 30 seconds? Multimedia (attachments) was https, with resumption. I don't remember the full history, originally I don't think we had resumption on uploads, there's some coordination required for that, which IIRC started as more or less send an IQ that you want to upload a file with a hash of the file, and get a response of either what the download url is if the file was complete, or where to upload and what byte to start with if not. I think it's likely different now, but probably still https based. I wanted to move it so multimedia would be either multiplexed on the chat channel or using a similar protocol to the chat channel, but I didn't have the pull, and I got redirected into pushing TLS 1.3 into our Android client's mms upload/download instead; I didn't do the code there, just prototyping to show it could be possible, and then was more of a facilitator than a contributor. I'm not sure I got all the benefits I was looking for, but there were some, and it kept me busy while I was wrapping up our pre-FB hosting and my time at WA.

by toast0 2y ago

ectospheno 2y ago

> I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. I’ve run a min ttl of 3600 on my home network for over a year. No one has complained yet.

toast0 2y ago
That's only because there's no way for service operators to effectively complain when your clients continue to hit service ips for 55 minutes after you should. And if there was, we'd first yell at all the people who continue to hit service ips for weeks and months after a change... by the time we get to complaining about one home using an hour ttl, it's not a big deal.
- ectospheno 2y ago
  I take the point of view that if me not honoring your 60 second ttl breaks your site for me then I want to know so I stop going there.

ignoramous 2y ago

> you control the clients, it's reasonable to call the platform dns api to get a list of ips and shuffle and iterate through in an appropriate way. Even better if you have a few stable allocated IPs you can distribute in client binaries for when DNS is broken You know, not many apps do this but in particular WhatsApp does! Was it you?

toast0 2y ago
Not my idea, but I supported it. Originally, client build scripts resolved the service names at build time, and that worked ok because our hosts tended to have a lot of longevity, and DNS tends to work, but things got a little better when we were more intentional about selecting the servers to be in the list, and keep track of which ones were in the list, so retirements could be managed a bit better. And I pushed until we got agreement on a set of FB load balancer IPs to include as well.
- ignoramous 2y ago
  Nice. Thanks! Another peculiar thing I observed (way back when) was... in the most losiest of lossy EDGE/2G environments in rural India, only WhatsApp managed to work (email clients, browsers, other chat apps didn't). Not only was WhatsApp able to send/recieve messages but also upload/download ~100KB PDFs (over what seemed like a 20m to 30m slow process, but it did complete alright). If it is okay to disclose, did WhatsApp build its own protocol/impl atop TCP/UDP for such scenarios?
  - toast0 2y ago
    The EU marketplace disclosures on protocol seem to be pretty close to what I remember of the non-public protocols. Chat is basically binary encoded XMPP, with essentially a compression dictionary, so per iq overhead is minimal. Especially for the start of connection stuff (login, offline message delivery), we counted bytes and made accomidations for typical network issues we would see. Not acking a big chunk of offline messages after a few tries? Let's send one at a time and see if that works, etc. Our socket timeouts were rather long as well. Before the move into Facebook infra, servers were in the US only, and rural India is a long ways from the US; and last mile contention on 2G gets real rough out there too... I want to say timeouts were on the order of 30 seconds? Multimedia (attachments) was https, with resumption. I don't remember the full history, originally I don't think we had resumption on uploads, there's some coordination required for that, which IIRC started as more or less send an IQ that you want to upload a file with a hash of the file, and get a response of either what the download url is if the file was complete, or where to upload and what byte to start with if not. I think it's likely different now, but probably still https based. I wanted to move it so multimedia would be either multiplexed on the chat channel or using a similar protocol to the chat channel, but I didn't have the pull, and I got redirected into pushing TLS 1.3 into our Android client's mms upload/download instead; I didn't do the code there, just prototyping to show it could be possible, and then was more of a facilitator than a contributor. I'm not sure I got all the benefits I was looking for, but there were some, and it kept me busy while I was wrapping up our pre-FB hosting and my time at WA.