5 ms·

> In Australia you're just 200ms from everything most of the time. (...) > The only app that I am always impressed with is WhatsApp. It's always the first app

> In Australia you're just 200ms from everything most of the time. (...) > The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast. The 200ms is telling. I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. Most global companies deploy only to at most three regions: * the US (us-east, us-central, us-east+us-east) * Europe (west-europe), * and somewhat rarely far-east (meither us-west or Japan) This means that places such as south Africa, south America, and of course Australia typically have to pull data from one of these regions, which means latencies of at least 200ms due to physics. Australia is particularly hit because, even with dedicated deployments in their theoretical catchment area, often these servers are actualy located in an entirely separate continent (west-us or Japan) and thus users do experience the performance impact of having packets cross half a globe.

luuurker 2y ago
Part of the problem seems to be the cost of peering in Australia: https://blog.cloudflare.com/bandwidth-costs-around-the-world https://blog.cloudflare.com/bandwidth-costs-around-the-world
2143 2y ago
I kinda chuckled that you left out most of Asia. Most global companies (at least the US-based ones) have deployed in India, where I'm at right now. I suppose a billion people online is too big of a market to ignore (or not; I really don't know). Or there are services that I'm completely unaware of that's not in India. Internet's pretty fast as well. Much faster than a certain conspicuous European country you'd expect to have fast internet ;)
toast0 2y ago
> I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. So, I used to work at WhatsApp. And we got this kind of praise when we only had servers in Reston, Virginia (not at aws us-east1, but in the same neighborhood). Nowadays, Facebook is most likely terminating connections in Australia, but messaging most likely goes through another continent. Calling within Australia should stay local though (either p2p or through a nearby relay). There's lots of things WhatsApp does to improve experience on low quality networks that other services don't (even when we worked in the same buildings and told them they should consider things!) In no particular order: 0) offline first, phone is the source of truth, although there's multi-device now. You don't need to be online to read messages you have, or to write messages to be sent whenever you're online. Email used to work like this for everyone; and it was no big deal to grab mail once in a while, read it and reply, and then send in a batch. Online messaging is great, if you can, but for things like being on a commuter train where connectivity ebbs and flows, it's nice to pick up messages when you can. a) hardcode fallback ips for when DNS doesn't work (not if) b) setup "0rtt" fast resume, so you can start getting messages on the second round trip. This is part of noise pipes or whatever they're called, and tls 1.3 c) do reasonable-ish things to work with MTU. In the old days, FreeBSD reflected the client MSS back to it, which helps when there's a tunnel like PPPoE and it only modifies outgoing syns and not incoming syn+ack. Linux never did that, and afaik, FreeBSD took it out. Behind Facebook infrastructure, they just hardcode the mss for i think 1480 MTU (you can/should check with tcpdump). I did some limited testing, and really the best results come from monitoring for /24's with bad behavior (it's pretty easy, if you look for it --- never got any large packets and packet gaps are a multiple of MSS - space for tcp timestamps) and then sending back client - 20 to those; you could also just always send back client - 20. I think Android finally started doing pMTUD blackhole detection stuff a couple years back, Apple has been doing it really well for longer. Path MTU Discovery is still an issue, and anything you can do to make it happier is good. d) connect in the background to exchange messages when possible. Don't post notifications unless the message content is on the device. Don't be one of those apps that can only load messsages from the network when the app is in the foreground, because the user might not have connectivity then e) prioritize messages over telemetry. Don't measure everything, only measure things when you know what you'll do with the numbers. Everybody hates telemetry, but it can be super useful as a developer. But if you've got giant telemetry packs to upload, that's bad by itself, and if you do them before you get messages in and out, you're failing the user. f) pay attention to how big things are on the wire. Not everything needs to get shrunk as much as possible, but login needs to be very tight, and message sending should be too. IMHO, http and json and xml are too bulky for those, but are ok for multimedia because the payload is big so framing doesn't matter as much, and they're ok for low volume services because they're low volume.
- cheema33 2y ago
  > I used to work at WhatsApp.. Do you know why there is a 4 device limit? I run into this limit quite a bit, because I have a lot more devices. And... Why is there is WhatsApp for most commonly used devices, but iPads?
  - swiftcoder 2y ago
    > Why is there is WhatsApp for most commonly used devices, but iPads? I was frustrated by this a while back, so I asked the PMs. Basically when investing engineering effort WhatsApp prioritises the overall number of users connected, and supporting iPads doesn't really move that metric, because (a) the vast majority of iPad owners also own a smartphone, and (b) iPads are pretty rare outside of wealthy western cities.
  - toast0 2y ago
    I've been gone too long for accurate answers, but I can guess. For iPad, I think it's like the sibling notes; expected use is very low, so it didm't justify the engineering cost while I was there. But I see some signs it might happen eventually [1]; WhatsApp for Android Tablets wasn't a thing when I was there either, but it is now. For the four device limit, there's a few things going on IMHO. Synchronization is hard and the more devices are playing, the harder it is. Independent devices makes it easier in some ways because the user devices don't have to be online together to communicate (like when whatsapp web was essentially a remote control for your phone), but it does mean that all of your communications partner's devices have to work harder and the servers have to work harder, too. Four deviced covers your phone, a desktop at home and work, and a laptop; but really most of the users only have a phone. Allowing more devices makes it more likely that you'll lose track of one or not use it for long enough that it's lost sync, etc. WhatsApp has usually focused on product features that benefit the most users, and more than 4 devices isn't going to benefit many people, and 4 is plenty for internal use (phone, prod build, dev build, home computer). I'm sure they've got metrics of how many devices are used, and if there's a lot of 4 device users and enough requests, it's a #define somewhere. [1] https://www.macworld.com/article/668638/how-to-get-whatsapp-on-ipad.html https://www.macworld.com/article/668638/how-to-get-whatsapp-...
- 2y ago

luuurker 2y ago

Part of the problem seems to be the cost of peering in Australia: https://blog.cloudflare.com/bandwidth-costs-around-the-world https://blog.cloudflare.com/bandwidth-costs-around-the-world

2143 2y ago

I kinda chuckled that you left out most of Asia. Most global companies (at least the US-based ones) have deployed in India, where I'm at right now. I suppose a billion people online is too big of a market to ignore (or not; I really don't know). Or there are services that I'm completely unaware of that's not in India. Internet's pretty fast as well. Much faster than a certain conspicuous European country you'd expect to have fast internet ;)

toast0 2y ago

> I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. So, I used to work at WhatsApp. And we got this kind of praise when we only had servers in Reston, Virginia (not at aws us-east1, but in the same neighborhood). Nowadays, Facebook is most likely terminating connections in Australia, but messaging most likely goes through another continent. Calling within Australia should stay local though (either p2p or through a nearby relay). There's lots of things WhatsApp does to improve experience on low quality networks that other services don't (even when we worked in the same buildings and told them they should consider things!) In no particular order: 0) offline first, phone is the source of truth, although there's multi-device now. You don't need to be online to read messages you have, or to write messages to be sent whenever you're online. Email used to work like this for everyone; and it was no big deal to grab mail once in a while, read it and reply, and then send in a batch. Online messaging is great, if you can, but for things like being on a commuter train where connectivity ebbs and flows, it's nice to pick up messages when you can. a) hardcode fallback ips for when DNS doesn't work (not if) b) setup "0rtt" fast resume, so you can start getting messages on the second round trip. This is part of noise pipes or whatever they're called, and tls 1.3 c) do reasonable-ish things to work with MTU. In the old days, FreeBSD reflected the client MSS back to it, which helps when there's a tunnel like PPPoE and it only modifies outgoing syns and not incoming syn+ack. Linux never did that, and afaik, FreeBSD took it out. Behind Facebook infrastructure, they just hardcode the mss for i think 1480 MTU (you can/should check with tcpdump). I did some limited testing, and really the best results come from monitoring for /24's with bad behavior (it's pretty easy, if you look for it --- never got any large packets and packet gaps are a multiple of MSS - space for tcp timestamps) and then sending back client - 20 to those; you could also just always send back client - 20. I think Android finally started doing pMTUD blackhole detection stuff a couple years back, Apple has been doing it really well for longer. Path MTU Discovery is still an issue, and anything you can do to make it happier is good. d) connect in the background to exchange messages when possible. Don't post notifications unless the message content is on the device. Don't be one of those apps that can only load messsages from the network when the app is in the foreground, because the user might not have connectivity then e) prioritize messages over telemetry. Don't measure everything, only measure things when you know what you'll do with the numbers. Everybody hates telemetry, but it can be super useful as a developer. But if you've got giant telemetry packs to upload, that's bad by itself, and if you do them before you get messages in and out, you're failing the user. f) pay attention to how big things are on the wire. Not everything needs to get shrunk as much as possible, but login needs to be very tight, and message sending should be too. IMHO, http and json and xml are too bulky for those, but are ok for multimedia because the payload is big so framing doesn't matter as much, and they're ok for low volume services because they're low volume.

cheema33 2y ago
> I used to work at WhatsApp.. Do you know why there is a 4 device limit? I run into this limit quite a bit, because I have a lot more devices. And... Why is there is WhatsApp for most commonly used devices, but iPads?
- swiftcoder 2y ago
  > Why is there is WhatsApp for most commonly used devices, but iPads? I was frustrated by this a while back, so I asked the PMs. Basically when investing engineering effort WhatsApp prioritises the overall number of users connected, and supporting iPads doesn't really move that metric, because (a) the vast majority of iPad owners also own a smartphone, and (b) iPads are pretty rare outside of wealthy western cities.
- toast0 2y ago
  I've been gone too long for accurate answers, but I can guess. For iPad, I think it's like the sibling notes; expected use is very low, so it didm't justify the engineering cost while I was there. But I see some signs it might happen eventually [1]; WhatsApp for Android Tablets wasn't a thing when I was there either, but it is now. For the four device limit, there's a few things going on IMHO. Synchronization is hard and the more devices are playing, the harder it is. Independent devices makes it easier in some ways because the user devices don't have to be online together to communicate (like when whatsapp web was essentially a remote control for your phone), but it does mean that all of your communications partner's devices have to work harder and the servers have to work harder, too. Four deviced covers your phone, a desktop at home and work, and a laptop; but really most of the users only have a phone. Allowing more devices makes it more likely that you'll lose track of one or not use it for long enough that it's lost sync, etc. WhatsApp has usually focused on product features that benefit the most users, and more than 4 devices isn't going to benefit many people, and 4 is plenty for internal use (phone, prod build, dev build, home computer). I'm sure they've got metrics of how many devices are used, and if there's a lot of 4 device users and enough requests, it's a #define somewhere. [1] https://www.macworld.com/article/668638/how-to-get-whatsapp-on-ipad.html https://www.macworld.com/article/668638/how-to-get-whatsapp-...
2y ago