5 ms·

Having a lot of experience commuting on underground public transport (intermittent, congested), and living/working in Australia (remote), I can safely say that

Having a lot of experience commuting on underground public transport (intermittent, congested), and living/working in Australia (remote), I can safely say that most services are terrible for people without "ideal" network conditions. On the London Underground it's particularly noticeable that most apps are terrible at handling network that comes and goes every ~2 minutes (between stops), and which takes ~15s to connect to each AP as a train with 500 people on it all try to connect at the same time. In Australia you're just 200ms from everything most of the time. That might not seem like much, but it really highlights which apps trip up on the N+1 request problem. The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast.

pimeys 2y ago
I guess in London you get wifi only on stops, it's the same in Berlin. In Helsinki the wifi connection is available inside the trains, and in the stations. So you never get a connection loss when moving. I never understood the decision in Berlin to do this, why not just provide internet inside the train... And yeah, most of the internet works very badly when you drop the network all the time...
- pintxo 2y ago
  Berlin did not have mobile connections inside the tunnels until very recently (this year, I believe). This included the trains not being connected to any outside network. Thus wifi on the subway was useless to implement.
  - saagarjha 2y ago
    I was in Berlin earlier this month and the cellular connections underground were quite good now. So maybe this is less of a problem?
    - nicbou 2y ago
      It's provider-specific
      - alanpearce 2y ago
        Not any more? https://unternehmen.bvg.de/pressemitteilung/grossprojekt-erfolgreich-beendet-highspeed-mobilfunk-fuer-alle-in-der-berliner-u-bahn/ https://unternehmen.bvg.de/pressemitteilung/grossprojekt-erf... Summary: since 2024-05-06, users of all networks also get LTE in the U-Bahn thanks to a project between BVG and Teléfonica (not surprising that Teléfonica deployed the infra because they had the best U-Bahn LTE coverage beforehand)
  - amaccuish 2y ago
    They did if you were on o2, that's why I'm still with Aldi Talk (they use the o2 network); they've had LTE through the entire network for a while now. The new thing is 5G for everyone.
  - rkachowski 2y ago
    Despite Berlin's general lack of parity with modern technology, I've never actually had a problem with internet access across the ubahn network in the past decade. I noticed that certain carriers used to have very different availability when travelling and so switched to a better one, but I was always surprised at being able to handle mobile data whilst underground.
    - nicbou 2y ago
      Really? I don't even get consistent internet on the Ringbahn. There are lots of holes in the coverage in Berlin. Which provider are you with? Vodafone is still dead in large parts of the U-Bahn, but I know that one of them works much better.
      - avh02 2y ago
        used to have spotty coverage underground with vodafone, when i switched to telekom, internet suddenly magically worked underground on the routes i used. I believe someone published a map of the data coverage of different providers on the berlin ubahn, but probably outdated now
      - rkachowski 2y ago
        Yeah, admittedly this year I've also started experiencing holes on the ringbahn (strangely and consistently around frankfurter allee), but the ubahn has been fine. I'm with sim.de which I believe is essentially an O2 reseller (apn references o2)
        immibis 2y ago
        Gesundbrunnen too.
      - brabel 2y ago
        Wow! I was in Berlin last week and kept losing connection... like all the time. I use 3 with a Swedish plan. In Sweden, it literally never drops, not on trains, not on metro, not on faraway mountains... it works everywhere.
- Liskni_si 2y ago
  Yes, right now it's mostly just wifi at stations only. However, they're deploying 4G/5G coverage in the tunnels and expect 80% coverage by the end of 2024 [1]. So… you can expect apps developed by engineers in London to get much worse on slow internet in 2025. :-) [1]: https://tfl.gov.uk/campaign/station-wifi https://tfl.gov.uk/campaign/station-wifi
- bombcar 2y ago
  WiFi at a stop is as easy as putting up a few wireless routers, it's a bit more complex than at home but the same general idea. Wifi inside the trains involves much more work, and to get them to ALSO be seamless across the entire setup - even harder. Easily 10x or 100x the cost. It's sad, because the Internet shouldn't be that bad when the network drops all the time; it should just be slower as it waits to send good data.
  - Anotheroneagain 2y ago
    I think they just put a wire in the tunnel.
    - greenish_shores 2y ago
      Yes, that's the best way which is often used. A "leaky cable" aka "leaky feeder", to be particular.
chipdart 2y ago
> In Australia you're just 200ms from everything most of the time. (...) > The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast. The 200ms is telling. I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. Most global companies deploy only to at most three regions: * the US (us-east, us-central, us-east+us-east) * Europe (west-europe), * and somewhat rarely far-east (meither us-west or Japan) This means that places such as south Africa, south America, and of course Australia typically have to pull data from one of these regions, which means latencies of at least 200ms due to physics. Australia is particularly hit because, even with dedicated deployments in their theoretical catchment area, often these servers are actualy located in an entirely separate continent (west-us or Japan) and thus users do experience the performance impact of having packets cross half a globe.
- luuurker 2y ago
  Part of the problem seems to be the cost of peering in Australia: https://blog.cloudflare.com/bandwidth-costs-around-the-world https://blog.cloudflare.com/bandwidth-costs-around-the-world
- 2143 2y ago
  I kinda chuckled that you left out most of Asia. Most global companies (at least the US-based ones) have deployed in India, where I'm at right now. I suppose a billion people online is too big of a market to ignore (or not; I really don't know). Or there are services that I'm completely unaware of that's not in India. Internet's pretty fast as well. Much faster than a certain conspicuous European country you'd expect to have fast internet ;)
- toast0 2y ago
  > I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. So, I used to work at WhatsApp. And we got this kind of praise when we only had servers in Reston, Virginia (not at aws us-east1, but in the same neighborhood). Nowadays, Facebook is most likely terminating connections in Australia, but messaging most likely goes through another continent. Calling within Australia should stay local though (either p2p or through a nearby relay). There's lots of things WhatsApp does to improve experience on low quality networks that other services don't (even when we worked in the same buildings and told them they should consider things!) In no particular order: 0) offline first, phone is the source of truth, although there's multi-device now. You don't need to be online to read messages you have, or to write messages to be sent whenever you're online. Email used to work like this for everyone; and it was no big deal to grab mail once in a while, read it and reply, and then send in a batch. Online messaging is great, if you can, but for things like being on a commuter train where connectivity ebbs and flows, it's nice to pick up messages when you can. a) hardcode fallback ips for when DNS doesn't work (not if) b) setup "0rtt" fast resume, so you can start getting messages on the second round trip. This is part of noise pipes or whatever they're called, and tls 1.3 c) do reasonable-ish things to work with MTU. In the old days, FreeBSD reflected the client MSS back to it, which helps when there's a tunnel like PPPoE and it only modifies outgoing syns and not incoming syn+ack. Linux never did that, and afaik, FreeBSD took it out. Behind Facebook infrastructure, they just hardcode the mss for i think 1480 MTU (you can/should check with tcpdump). I did some limited testing, and really the best results come from monitoring for /24's with bad behavior (it's pretty easy, if you look for it --- never got any large packets and packet gaps are a multiple of MSS - space for tcp timestamps) and then sending back client - 20 to those; you could also just always send back client - 20. I think Android finally started doing pMTUD blackhole detection stuff a couple years back, Apple has been doing it really well for longer. Path MTU Discovery is still an issue, and anything you can do to make it happier is good. d) connect in the background to exchange messages when possible. Don't post notifications unless the message content is on the device. Don't be one of those apps that can only load messsages from the network when the app is in the foreground, because the user might not have connectivity then e) prioritize messages over telemetry. Don't measure everything, only measure things when you know what you'll do with the numbers. Everybody hates telemetry, but it can be super useful as a developer. But if you've got giant telemetry packs to upload, that's bad by itself, and if you do them before you get messages in and out, you're failing the user. f) pay attention to how big things are on the wire. Not everything needs to get shrunk as much as possible, but login needs to be very tight, and message sending should be too. IMHO, http and json and xml are too bulky for those, but are ok for multimedia because the payload is big so framing doesn't matter as much, and they're ok for low volume services because they're low volume.
camel-cdr 2y ago
It's not only the services them self. I have a very slow mobile connection, and one thing that bothered me immensly is downloading images in the browser: How is it, than when I go to a .jpg url to view an image in the browser it takes way longer and sometimes times out, than hopping over to termux and running wget. I had this problem with both firefox and chrome based browsers. Note that even the wget download usually takes 10-30 seconds on my mobile connection.
- giantrobot 2y ago
  Too many services today do stupid image transcoding today. While the URL says jpg it will decide that because your browser supports WebP that what you really must have wanted was a WebP. It'll then either transcode or just send you WebP data for the image or send you a redirect. This is rarely what you actually want. With wget it sends you the source you actually requested and doesn't try to get clever (stupid). Google likes WebP so that means everyone needs to join the WebP cargo cult even if it means transcoding a lossy format to another lossy format.
- chris_pie 2y ago
  I have the same issue with nearly every static asset.
- bombcar 2y ago
  Browsers usually try to multiplex things, sometimes even the same image if the server supports "get specific byte range" or whatever. There may be a setting to turn a browser back into a dumb wget visual displayer.
- armchair_expert 2y ago
  You can try going into proxy settings and setting to "none" instead of autodetect. Also, the dns server used by the browser could be different (and slower).
kylehotchkiss 2y ago
WhatsApp has a massive audience in developing countries where it's normal for people to have slower internet and much slower devices. That perspective being so embedded in their development goals certainly has given WhatsApp good reason to be the leading messaging platform in many countries around the world
- nicbou 2y ago
  It works remarkably well when your phone runs out of data and you get capped at 8 kbps. Even voice calls work smoothly.
  - qingcharles 2y ago
    LOL 8kbps. Damn. That takes me back. I built the first version of one of the world's largest music streaming sites on a 9.6kbps connection. I was working from home (we had no offices yet) and my cable Internet got cut off. My only back up was a serial cable to a 2G Nokia 9000i. I had to re-encode a chunk of the music catalog at 8kbps so I could test it from home before I pushed the code to production. Psychoacoustic compression is a miracle.
    - greenish_shores 2y ago
      Nokia 9000i, so you had to work on CSD (which is usually billed per-minute, like dial-up), not even GPRS. How much did that cost you? :P BTW, an interesting thing is that some/most carriers allow you to use CSD/HSCSD over 3G these days, and you can establish data CSD connection between two phone numbers, yielding essentially a dedicated L2 pipe which isn't routed over internet. Can have much lower latency and jitter if that's what you need. Some specialized telemetry is still using that, however as 3G is slowly getting phased out, it will probably have to change.
      - qingcharles 2y ago
        God, the cost was probably horrid, but I was connecting in, setting tasks running and logging out. This was late 1999 in the UK, so per-minute prices were high. Also, these were Windows servers, so I had to sluggishly RDP into them, no nice low-bandwidth terminals.
throwaway211 2y ago
The London Underground not having any connectivity for decades after other metro systems showed only that high connectivity during a commute isn't necessary.
- HPsquared 2y ago
  London fails to provide a lot of essentials.
  - throwaway211 2y ago
    Of which the need for status updates and short video isn't one.

pimeys 2y ago

I guess in London you get wifi only on stops, it's the same in Berlin. In Helsinki the wifi connection is available inside the trains, and in the stations. So you never get a connection loss when moving. I never understood the decision in Berlin to do this, why not just provide internet inside the train... And yeah, most of the internet works very badly when you drop the network all the time...

pintxo 2y ago
Berlin did not have mobile connections inside the tunnels until very recently (this year, I believe). This included the trains not being connected to any outside network. Thus wifi on the subway was useless to implement.
- saagarjha 2y ago
  I was in Berlin earlier this month and the cellular connections underground were quite good now. So maybe this is less of a problem?
  - nicbou 2y ago
    It's provider-specific
    - alanpearce 2y ago
      Not any more? https://unternehmen.bvg.de/pressemitteilung/grossprojekt-erfolgreich-beendet-highspeed-mobilfunk-fuer-alle-in-der-berliner-u-bahn/ https://unternehmen.bvg.de/pressemitteilung/grossprojekt-erf... Summary: since 2024-05-06, users of all networks also get LTE in the U-Bahn thanks to a project between BVG and Teléfonica (not surprising that Teléfonica deployed the infra because they had the best U-Bahn LTE coverage beforehand)
- amaccuish 2y ago
  They did if you were on o2, that's why I'm still with Aldi Talk (they use the o2 network); they've had LTE through the entire network for a while now. The new thing is 5G for everyone.
- rkachowski 2y ago
  Despite Berlin's general lack of parity with modern technology, I've never actually had a problem with internet access across the ubahn network in the past decade. I noticed that certain carriers used to have very different availability when travelling and so switched to a better one, but I was always surprised at being able to handle mobile data whilst underground.
  - nicbou 2y ago
    Really? I don't even get consistent internet on the Ringbahn. There are lots of holes in the coverage in Berlin. Which provider are you with? Vodafone is still dead in large parts of the U-Bahn, but I know that one of them works much better.
    - avh02 2y ago
      used to have spotty coverage underground with vodafone, when i switched to telekom, internet suddenly magically worked underground on the routes i used. I believe someone published a map of the data coverage of different providers on the berlin ubahn, but probably outdated now
    - rkachowski 2y ago
      Yeah, admittedly this year I've also started experiencing holes on the ringbahn (strangely and consistently around frankfurter allee), but the ubahn has been fine. I'm with sim.de which I believe is essentially an O2 reseller (apn references o2)
      - immibis 2y ago
        Gesundbrunnen too.
    - brabel 2y ago
      Wow! I was in Berlin last week and kept losing connection... like all the time. I use 3 with a Swedish plan. In Sweden, it literally never drops, not on trains, not on metro, not on faraway mountains... it works everywhere.
Liskni_si 2y ago
Yes, right now it's mostly just wifi at stations only. However, they're deploying 4G/5G coverage in the tunnels and expect 80% coverage by the end of 2024 [1]. So… you can expect apps developed by engineers in London to get much worse on slow internet in 2025. :-) [1]: https://tfl.gov.uk/campaign/station-wifi https://tfl.gov.uk/campaign/station-wifi
bombcar 2y ago
WiFi at a stop is as easy as putting up a few wireless routers, it's a bit more complex than at home but the same general idea. Wifi inside the trains involves much more work, and to get them to ALSO be seamless across the entire setup - even harder. Easily 10x or 100x the cost. It's sad, because the Internet shouldn't be that bad when the network drops all the time; it should just be slower as it waits to send good data.
- Anotheroneagain 2y ago
  I think they just put a wire in the tunnel.
  - greenish_shores 2y ago
    Yes, that's the best way which is often used. A "leaky cable" aka "leaky feeder", to be particular.

chipdart 2y ago

> In Australia you're just 200ms from everything most of the time. (...) > The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast. The 200ms is telling. I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. Most global companies deploy only to at most three regions: * the US (us-east, us-central, us-east+us-east) * Europe (west-europe), * and somewhat rarely far-east (meither us-west or Japan) This means that places such as south Africa, south America, and of course Australia typically have to pull data from one of these regions, which means latencies of at least 200ms due to physics. Australia is particularly hit because, even with dedicated deployments in their theoretical catchment area, often these servers are actualy located in an entirely separate continent (west-us or Japan) and thus users do experience the performance impact of having packets cross half a globe.

luuurker 2y ago
Part of the problem seems to be the cost of peering in Australia: https://blog.cloudflare.com/bandwidth-costs-around-the-world https://blog.cloudflare.com/bandwidth-costs-around-the-world
2143 2y ago
I kinda chuckled that you left out most of Asia. Most global companies (at least the US-based ones) have deployed in India, where I'm at right now. I suppose a billion people online is too big of a market to ignore (or not; I really don't know). Or there are services that I'm completely unaware of that's not in India. Internet's pretty fast as well. Much faster than a certain conspicuous European country you'd expect to have fast internet ;)
toast0 2y ago
> I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic. So, I used to work at WhatsApp. And we got this kind of praise when we only had servers in Reston, Virginia (not at aws us-east1, but in the same neighborhood). Nowadays, Facebook is most likely terminating connections in Australia, but messaging most likely goes through another continent. Calling within Australia should stay local though (either p2p or through a nearby relay). There's lots of things WhatsApp does to improve experience on low quality networks that other services don't (even when we worked in the same buildings and told them they should consider things!) In no particular order: 0) offline first, phone is the source of truth, although there's multi-device now. You don't need to be online to read messages you have, or to write messages to be sent whenever you're online. Email used to work like this for everyone; and it was no big deal to grab mail once in a while, read it and reply, and then send in a batch. Online messaging is great, if you can, but for things like being on a commuter train where connectivity ebbs and flows, it's nice to pick up messages when you can. a) hardcode fallback ips for when DNS doesn't work (not if) b) setup "0rtt" fast resume, so you can start getting messages on the second round trip. This is part of noise pipes or whatever they're called, and tls 1.3 c) do reasonable-ish things to work with MTU. In the old days, FreeBSD reflected the client MSS back to it, which helps when there's a tunnel like PPPoE and it only modifies outgoing syns and not incoming syn+ack. Linux never did that, and afaik, FreeBSD took it out. Behind Facebook infrastructure, they just hardcode the mss for i think 1480 MTU (you can/should check with tcpdump). I did some limited testing, and really the best results come from monitoring for /24's with bad behavior (it's pretty easy, if you look for it --- never got any large packets and packet gaps are a multiple of MSS - space for tcp timestamps) and then sending back client - 20 to those; you could also just always send back client - 20. I think Android finally started doing pMTUD blackhole detection stuff a couple years back, Apple has been doing it really well for longer. Path MTU Discovery is still an issue, and anything you can do to make it happier is good. d) connect in the background to exchange messages when possible. Don't post notifications unless the message content is on the device. Don't be one of those apps that can only load messsages from the network when the app is in the foreground, because the user might not have connectivity then e) prioritize messages over telemetry. Don't measure everything, only measure things when you know what you'll do with the numbers. Everybody hates telemetry, but it can be super useful as a developer. But if you've got giant telemetry packs to upload, that's bad by itself, and if you do them before you get messages in and out, you're failing the user. f) pay attention to how big things are on the wire. Not everything needs to get shrunk as much as possible, but login needs to be very tight, and message sending should be too. IMHO, http and json and xml are too bulky for those, but are ok for multimedia because the payload is big so framing doesn't matter as much, and they're ok for low volume services because they're low volume.

camel-cdr 2y ago

It's not only the services them self. I have a very slow mobile connection, and one thing that bothered me immensly is downloading images in the browser: How is it, than when I go to a .jpg url to view an image in the browser it takes way longer and sometimes times out, than hopping over to termux and running wget. I had this problem with both firefox and chrome based browsers. Note that even the wget download usually takes 10-30 seconds on my mobile connection.

giantrobot 2y ago
Too many services today do stupid image transcoding today. While the URL says jpg it will decide that because your browser supports WebP that what you really must have wanted was a WebP. It'll then either transcode or just send you WebP data for the image or send you a redirect. This is rarely what you actually want. With wget it sends you the source you actually requested and doesn't try to get clever (stupid). Google likes WebP so that means everyone needs to join the WebP cargo cult even if it means transcoding a lossy format to another lossy format.
chris_pie 2y ago
I have the same issue with nearly every static asset.
bombcar 2y ago
Browsers usually try to multiplex things, sometimes even the same image if the server supports "get specific byte range" or whatever. There may be a setting to turn a browser back into a dumb wget visual displayer.
armchair_expert 2y ago
You can try going into proxy settings and setting to "none" instead of autodetect. Also, the dns server used by the browser could be different (and slower).

kylehotchkiss 2y ago

WhatsApp has a massive audience in developing countries where it's normal for people to have slower internet and much slower devices. That perspective being so embedded in their development goals certainly has given WhatsApp good reason to be the leading messaging platform in many countries around the world

nicbou 2y ago
It works remarkably well when your phone runs out of data and you get capped at 8 kbps. Even voice calls work smoothly.
- qingcharles 2y ago
  LOL 8kbps. Damn. That takes me back. I built the first version of one of the world's largest music streaming sites on a 9.6kbps connection. I was working from home (we had no offices yet) and my cable Internet got cut off. My only back up was a serial cable to a 2G Nokia 9000i. I had to re-encode a chunk of the music catalog at 8kbps so I could test it from home before I pushed the code to production. Psychoacoustic compression is a miracle.
  - greenish_shores 2y ago
    Nokia 9000i, so you had to work on CSD (which is usually billed per-minute, like dial-up), not even GPRS. How much did that cost you? :P BTW, an interesting thing is that some/most carriers allow you to use CSD/HSCSD over 3G these days, and you can establish data CSD connection between two phone numbers, yielding essentially a dedicated L2 pipe which isn't routed over internet. Can have much lower latency and jitter if that's what you need. Some specialized telemetry is still using that, however as 3G is slowly getting phased out, it will probably have to change.
    - qingcharles 2y ago
      God, the cost was probably horrid, but I was connecting in, setting tasks running and logging out. This was late 1999 in the UK, so per-minute prices were high. Also, these were Windows servers, so I had to sluggishly RDP into them, no nice low-bandwidth terminals.

throwaway211 2y ago

The London Underground not having any connectivity for decades after other metro systems showed only that high connectivity during a commute isn't necessary.

HPsquared 2y ago
London fails to provide a lot of essentials.
- throwaway211 2y ago
  Of which the need for status updates and short video isn't one.