4 ms·

Latency is a cruel mistress. Had a Customer who was using an old Win32 app that did a ton of individual SELECT queries against the database server to render the

Latency is a cruel mistress. Had a Customer who was using an old Win32 app that did a ton of individual SELECT queries against the database server to render the UI. They tried to put it on the end of a VPN connection and it was excruciating. The old vendor kept trying to convince them to add bandwidth. Fortunately the Customer asked the question "Why does the app work fine across the 1.544Mbps T1 to our other office?" (The T1 had sub-5ms latency.)

chrismorgan 2y ago
I was involved in some physical network inventory software a dozen years ago. One team produced some new web-based software for it, for use by field agents and such. The first version we got to review was rather bad in various important areas; my favourite was that search would take over thirty seconds in common real-world deployment environment: it was implemented in .NET stuff that makes server calls easy to do by accident and in unnecessarily blocking fashion, and searched by each of the 34 entity types individually, in sequence; and some .NET session thing meant that if the connection had been idle for ten minutes or something, it would even then need to retry every request that got queued while the session was “stale”, which was all of them. So you ended up with 68 sequential requests, on up to half a second’s latency (high-latency 3G or 2G or geostationary satellite)… so yeah, 30 seconds. They’d only developed it with sub-millisecond latency to the server, so they never noticed this. I don’t think it was a coincidence that the team was US-based: in Australia, we’re used to internet stuff having hundreds of milliseconds of latency, since so much of the internet is US-hosted, so I think Australians would be more likely to notice such issues early on. All those studies about people abandoning pages if they take more than two seconds to load… at those times, it was a rare page that has even started rendering that soon, because of request waterfalls and high latency. (These days, it’s somewhat more common to have CDNs fix the worst of the problem.)
- dietr1ch 2y ago
  I think that latency people grow up with has a huge impact on how careful they are when using the internet. Having gotten my hands on an experimental 128kbps link early on, but later and moving to the countryside with a 56kb-1Mbps really spotty connection made me really appreciate local state as every time things blocked on the internet made it pretty notorious. I'm glad there's a push for synchronized, local-first state now, as roaming around on mobile or with a laptop hopping on wifi can only perform nicely with local state.
  - EvanAnderson 2y ago
    > I think that latency people grow up with has a huge impact on how careful they are when using the internet. This reminds me a ton about the studies showing harm to children's future mental health as a result of growing up poor. I definitely "hoard" data locally because I grew up with accessing BBSs at 1200 baud, moving up thru dial-uo Internet, and eventually shitty DSL. Services and products that rely on constant access to the Internet strike me as dodgy and likely to fail as a result of my "upbringing".
akira2501 2y ago
I test all my web dev on the other side of a 4G modem connected to an MVNO. It forces you to be considerate of both bandwidth and latency as it's about 5-20Mbps in the city with 120ms average latency. It's not at all impossible to design fast and responsive sites and single page applications under these constraints, you just have to be aware of them, and actively target it during the full course of development.
- slt2021 2y ago
  good for you, you are doing your job very well. part of the reason why modern software is so crappy, is because developers often have thee most powerful machines (MacBook pro like) and don't even realize how resource hungry and crrappy their software at lower end devices
  - Escapado 2y ago
    Another part of the reason is that in every company I have ever worked for some SEO dude insists we need to add literally 10 different tracking scripts and insists they need to load first and consume megabytes of data. At my last gig the landing page was 190kb of HTML, CSS and JS from us, 800kb of images and literally 8mb in vendor scripts we were mandated to load as early as possible. Of course we fought it, of course nobody cared.
    - justmarc 2y ago
      Try an e-banking website which literally loads over 25MB of crap just to show the login page. This is by far the worst offender I've seen. Madness.
  - huijzer 2y ago
    Yes MacBook pro with a glass fibre connection with a ping below 10 ms to the server. I’m usually on an old copper line (16 ms ping to Amsterdam) in the Netherlands (130 ms to San Francisco). Some sites are just consistently slow. Especially GitHub or cloud dashboards. My theory is that the round trip to the database slows things down.
Hikikomori 2y ago
Had some devs in another country complaining that their database query was taking hours to complete but doing it from a server in the same datacenter took a few minutes. Took some weeks of emails and a meeting or two until they understood that we couldn't do anything, I had to actually say that we couldn't do anything about latency unless they physically move their country closer to us.
- jstanley 2y ago
  Could you replicate the data to their country and let them run queries locally? Could they run their client from your country and operate the UI remotely? There are more options than moving the country!
  - Hikikomori 2y ago
    I was the network engineer. It was their server and database, I couldn't solve the latency problem for them.
- rodrigodlu 2y ago
  It's easier to have an replica. Doesn't matter if it's "realtime" sync or using CDC, from backups, etc. You can even ask one of these guys to do the setup for you. They'll do in a pinch with a happy face. I know because I did.
  - Hikikomori 2y ago
    It is, but not my problem as a network engineer. We did suggest that though but they refused to believe that we couldn't solve the latency "problem".
    - martyvis 2y ago
      I have been there many times. While the network guy can solve some of the latency in things like TCP handshakes and use compression and caching with magic black boxes you can't fix the actual application query and acknowledgement requirements that might be there.
chickenbig 2y ago
> Latency is a cruel mistress. Yes, Bloomberg had fun with latency because of their datacenter locations (about a decade ago they still only had two and a half close to New York). Pages that would paint acceptably in London would be unacceptable in Tokyo as when poorly designed they would require several round trips to render. Once the page rendered there was still the matter of updating the prices, which was handled by separately streaming data from servers close to the markets to the terminals. A very different architecture but rather difficult to test because of the significant terminal-side functionality.
mschuster91 2y ago
> Fortunately the Customer asked the question "Why does the app work fine across the 1.544Mbps T1 to our other office?" (The T1 had sub-5ms latency.) That reminds me on the atrocious performance of Apple's TimeMachine with small files. Running backups on SSDs is fast, but cable ethernet is noticeably worse, and even WiFi 6 is utterly disgraceful. To my knowledge you can't even go and say "do not include any folder named vendor (PHP) or node_modules (JS)", because (at least on my machine) these stacks tend to be the worst offenders in creating hundreds of thousands of small files.
rahimnathwani 2y ago
Multiple requests sent in series (e.g. because each depends on the previous one) is also a problem with web apps. When you're developing locally or testing on a staging server near you, the latency is negligible and you might not notice. Chrome's developer tools allow you to disable caching and choose to simulate a specific network type. Most people know those settings restrict the throughput. But they also increase the request latency. They do this on a request-by-request basis, not at packet level, so it's only an approximation. Still a good test.
therein 2y ago
Latency to the database especially a cruel mistress.

chrismorgan 2y ago

I was involved in some physical network inventory software a dozen years ago. One team produced some new web-based software for it, for use by field agents and such. The first version we got to review was rather bad in various important areas; my favourite was that search would take over thirty seconds in common real-world deployment environment: it was implemented in .NET stuff that makes server calls easy to do by accident and in unnecessarily blocking fashion, and searched by each of the 34 entity types individually, in sequence; and some .NET session thing meant that if the connection had been idle for ten minutes or something, it would even then need to retry every request that got queued while the session was “stale”, which was all of them. So you ended up with 68 sequential requests, on up to half a second’s latency (high-latency 3G or 2G or geostationary satellite)… so yeah, 30 seconds. They’d only developed it with sub-millisecond latency to the server, so they never noticed this. I don’t think it was a coincidence that the team was US-based: in Australia, we’re used to internet stuff having hundreds of milliseconds of latency, since so much of the internet is US-hosted, so I think Australians would be more likely to notice such issues early on. All those studies about people abandoning pages if they take more than two seconds to load… at those times, it was a rare page that has even started rendering that soon, because of request waterfalls and high latency. (These days, it’s somewhat more common to have CDNs fix the worst of the problem.)

dietr1ch 2y ago
I think that latency people grow up with has a huge impact on how careful they are when using the internet. Having gotten my hands on an experimental 128kbps link early on, but later and moving to the countryside with a 56kb-1Mbps really spotty connection made me really appreciate local state as every time things blocked on the internet made it pretty notorious. I'm glad there's a push for synchronized, local-first state now, as roaming around on mobile or with a laptop hopping on wifi can only perform nicely with local state.
- EvanAnderson 2y ago
  > I think that latency people grow up with has a huge impact on how careful they are when using the internet. This reminds me a ton about the studies showing harm to children's future mental health as a result of growing up poor. I definitely "hoard" data locally because I grew up with accessing BBSs at 1200 baud, moving up thru dial-uo Internet, and eventually shitty DSL. Services and products that rely on constant access to the Internet strike me as dodgy and likely to fail as a result of my "upbringing".

akira2501 2y ago

I test all my web dev on the other side of a 4G modem connected to an MVNO. It forces you to be considerate of both bandwidth and latency as it's about 5-20Mbps in the city with 120ms average latency. It's not at all impossible to design fast and responsive sites and single page applications under these constraints, you just have to be aware of them, and actively target it during the full course of development.

slt2021 2y ago
good for you, you are doing your job very well. part of the reason why modern software is so crappy, is because developers often have thee most powerful machines (MacBook pro like) and don't even realize how resource hungry and crrappy their software at lower end devices
- Escapado 2y ago
  Another part of the reason is that in every company I have ever worked for some SEO dude insists we need to add literally 10 different tracking scripts and insists they need to load first and consume megabytes of data. At my last gig the landing page was 190kb of HTML, CSS and JS from us, 800kb of images and literally 8mb in vendor scripts we were mandated to load as early as possible. Of course we fought it, of course nobody cared.
  - justmarc 2y ago
    Try an e-banking website which literally loads over 25MB of crap just to show the login page. This is by far the worst offender I've seen. Madness.
- huijzer 2y ago
  Yes MacBook pro with a glass fibre connection with a ping below 10 ms to the server. I’m usually on an old copper line (16 ms ping to Amsterdam) in the Netherlands (130 ms to San Francisco). Some sites are just consistently slow. Especially GitHub or cloud dashboards. My theory is that the round trip to the database slows things down.

Hikikomori 2y ago

Had some devs in another country complaining that their database query was taking hours to complete but doing it from a server in the same datacenter took a few minutes. Took some weeks of emails and a meeting or two until they understood that we couldn't do anything, I had to actually say that we couldn't do anything about latency unless they physically move their country closer to us.

jstanley 2y ago
Could you replicate the data to their country and let them run queries locally? Could they run their client from your country and operate the UI remotely? There are more options than moving the country!
- Hikikomori 2y ago
  I was the network engineer. It was their server and database, I couldn't solve the latency problem for them.
rodrigodlu 2y ago
It's easier to have an replica. Doesn't matter if it's "realtime" sync or using CDC, from backups, etc. You can even ask one of these guys to do the setup for you. They'll do in a pinch with a happy face. I know because I did.
- Hikikomori 2y ago
  It is, but not my problem as a network engineer. We did suggest that though but they refused to believe that we couldn't solve the latency "problem".
  - martyvis 2y ago
    I have been there many times. While the network guy can solve some of the latency in things like TCP handshakes and use compression and caching with magic black boxes you can't fix the actual application query and acknowledgement requirements that might be there.

chickenbig 2y ago

> Latency is a cruel mistress. Yes, Bloomberg had fun with latency because of their datacenter locations (about a decade ago they still only had two and a half close to New York). Pages that would paint acceptably in London would be unacceptable in Tokyo as when poorly designed they would require several round trips to render. Once the page rendered there was still the matter of updating the prices, which was handled by separately streaming data from servers close to the markets to the terminals. A very different architecture but rather difficult to test because of the significant terminal-side functionality.

mschuster91 2y ago

> Fortunately the Customer asked the question "Why does the app work fine across the 1.544Mbps T1 to our other office?" (The T1 had sub-5ms latency.) That reminds me on the atrocious performance of Apple's TimeMachine with small files. Running backups on SSDs is fast, but cable ethernet is noticeably worse, and even WiFi 6 is utterly disgraceful. To my knowledge you can't even go and say "do not include any folder named vendor (PHP) or node_modules (JS)", because (at least on my machine) these stacks tend to be the worst offenders in creating hundreds of thousands of small files.

rahimnathwani 2y ago

Multiple requests sent in series (e.g. because each depends on the previous one) is also a problem with web apps. When you're developing locally or testing on a staging server near you, the latency is negligible and you might not notice. Chrome's developer tools allow you to disable caching and choose to simulate a specific network type. Most people know those settings restrict the throughput. But they also increase the request latency. They do this on a request-by-request basis, not at packet level, so it's only an approximation. Still a good test.

therein 2y ago

Latency to the database especially a cruel mistress.