5 ms·

I never really appreciated the relationship and importance of bandwidth, latency and packet loss until I had to move a small office from all on-prem infrastruct

I never really appreciated the relationship and importance of bandwidth, latency and packet loss until I had to move a small office from all on-prem infrastructure to completely cloud based at a regional data centre, about 10 years ago. And I mean completely. locally we had firewall, router, riverbed, UPS that's it for max 40 people. No data at rest (besides riverbed cache) due to security. We had ridiculous bandwidth for the time, which users knew, and they would often come and ask "why can I stream youtube in HD but accessing my document/file takes ages" (ages being anything more than 2-3 seconds). A perfectly valid question. The answer was usually "illegal fishing", or "sharks". If the primary sub-sea cable was cut, we'd have to route through another which would add at least 400ms round trip extra. Luckily this was a well known risk/issue for our parent company who had a team dedicated to negotiating peering and routing between office locations and their nearest regional DCs (these were not azure, aws, etc. usually local IX or Teclco). But they couldn't do anything about the sharks :(.

club_tropical 4y ago
> The answer was usually "illegal fishing", or "sharks". You make it sound like every other day some UNDERSEA CABLE gets cut - that's not a once-in-a-blue-moon thing?
- midasuni 4y ago
  I haven’t found a good source of news on submarine cable cuts/fixes, but anecdotally it’s not that often - maybe once a year on cables through suez. Atlantic you never notice. Even rarer is something like the recent Shetland outage where virg cables were lost.
  - triggercut 4y ago
    In this case it was about once a year and it was the only cable directly connecting the two locations. There was one occasion where there was a break only months after one was finally repaired, really frustrating. There are more cables taking that route now, so a bit of redundancy. Most of the time I'd find out officially via network operators bulletin boards (i.e. those looking after AS and IXs) there wouldn't be anything reported in the local news because, for most people, as this article alludes to, the content edge servers they actually connect to are pretty close, both physically (distance) and virtually (hops).
  - harywilke 4y ago
    https://www2.telegeography.com/submarine-cable-faqs-frequently-asked-questions https://www2.telegeography.com/submarine-cable-faqs-frequent... "Don't these cables ever break? Yes! Cable faults are common. On average, there are over 100 each year. You rarely hear about these cable faults because most companies that use cables follow a “safety in numbers” approach to usage, spreading their networks' capacity over multiple cables so that if one breaks, their network will run smoothly over other cables while service is restored on the damaged one. Accidents like fishing vessels and ships dragging anchors account for two-thirds of all cable faults. Environmental factors like earthquakes also contribute to damage. Less commonly, underwater components can fail. Deliberate sabotage and shark bites are exceedingly rare. "
    - midasuni 4y ago
      Sure but there’s 200 cables so that’s one break every 2 years on average.
- jeroenhd 4y ago
  Worldwide, there's a cable break about once every three days. How much that affects you depends on the backup infrastructure and your current location. Submarine cables getting damaged is all but uncommon. I'd expect cables through the Atlantic or Pacific ocean to get damaged much more rarely than cables through your local lake. Shallow waters and sports/amateur boating can damage the cables in shallower waters more easily than in the humongous depths of the greater oceans in the world.
alexeldeib 4y ago
A great example too -- initial page/content load for text/docs, vs continuous streaming which could be workable with arbitrary latency compensation as long as you do really have sufficient bandwidth. The start time might suck, but once you're going, you're going. I had to double take when I saw illegal fishing, I thought you meant the digital kind and my mind went to ddos/similar :P
- midoridensha 4y ago
  >The start time might suck, but once you're going, you're going. Average users will barely even notice a start time of up to 1 second. 500ms will seem fast to them, for starting a video. Typing a document in Google Docs and having a 1000ms round-trip latency for every letter they type, however, will quickly become annoying.
  - alexeldeib 4y ago
    I was thinking of streaming video, where the first load might be a lot heavier to buffer you to a state where you don't get choppy playback. Yeah, regardless of throughput, you're going to have a bad time with interactive workloads with high latency.
StillBored 4y ago
Yah, if you really want it hammered home, find something that is considered a long fat pipe and try to max it out. About a decade ago, one of our customers had a 25Gbit (IIRC) line that went cross country and had something like 150ms of latency. Then after hours they would basically cross replicate their entire data center through a couple machines they purchased from us (running some of our software). While their packet loss was fairly trivial, it was enough to cause some serious pain. Explaining why, if they insisted on TCP they were going to need an enormous amount of ram (for the time) in order to keep the thing running at line rate took about 6 months... Eventually someone ponied up the $$$$ for the ram, and the problem went away, and it sat there happily doing its thing, with only the occasional problems caused by other departments trying their own experiments and killing its throughput.
- eru 4y ago
  BBR for TCP should improve things?
- quickthrower2 4y ago
  Would UDP and something like bittorrent help?
  - dumpsterdiver 4y ago
    (freezes) Did anyone else just hear something? No? Must have been the wind...
    - nix23 4y ago
      Why you freeze?
      - worthless-trash 4y ago
        Roleplaying an NPC changing state maybe.
  - nix23 4y ago
thejosh 4y ago
Yes. It's all well and good having gigabit+ connections, but latency is the killer. Here in Perth, Western Australia we are very far from everything. In the 90s/early 2000's our routing and latency was pretty terrible, especially over dialup and slow connections and that's how I thought online gaming was (300ms+++). When there started to be "local" servers in Adelaide (thanks Internode!) and Sydney, this went down to 25-30ms for adelaide and 50ms for Sydney. Now we have undersea cables to Singapore from Perth, so we actually get 2ms lower latency from PER->SNG than PER->SYD! Sometimes I get weird routing through Asia if a service determines I'm better suited for that. There is apparently also a cable from PER->Europe now, where we now get less latency to EU (it's now less than US, crazy).
- djhn 4y ago
  Out of curiosity, what kind of ping do you have now in absolute terms? I think mine is in the region of 1-2ms (Europe), but I do remember having double digit ping.
  - thejosh 4y ago
    I have FTTP, so I get ~1ms-2ms to my ISP.
  - djhn 4y ago
    * to local servers. Now that I've tried out blip on a 4g LTE connection it's around 50.
  - adamcharnock 4y ago
    Side note: if you want to get a ballpark for one way latency, just measure the approximate distance of the undersea cable on Google Maps (in km) and divide by 155. That will give latency (in ms) to +/-10%, excluding any additional last-mile latency (Wi-Fi etc)
- aussieshibe 4y ago
  It's a shame Internode went the way it did after iiNet/TPG took over. Old Node was one of those rare companies where I actually believed they put their customers and employees before their bottom line. It's just TPG with different branding now. (by the way, has anyone seen my orange watch?)

club_tropical 4y ago

> The answer was usually "illegal fishing", or "sharks". You make it sound like every other day some UNDERSEA CABLE gets cut - that's not a once-in-a-blue-moon thing?

midasuni 4y ago
I haven’t found a good source of news on submarine cable cuts/fixes, but anecdotally it’s not that often - maybe once a year on cables through suez. Atlantic you never notice. Even rarer is something like the recent Shetland outage where virg cables were lost.
- triggercut 4y ago
  In this case it was about once a year and it was the only cable directly connecting the two locations. There was one occasion where there was a break only months after one was finally repaired, really frustrating. There are more cables taking that route now, so a bit of redundancy. Most of the time I'd find out officially via network operators bulletin boards (i.e. those looking after AS and IXs) there wouldn't be anything reported in the local news because, for most people, as this article alludes to, the content edge servers they actually connect to are pretty close, both physically (distance) and virtually (hops).
- harywilke 4y ago
  https://www2.telegeography.com/submarine-cable-faqs-frequently-asked-questions https://www2.telegeography.com/submarine-cable-faqs-frequent... "Don't these cables ever break? Yes! Cable faults are common. On average, there are over 100 each year. You rarely hear about these cable faults because most companies that use cables follow a “safety in numbers” approach to usage, spreading their networks' capacity over multiple cables so that if one breaks, their network will run smoothly over other cables while service is restored on the damaged one. Accidents like fishing vessels and ships dragging anchors account for two-thirds of all cable faults. Environmental factors like earthquakes also contribute to damage. Less commonly, underwater components can fail. Deliberate sabotage and shark bites are exceedingly rare. "
  - midasuni 4y ago
    Sure but there’s 200 cables so that’s one break every 2 years on average.
jeroenhd 4y ago
Worldwide, there's a cable break about once every three days. How much that affects you depends on the backup infrastructure and your current location. Submarine cables getting damaged is all but uncommon. I'd expect cables through the Atlantic or Pacific ocean to get damaged much more rarely than cables through your local lake. Shallow waters and sports/amateur boating can damage the cables in shallower waters more easily than in the humongous depths of the greater oceans in the world.

alexeldeib 4y ago

A great example too -- initial page/content load for text/docs, vs continuous streaming which could be workable with arbitrary latency compensation as long as you do really have sufficient bandwidth. The start time might suck, but once you're going, you're going. I had to double take when I saw illegal fishing, I thought you meant the digital kind and my mind went to ddos/similar :P

midoridensha 4y ago
>The start time might suck, but once you're going, you're going. Average users will barely even notice a start time of up to 1 second. 500ms will seem fast to them, for starting a video. Typing a document in Google Docs and having a 1000ms round-trip latency for every letter they type, however, will quickly become annoying.
- alexeldeib 4y ago
  I was thinking of streaming video, where the first load might be a lot heavier to buffer you to a state where you don't get choppy playback. Yeah, regardless of throughput, you're going to have a bad time with interactive workloads with high latency.

StillBored 4y ago

Yah, if you really want it hammered home, find something that is considered a long fat pipe and try to max it out. About a decade ago, one of our customers had a 25Gbit (IIRC) line that went cross country and had something like 150ms of latency. Then after hours they would basically cross replicate their entire data center through a couple machines they purchased from us (running some of our software). While their packet loss was fairly trivial, it was enough to cause some serious pain. Explaining why, if they insisted on TCP they were going to need an enormous amount of ram (for the time) in order to keep the thing running at line rate took about 6 months... Eventually someone ponied up the $$$$ for the ram, and the problem went away, and it sat there happily doing its thing, with only the occasional problems caused by other departments trying their own experiments and killing its throughput.

eru 4y ago
BBR for TCP should improve things?
quickthrower2 4y ago
Would UDP and something like bittorrent help?
- dumpsterdiver 4y ago
  (freezes) Did anyone else just hear something? No? Must have been the wind...
  - nix23 4y ago
    Why you freeze?
    - worthless-trash 4y ago
      Roleplaying an NPC changing state maybe.
- nix23 4y ago

thejosh 4y ago

Yes. It's all well and good having gigabit+ connections, but latency is the killer. Here in Perth, Western Australia we are very far from everything. In the 90s/early 2000's our routing and latency was pretty terrible, especially over dialup and slow connections and that's how I thought online gaming was (300ms+++). When there started to be "local" servers in Adelaide (thanks Internode!) and Sydney, this went down to 25-30ms for adelaide and 50ms for Sydney. Now we have undersea cables to Singapore from Perth, so we actually get 2ms lower latency from PER->SNG than PER->SYD! Sometimes I get weird routing through Asia if a service determines I'm better suited for that. There is apparently also a cable from PER->Europe now, where we now get less latency to EU (it's now less than US, crazy).

djhn 4y ago
Out of curiosity, what kind of ping do you have now in absolute terms? I think mine is in the region of 1-2ms (Europe), but I do remember having double digit ping.
- thejosh 4y ago
  I have FTTP, so I get ~1ms-2ms to my ISP.
- djhn 4y ago
  * to local servers. Now that I've tried out blip on a 4g LTE connection it's around 50.
- adamcharnock 4y ago
  Side note: if you want to get a ballpark for one way latency, just measure the approximate distance of the undersea cable on Google Maps (in km) and divide by 155. That will give latency (in ms) to +/-10%, excluding any additional last-mile latency (Wi-Fi etc)
aussieshibe 4y ago
It's a shame Internode went the way it did after iiNet/TPG took over. Old Node was one of those rare companies where I actually believed they put their customers and employees before their bottom line. It's just TPG with different branding now. (by the way, has anyone seen my orange watch?)