5 ms·

My isp got the brilliant idea of rolling their own YouTube cache servers. It's great in theory but in or active they're under powered and so at peak hours I can

My isp got the brilliant idea of rolling their own YouTube cache servers. It's great in theory but in or active they're under powered and so at peak hours I can't even stream 240p on my 500mbits connection. I've had to block their cache servers in my firewall for YouTube to be butter smooth at 1080p consistently. Another example is bell Canada who used to mine your DNS queries to profile you for ads, or ISPs that high jack the nxdomain result to send you sponsored results of vaguely similar sounding websites.

gruez 9y ago
>My isp got the brilliant idea of rolling their own YouTube cache servers Are YouTube videos not served using https? Sounds like it's google/YouTube servers but they're underprovsioned
mino 9y ago
It is common for ISP to host instances of the Google Global Cache (GGC, see https://peering.google.com/ https://peering.google.com/) which are used for many Google services, most importantly YouTube. In fact, in many cases Google itself "suggests" to ISP that they host a few GGC servers. They are directly monitored by Google, and the ISP has basically no say in how they are run. Capacity is managed by Google directly.
- danpalmer 9y ago
  Yep, Virgin Media did this in the UK and messed it up badly. Every day at 6pm YouTube would stop working until the following morning. It appeared to work by inspecting DNS packets and replying with overrides if necessary. I didn’t like it but I could understand that. What I did not agree with was the fact that this also happened for other DNS services. Google DNS and OpenDNS both experienced the same issue, as did a few other “famous” DNS providers. Random little ones wouldn’t return the caching servers, and also enabling encrypted DNS for Google/OpenDNS would stop it happening too. I’m fairly sure it was some badly thought out deep packet inspection.
  - mino 9y ago
    Again, it's probably _not_ Virgin's fault or responsibility. Capacity planning is handled by Google directly. Also, I think that virtually every ISP with more than a few tens of thousands users is hosting a GGC instance nowadays (and a Netflix OpenCache, etc. etc.). Nowadays, the vast majority of the transit&peering of ISPs is not going to the Internet, but to a few racks of local caching servers managed by OTT operators. Bandwidth-wise, at least in prime time, the Internet is much less connected/realtime than people think :)
    - deleted 9y ago
      [deleted]
    - danpalmer 9y ago
      I would suggest that the deep packet inspection is Virgin's fault. Perhaps it was Google under provisioning the CDN inside VM's network, but I would suggest that's also probably Virgin's fault in part, as they will likely have more network information available to them that may have suggested network congestion, but that was not acted upon.
takeda 9y ago
The YouTube cache is probably there not to increase performance for you, but to reduce ISP cost (ISP pays for all data received from outside of their network)
- oasisbob 9y ago
  That's a strange way of looking at it. ISPs take all sorts of actions to both increase performance to customers, and to reduce their costs. A competent ISP operating in good faith will execute and expand peering agreements when it makes economic sense to do so.
  - Sohcahtoa82 9y ago
    > A competent ISP operating in good faith That's probably the funniest thing I've read all year
  - takeda 9y ago
    An ISPs #1 goal is to generate money. When there's a competition this often equates with making customers happy so they will stick with the ISP and perhaps more will switch from competitors. When there's no competition and customers have no choice who to use it's purely all about squeezing as much money as they can. This means raising prices for the service and reducing cost by paying less to others services even if it would reduce quality of service. The goal is to have service that's good enough to keep people still use them (I mean there is price point and/or quality that people would just not to have service at all). If a specific action that ISP with monopoly does and it improves quality for customer that's just a coincidence.
- dx034 9y ago
  Even for data not delivered via their cache they'll get free peering with Google. But having cache servers at several locations reduces the capacity they need to peering points. E.g. in the UK, most traffic gets peered in London so having cache servers in Northern England reduces the bandwidth they need for their internal network within the country. I don't think ISPs pay much for peering in Europe.
crschmidt 9y ago
As someone else suggested, this is likely part of our GGC program. If you can give me info on which ISP + Geographic region you're in, I can take a look and see if there's anything in our logs to indicate a problems; you can email me details at myusernamehere at google dot com with details, since I don't routinely check Hacker News. If you can reproduce a bad experience, and right click on the player, click "Get Debug Info", and share that result, it's the most helpful thing for us to dig into problems.
- Cogito 9y ago
  FYI, something that makes my Hacker News experience much nicer is email notifications when my comments are replied to. Get your own at http://hnreplies.com/ http://hnreplies.com/ - built by Dan Grossman

gruez 9y ago

>My isp got the brilliant idea of rolling their own YouTube cache servers Are YouTube videos not served using https? Sounds like it's google/YouTube servers but they're underprovsioned

mino 9y ago

It is common for ISP to host instances of the Google Global Cache (GGC, see https://peering.google.com/ https://peering.google.com/) which are used for many Google services, most importantly YouTube. In fact, in many cases Google itself "suggests" to ISP that they host a few GGC servers. They are directly monitored by Google, and the ISP has basically no say in how they are run. Capacity is managed by Google directly.

danpalmer 9y ago
Yep, Virgin Media did this in the UK and messed it up badly. Every day at 6pm YouTube would stop working until the following morning. It appeared to work by inspecting DNS packets and replying with overrides if necessary. I didn’t like it but I could understand that. What I did not agree with was the fact that this also happened for other DNS services. Google DNS and OpenDNS both experienced the same issue, as did a few other “famous” DNS providers. Random little ones wouldn’t return the caching servers, and also enabling encrypted DNS for Google/OpenDNS would stop it happening too. I’m fairly sure it was some badly thought out deep packet inspection.
- mino 9y ago
  Again, it's probably _not_ Virgin's fault or responsibility. Capacity planning is handled by Google directly. Also, I think that virtually every ISP with more than a few tens of thousands users is hosting a GGC instance nowadays (and a Netflix OpenCache, etc. etc.). Nowadays, the vast majority of the transit&peering of ISPs is not going to the Internet, but to a few racks of local caching servers managed by OTT operators. Bandwidth-wise, at least in prime time, the Internet is much less connected/realtime than people think :)
  - deleted 9y ago
    [deleted]
  - danpalmer 9y ago
    I would suggest that the deep packet inspection is Virgin's fault. Perhaps it was Google under provisioning the CDN inside VM's network, but I would suggest that's also probably Virgin's fault in part, as they will likely have more network information available to them that may have suggested network congestion, but that was not acted upon.

takeda 9y ago

The YouTube cache is probably there not to increase performance for you, but to reduce ISP cost (ISP pays for all data received from outside of their network)

oasisbob 9y ago
That's a strange way of looking at it. ISPs take all sorts of actions to both increase performance to customers, and to reduce their costs. A competent ISP operating in good faith will execute and expand peering agreements when it makes economic sense to do so.
- Sohcahtoa82 9y ago
  > A competent ISP operating in good faith That's probably the funniest thing I've read all year
- takeda 9y ago
  An ISPs #1 goal is to generate money. When there's a competition this often equates with making customers happy so they will stick with the ISP and perhaps more will switch from competitors. When there's no competition and customers have no choice who to use it's purely all about squeezing as much money as they can. This means raising prices for the service and reducing cost by paying less to others services even if it would reduce quality of service. The goal is to have service that's good enough to keep people still use them (I mean there is price point and/or quality that people would just not to have service at all). If a specific action that ISP with monopoly does and it improves quality for customer that's just a coincidence.
dx034 9y ago
Even for data not delivered via their cache they'll get free peering with Google. But having cache servers at several locations reduces the capacity they need to peering points. E.g. in the UK, most traffic gets peered in London so having cache servers in Northern England reduces the bandwidth they need for their internal network within the country. I don't think ISPs pay much for peering in Europe.

crschmidt 9y ago

As someone else suggested, this is likely part of our GGC program. If you can give me info on which ISP + Geographic region you're in, I can take a look and see if there's anything in our logs to indicate a problems; you can email me details at myusernamehere at google dot com with details, since I don't routinely check Hacker News. If you can reproduce a bad experience, and right click on the player, click "Get Debug Info", and share that result, it's the most helpful thing for us to dig into problems.

Cogito 9y ago
FYI, something that makes my Hacker News experience much nicer is email notifications when my comments are replied to. Get your own at http://hnreplies.com/ http://hnreplies.com/ - built by Dan Grossman