18 ms·

I'd swear Verizon FIOS is throttling YouTube recently for me, the quality is getting worse and worse. Constant buffering even at 720p. Forget 4K these days, whi

I'd swear Verizon FIOS is throttling YouTube recently for me, the quality is getting worse and worse. Constant buffering even at 720p. Forget 4K these days, which used to be perfectly fine a year ago. With the current FCC I'm afraid it's just going to get worse.

knowaveragejoe 9y ago
I've had a similar experience recently as well. I pay for 150/150, yet youtube seems routinely slow.
NathanKP 9y ago
In NYC Verizon has an issue where they haven't purchased enough bandwidth into NYC from the Youtube servers. I just turn on my VPN (VyperVPN) which has great bandwidth to the YouTube servers and I get crystal clear 4K streaming. It's kind of pathetic. I don't think they are deliberately throttling, its just that Verizon is being cheap and there is plenty of local bandwidth between VyperVPN's servers and my home, but not enough bandwidth between YouTube's servers and FIOS network in NYC.
- tw04 9y ago
  You've got that backwards. Verizon is trying to extort Google by getting them to pay for interconnects and Google wants nothing to do with it. It's the exact same thing that was going on with Comcast and Netflix. It's complete and utter garbage that they continually double dip, and is the EXACT reason ISPs shouldn't be allowed to own ANY content. It's a direct conflict of interest.
  - mrkrabo 9y ago
    Why shouldn't Verizon and Google split the bill?
    - jmuguy 9y ago
      My understanding is that its not an issue of the cost to "upgrade" the interconnection infrastructure. That cost is likely negligible. What's happening is that VZ is refusing to do it, and allowing service to degrade, unless Google pays X. So yeah, basically extortion. Google has already paid for bandwidth, and this is traffic that VZ's paying customers are requesting.
      - mrkrabo 9y ago
        That's what splitting the bill means. You don't pay "for the bandwidth", you pay for a link between two ISPs (Google and Verizon). The bill between then should be split, otherwise the traffic will have to pass through somewhere else, and that will cause congestion problems.
        jmuguy 9y ago
        Google has most certainly paid for bandwidth/traffic from their end. Whether that's to a CDN or some other arrangement, and there's likely already peering agreements in place. In these cases what Verizon is doing is asking for money above and beyond what it actually costs to handle the interconnection. They use declining service to their own customers as leverage. Its ridiculous, and even more worrying when you see ISPs and content providers merging as is the case with VZ buying Yahoo and AOL, Comcast and NBC merger, etc.
      - jerkstate 9y ago
        let's do a little math.. the router ports on both sides, let's say you are using 4 ports across 4 fully loaded nexus 9508 (let's call those $500k per incl. optics, so $2M / 384 x 4 ports / 36 months = $600/port/month x 4 = $2400/4-ports/month) Depending on how the connection between the two worked or was paid for, it could be $1000-$20000 for 2 pairs of fiber per month depending on distance. Let's take a middle of the road $10k average cost. So for 10Gbps peak redundant capacity, you are at $12,400 per month. Netflix 4k stream is about 16 megabits, so you can fit 625 4k streams in $12,400 of cross-connect capacity. If you don't share costs, that's about $20 per user to support Netflix's business model that people seem to think it's Comcast's responsibility to pay. That's not even considering Comcast's last mile distribution cost or paying any salaries. Streaming video services soak up a lot of network resources and putting all of the cost on ISPs is going to increase your ISP's cost and price. How many low-income families do you think would lose broadband if it went from $50/mo to $100/mo?
  - Androider 9y ago
    Google should instead throw a couple more billions at the SpaceX mesh Internet satellite project, bypassing these asshats completely once and for all. I'd switch in a heartbeat, even if it was a bit more expensive.
    - jerkstate 9y ago
      surely once google runs the pipes they'll be incentivized to pay for netflix's traffic on both sides, or maybe we'll just be back to square 1..
      - tw04 9y ago
        Unlikely. Even if Google were to go hard into the video space (unlikely but not impossible), they still wouldn't benefit from throttling/capping. It would just give users a reason to go back to their original providers. Furthermore we've already seen the governmental backlash when facebook tried similar in developing nations. *I should clarify when I say "the video space" I mean original AAA content like Netflix is doing. I could be underestimating them but that's a LONG way from their market sweet spot.
    - Orangeair 9y ago
      Satellite is nice, but it's not a solution for every use case. Specifically, it's not great for anything that requires low latency (like gaming).
      - Androider 9y ago
        The proposed SpaceX satellite network is in low earth orbit unlike traditional satellite Internet, so it should provide decent latencies (25ms vs 600+ms): https://arstechnica.com/information-technology/2016/11/spacex-plans-worldwide-satellite-internet-with-low-latency-gigabit-speed/ https://arstechnica.com/information-technology/2016/11/space...
    - matt4077 9y ago
      If only there were a legal solution for such problems, something that any sane person would consider obvious. We could solve it by tomorrow 6pm, without a single satellite launch.
  - AndrewKemendo 9y ago
    Verizon is trying to extort Google by getting them to pay for interconnects and Google wants nothing to do with it. I don't think that's right. At its most basic, an interconnection agreement says “You carry some traffic for me, in return for which I’ll do something—either carry traffic for you, or pay you, or some combination of the two.” With Netflix and Google, both were basically saying, "we're not going to pay you (Comcast/Verizon) for access to your network, because we're important enough that we shouldn't have to." They don't have their own ISP networks to exchange traffic at the same rate, all they have is their services, so they don't have anything to offer the ISP in return. So it's disingenuous to say that service providers (Google, Netflix), are extorting anyone. They don't see themselves as ISPs, but they are setting up their own interconnects to provide faster access to customers, so they assume they are exempt from what were traditionally informal interconnect rules. [1]http://www.interisle.net/sub/ISP%20Interconnection.pdf http://www.interisle.net/sub/ISP%20Interconnection.pdf
    - kuschku 9y ago
      This is breaking a major concept of the internet. You pay for peering ports, not for bandwidth.
      - kbenson 9y ago
        That's somewhat simplistic. Peering that's somewhat one-sided can have fees built into one side. Still, getting a large chunk of your content quicker and without overloading your other peers or upstreams means that Netflix and the like should have the power here, and if they desired could charge for this better access (it is one-sided, as mentioned before). It's totally backwards that Comcast/Verizon would be throttling them instead, and is only possible because they don't really compete in a fully open market.
    - kbenson 9y ago
      > So it's disingenuous to say that service providers (Google, Netflix), are extorting anyone. I don't think anyone was saying that. > With Netflix and Google, both were basically saying, "we're not going to pay you (Comcast/Verizon) for access to your network, because we're important enough that we shouldn't have to." They don't have their own ISP networks to exchange traffic at the same rate, all they have is their services, so they don't have anything to offer the ISP in return. Generally peering agreements work on total bandwidth and generally you try to make it as balanced as possible, so there's no cost to either side, as traffic may traverse your network from the peer but not terminate there (it continues through another peer), and that's just a load you bear, but the other side has the same risk. For an end service peering, that's not really as much of a risk, to my knowledge, so what you have is purely a win-win, where Netflix delivers content directly to your network so it's quicker, and you aren't using up peer bandwidth and backbone connections to serve that same content. In any market where Comcast/Verizon didn't have near monopolies over large areas, Netflix could easily charge for this access given their size and ubiquity. That some large ISPs are actually throttling the content only highlights the perverse incentives at play.
  - wbl 9y ago
    If Google was paying Verizon for transit, and you were paying them for transit, that wouldn't be double dipping. Why should one side of the transaction be the only ones paying for a network that connects both parties?
- bogomipz 9y ago
  They don't need to purchase bandwidth, Youtube is available via public peering exchanges. The way these generally work is you just pay for the port on the switch at the exchange and traffic directly. See: https://www.peeringdb.com/asn/15169 https://www.peeringdb.com/asn/15169 Youtube is available via public peering at the Equinix Internet Exchange New York: https://www.peeringdb.com/ix/12 https://www.peeringdb.com/ix/12 Purchasing a port on public peering fabric isn't the same as buying bandwidth from a transit provider. It's a great way to get traffic to your users much cheaper than you could via transit which is why people do it. The issue is eyeball networks - Comcast, FIOS, Time Warner etc. may choose not to do so because they went to seek rent in the form of "paid peering" from these same content providers that their users paid them to get access to in the first place.
mallaidh 9y ago
That's exactly what they're doing: http://www.businessinsider.com/verizon-netflix-youtube-throttling-speeds-2017-7 http://www.businessinsider.com/verizon-netflix-youtube-throt...
nfriedly 9y ago
If you configured an alternative DNS provider, such as Google Public DNS or OpenDNS, that could possibility cause something like that. Or maybe they are throttling it...
- JshWright 9y ago
  How would an 'alternative DNS provider' (also known as a 'DNS provider') cause streaming video to buffer?
  - jlgaddis 9y ago
    Let's say you live in NYC but you're using a DNS server (resolver) in Seattle. When your client performs a DNS query, you may get a response that directs you to a server/CDN/whatever in or near Seattle as opposed to one closer to you in NYC. Then, the data (streaming video) is traversing the country to get to you, passing through many more (potentially congested) interconnects/links on the way. Reality is a bit more complicated than this simple example but such an issue is certainly not unheard of (and inspired mitigations such as RFC7871, for example).
    - JshWright 9y ago
      Both Google and OpenDNS use anycast addresses for their DNS servers that are very likely to route though a 'local' server. I think ISP games are far more likely to be the root cause here...
      - notyourday 9y ago
        Google DNS requests enter Google network in multiple places. DNS itself is not served from the edges.
wil421 9y ago
I have a similar issue with AT&T fiber. YouTube is terrible and streams at sub 480 but if I jump on my VPN it streams at 1080p within a second.
- ryandrake 9y ago
  Forgive me if this is a dumb question, but how would the ISP take a 1080p stream and downsample it to 480 on the fly, without man-in-the-middling HTTPS?
  - jdmichal 9y ago
    They don't. HTTPS doesn't hide who the connection is to. How would a network route packets if it didn't know the recipient? Once they know you're connected to Youtube, they just limit the bandwidth on that connection. The Youtube video player will detect this and automatically downgrade to a lower quality until it finds one that fits within the bandwidth profile. Netflix does the same thing, which is why after buffering sometimes the video looks like crap and then gets better after 15 seconds or so. It buffered a lower quality until it figured out it could send higher quality again. EDIT: I should add, this is why VPNs are sometimes used as a "solution" to such bandwidth shaping. You are hiding the true recipient by purposefully man-in-the-middling the connection with a peer you (hopefully) trust. VPNs are also popular for connecting with business networks, so it's generally not "safe" for the ISP to shape bandwidth to any VPN as aggressively.
    - ryandrake 9y ago
      > The Youtube video player will detect this and automatically downgrade to a lower quality until it finds one that fits within the bandwidth profile Ahh, this is the key point I missed, thanks. So it's a problem with the video player itself trying to "decide" for you the quality you want. Couldn't this be worked around with things like youtube-dl?
      - wil421 9y ago
        The other poster has already pointed out the correct answer. Other users are reporting the same issue on the AT&T forums[1] and on DSLreports. I'd imagine this is due to politics and AT&T betting net neutrality will go away. Once YouTube forks over money a peering point will be upgraded. [1]https://forums.att.com/t5/AT-T-Fiber-Equipment/Gigapower-Youtube-Throttling/td-p/5041348 https://forums.att.com/t5/AT-T-Fiber-Equipment/Gigapower-You...
shados 9y ago
I used to have FiOS is one of the few places in Cambridge that offered it. It was the first time in my life that I'd been wishing for Comcast. 320p videos could barely load, and the lag in certain MMORPGs was unplayable (we're talking 500-1000ms delays) It got better after a while, but google had some stats by ISP for Youtube, and FiOS in my area was like 1/10th of the speed of Comcast on average.
zippergz 9y ago
It's probably a coincidence, but just this weekend for the first time ever, I noticed my YouTube streaming at visibly lower quality (my provider is Cox). It happened all weekened, intermittently. Nothing else seemed slow, and numerous speed tests (both from official "speed test" sites including Netflix's own Fast.com, and by moving large files between my network and AWS) all showed well over 100Mbps. I don't know if it was being throttled somewhere, Netflix's service was overloaded, or if there was some other weird problem happening on my network that affected Netflix and nothing else. But it was bad enough that my wife who is usually oblivious to audio/video quality actually brought it up.
chasing 9y ago
This is the downside of not being network neutral for companies like Verizon: Now Verizon gets the blame if something's slow.
deagle50 9y ago
+1. Youtube was terrible the past week and I have 100/100 FIOS with speed tests exceeding that. Google needs to take gloves off.

knowaveragejoe 9y ago

I've had a similar experience recently as well. I pay for 150/150, yet youtube seems routinely slow.

NathanKP 9y ago

In NYC Verizon has an issue where they haven't purchased enough bandwidth into NYC from the Youtube servers. I just turn on my VPN (VyperVPN) which has great bandwidth to the YouTube servers and I get crystal clear 4K streaming. It's kind of pathetic. I don't think they are deliberately throttling, its just that Verizon is being cheap and there is plenty of local bandwidth between VyperVPN's servers and my home, but not enough bandwidth between YouTube's servers and FIOS network in NYC.

tw04 9y ago
You've got that backwards. Verizon is trying to extort Google by getting them to pay for interconnects and Google wants nothing to do with it. It's the exact same thing that was going on with Comcast and Netflix. It's complete and utter garbage that they continually double dip, and is the EXACT reason ISPs shouldn't be allowed to own ANY content. It's a direct conflict of interest.
- mrkrabo 9y ago
  Why shouldn't Verizon and Google split the bill?
  - jmuguy 9y ago
    My understanding is that its not an issue of the cost to "upgrade" the interconnection infrastructure. That cost is likely negligible. What's happening is that VZ is refusing to do it, and allowing service to degrade, unless Google pays X. So yeah, basically extortion. Google has already paid for bandwidth, and this is traffic that VZ's paying customers are requesting.
    - mrkrabo 9y ago
      That's what splitting the bill means. You don't pay "for the bandwidth", you pay for a link between two ISPs (Google and Verizon). The bill between then should be split, otherwise the traffic will have to pass through somewhere else, and that will cause congestion problems.
      - jmuguy 9y ago
        Google has most certainly paid for bandwidth/traffic from their end. Whether that's to a CDN or some other arrangement, and there's likely already peering agreements in place. In these cases what Verizon is doing is asking for money above and beyond what it actually costs to handle the interconnection. They use declining service to their own customers as leverage. Its ridiculous, and even more worrying when you see ISPs and content providers merging as is the case with VZ buying Yahoo and AOL, Comcast and NBC merger, etc.
    - jerkstate 9y ago
      let's do a little math.. the router ports on both sides, let's say you are using 4 ports across 4 fully loaded nexus 9508 (let's call those $500k per incl. optics, so $2M / 384 x 4 ports / 36 months = $600/port/month x 4 = $2400/4-ports/month) Depending on how the connection between the two worked or was paid for, it could be $1000-$20000 for 2 pairs of fiber per month depending on distance. Let's take a middle of the road $10k average cost. So for 10Gbps peak redundant capacity, you are at $12,400 per month. Netflix 4k stream is about 16 megabits, so you can fit 625 4k streams in $12,400 of cross-connect capacity. If you don't share costs, that's about $20 per user to support Netflix's business model that people seem to think it's Comcast's responsibility to pay. That's not even considering Comcast's last mile distribution cost or paying any salaries. Streaming video services soak up a lot of network resources and putting all of the cost on ISPs is going to increase your ISP's cost and price. How many low-income families do you think would lose broadband if it went from $50/mo to $100/mo?
- Androider 9y ago
  Google should instead throw a couple more billions at the SpaceX mesh Internet satellite project, bypassing these asshats completely once and for all. I'd switch in a heartbeat, even if it was a bit more expensive.
  - jerkstate 9y ago
    surely once google runs the pipes they'll be incentivized to pay for netflix's traffic on both sides, or maybe we'll just be back to square 1..
    - tw04 9y ago
      Unlikely. Even if Google were to go hard into the video space (unlikely but not impossible), they still wouldn't benefit from throttling/capping. It would just give users a reason to go back to their original providers. Furthermore we've already seen the governmental backlash when facebook tried similar in developing nations. *I should clarify when I say "the video space" I mean original AAA content like Netflix is doing. I could be underestimating them but that's a LONG way from their market sweet spot.
  - Orangeair 9y ago
    Satellite is nice, but it's not a solution for every use case. Specifically, it's not great for anything that requires low latency (like gaming).
    - Androider 9y ago
      The proposed SpaceX satellite network is in low earth orbit unlike traditional satellite Internet, so it should provide decent latencies (25ms vs 600+ms): https://arstechnica.com/information-technology/2016/11/spacex-plans-worldwide-satellite-internet-with-low-latency-gigabit-speed/ https://arstechnica.com/information-technology/2016/11/space...
  - matt4077 9y ago
    If only there were a legal solution for such problems, something that any sane person would consider obvious. We could solve it by tomorrow 6pm, without a single satellite launch.
- AndrewKemendo 9y ago
  Verizon is trying to extort Google by getting them to pay for interconnects and Google wants nothing to do with it. I don't think that's right. At its most basic, an interconnection agreement says “You carry some traffic for me, in return for which I’ll do something—either carry traffic for you, or pay you, or some combination of the two.” With Netflix and Google, both were basically saying, "we're not going to pay you (Comcast/Verizon) for access to your network, because we're important enough that we shouldn't have to." They don't have their own ISP networks to exchange traffic at the same rate, all they have is their services, so they don't have anything to offer the ISP in return. So it's disingenuous to say that service providers (Google, Netflix), are extorting anyone. They don't see themselves as ISPs, but they are setting up their own interconnects to provide faster access to customers, so they assume they are exempt from what were traditionally informal interconnect rules. [1]http://www.interisle.net/sub/ISP%20Interconnection.pdf http://www.interisle.net/sub/ISP%20Interconnection.pdf
  - kuschku 9y ago
    This is breaking a major concept of the internet. You pay for peering ports, not for bandwidth.
    - kbenson 9y ago
      That's somewhat simplistic. Peering that's somewhat one-sided can have fees built into one side. Still, getting a large chunk of your content quicker and without overloading your other peers or upstreams means that Netflix and the like should have the power here, and if they desired could charge for this better access (it is one-sided, as mentioned before). It's totally backwards that Comcast/Verizon would be throttling them instead, and is only possible because they don't really compete in a fully open market.
  - kbenson 9y ago
    > So it's disingenuous to say that service providers (Google, Netflix), are extorting anyone. I don't think anyone was saying that. > With Netflix and Google, both were basically saying, "we're not going to pay you (Comcast/Verizon) for access to your network, because we're important enough that we shouldn't have to." They don't have their own ISP networks to exchange traffic at the same rate, all they have is their services, so they don't have anything to offer the ISP in return. Generally peering agreements work on total bandwidth and generally you try to make it as balanced as possible, so there's no cost to either side, as traffic may traverse your network from the peer but not terminate there (it continues through another peer), and that's just a load you bear, but the other side has the same risk. For an end service peering, that's not really as much of a risk, to my knowledge, so what you have is purely a win-win, where Netflix delivers content directly to your network so it's quicker, and you aren't using up peer bandwidth and backbone connections to serve that same content. In any market where Comcast/Verizon didn't have near monopolies over large areas, Netflix could easily charge for this access given their size and ubiquity. That some large ISPs are actually throttling the content only highlights the perverse incentives at play.
- wbl 9y ago
  If Google was paying Verizon for transit, and you were paying them for transit, that wouldn't be double dipping. Why should one side of the transaction be the only ones paying for a network that connects both parties?
bogomipz 9y ago
They don't need to purchase bandwidth, Youtube is available via public peering exchanges. The way these generally work is you just pay for the port on the switch at the exchange and traffic directly. See: https://www.peeringdb.com/asn/15169 https://www.peeringdb.com/asn/15169 Youtube is available via public peering at the Equinix Internet Exchange New York: https://www.peeringdb.com/ix/12 https://www.peeringdb.com/ix/12 Purchasing a port on public peering fabric isn't the same as buying bandwidth from a transit provider. It's a great way to get traffic to your users much cheaper than you could via transit which is why people do it. The issue is eyeball networks - Comcast, FIOS, Time Warner etc. may choose not to do so because they went to seek rent in the form of "paid peering" from these same content providers that their users paid them to get access to in the first place.

mallaidh 9y ago

That's exactly what they're doing: http://www.businessinsider.com/verizon-netflix-youtube-throttling-speeds-2017-7 http://www.businessinsider.com/verizon-netflix-youtube-throt...

nfriedly 9y ago

If you configured an alternative DNS provider, such as Google Public DNS or OpenDNS, that could possibility cause something like that. Or maybe they are throttling it...

JshWright 9y ago
How would an 'alternative DNS provider' (also known as a 'DNS provider') cause streaming video to buffer?
- jlgaddis 9y ago
  Let's say you live in NYC but you're using a DNS server (resolver) in Seattle. When your client performs a DNS query, you may get a response that directs you to a server/CDN/whatever in or near Seattle as opposed to one closer to you in NYC. Then, the data (streaming video) is traversing the country to get to you, passing through many more (potentially congested) interconnects/links on the way. Reality is a bit more complicated than this simple example but such an issue is certainly not unheard of (and inspired mitigations such as RFC7871, for example).
  - JshWright 9y ago
    Both Google and OpenDNS use anycast addresses for their DNS servers that are very likely to route though a 'local' server. I think ISP games are far more likely to be the root cause here...
    - notyourday 9y ago
      Google DNS requests enter Google network in multiple places. DNS itself is not served from the edges.

wil421 9y ago

I have a similar issue with AT&T fiber. YouTube is terrible and streams at sub 480 but if I jump on my VPN it streams at 1080p within a second.

ryandrake 9y ago
Forgive me if this is a dumb question, but how would the ISP take a 1080p stream and downsample it to 480 on the fly, without man-in-the-middling HTTPS?
- jdmichal 9y ago
  They don't. HTTPS doesn't hide who the connection is to. How would a network route packets if it didn't know the recipient? Once they know you're connected to Youtube, they just limit the bandwidth on that connection. The Youtube video player will detect this and automatically downgrade to a lower quality until it finds one that fits within the bandwidth profile. Netflix does the same thing, which is why after buffering sometimes the video looks like crap and then gets better after 15 seconds or so. It buffered a lower quality until it figured out it could send higher quality again. EDIT: I should add, this is why VPNs are sometimes used as a "solution" to such bandwidth shaping. You are hiding the true recipient by purposefully man-in-the-middling the connection with a peer you (hopefully) trust. VPNs are also popular for connecting with business networks, so it's generally not "safe" for the ISP to shape bandwidth to any VPN as aggressively.
  - ryandrake 9y ago
    > The Youtube video player will detect this and automatically downgrade to a lower quality until it finds one that fits within the bandwidth profile Ahh, this is the key point I missed, thanks. So it's a problem with the video player itself trying to "decide" for you the quality you want. Couldn't this be worked around with things like youtube-dl?
    - wil421 9y ago
      The other poster has already pointed out the correct answer. Other users are reporting the same issue on the AT&T forums[1] and on DSLreports. I'd imagine this is due to politics and AT&T betting net neutrality will go away. Once YouTube forks over money a peering point will be upgraded. [1]https://forums.att.com/t5/AT-T-Fiber-Equipment/Gigapower-Youtube-Throttling/td-p/5041348 https://forums.att.com/t5/AT-T-Fiber-Equipment/Gigapower-You...

shados 9y ago

I used to have FiOS is one of the few places in Cambridge that offered it. It was the first time in my life that I'd been wishing for Comcast. 320p videos could barely load, and the lag in certain MMORPGs was unplayable (we're talking 500-1000ms delays) It got better after a while, but google had some stats by ISP for Youtube, and FiOS in my area was like 1/10th of the speed of Comcast on average.

zippergz 9y ago

It's probably a coincidence, but just this weekend for the first time ever, I noticed my YouTube streaming at visibly lower quality (my provider is Cox). It happened all weekened, intermittently. Nothing else seemed slow, and numerous speed tests (both from official "speed test" sites including Netflix's own Fast.com, and by moving large files between my network and AWS) all showed well over 100Mbps. I don't know if it was being throttled somewhere, Netflix's service was overloaded, or if there was some other weird problem happening on my network that affected Netflix and nothing else. But it was bad enough that my wife who is usually oblivious to audio/video quality actually brought it up.

chasing 9y ago

This is the downside of not being network neutral for companies like Verizon: Now Verizon gets the blame if something's slow.

deagle50 9y ago

+1. Youtube was terrible the past week and I have 100/100 FIOS with speed tests exceeding that. Google needs to take gloves off.