9 ms·

When Your Fiber ISP's 'Dumb' Device Screws Your TCP Sessions

63 points by neelc 5y ago

jcvhaarst 5y ago
So ISP delivers router that breaks your internet, and they won't replace it with a real ONT? Then why not simply replace it yourself? As long as it isn't PON, but just plain AON, that should be relatively straight forward.
- loeg 5y ago
  I believe CenturyLink in Seattle is a PON.
  - neelc 5y ago
    Yes, it is GPON.
- OldTimeCoffee 5y ago
  There's authentication between the ONT and OLT that you would have to either implement or relay. This is an edge case because of running Tor. The average user isn't going to run into these problems.
  - notwedtm 5y ago
    I think that's the crux here. "But I need to run an open Tor proxy!" is going to get some weird looks, especially on residential connections.
    - xxpor 5y ago
      An ISP is selling me fiber to transmit bits and an IP address to talk to the rest of the world. How many TCP connections I'm establishing is exactly none of their business unless they start receiving abuse reports (or run CGNAT, but that's not the issue here). Whoever thought a *stateful ONT* was a good idea should be shot out of a canon. Just wait until the connection timers in the ONT don't match your firewall. Then you'll have real fun.
      - mise_en_place 5y ago
        What a wild take. I was going to start a WISP but realized early on I would get potential customers like this. Let’s be real, you’re being cheap and don’t want to shell out extra cash per month for a business line or colocation.
        ikiris 5y ago
        It is indeed better that you didn't start a WISP with this additude. There's enough bad ones out there as it is.
        jevoten 5y ago
        Why should he need a business line? He's not using excessive bandwidth. The restriction imposed on him is completely arbitrary.
      - jcrawfordor 5y ago
        The service that you're describing is usually called dedicated internet access or DIA. It is a distinct service from residential ISPs, and a more costly one for good reasons. Residential and business ISPs operate a shared resource on which they must impose limits to avoid impacts on other customers. This is as true of PON as other last-mile technologies. Total ballpark, because it depends plenty on your market, proximity to carrier resources, etc, gigabit symmetric DIA tends to be in the neighborhood of $1000-2000 per month. A lot of the variance comes from the fact that it will be delivered by conventional fiber, not PON, in order to avoid resource contention. So trenching is usually involved in the installation, but the price of that is usually amortized into your 3-year contract.
      - bcrl 5y ago
        An ISP (I run one) sells a residential connection to you as a user under a number of assumptions that you are like other residential users. That means that your usage these days is roughly 4 Mbps measured at the 95th percentile (in aggregate). When you run a Tor node you cause the following problems: - your 95th percentile usage is now likely going to be substantially more than 4 Mbps - your usage is likely to be much more constant (less bursty). This breaks statistical multiplexing amongst residential users. For reference, Netflix with HD video streams tends to burst to 25Mbps for a second and is then idle for 4-5 seconds. - your usage is now exposing the ISP to DoS attacks and other interesting (read as expensive) problems caused by running a Tor node. This includes legal costs when dealing with investigations into malicious use of the network by nefarious people trying to hide illegal activities via Tor. Yes, your ISP has to bear the cost for legal issues that arise when its users engage in illegal activity over their internet connections. - your Tor usage is likely to result in the IPs that are used by you to get added to various blacklists. This results in support costs for the ISP when your dynamic IP gets assigned to another user and causes problems for an unrelated. If you really want to do this, colocate a Tor node in a data center. This kind of traffic is perfectly appropriate in commercial circumstances, and the price you pay will reflect the actual cost of the service being delivered. You're not going to cause nearly as much collateral damage with a dedicated internet connection as you will on a residential network. Yes, Tor has its place, and if you're going to run a Tor node, think long and hard about the impact it will have before doing so. Many smaller ISPs are not at a scale where the company can afford to carry the costs needed to support traffic patterns that are generated by Tor. Small ISPs have to be very careful to balance the line between expanding to serve the needs of our customers and breaking even. Legal budgets only become a thing after an ISP has hundreds of thousands of dollars a month in revenue. Please, don't do something like this to a small ISP that's trying to help bridge the broadband divide. At the very least, run it by them before doing so.
tentacleuno 5y ago
I remember something about this from a few years back. Can't recall the link now though. His ssh sessions were constantly timing out. It only happened when he left the SSH session to idle. It turns out his router was dropping the TCP sessions because it considered them dead. He got around it by implementing a "keep alive" packet, of sorts. Very interesting stuff. I don't really work at such a low level in the stack regularly, so it's quite fascinating to see the strange issues people encounter with these tools. Especially when ISP's meddle around with stable protocols. Also reminds me of how some ISP DNS servers totally ignore TTL values from DNS records[0]. [0]: https://news.ycombinator.com/item?id=29568510 https://news.ycombinator.com/item?id=29568510
- viraptor 5y ago
  This is a pretty common issue. See https://access.redhat.com/solutions/23874 https://access.redhat.com/solutions/23874 The keep alive pings can be added on both the TCP and app level. If you ever cross a NAT, you will have some expiry on your connection. It's not really "meddling".
  - fragmede 5y ago
    Yes it is. A packet being sent isn't reaching its destination because your ISP is choosing not to forward it? That we've come to expect that broken behavior is the reality that we live in, but a different route would be for the firewall/NAT device to forge an RST to both ends, since it will no longer be forwarding said packets on that TCP connection. Given all the advances in technology, I don't think that's as bad an idea as it once was.
    - viraptor 5y ago
      "choosing not to forward it" is an interesting phrase. NAT needs to have some expiry for each entry, because we don't have unlimited space for that table. Dropping the mapping entry has the same result as the other side becoming unavailable and is an understood state. You can't just produce RST out of nowhere on the NAT expiry, because that connection may actually be active somewhere else. Consider a replicating pair of NATs - your connection gets moved from one to the other because (network reasons), but the previous one does not get a message about it because (network reasons). If it sent the RST packets it would actively kill live connections which it should not touch anymore.
kevingadd 5y ago
Glad I didn't pick CenturyLink for fiber when I moved here, but Wave G's incredibly unreliable in its own way which makes me wonder if they're using the same hardware. Kinda wish I picked Google Fiber.
- lotsofpulp 5y ago
  Where is this place that has 3 fiber ISP choices? It is hard enough to find residences with 1 choice of fiber ISP. I have yet to see a single residential location in the US that has more than one option for a fiber ISP.
  - lozaning 5y ago
    I can get xfinity, CenturyLink, and USI fibre at my house in Minneapolis. It's great, the competition is such that I pay $50 a month for symmetrical gig fibre.
    - loeg 5y ago
      OP (neelc) and GPP (kevingadd) are both talking about Seattle. Having any ISP competition at all is relatively new for the Seattle metro. It was essentially only Comcast for a decade. Now we have Centurylink in large parts of the city. I don’t think Google Fiber is widely available here. Centurylink rates for 940/940 Mbps in Seattle are $65/mo, which isn’t as nice as $50, but not bad. Comcast charged like $80/mo (coax) for ~200/20 as recently as 2017.
- loeg 5y ago
  Centurylink works great for me; it costs less than Comcast (which enjoyed a decade-long monopoly until recently, and is the only other option where I live), and delivers superior performance. I wish the ONTs didn't do this sort of thing, but I can't say I've noticed it.
- neelc 5y ago
  I am the author. Wave G is a crap service. Wave took CondoInternet and threw it down the drain. My "Gigabit" there was actually 10-20 Mbps most of the time. And where I had it, the building had wiring exclusivity to Comcast and Wave, so no Ziply Fiber or Atlas Networks. In fact, if I had to choose between Wave G or AT&T Fiber and it's forced router/802.1X, I'd pick AT&T any day. From what I know, Wave G uses Layer3 Ethernet switches whereas CenturyLink uses GPON. CenturyLink has a much better service which is sadly held back by the crappy ONT. But if you don't max out your TCP connection count and don't need IPv6 or can live with 6rd/HE.net tunnels, IMHO CenturyLink Fiber beats Wave G. Google Fiber (actually Webpass) was real solid in my previous place, plus it has native IPv6 which CenturyLink lacks. CenturyLink 6rd sucks, and Hurricane Electric tunnels are getting blocked by sites now. Most annoyingly, the FreeBSD mirrors are slow via CenturyLink 6rd so I have to route that via Tor since FreeBSD pkg lacks a IPv4-only flag. CL does have slightly better IPv4 peering and throughput, but Google Fiber/Webpass is totally worth it for $5 more. Yes, Webpass is slightly "slower" on IPv4, due to microwave links, but the real IPv6 and support and even sticky IPs make up for me.
lipnitsk 5y ago
FWIW I'm also on CenturyLink FTTH and just a week or two ago noticed latency spikes and packet loss which magically went away after 15 minutes. Good to read this analysis for future reference. I really wish end users had more control over ONT boxes similar to how we can use own modems for cable/DSL. A DOCSIS-like provisioning by ISP should be possible. Off topic, but CenturyLink Fiber still uses PPPoE and 6rd instead of native dual stack in many markets and are unwilling to upgrade to more modern configurations. EDIT: I do not use Tor at all.
- zokier 5y ago
  > A DOCSIS-like provisioning by ISP should be possible GSM solved provisioning 30 years ago with SIM cards, any reason why ONTs couldn't employ similar system?
  - lipnitsk 5y ago
    Good suggestion and question. Another challenge for bring-your-own-ONT is making a clean fiber connection without expensive tools, but I would imagine that's also solvable.
    - mindslight 5y ago
      My ONT has a standard single SC connector. The only custom splicework on the install is the run from the street to the service entrance. From there it's an off the shelf single mode SC-SC cable to the ONT. Knowing little about the GPON protocol, what does the ONT actually contain to authenticate to the network? With some quick web research, it seems like it's a serial number and/or a static password Would it be possible to replace the ONT with a well documented model that you have flashed with the appropriate identifiers? You might have to figure out how to take the ISP's provisioning profile and make your own device use those parameters? Then again if the ISP didn't want dodgy devices on their TDM network they should remove the motivation by deploying non-broken gear in the first place.
      - loeg 5y ago
PeterisP 5y ago
Quoting the article, the cause is identified "The Calix 716GE-I ONT device is working as designed by activating Denial of Service (DOS) attack prevention when too many connections are established, which includes jumbo or small packets". Sounds like a reasonable feature for residental devices, even if it isn't compatible with the niche usecase of running a Tor relay.
- lipnitsk 5y ago
  Why not make it configurable by advanced users though?
  - PeterisP 5y ago
    Probably the expected market for advanced users who would need this particular feature is tiny. Like, for the Tor relay usecase, there are something like 6000 relays worldwide, most of them probably provided by various organizations (where a single operator runs many relays) instead of hobbyists, most of them outside USA, and the vast majority of them using some entirely different network connection not affected by this particular device model in any way. The described scenario ("10000s of concurrent TCP sessions") is literally an edge case for residental use; the article does follow up with "What about BitTorrent or cryptocurrency and Web 3.0 apps?" but none of those have network behavior like that. Like, perhaps this problem is also affecting other kinds of usage, but the original article does not attempt to claim that, and purely from their example it would be generous to assume that literally dozens of individuals would need this feature and, well, it's not worth to make and test features (even if they're just a configuration option) in this case.
    - AnthonyMouse 5y ago
      The problem with this logic is that ordinary users don't become the target of a denial of service attack either. If it should exist at all, the default should be off. And if then no one would turn it on, it could just as well not exist.
      - 5y ago
josteink 5y ago
Who’d think I’d be happy to have a Huawei ONT for my FTTH setup? But reading this, clearly one can have much worse.
throwaway984393 5y ago
> But what if a large number of TCP connections is intentional? Sorry, that ship sailed long ago. Carriers have forever put restrictions on how their customers can use their internet connections, such as "no hosting servers" or even not getting a routable IP address. Traffic shaping is part of the deal too. I think the only means we have to change the situation (in the face of a lack of competition) is to lobby for municipal internet. Or start a company.
- superkuh 5y ago
  It seems like all the real Internet Service Providers have died and all we're left with is web service providers with an incomplete internet implementation. This started with the wireless telcos where it was almost justified; they were late to the game and didn't have enough IPv4. But for established holders of large IP spaces this is exploitation if not outright fraud.
- rubatuga 5y ago
  We started a company called Hoppy Network that does away with ISP bullshit like CGNAT. As long as your ISP doesn’t block UDP packets, you’re set. I talk about the rationale here: https://www.naut.ca/blog/2020/12/30/launching-a-new-service/ https://www.naut.ca/blog/2020/12/30/launching-a-new-service/
  - jevoten 5y ago
    > As long as your ISP doesn’t block UDP packets Can you explain? I thought TCP was implemented on top of UDP.
    - rubatuga 5y ago
      Nope, they’re both layer 4 protocols.
    - jmull 5y ago
      I believe technically both UDP and TCP are implemented on top of IP. But UDP is basically IP + a port number and a checksum, while TCP is IP + a port number and includes a checksum as well, and a bunch of other stuff. So while TCP isn’t exactly implemented on top of UDP, it’s pretty close. Something very much like TCP can be implemented on top of UDP, with whatever improvements or differences you might want to implement. (It’s just that both sides need to understand the custom TCP-like protocol.)
mise_en_place 5y ago
I’m not sympathetic to the author at all. You’re essentially using a home ISP for commercial purposes by hosting Tor relays. If you need resilience, then you really ought to colocate at a DC. 10 gbit is not that expensive these days, and you would provide your own switch like mikrotik.
- Taniwha 5y ago
  What he does with the bandwidth he pays for is nobody's business
- superkuh 5y ago
  An ISP provides an internet connection. When it doesn't provide an internet connection and only provides a web service with some internet features it isn't upholding it's side of the contract or the advertising. This is far worse than any "speeds up to $x!" lie. And it's not just tor relays that use a lot of TCP sessions. Pretty much all distributed protocols are going to hold open a lot of TCP connections. This is not a bad thing and it isn't a heavy resource usage. It's normal. What's abnormal are wireless telco style restrictions being applied in contexts where there is no justification for them. Saying everyone who does more than use a browser should colocate at a DC is disconnected from reality.
- psKama 5y ago
  I am not sympathetic to you at all. Running a Tor relay shouldn't require a commercial infrastructure for anyone who wants to. Also, Tor is not the only service he mentions. What I am going to do with my internet is my business, not anyone else's. I shouldn't be limited in any way or form the way I want to use the internet as long as I stay within the limits of law.
jrockway 5y ago
I was never happy with the performance of Calix CPE. We used them heavily at my last job, and indeed customers would have all sorts of trouble that we could never reproduce when we sent a tech. My favorite little hack was that I wanted live stats from the OLTs to be in our own database so that it could show up in our support portal and internal CRM and be aggregated for general network health statistics. (i.e. when someone went out to repair a fiber, they could instantly see the customers come back online, or more often... know while they were still out in the field that they didn't fix it) I wrote a program to scrape it (by ssh-ing in, thanks golang.org/x/crypto/ssh!, because their SOAP API returned no useful details), and after running for many days... it caused the OLT to stop routing packets entirely. No Internet routing, no management interface, it just flat out died unrecoverably. Anyway, they blamed my app, so I built them a static binary of the scraper that could run on Windows (they didn't have any Linux boxes) and after much back and forth they traced it down to a race condition between the two redundant processor modules in the OLT. So much whining how it was my fault, when it was their fault. At the ISP before that we made our own CPE. The leads on that project really understood the Internet and managed to get reasonable latency, even over WiFi. But the incumbents still seem to not know about fq_codel, or how to put more than 4MB of RAM in their devices, and the users suffer as a result. This article reminded me of how mad it makes me, sorry for the rant. (I switched to a different industry where less lasers are involved.)
- toast0 5y ago
  The CenturyLink CPE for DSL is pretty crap too. I had one that would reboot if you sent a fragmented IPv6 packet, among other problems like the web UI refusing to work if it was on long enough (thankfully EOL and they replaced it without much questions). The replacement didn't reboot, but I was seeing ping times go up to seconds, so I gave up and do PPPoE on my equipment now (I didn't want to ever run PPPoE cause it's stupid, but now I have to)
  - jrockway 5y ago
    Yeah. I feel like these products were always designed to be ultra short lifespan devices, but in practice, they live forever. ONTs from 2013 are still being used and sold new, so clearly investing in fixing bugs and testing would have paid dividends. But I guess it's that case where you can't go to Amazon and buy the best ONT; Calix or whomever sells them to an ISP once, everyone has a good round of golf and dinner or whatever, and the customers are stuck being sad (and maybe the ISPs give out token credits; we gave out a lot of token credits). The ISP mostly competes by being available to a customer; if you dig up the street, then they don't really have an option to go elsewhere unless someone else digs up the street. Hopefully some crazy person will launch an astronomically expensive fleet of satellites that solves these problems once and for all. All you need are a few engineers on the CPE team that give a damn, and you can fix the Internet for everyone in the world.
rdtwo 5y ago
How do I check what the limit is on century link? I routinely use more than 1000 connections and want to know what my cap is
- loeg 5y ago
  OP uses a web server benchmarking tool, “ab”, in the article.

63 points by neelc 5y ago

So ISP delivers router that breaks your internet, and they won't replace it with a real ONT? Then why not simply replace it yourself? As long as it isn't PON, but just plain AON, that should be relatively straight forward.

loeg 5y ago
I believe CenturyLink in Seattle is a PON.
- neelc 5y ago
  Yes, it is GPON.
OldTimeCoffee 5y ago
There's authentication between the ONT and OLT that you would have to either implement or relay. This is an edge case because of running Tor. The average user isn't going to run into these problems.
- notwedtm 5y ago
  I think that's the crux here. "But I need to run an open Tor proxy!" is going to get some weird looks, especially on residential connections.
  - xxpor 5y ago
    An ISP is selling me fiber to transmit bits and an IP address to talk to the rest of the world. How many TCP connections I'm establishing is exactly none of their business unless they start receiving abuse reports (or run CGNAT, but that's not the issue here). Whoever thought a *stateful ONT* was a good idea should be shot out of a canon. Just wait until the connection timers in the ONT don't match your firewall. Then you'll have real fun.
    - mise_en_place 5y ago
      What a wild take. I was going to start a WISP but realized early on I would get potential customers like this. Let’s be real, you’re being cheap and don’t want to shell out extra cash per month for a business line or colocation.
      - ikiris 5y ago
        It is indeed better that you didn't start a WISP with this additude. There's enough bad ones out there as it is.
      - jevoten 5y ago
        Why should he need a business line? He's not using excessive bandwidth. The restriction imposed on him is completely arbitrary.
    - jcrawfordor 5y ago
      The service that you're describing is usually called dedicated internet access or DIA. It is a distinct service from residential ISPs, and a more costly one for good reasons. Residential and business ISPs operate a shared resource on which they must impose limits to avoid impacts on other customers. This is as true of PON as other last-mile technologies. Total ballpark, because it depends plenty on your market, proximity to carrier resources, etc, gigabit symmetric DIA tends to be in the neighborhood of $1000-2000 per month. A lot of the variance comes from the fact that it will be delivered by conventional fiber, not PON, in order to avoid resource contention. So trenching is usually involved in the installation, but the price of that is usually amortized into your 3-year contract.
    - bcrl 5y ago
      An ISP (I run one) sells a residential connection to you as a user under a number of assumptions that you are like other residential users. That means that your usage these days is roughly 4 Mbps measured at the 95th percentile (in aggregate). When you run a Tor node you cause the following problems: - your 95th percentile usage is now likely going to be substantially more than 4 Mbps - your usage is likely to be much more constant (less bursty). This breaks statistical multiplexing amongst residential users. For reference, Netflix with HD video streams tends to burst to 25Mbps for a second and is then idle for 4-5 seconds. - your usage is now exposing the ISP to DoS attacks and other interesting (read as expensive) problems caused by running a Tor node. This includes legal costs when dealing with investigations into malicious use of the network by nefarious people trying to hide illegal activities via Tor. Yes, your ISP has to bear the cost for legal issues that arise when its users engage in illegal activity over their internet connections. - your Tor usage is likely to result in the IPs that are used by you to get added to various blacklists. This results in support costs for the ISP when your dynamic IP gets assigned to another user and causes problems for an unrelated. If you really want to do this, colocate a Tor node in a data center. This kind of traffic is perfectly appropriate in commercial circumstances, and the price you pay will reflect the actual cost of the service being delivered. You're not going to cause nearly as much collateral damage with a dedicated internet connection as you will on a residential network. Yes, Tor has its place, and if you're going to run a Tor node, think long and hard about the impact it will have before doing so. Many smaller ISPs are not at a scale where the company can afford to carry the costs needed to support traffic patterns that are generated by Tor. Small ISPs have to be very careful to balance the line between expanding to serve the needs of our customers and breaking even. Legal budgets only become a thing after an ISP has hundreds of thousands of dollars a month in revenue. Please, don't do something like this to a small ISP that's trying to help bridge the broadband divide. At the very least, run it by them before doing so.

tentacleuno 5y ago

I remember something about this from a few years back. Can't recall the link now though. His ssh sessions were constantly timing out. It only happened when he left the SSH session to idle. It turns out his router was dropping the TCP sessions because it considered them dead. He got around it by implementing a "keep alive" packet, of sorts. Very interesting stuff. I don't really work at such a low level in the stack regularly, so it's quite fascinating to see the strange issues people encounter with these tools. Especially when ISP's meddle around with stable protocols. Also reminds me of how some ISP DNS servers totally ignore TTL values from DNS records[0]. [0]: https://news.ycombinator.com/item?id=29568510 https://news.ycombinator.com/item?id=29568510

viraptor 5y ago
This is a pretty common issue. See https://access.redhat.com/solutions/23874 https://access.redhat.com/solutions/23874 The keep alive pings can be added on both the TCP and app level. If you ever cross a NAT, you will have some expiry on your connection. It's not really "meddling".
- fragmede 5y ago
  Yes it is. A packet being sent isn't reaching its destination because your ISP is choosing not to forward it? That we've come to expect that broken behavior is the reality that we live in, but a different route would be for the firewall/NAT device to forge an RST to both ends, since it will no longer be forwarding said packets on that TCP connection. Given all the advances in technology, I don't think that's as bad an idea as it once was.
  - viraptor 5y ago
    "choosing not to forward it" is an interesting phrase. NAT needs to have some expiry for each entry, because we don't have unlimited space for that table. Dropping the mapping entry has the same result as the other side becoming unavailable and is an understood state. You can't just produce RST out of nowhere on the NAT expiry, because that connection may actually be active somewhere else. Consider a replicating pair of NATs - your connection gets moved from one to the other because (network reasons), but the previous one does not get a message about it because (network reasons). If it sent the RST packets it would actively kill live connections which it should not touch anymore.

kevingadd 5y ago

Glad I didn't pick CenturyLink for fiber when I moved here, but Wave G's incredibly unreliable in its own way which makes me wonder if they're using the same hardware. Kinda wish I picked Google Fiber.

lotsofpulp 5y ago
Where is this place that has 3 fiber ISP choices? It is hard enough to find residences with 1 choice of fiber ISP. I have yet to see a single residential location in the US that has more than one option for a fiber ISP.
- lozaning 5y ago
  I can get xfinity, CenturyLink, and USI fibre at my house in Minneapolis. It's great, the competition is such that I pay $50 a month for symmetrical gig fibre.
  - loeg 5y ago
    OP (neelc) and GPP (kevingadd) are both talking about Seattle. Having any ISP competition at all is relatively new for the Seattle metro. It was essentially only Comcast for a decade. Now we have Centurylink in large parts of the city. I don’t think Google Fiber is widely available here. Centurylink rates for 940/940 Mbps in Seattle are $65/mo, which isn’t as nice as $50, but not bad. Comcast charged like $80/mo (coax) for ~200/20 as recently as 2017.
loeg 5y ago
Centurylink works great for me; it costs less than Comcast (which enjoyed a decade-long monopoly until recently, and is the only other option where I live), and delivers superior performance. I wish the ONTs didn't do this sort of thing, but I can't say I've noticed it.
neelc 5y ago
I am the author. Wave G is a crap service. Wave took CondoInternet and threw it down the drain. My "Gigabit" there was actually 10-20 Mbps most of the time. And where I had it, the building had wiring exclusivity to Comcast and Wave, so no Ziply Fiber or Atlas Networks. In fact, if I had to choose between Wave G or AT&T Fiber and it's forced router/802.1X, I'd pick AT&T any day. From what I know, Wave G uses Layer3 Ethernet switches whereas CenturyLink uses GPON. CenturyLink has a much better service which is sadly held back by the crappy ONT. But if you don't max out your TCP connection count and don't need IPv6 or can live with 6rd/HE.net tunnels, IMHO CenturyLink Fiber beats Wave G. Google Fiber (actually Webpass) was real solid in my previous place, plus it has native IPv6 which CenturyLink lacks. CenturyLink 6rd sucks, and Hurricane Electric tunnels are getting blocked by sites now. Most annoyingly, the FreeBSD mirrors are slow via CenturyLink 6rd so I have to route that via Tor since FreeBSD pkg lacks a IPv4-only flag. CL does have slightly better IPv4 peering and throughput, but Google Fiber/Webpass is totally worth it for $5 more. Yes, Webpass is slightly "slower" on IPv4, due to microwave links, but the real IPv6 and support and even sticky IPs make up for me.

lipnitsk 5y ago

FWIW I'm also on CenturyLink FTTH and just a week or two ago noticed latency spikes and packet loss which magically went away after 15 minutes. Good to read this analysis for future reference. I really wish end users had more control over ONT boxes similar to how we can use own modems for cable/DSL. A DOCSIS-like provisioning by ISP should be possible. Off topic, but CenturyLink Fiber still uses PPPoE and 6rd instead of native dual stack in many markets and are unwilling to upgrade to more modern configurations. EDIT: I do not use Tor at all.

zokier 5y ago
> A DOCSIS-like provisioning by ISP should be possible GSM solved provisioning 30 years ago with SIM cards, any reason why ONTs couldn't employ similar system?
- lipnitsk 5y ago
  Good suggestion and question. Another challenge for bring-your-own-ONT is making a clean fiber connection without expensive tools, but I would imagine that's also solvable.
  - mindslight 5y ago
    My ONT has a standard single SC connector. The only custom splicework on the install is the run from the street to the service entrance. From there it's an off the shelf single mode SC-SC cable to the ONT. Knowing little about the GPON protocol, what does the ONT actually contain to authenticate to the network? With some quick web research, it seems like it's a serial number and/or a static password Would it be possible to replace the ONT with a well documented model that you have flashed with the appropriate identifiers? You might have to figure out how to take the ISP's provisioning profile and make your own device use those parameters? Then again if the ISP didn't want dodgy devices on their TDM network they should remove the motivation by deploying non-broken gear in the first place.
    - loeg 5y ago

PeterisP 5y ago

Quoting the article, the cause is identified "The Calix 716GE-I ONT device is working as designed by activating Denial of Service (DOS) attack prevention when too many connections are established, which includes jumbo or small packets". Sounds like a reasonable feature for residental devices, even if it isn't compatible with the niche usecase of running a Tor relay.

lipnitsk 5y ago
Why not make it configurable by advanced users though?
- PeterisP 5y ago
  Probably the expected market for advanced users who would need this particular feature is tiny. Like, for the Tor relay usecase, there are something like 6000 relays worldwide, most of them probably provided by various organizations (where a single operator runs many relays) instead of hobbyists, most of them outside USA, and the vast majority of them using some entirely different network connection not affected by this particular device model in any way. The described scenario ("10000s of concurrent TCP sessions") is literally an edge case for residental use; the article does follow up with "What about BitTorrent or cryptocurrency and Web 3.0 apps?" but none of those have network behavior like that. Like, perhaps this problem is also affecting other kinds of usage, but the original article does not attempt to claim that, and purely from their example it would be generous to assume that literally dozens of individuals would need this feature and, well, it's not worth to make and test features (even if they're just a configuration option) in this case.
  - AnthonyMouse 5y ago
    The problem with this logic is that ordinary users don't become the target of a denial of service attack either. If it should exist at all, the default should be off. And if then no one would turn it on, it could just as well not exist.
    - 5y ago

josteink 5y ago

Who’d think I’d be happy to have a Huawei ONT for my FTTH setup? But reading this, clearly one can have much worse.

throwaway984393 5y ago

> But what if a large number of TCP connections is intentional? Sorry, that ship sailed long ago. Carriers have forever put restrictions on how their customers can use their internet connections, such as "no hosting servers" or even not getting a routable IP address. Traffic shaping is part of the deal too. I think the only means we have to change the situation (in the face of a lack of competition) is to lobby for municipal internet. Or start a company.

superkuh 5y ago
It seems like all the real Internet Service Providers have died and all we're left with is web service providers with an incomplete internet implementation. This started with the wireless telcos where it was almost justified; they were late to the game and didn't have enough IPv4. But for established holders of large IP spaces this is exploitation if not outright fraud.
rubatuga 5y ago
We started a company called Hoppy Network that does away with ISP bullshit like CGNAT. As long as your ISP doesn’t block UDP packets, you’re set. I talk about the rationale here: https://www.naut.ca/blog/2020/12/30/launching-a-new-service/ https://www.naut.ca/blog/2020/12/30/launching-a-new-service/
- jevoten 5y ago
  > As long as your ISP doesn’t block UDP packets Can you explain? I thought TCP was implemented on top of UDP.
  - rubatuga 5y ago
    Nope, they’re both layer 4 protocols.
  - jmull 5y ago
    I believe technically both UDP and TCP are implemented on top of IP. But UDP is basically IP + a port number and a checksum, while TCP is IP + a port number and includes a checksum as well, and a bunch of other stuff. So while TCP isn’t exactly implemented on top of UDP, it’s pretty close. Something very much like TCP can be implemented on top of UDP, with whatever improvements or differences you might want to implement. (It’s just that both sides need to understand the custom TCP-like protocol.)

mise_en_place 5y ago

I’m not sympathetic to the author at all. You’re essentially using a home ISP for commercial purposes by hosting Tor relays. If you need resilience, then you really ought to colocate at a DC. 10 gbit is not that expensive these days, and you would provide your own switch like mikrotik.

Taniwha 5y ago
What he does with the bandwidth he pays for is nobody's business
superkuh 5y ago
An ISP provides an internet connection. When it doesn't provide an internet connection and only provides a web service with some internet features it isn't upholding it's side of the contract or the advertising. This is far worse than any "speeds up to $x!" lie. And it's not just tor relays that use a lot of TCP sessions. Pretty much all distributed protocols are going to hold open a lot of TCP connections. This is not a bad thing and it isn't a heavy resource usage. It's normal. What's abnormal are wireless telco style restrictions being applied in contexts where there is no justification for them. Saying everyone who does more than use a browser should colocate at a DC is disconnected from reality.
psKama 5y ago
I am not sympathetic to you at all. Running a Tor relay shouldn't require a commercial infrastructure for anyone who wants to. Also, Tor is not the only service he mentions. What I am going to do with my internet is my business, not anyone else's. I shouldn't be limited in any way or form the way I want to use the internet as long as I stay within the limits of law.

jrockway 5y ago

I was never happy with the performance of Calix CPE. We used them heavily at my last job, and indeed customers would have all sorts of trouble that we could never reproduce when we sent a tech. My favorite little hack was that I wanted live stats from the OLTs to be in our own database so that it could show up in our support portal and internal CRM and be aggregated for general network health statistics. (i.e. when someone went out to repair a fiber, they could instantly see the customers come back online, or more often... know while they were still out in the field that they didn't fix it) I wrote a program to scrape it (by ssh-ing in, thanks golang.org/x/crypto/ssh!, because their SOAP API returned no useful details), and after running for many days... it caused the OLT to stop routing packets entirely. No Internet routing, no management interface, it just flat out died unrecoverably. Anyway, they blamed my app, so I built them a static binary of the scraper that could run on Windows (they didn't have any Linux boxes) and after much back and forth they traced it down to a race condition between the two redundant processor modules in the OLT. So much whining how it was my fault, when it was their fault. At the ISP before that we made our own CPE. The leads on that project really understood the Internet and managed to get reasonable latency, even over WiFi. But the incumbents still seem to not know about fq_codel, or how to put more than 4MB of RAM in their devices, and the users suffer as a result. This article reminded me of how mad it makes me, sorry for the rant. (I switched to a different industry where less lasers are involved.)

toast0 5y ago
The CenturyLink CPE for DSL is pretty crap too. I had one that would reboot if you sent a fragmented IPv6 packet, among other problems like the web UI refusing to work if it was on long enough (thankfully EOL and they replaced it without much questions). The replacement didn't reboot, but I was seeing ping times go up to seconds, so I gave up and do PPPoE on my equipment now (I didn't want to ever run PPPoE cause it's stupid, but now I have to)
- jrockway 5y ago
  Yeah. I feel like these products were always designed to be ultra short lifespan devices, but in practice, they live forever. ONTs from 2013 are still being used and sold new, so clearly investing in fixing bugs and testing would have paid dividends. But I guess it's that case where you can't go to Amazon and buy the best ONT; Calix or whomever sells them to an ISP once, everyone has a good round of golf and dinner or whatever, and the customers are stuck being sad (and maybe the ISPs give out token credits; we gave out a lot of token credits). The ISP mostly competes by being available to a customer; if you dig up the street, then they don't really have an option to go elsewhere unless someone else digs up the street. Hopefully some crazy person will launch an astronomically expensive fleet of satellites that solves these problems once and for all. All you need are a few engineers on the CPE team that give a damn, and you can fix the Internet for everyone in the world.

rdtwo 5y ago

How do I check what the limit is on century link? I routinely use more than 1000 connections and want to know what my cap is

loeg 5y ago
OP uses a web server benchmarking tool, “ab”, in the article.