7 ms·

As an ex-Comcast employee. Seeing stuff like that happen really doesn't surprise me When Comcast first rolled out that data cap nation wide, I started prodding

As an ex-Comcast employee. Seeing stuff like that happen really doesn't surprise me When Comcast first rolled out that data cap nation wide, I started prodding at it one night out of morbid curiosity Turned out that it would silently slurp all HTTP traffic! Once you hit some arbitrary measurement (EG: 50%) it'll immediately start hijacking all HTTP websites you visit and inject a ton of Javascript to put a message over the web page forcing you to acknowledge your cap Nmapping the server they used caused the messages to immediately disappear. As well as the server to seemingly vanish. Turned out the firewall was just blanket banning the entire IP range when it saw a portscan! The upside being that Comcast would stop MITM'ing HTTP traffic for about 72 hours

walrus01 9y ago
And people wonder why https everywhere is such a necessity now. It should not be necessary to treat your last mile ISP as a hostile entity , but sadly, it often is.
- jakeogh 9y ago
  I canceled my decade old COX account last time that happened. Even asked nicely not to "help" by editing traffic, the runaround was fun. At the end they offered to take my ~$90/mo to ~$70; re-confirming they had no idea what I was unhappy about.
  - walrus01 9y ago
    A $12/hour call center customer retention worker in rural TN has no idea what you're complaining about, their job is simply to meet some retention metrics on a weekly basis. Even if you can actually reach the people who run the ASN of your ISPs, if it something big like Cox, charter, Shaw, etc, they'll be politically unable to confirm or deny anything, and won't want to talk to you. You might get a straight answer if you are in a similarly senior position at an equivalent sized ISP that has mutual settlement free peering, such as between RCN and Charter.
    - username223 9y ago
      Such are the joys of modern "customer support" -- human beings don't scale, because they need to sleep and can only talk to one other human at a time. So you hire the cheapest ones you can find, and instruct them to be minimally helpful. Even better, make them all "managers," so "can I please speak to your manager" will just take you to another minimum-wage employee. If you want actual customer support these days, your best bet is to create a PR problem for the company via social media, because PR flacks are paid enough to matter.
      - acdha 9y ago
        In cases without competition. I live in an area with three gigabit-capable ISPs and the difference in support quality is unbelievable, even for the Comcast and Verizon customers – just calling from a competitive neighborhood gets your call processed differently.
        walrus01 9y ago
        Which city?
        goldenkey 9y ago
        Sounds like Seattle
        acdha 9y ago
        Washington DC - my neighborhood has Comcast, RCN, and Verizon. Our neighbors report better service on all three — and when you go a couple blocks south where the FIOS rollout stopped, regression to normal sets in for Comcast. We also have municipal fiber but they’ve chosen not to make that available for residential service which is really disappointing but … politics.
        walrus01 9y ago
        So, channel bonded docsis3.0 and 3.1 on Comcast and rcn, and gpon fiber from Verizon? Or are the cable operators also doing singlemode to the house now?
- lumisota 9y ago
  Your last mile ISP is almost certainly a hostile entity, but HTTPS alone isn't going to save you [0]. [0]: www.cs.umd.edu/class/fall2017/cmsc818O/papers/tangled-mass.pdf
  - NoGravitas 9y ago
    I suppose the only alternatives are to use a VPN (and hope your VPN provider isn't a hostile entity), or to use TOR (not appropriate for all kinds of traffic). This is the world we've made for ourselves...
  - tialaramex 9y ago
    That paper is from 2014 and some of the circumstances described have since changed. They've also done some things that I assume fell out of operating ICSI's Notary but don't make any real sense for this paper. For example: For a real user what we care about is this cert the end user was presented for a site: Would that be trusted in (Internet Explorer on XP, Safari on iOS, a Python script on a Debian machine, etcetera) and would it be trusted in this smartphone. And what they've looked at is, were the same Trust Stores baked into an Android phone as the above systems? But that's subtly different in a way that fogs the issue here. Example: Suppose phone X trusts ISRG Root X1, XP trusts DST Root CA X 3, and a Debian system trusts Lets Encrypt Authority X3. Those are, to the naked eye, and this study, three completely different things. But in _practice_ for an end user it'd turn out any of the three work for trusting a vast number of certificates used on the web. Trusting one or another _does_ matter, but this paper isn't about why that is, and doesn't really explain what's going on here, it treats that sort of scenario as anomalous and potentially alarming without explaining. The paper did remind me that ICSI's Notary won't work with TLS 1.3, which I have sort of known but not ever mentally addressed. The ICSI Notary works by peeking inside TLS sessions. In versions up to TLS 1.2, the server's Certificate is delivered unencrypted, just before both peers encryption switches on and their communications are unintelligible. This is used by the Notary and by lots of crappy middleboxes, but in TLS 1.3 the encryption has switched on earlier, before the certificate is sent, so the Notary can't see certificates any more.
- ams6110 9y ago
  At my work, we treat our own internal network as a hostile entity. Defense in depth.
  - sli 9y ago
    Same here. Devs can't even mount thumbdrives. Which is fine for us, we don't need them, but it prevents some funny business should one of our laptops get stolen.
    - reaperducer 9y ago
      Ditto. No flash drives except for special ones distributed by the IT department, and even then it takes a few months of reviews to get one. But we go one further — no development on the LAN. Develop on another server at a different hosting company, and only deploy on the production host when thoroughly vetted.
      - B1FF_PSUVM 9y ago
        For some reason, I get this mental image of everybody in a city street dressed in hazmat gear.
      - Macha 9y ago
        > no development on the LAN Does this just mean staging/shared servers are not deployed on machines inside the corporate network, or that developers can't have their dev environment locally and instead remote in to some other machine to do their work? I've heard of the latter in a few companies and it's always the kind of thing that makes me nope out of every applying to them.
        reaperducer 9y ago
        Devs do most of their work on localhost. Changes are pushed to an independent server on an independent network in another city. Once those changes are vetted, then the work is mirrored to production. Each dev has his own cellular connection for internet on his development box. Again, no LAN access. Corporate e-mail and cross-department file servers are on another box on each dev's desk. The downside is massive over-usage charges for each dev's cell data (40-50 GB/month/dev over). The upside is that devs are effectively airgapped from the company, which I assume is the primary goal of all this. Also, chat is banned. There have been efforts in the past to bring in tools like Slack, but the honchos believe that it makes the company better as a whole if we speak to each other like human beings, especially cross-department. Even if that means slowing down a project. And even if that means having to walk across campus, or occasionally driving to another part of town to another building. Phones are largely only used when off-site, or we need to talk to someone in another city. It all sounds tremendously inefficient, but I try to think of it as being like working at IBM or Sperry in the 60's. It seems to work. The company is profitable and expanding, and has been for 40+ years.
rconti 9y ago
They still inject that message now. I hit 90% almost monthly.

walrus01 9y ago

And people wonder why https everywhere is such a necessity now. It should not be necessary to treat your last mile ISP as a hostile entity , but sadly, it often is.

jakeogh 9y ago
I canceled my decade old COX account last time that happened. Even asked nicely not to "help" by editing traffic, the runaround was fun. At the end they offered to take my ~$90/mo to ~$70; re-confirming they had no idea what I was unhappy about.
- walrus01 9y ago
  A $12/hour call center customer retention worker in rural TN has no idea what you're complaining about, their job is simply to meet some retention metrics on a weekly basis. Even if you can actually reach the people who run the ASN of your ISPs, if it something big like Cox, charter, Shaw, etc, they'll be politically unable to confirm or deny anything, and won't want to talk to you. You might get a straight answer if you are in a similarly senior position at an equivalent sized ISP that has mutual settlement free peering, such as between RCN and Charter.
  - username223 9y ago
    Such are the joys of modern "customer support" -- human beings don't scale, because they need to sleep and can only talk to one other human at a time. So you hire the cheapest ones you can find, and instruct them to be minimally helpful. Even better, make them all "managers," so "can I please speak to your manager" will just take you to another minimum-wage employee. If you want actual customer support these days, your best bet is to create a PR problem for the company via social media, because PR flacks are paid enough to matter.
    - acdha 9y ago
      In cases without competition. I live in an area with three gigabit-capable ISPs and the difference in support quality is unbelievable, even for the Comcast and Verizon customers – just calling from a competitive neighborhood gets your call processed differently.
      - walrus01 9y ago
        Which city?
        goldenkey 9y ago
        Sounds like Seattle
        acdha 9y ago
        Washington DC - my neighborhood has Comcast, RCN, and Verizon. Our neighbors report better service on all three — and when you go a couple blocks south where the FIOS rollout stopped, regression to normal sets in for Comcast. We also have municipal fiber but they’ve chosen not to make that available for residential service which is really disappointing but … politics.
        walrus01 9y ago
        So, channel bonded docsis3.0 and 3.1 on Comcast and rcn, and gpon fiber from Verizon? Or are the cable operators also doing singlemode to the house now?
lumisota 9y ago
Your last mile ISP is almost certainly a hostile entity, but HTTPS alone isn't going to save you [0]. [0]: www.cs.umd.edu/class/fall2017/cmsc818O/papers/tangled-mass.pdf
- NoGravitas 9y ago
  I suppose the only alternatives are to use a VPN (and hope your VPN provider isn't a hostile entity), or to use TOR (not appropriate for all kinds of traffic). This is the world we've made for ourselves...
- tialaramex 9y ago
  That paper is from 2014 and some of the circumstances described have since changed. They've also done some things that I assume fell out of operating ICSI's Notary but don't make any real sense for this paper. For example: For a real user what we care about is this cert the end user was presented for a site: Would that be trusted in (Internet Explorer on XP, Safari on iOS, a Python script on a Debian machine, etcetera) and would it be trusted in this smartphone. And what they've looked at is, were the same Trust Stores baked into an Android phone as the above systems? But that's subtly different in a way that fogs the issue here. Example: Suppose phone X trusts ISRG Root X1, XP trusts DST Root CA X 3, and a Debian system trusts Lets Encrypt Authority X3. Those are, to the naked eye, and this study, three completely different things. But in _practice_ for an end user it'd turn out any of the three work for trusting a vast number of certificates used on the web. Trusting one or another _does_ matter, but this paper isn't about why that is, and doesn't really explain what's going on here, it treats that sort of scenario as anomalous and potentially alarming without explaining. The paper did remind me that ICSI's Notary won't work with TLS 1.3, which I have sort of known but not ever mentally addressed. The ICSI Notary works by peeking inside TLS sessions. In versions up to TLS 1.2, the server's Certificate is delivered unencrypted, just before both peers encryption switches on and their communications are unintelligible. This is used by the Notary and by lots of crappy middleboxes, but in TLS 1.3 the encryption has switched on earlier, before the certificate is sent, so the Notary can't see certificates any more.
ams6110 9y ago
At my work, we treat our own internal network as a hostile entity. Defense in depth.
- sli 9y ago
  Same here. Devs can't even mount thumbdrives. Which is fine for us, we don't need them, but it prevents some funny business should one of our laptops get stolen.
  - reaperducer 9y ago
    Ditto. No flash drives except for special ones distributed by the IT department, and even then it takes a few months of reviews to get one. But we go one further — no development on the LAN. Develop on another server at a different hosting company, and only deploy on the production host when thoroughly vetted.
    - B1FF_PSUVM 9y ago
      For some reason, I get this mental image of everybody in a city street dressed in hazmat gear.
    - Macha 9y ago
      > no development on the LAN Does this just mean staging/shared servers are not deployed on machines inside the corporate network, or that developers can't have their dev environment locally and instead remote in to some other machine to do their work? I've heard of the latter in a few companies and it's always the kind of thing that makes me nope out of every applying to them.
      - reaperducer 9y ago
        Devs do most of their work on localhost. Changes are pushed to an independent server on an independent network in another city. Once those changes are vetted, then the work is mirrored to production. Each dev has his own cellular connection for internet on his development box. Again, no LAN access. Corporate e-mail and cross-department file servers are on another box on each dev's desk. The downside is massive over-usage charges for each dev's cell data (40-50 GB/month/dev over). The upside is that devs are effectively airgapped from the company, which I assume is the primary goal of all this. Also, chat is banned. There have been efforts in the past to bring in tools like Slack, but the honchos believe that it makes the company better as a whole if we speak to each other like human beings, especially cross-department. Even if that means slowing down a project. And even if that means having to walk across campus, or occasionally driving to another part of town to another building. Phones are largely only used when off-site, or we need to talk to someone in another city. It all sounds tremendously inefficient, but I try to think of it as being like working at IBM or Sperry in the 60's. It seems to work. The company is profitable and expanding, and has been for 40+ years.

rconti 9y ago

They still inject that message now. I hit 90% almost monthly.