14 ms·

They definitely are. I know for a fact that they are running massive Hadoop clusters storing information on DNS records involved in their customer traffic. If I

by z9e 7y ago

They definitely are. I know for a fact that they are running massive Hadoop clusters storing information on DNS records involved in their customer traffic. If I recall correctly they mirror a lot of the traffic to analytics environments.

everdrive 7y ago
How do you know this fact?
- z9e 7y ago
  I’m friends with solutions engineers at Hortonworks and Cloudera. It’s possible I’m wrong, and since this is anecdotal evidence I see “fact” isn’t a valid use here.
- windexh8er 7y ago
  I worked for a Comcast subsidiary in 2010-2011 timeframe. The company owned the network end to end. They had about 250k subscribers at the time across 4 states and at the time was the first DOCIS 3.0 network in the US. They were collecting DNS log data back then. I've been told that hasn't stopped and has progressed. Don't trust your ISP to not be passively monitoring. This particular ISP had closets full of old Sandvines [0] hardware as well that I ran across one day. I asked what the hardware had been used for and the answer was simply: "network monitoring for law enforcement". At the time all of that old gear had been decomm'd. But as I've said in older posts the DC had a hands off, tamper taped mobile rack that was plugged into core routing installed by a 3 letter agency while I was employed. This was pre-Snowden and post 9/11, likely courtesy of all those fun programs we found out about that Clapper denied. [0] https://en.m.wikipedia.org/wiki/Sandvine https://en.m.wikipedia.org/wiki/Sandvine
  - cabaalis 7y ago
    We'll know how much this will affect those TLAs when the government suddenly gets involved for some altruistic reason to block DNS over HTTPS. "Don't let google take over the internet!" "Time to break up big tech!"
    - windexh8er 7y ago
      I find it interesting that Google and CloudFlare are now the scapegoats. I mean, it's not like DoH isn't configurable and we don't have a choice.
      - topranks 7y ago
        Not attacking Google - their approach here is fine. But Mozilla switching people is a worry for me. Sure people “have a choice” but in reality expecting the average Joe who doesn’t even know what DNS is to make an informed decision about it is unrealistic. Meanwhile Mozilla has started sending a list of every domain you visit to a US company subject to US law enforcement. Not ideal.
        everdrive 7y ago
        >Meanwhile Mozilla has started sending a list of every domain you visit to a US company subject to US law enforcement. Not ideal. Do you have an article which explains this?
        icebraining 7y ago
        Mozilla only started to doing that for US users.
        teddyh 7y ago
        “started”
  - everdrive 7y ago
    Thanks for the explanation. I was curious about some more specifics: - How was the DNS logged? - Was every query logged, or only unique queries? - Was it combined with other data? - How long was it searchable for? - What were the DNS queries used for? Simply sold to 3rd parties? If so, who was buying?
gnu8 7y ago
I wonder why someone who knows how to do any of that, would think it is a good idea or go along with implementing that. The shitbirds who actually want to do this type of thing are not smart enough to execute it.
- deleted 7y ago
  [deleted]
- etaioinshrdlu 7y ago
  To make money on analytics?
  - jdironman 7y ago
    He is speaking of the developers and engineers who have the technical expertise and should know it's a bad idea but still agree to implement it regardless of their moral compass.
    - gilrain 7y ago
      Capitalism requires corruption of everyone.
      - afiori 7y ago
        There are many different kinds of capitalism
    - tidepod12 7y ago
      Why the implication that devs/engineers who have such technical expertise "know it's a bad idea"? There are plenty of devs and engineers who would have no moral issue with mass data collection and analytics. You don't magically become a paragon of morality just because you got a CS degree. Just ask Zuckerberg.
      - Trisell 7y ago
        I’m working at a company and they want to do a massive amount of logging from our companies IOS app. Basically log everything in the name of security. I made the statement today what does legal think about the data we would be now storing? It has user locations, gps coordinates, all the other fun stuff you can get from a users phone. They all looked at me like I was crazy for even asking that question. And I don’t think a single person in the room the devs included had even though about the personal data we were going to be able to collect. And if we SHOULD be collecting it.
deleted 7y ago
[deleted]
mirimir 7y ago
I assume that all US TLAs have (or could have) access to all data that my ISP logs (or could log). That's just how it is. Given government ~monopoly on force. And that's why I use VPN services. But the same is true for VPN services, regarding US and/or other TLAs. So I use nested VPN chains, to make it harder to get complete data. And when it really matters, I add Tor to the mix. Even if it's heavily infiltrated by US TLAs, there's at least the chance that it's also heavily infiltrated by TLAs of US adversaries. So, Dog willing, maybe they cancel each other out, at least somewhat.
- Santosh83 7y ago
  So if VPN over Tor (or Tor over VPN) increases anonymity then why is it the popular advice on the Net is not to do it?
  - NateEag 7y ago
    Perhaps because downloading Tor (or even searching for it / visiting its website) demonstrates an active interest in thwarting surveillance. Almost by definition, that means you're worth taking a closer look at. Once you're under the microscope, you'd better hope your opsec is flawless or that your activities are completely boring, or else the $TLA knows exactly what you've been up to, TOR or not. Disclosure: my activities are completely boring, and I don't use Tor, VPNs, or anything like them.
    - auslander 7y ago
      > Perhaps because downloading Tor (or even searching for it / visiting its website) demonstrates an active interest in thwarting surveillance. Not if you access tor over VPN. VPN hides all traffic from ISP and Gov. Obviously, make sure your browser does not use Google or Cloudflare DNS.
      - edf13 7y ago
        Only hides it at the VPN entry point ~ you have to trust the VPN endpoint isn’t giving up your info too!
foobiekr 7y ago
Netflow data, DNS capture, enrichment of cell tower access data (location), reporting on non-usage (idle time, tracking), Bill and household information, credit account usage, etc. SPs are huge sellers in this market. We still need to encrypt the accessed resource and DNS queries everywhere. Even once that’s done, things like opencaching will be used by SPs to gather tons of data where they participate.
- dmix 7y ago
  What about 8.8.8.8? I'm guessing we're just trusting Google here (and Cloudflare 1.1.1.1 who now also does 10gb free VPNs) + the good will of engineers with access to this information within Google.
  - foobiekr 7y ago
    I think google is evil. But I know AT&T is.
    - judge2020 7y ago
      Big G is driven by money like any other company, but they lose more money if they don't employ top security practices and prevent others from getting their data. ATT's main business isn't selling the data, it's selling the pipes that carry data, so security on their data lakes is probably less of a priority.
      - bregma 7y ago
        > ATT's main business isn't selling the data, it's selling the pipes that carry data ... and I'm sure their shareholders are pressing them to forego the greater revenue from subscriber data so they can keep their dividend cheques comfortably small.
      - foobiekr 7y ago
        With the SPs, all you have to do is look at ANYTHING security related around their business and observe what shambles it is. The last time I looked, the csrf token issued by the homepage of one of the big three mobile carriers was "undefined."
tjpnz 7y ago
What's their endgame in doing this?
lucb1e 7y ago
I recall wanting to do research in university and needing this data. It would tell me how frequent bit flips are in dns traffic. And in anonymous, aggregated form (e.g. only include domains that were accessed by multiple customers, the frequency per day and domain name, maybe geographical data precise to a region corresponding to a million people), I would be perfectly fine with this, even if it gets sent to ad companies. I'd not like it, but I'd also not see the harm and there is a lot of money involved, so if we can't stop it then I'll be fine with it in a basic form (aggregated). At least until we decide that trying to optimize manipulating/influencing people is brainwashing (I'm undecided whether playing psychological tricks on people while they try to get groceries or look for information online or whatever is morally okay).
kitteh 7y ago
In the mid 2000s it was common that several large broadband isps would have folks on their DNS team selling DNS traffic data under the table to people engaging in domain tasting and other things. It was only a matter of time until the isps realized they could wet their beak with the same info.

by z9e 7y ago

everdrive 7y ago

How do you know this fact?

z9e 7y ago
I’m friends with solutions engineers at Hortonworks and Cloudera. It’s possible I’m wrong, and since this is anecdotal evidence I see “fact” isn’t a valid use here.
windexh8er 7y ago
I worked for a Comcast subsidiary in 2010-2011 timeframe. The company owned the network end to end. They had about 250k subscribers at the time across 4 states and at the time was the first DOCIS 3.0 network in the US. They were collecting DNS log data back then. I've been told that hasn't stopped and has progressed. Don't trust your ISP to not be passively monitoring. This particular ISP had closets full of old Sandvines [0] hardware as well that I ran across one day. I asked what the hardware had been used for and the answer was simply: "network monitoring for law enforcement". At the time all of that old gear had been decomm'd. But as I've said in older posts the DC had a hands off, tamper taped mobile rack that was plugged into core routing installed by a 3 letter agency while I was employed. This was pre-Snowden and post 9/11, likely courtesy of all those fun programs we found out about that Clapper denied. [0] https://en.m.wikipedia.org/wiki/Sandvine https://en.m.wikipedia.org/wiki/Sandvine
- cabaalis 7y ago
  We'll know how much this will affect those TLAs when the government suddenly gets involved for some altruistic reason to block DNS over HTTPS. "Don't let google take over the internet!" "Time to break up big tech!"
  - windexh8er 7y ago
    I find it interesting that Google and CloudFlare are now the scapegoats. I mean, it's not like DoH isn't configurable and we don't have a choice.
    - topranks 7y ago
      Not attacking Google - their approach here is fine. But Mozilla switching people is a worry for me. Sure people “have a choice” but in reality expecting the average Joe who doesn’t even know what DNS is to make an informed decision about it is unrealistic. Meanwhile Mozilla has started sending a list of every domain you visit to a US company subject to US law enforcement. Not ideal.
      - everdrive 7y ago
        >Meanwhile Mozilla has started sending a list of every domain you visit to a US company subject to US law enforcement. Not ideal. Do you have an article which explains this?
      - icebraining 7y ago
        Mozilla only started to doing that for US users.
        teddyh 7y ago
        “started”
- everdrive 7y ago
  Thanks for the explanation. I was curious about some more specifics: - How was the DNS logged? - Was every query logged, or only unique queries? - Was it combined with other data? - How long was it searchable for? - What were the DNS queries used for? Simply sold to 3rd parties? If so, who was buying?

gnu8 7y ago

I wonder why someone who knows how to do any of that, would think it is a good idea or go along with implementing that. The shitbirds who actually want to do this type of thing are not smart enough to execute it.

deleted 7y ago
[deleted]
etaioinshrdlu 7y ago
To make money on analytics?
- jdironman 7y ago
  He is speaking of the developers and engineers who have the technical expertise and should know it's a bad idea but still agree to implement it regardless of their moral compass.
  - gilrain 7y ago
    Capitalism requires corruption of everyone.
    - afiori 7y ago
      There are many different kinds of capitalism
  - tidepod12 7y ago
    Why the implication that devs/engineers who have such technical expertise "know it's a bad idea"? There are plenty of devs and engineers who would have no moral issue with mass data collection and analytics. You don't magically become a paragon of morality just because you got a CS degree. Just ask Zuckerberg.
    - Trisell 7y ago
      I’m working at a company and they want to do a massive amount of logging from our companies IOS app. Basically log everything in the name of security. I made the statement today what does legal think about the data we would be now storing? It has user locations, gps coordinates, all the other fun stuff you can get from a users phone. They all looked at me like I was crazy for even asking that question. And I don’t think a single person in the room the devs included had even though about the personal data we were going to be able to collect. And if we SHOULD be collecting it.

deleted 7y ago

[deleted]

mirimir 7y ago

I assume that all US TLAs have (or could have) access to all data that my ISP logs (or could log). That's just how it is. Given government ~monopoly on force. And that's why I use VPN services. But the same is true for VPN services, regarding US and/or other TLAs. So I use nested VPN chains, to make it harder to get complete data. And when it really matters, I add Tor to the mix. Even if it's heavily infiltrated by US TLAs, there's at least the chance that it's also heavily infiltrated by TLAs of US adversaries. So, Dog willing, maybe they cancel each other out, at least somewhat.

Santosh83 7y ago
So if VPN over Tor (or Tor over VPN) increases anonymity then why is it the popular advice on the Net is not to do it?
- NateEag 7y ago
  Perhaps because downloading Tor (or even searching for it / visiting its website) demonstrates an active interest in thwarting surveillance. Almost by definition, that means you're worth taking a closer look at. Once you're under the microscope, you'd better hope your opsec is flawless or that your activities are completely boring, or else the $TLA knows exactly what you've been up to, TOR or not. Disclosure: my activities are completely boring, and I don't use Tor, VPNs, or anything like them.
  - auslander 7y ago
    > Perhaps because downloading Tor (or even searching for it / visiting its website) demonstrates an active interest in thwarting surveillance. Not if you access tor over VPN. VPN hides all traffic from ISP and Gov. Obviously, make sure your browser does not use Google or Cloudflare DNS.
    - edf13 7y ago
      Only hides it at the VPN entry point ~ you have to trust the VPN endpoint isn’t giving up your info too!

foobiekr 7y ago

Netflow data, DNS capture, enrichment of cell tower access data (location), reporting on non-usage (idle time, tracking), Bill and household information, credit account usage, etc. SPs are huge sellers in this market. We still need to encrypt the accessed resource and DNS queries everywhere. Even once that’s done, things like opencaching will be used by SPs to gather tons of data where they participate.

dmix 7y ago
What about 8.8.8.8? I'm guessing we're just trusting Google here (and Cloudflare 1.1.1.1 who now also does 10gb free VPNs) + the good will of engineers with access to this information within Google.
- foobiekr 7y ago
  I think google is evil. But I know AT&T is.
  - judge2020 7y ago
    Big G is driven by money like any other company, but they lose more money if they don't employ top security practices and prevent others from getting their data. ATT's main business isn't selling the data, it's selling the pipes that carry data, so security on their data lakes is probably less of a priority.
    - bregma 7y ago
      > ATT's main business isn't selling the data, it's selling the pipes that carry data ... and I'm sure their shareholders are pressing them to forego the greater revenue from subscriber data so they can keep their dividend cheques comfortably small.
    - foobiekr 7y ago
      With the SPs, all you have to do is look at ANYTHING security related around their business and observe what shambles it is. The last time I looked, the csrf token issued by the homepage of one of the big three mobile carriers was "undefined."

tjpnz 7y ago

What's their endgame in doing this?

lucb1e 7y ago

I recall wanting to do research in university and needing this data. It would tell me how frequent bit flips are in dns traffic. And in anonymous, aggregated form (e.g. only include domains that were accessed by multiple customers, the frequency per day and domain name, maybe geographical data precise to a region corresponding to a million people), I would be perfectly fine with this, even if it gets sent to ad companies. I'd not like it, but I'd also not see the harm and there is a lot of money involved, so if we can't stop it then I'll be fine with it in a basic form (aggregated). At least until we decide that trying to optimize manipulating/influencing people is brainwashing (I'm undecided whether playing psychological tricks on people while they try to get groceries or look for information online or whatever is morally okay).

kitteh 7y ago

In the mid 2000s it was common that several large broadband isps would have folks on their DNS team selling DNS traffic data under the table to people engaging in domain tasting and other things. It was only a matter of time until the isps realized they could wet their beak with the same info.