16 ms·

How your data is collected and commoditised via “free” online services

217 points by matthewwarren 11y ago

hodwik 11y ago
He's on the right track, but absolutely wrong about where the bulk of this data comes from. It's not shady "Take a free trip!" websites. It's your Dominoes Pizza order, it's your trip to Target, it's everything you do. (A sales person for one of the big companies listed those specific groups, but said it was basically everyone.) The big companies in the industry (LexisNexis, Thompson Reuters' CLEAR) buy everyone's data. Anything they can get their hands on. As far as I can tell, everyone in the country's data is being stored; everywhere you have ever lived, all of your phone numbers, voting records, debts, legal history, marriages, past names and aliases, email addresses, everything. It's all in a handful of huge private systems. Systems that were originally marketed to law enforcement agencies, but are now being shopped to private companies as well. Insant CheckMate is just a tiny tiny fraction of what is actually captured by these data brokers. Source: You'll have to take my word for it. Interesting tidbit -- They have alerts setup in the system; E.g. if you search for the wrong person you'll get a call from headquarters asking why you're pulling up Kanye West's home phone number, etc.
- shostack 11y ago
  There's a lot more (Axciom, Neustar, etc.). Just check out one of these Lumascapes--the data companies are slightly off to the right near the top in the DB Marketing, DMP and Marketing Data sections [1]. Advertisers and publishers can also sell this data directly on exchanges (a relatively new thing) like Adobe's Audience Marketplace [2]. [1] http://www.lumapartners.com/lumascapes/marketing-technology-lumascape/ http://www.lumapartners.com/lumascapes/marketing-technology-... [2] https://blogs.adobe.com/digitalmarketing/advertising/source-sell-and-swap-audiences-with-ease-using-adobe-audience-manager/ https://blogs.adobe.com/digitalmarketing/advertising/source-...
- losteric 11y ago
  If it's legal to sell, it's for sale. Credit cards definitely sell your info. People also forget about "public data". Student organizations (sport tournaments, student bands, chess club, etc) and middle school/high school/college graduation lists are big ones for targeting young consumers. Facebook/Twitter is just the tip of the iceberg. Hyper-targeting is very real and companies know way more about you than you think. Pay cash and be mindful of what data gets online. * On a more positive note, I'm excited to see how digital personal assistants integrate this kind of data for personalization.
- fweespee_ch 11y ago
  > Source: You'll have to take my word for it. Fwiw, I agree with you. ;) Source: Someone tried to sell me this sort of thing once as part of an age verification service.
  - daodedickinson 11y ago
    Haha so why not sell this to dudebros worried about getting brought up on statutory rape charges? Or is there already an app for that?
    - fweespee_ch 11y ago
      Because I have something resembling a soul? Iirc, their marketing said they were ~95% accurate.
      - daodedickinson 11y ago
        Well I think it might be a valid public service to keep people from committing statutory rape. Or are your worried about the effect on your soul when 5% of the time someone has underage sex and another person goes to jail?
        fweespee_ch 11y ago
        > Well I think it might be a valid public service to keep people from committing statutory rape. Or are your worried about the effect on your soul when 5% of the time someone has underage sex and another person goes to jail? That is what I'd be worried about. I gave them false assurance and I'd feel responsible.
- state 11y ago
  OK, I'll take your word for it. Everything I do is being tracked, recorded and passed around by companies big and small. Yep, got it. Anyone care to speculate on how we actually change this situation? Or is it that there's really not much utility in privacy for most people, so we can just forget about it permanently. I realize that the paths to change are either regulatory or technical (and I find the latter more likely), but seriously: paint the picture. Should we just accept this as the norm? (edit: grammar)
  - bduerst 11y ago
    This isn't a popular question, but is there really a problem? I've heard the big-brother nightmare scenario type answers before, but I'm curious to see how they are more likely to negatively affect me than a car accident or slipping in the shower. Even with government agencies (NSA/FBI) overstepping or trying to overstep their limits, isn't the root problem with policy makers keeping them in line?
    - astazangasta 11y ago
      We're possibly months away from a Trump presidency, and you're not worried that this sort of power exists in the world? Trump aside, the general trend is increasingly fascist. Maybe you're safely ensconced in the glitterati, but I am a known leftist with brown skin; I'm going to worry.
      - bduerst 11y ago
        Come on, even this nightmare scenario isn't that nightmarish. The POTUS commands the nuclear arsenal, which is arguably more powerful than knowing my shopping habits.
        majewsky 11y ago
        Depends. The POTUS is not likely to use the nuclear arsenal against you. Your personal data, though, is much more actionable from his POV.
- bsilvereagle 11y ago
  > everywhere you have ever lived, all of your phone numbers, voting records, debts, legal history, marriages, past names and aliases, email addresses, everything. A lot of these things are publicly available from government entities. Voter records (available from sites like voterfind.com) include your address, and phone number in some locales. Marriage licenses are public record. If a debt goes to court, it becomes a public record. Any and all legal incidents are public record. Property taxes are public record.
  - majewsky 11y ago
    Depends on the law. Most or all of these examples are not public record in Germany.
  - hodwik 11y ago
    Yes, technically. In the US your SSN, your mother's maiden name, and all of your school records are public data under many circumstances. That doesn't mean they get used or treated as such.
- arafa 11y ago
  A lot of how personal data is collected and sold is detailed in Bruce Schneier's "Data and Goliath" (http://www.amazon.com/Data-Goliath-Battles-Collect-Control/dp/0393244814 http://www.amazon.com/Data-Goliath-Battles-Collect-Control/d...) and confirms what you're saying.
  - choosername 11y ago
    link to amazon kinda ironic
- webbore 11y ago
  I saw a (I believe) 60 minutes story about WalMart a while ago and they stated they stored everything at line-item resolution for 2 years of sales. With their volume it's an amazing amount of data. UPDATE - did some quick googling and that was 10 years ago. I have no idea how big their DW is now (was 583 TB then)
  - hammock 11y ago
    The one thing about Walmart though, and many of the largest retailers like Kroger, is that that market basket data is far too valuable to allow it into the hands of anyone else - and they won't/don't sell it. Other smaller retailers, including Target and hundreds of others, will pool their transaction data into blind coops like Abacus, or syndicate it via companies like dunnhumby.
- bigiain 11y ago
  That's true about "the bulk of the data ( … that is collected and comoditized by "free" websites … )" sense, but Troy's article is talking about a specific breach he'd been given and his investigation of it's providence. _This_ bunch of data had, with a fairly high degree of certainty, come from shady "Take a free trip!" websites. (Admittedly, the headline is misleading there, but that's typical of headlines...)
- Gustomaximus 11y ago
  I'd wouldn't say the bulk comes from large stores vs. 'win-a-trip' type sites. Its both. I buy these leads on a regular basis (many millions a year) and I'd say where the most comes from depends on country primarily. Also businesses like Dominoes typically dont sell identifiable customer data, they will often present your company as an offer either via EDM, newsletter or ever increasingly post-transaction via lead aggregators. When Dominoes/Target type organisations do sell data it's typically done in a non-PII (personally identifiable information) format, for legal, ethical and brand reasons. This way companies can utilise the data do things like understand purchase personalities/habits by postcode type thing but they dont directly know who you are. There is another subset of companies that claim they can take this non-PII data and re-match it to people but I'm skeptical about how accurate this is. Another an area not mentioned of personal darta storage that is every growing is programmatic media. In this ad servers attempt to identify you and call on stored information which is then used in a stock market like real-time environment to bid for ad views. Like when ads follow you around the internet, but for every sophisticated information/algorithmic rationale on who you are. Nothing nefarious but marketers want to show ads that are likely to be interesting and result in a sale (or other company goal).
  - jsonne 11y ago
    This is pretty spot on. It's also good to note that marketers like me don't get your name and everything about you. We get anonymous and aggregated data. Eg. I have no idea if you John Doe personally buy dominos. I'm target 50k people that like dominos, also like cars, and live in the Bay Area.
    - hodwik 11y ago
      I'm not talking about marketing systems. I'm talking about systems full of PII -- like Thomson Reuters CLEAR -- which was designed for law enforcement but can be purchased by private companies as well. http://r1.officer.com/files/base/image/OFCR/2013/04/16x9/640x360/clear-screenshot-addresses_10919281.jpg http://r1.officer.com/files/base/image/OFCR/2013/04/16x9/640...
  - hodwik 11y ago
- stephengillie 11y ago
  If you play games with Steam, someone is recording your play sessions. It's not actually much different from graphing when your Facebook friends are asleep[1]. Valve, being an NIH shop, use a UDP call instead of an API like most groups[2]. At a minimum, you can get the Steam name of every player on any server. [1] https://news.ycombinator.com/item?id=11130688 https://news.ycombinator.com/item?id=11130688 [2] https://developer.valvesoftware.com/wiki/Server_queries https://developer.valvesoftware.com/wiki/Server_queries
- wiradikusuma 11y ago
  Honest questions: How do you explain to your potential buyers (or govt agencies) when they ask you where you get the data from? "Various sources"? I mean, sure, you can collect data however you want, but I don't think any legitimate business want it unless you collected it legitimately. Or am I wrong?
TheCartographer 11y ago
He says it in the article but it walkways bears repeating: if you aren't the customer, you're the product.
- guelo 11y ago
  No, even if you are the customer, say at a supermarket, they're still going to take your purchase data and sell it. Supermarkets are especially sneaky, they'll charge you extra if you refuse to be tracked via a "loyalty" card, and then they market that extortion as savings!
  - ross-life 11y ago
    Ha, I don't use loyalty cards because of the tracking but I never thought of the price difference that way, it makes sense.
  - pravda 11y ago
    Supermarkets will give you as many 'loyalty' cards as you want. And you can put false info on them. I assume they can still track your purchases by credit card #.
    - spdustin 11y ago
      Handy tip: in the US, enter (nnn) 867-5309 (where nnn is the local area code). There's a real good chance someone already has a card assigned to that number.
    - newman314 11y ago
      Actually, BevMo is now being much more brazen about it. They scan your driver's license under the guise of age verification. I'm several decades past 21. There's no way in hell that I'm an underage drinker. When pressed the last time I was there, I was told that age verification up to the age of 50 is store policy.
      - hodwik 11y ago
textaural 11y ago
I have found https://donottrack-doc.com https://donottrack-doc.com [made by the NFB] to be helpful in thinking about data collection. If I have time at the end of this semester I'll be taking my first-year students through the first episode.
mirimir 11y ago
Just disappear. Gradually, a little at a time. Diversify identities. Take your friends with you. Make it a game. Many games. What's left public is just the boring stuff. Make it as normal and unremarkable as possible.
- bgar 11y ago
  I've toyed with the idea of vanishing completely, but nowadays employers often want to see a web presence.... if it weren't for that, and the convenience of Facebook Messenger, I might have disappeared already. I was just thinking about how since one usually uses the same identity for many different accounts and interests online, it would be very easy to find someone by looking for people that match those specific sets of interests. You would have to make separate identities for every sort of interest, and never mention the other from a different profile.
  - mirimir 11y ago
    Well, most people on Facebook just share links. So it's not hard to look normal. Maybe a little more boring than most, but hey, you work hard and all ;) If you're going to use multiple identities, they shouldn't have overlapping interests, or common friends, accounts, etc, etc. And they shouldn't ever refer to each other. However, it's fine to have a bunch of identities that break all of those rules, just for the fun of it. You have unassociated clouds of associated identities. Dig into Mirimir, if you're interested. There are several more-or-less associated identities. Some of them are obvious. Some would take a little work to find. But then, Mirimir is just a relatively superficial pseudonym.
  - deleted 11y ago
    [deleted]
gcb0 11y ago
most other countries listed have laws that completely drop the "we can sell your data" clause on the site. If you can find a presence of the company in such countries (and for most you have to provide some sort of tax id to buy national domain sulfix) then you can enter unique data there, and call a lawyer. profit.

217 points by matthewwarren 11y ago

He's on the right track, but absolutely wrong about where the bulk of this data comes from. It's not shady "Take a free trip!" websites. It's your Dominoes Pizza order, it's your trip to Target, it's everything you do. (A sales person for one of the big companies listed those specific groups, but said it was basically everyone.) The big companies in the industry (LexisNexis, Thompson Reuters' CLEAR) buy everyone's data. Anything they can get their hands on. As far as I can tell, everyone in the country's data is being stored; everywhere you have ever lived, all of your phone numbers, voting records, debts, legal history, marriages, past names and aliases, email addresses, everything. It's all in a handful of huge private systems. Systems that were originally marketed to law enforcement agencies, but are now being shopped to private companies as well. Insant CheckMate is just a tiny tiny fraction of what is actually captured by these data brokers. Source: You'll have to take my word for it. Interesting tidbit -- They have alerts setup in the system; E.g. if you search for the wrong person you'll get a call from headquarters asking why you're pulling up Kanye West's home phone number, etc.

shostack 11y ago
There's a lot more (Axciom, Neustar, etc.). Just check out one of these Lumascapes--the data companies are slightly off to the right near the top in the DB Marketing, DMP and Marketing Data sections [1]. Advertisers and publishers can also sell this data directly on exchanges (a relatively new thing) like Adobe's Audience Marketplace [2]. [1] http://www.lumapartners.com/lumascapes/marketing-technology-lumascape/ http://www.lumapartners.com/lumascapes/marketing-technology-... [2] https://blogs.adobe.com/digitalmarketing/advertising/source-sell-and-swap-audiences-with-ease-using-adobe-audience-manager/ https://blogs.adobe.com/digitalmarketing/advertising/source-...
losteric 11y ago
If it's legal to sell, it's for sale. Credit cards definitely sell your info. People also forget about "public data". Student organizations (sport tournaments, student bands, chess club, etc) and middle school/high school/college graduation lists are big ones for targeting young consumers. Facebook/Twitter is just the tip of the iceberg. Hyper-targeting is very real and companies know way more about you than you think. Pay cash and be mindful of what data gets online. * On a more positive note, I'm excited to see how digital personal assistants integrate this kind of data for personalization.
fweespee_ch 11y ago
> Source: You'll have to take my word for it. Fwiw, I agree with you. ;) Source: Someone tried to sell me this sort of thing once as part of an age verification service.
- daodedickinson 11y ago
  Haha so why not sell this to dudebros worried about getting brought up on statutory rape charges? Or is there already an app for that?
  - fweespee_ch 11y ago
    Because I have something resembling a soul? Iirc, their marketing said they were ~95% accurate.
    - daodedickinson 11y ago
      Well I think it might be a valid public service to keep people from committing statutory rape. Or are your worried about the effect on your soul when 5% of the time someone has underage sex and another person goes to jail?
      - fweespee_ch 11y ago
        > Well I think it might be a valid public service to keep people from committing statutory rape. Or are your worried about the effect on your soul when 5% of the time someone has underage sex and another person goes to jail? That is what I'd be worried about. I gave them false assurance and I'd feel responsible.
state 11y ago
OK, I'll take your word for it. Everything I do is being tracked, recorded and passed around by companies big and small. Yep, got it. Anyone care to speculate on how we actually change this situation? Or is it that there's really not much utility in privacy for most people, so we can just forget about it permanently. I realize that the paths to change are either regulatory or technical (and I find the latter more likely), but seriously: paint the picture. Should we just accept this as the norm? (edit: grammar)
- bduerst 11y ago
  This isn't a popular question, but is there really a problem? I've heard the big-brother nightmare scenario type answers before, but I'm curious to see how they are more likely to negatively affect me than a car accident or slipping in the shower. Even with government agencies (NSA/FBI) overstepping or trying to overstep their limits, isn't the root problem with policy makers keeping them in line?
  - astazangasta 11y ago
    We're possibly months away from a Trump presidency, and you're not worried that this sort of power exists in the world? Trump aside, the general trend is increasingly fascist. Maybe you're safely ensconced in the glitterati, but I am a known leftist with brown skin; I'm going to worry.
    - bduerst 11y ago
      Come on, even this nightmare scenario isn't that nightmarish. The POTUS commands the nuclear arsenal, which is arguably more powerful than knowing my shopping habits.
      - majewsky 11y ago
        Depends. The POTUS is not likely to use the nuclear arsenal against you. Your personal data, though, is much more actionable from his POV.
bsilvereagle 11y ago
> everywhere you have ever lived, all of your phone numbers, voting records, debts, legal history, marriages, past names and aliases, email addresses, everything. A lot of these things are publicly available from government entities. Voter records (available from sites like voterfind.com) include your address, and phone number in some locales. Marriage licenses are public record. If a debt goes to court, it becomes a public record. Any and all legal incidents are public record. Property taxes are public record.
- majewsky 11y ago
  Depends on the law. Most or all of these examples are not public record in Germany.
- hodwik 11y ago
  Yes, technically. In the US your SSN, your mother's maiden name, and all of your school records are public data under many circumstances. That doesn't mean they get used or treated as such.
arafa 11y ago
A lot of how personal data is collected and sold is detailed in Bruce Schneier's "Data and Goliath" (http://www.amazon.com/Data-Goliath-Battles-Collect-Control/dp/0393244814 http://www.amazon.com/Data-Goliath-Battles-Collect-Control/d...) and confirms what you're saying.
- choosername 11y ago
  link to amazon kinda ironic
webbore 11y ago
I saw a (I believe) 60 minutes story about WalMart a while ago and they stated they stored everything at line-item resolution for 2 years of sales. With their volume it's an amazing amount of data. UPDATE - did some quick googling and that was 10 years ago. I have no idea how big their DW is now (was 583 TB then)
- hammock 11y ago
  The one thing about Walmart though, and many of the largest retailers like Kroger, is that that market basket data is far too valuable to allow it into the hands of anyone else - and they won't/don't sell it. Other smaller retailers, including Target and hundreds of others, will pool their transaction data into blind coops like Abacus, or syndicate it via companies like dunnhumby.
bigiain 11y ago
That's true about "the bulk of the data ( … that is collected and comoditized by "free" websites … )" sense, but Troy's article is talking about a specific breach he'd been given and his investigation of it's providence. _This_ bunch of data had, with a fairly high degree of certainty, come from shady "Take a free trip!" websites. (Admittedly, the headline is misleading there, but that's typical of headlines...)
Gustomaximus 11y ago
I'd wouldn't say the bulk comes from large stores vs. 'win-a-trip' type sites. Its both. I buy these leads on a regular basis (many millions a year) and I'd say where the most comes from depends on country primarily. Also businesses like Dominoes typically dont sell identifiable customer data, they will often present your company as an offer either via EDM, newsletter or ever increasingly post-transaction via lead aggregators. When Dominoes/Target type organisations do sell data it's typically done in a non-PII (personally identifiable information) format, for legal, ethical and brand reasons. This way companies can utilise the data do things like understand purchase personalities/habits by postcode type thing but they dont directly know who you are. There is another subset of companies that claim they can take this non-PII data and re-match it to people but I'm skeptical about how accurate this is. Another an area not mentioned of personal darta storage that is every growing is programmatic media. In this ad servers attempt to identify you and call on stored information which is then used in a stock market like real-time environment to bid for ad views. Like when ads follow you around the internet, but for every sophisticated information/algorithmic rationale on who you are. Nothing nefarious but marketers want to show ads that are likely to be interesting and result in a sale (or other company goal).
- jsonne 11y ago
  This is pretty spot on. It's also good to note that marketers like me don't get your name and everything about you. We get anonymous and aggregated data. Eg. I have no idea if you John Doe personally buy dominos. I'm target 50k people that like dominos, also like cars, and live in the Bay Area.
  - hodwik 11y ago
    I'm not talking about marketing systems. I'm talking about systems full of PII -- like Thomson Reuters CLEAR -- which was designed for law enforcement but can be purchased by private companies as well. http://r1.officer.com/files/base/image/OFCR/2013/04/16x9/640x360/clear-screenshot-addresses_10919281.jpg http://r1.officer.com/files/base/image/OFCR/2013/04/16x9/640...
- hodwik 11y ago
stephengillie 11y ago
If you play games with Steam, someone is recording your play sessions. It's not actually much different from graphing when your Facebook friends are asleep[1]. Valve, being an NIH shop, use a UDP call instead of an API like most groups[2]. At a minimum, you can get the Steam name of every player on any server. [1] https://news.ycombinator.com/item?id=11130688 https://news.ycombinator.com/item?id=11130688 [2] https://developer.valvesoftware.com/wiki/Server_queries https://developer.valvesoftware.com/wiki/Server_queries
wiradikusuma 11y ago
Honest questions: How do you explain to your potential buyers (or govt agencies) when they ask you where you get the data from? "Various sources"? I mean, sure, you can collect data however you want, but I don't think any legitimate business want it unless you collected it legitimately. Or am I wrong?

TheCartographer 11y ago

He says it in the article but it walkways bears repeating: if you aren't the customer, you're the product.

guelo 11y ago
No, even if you are the customer, say at a supermarket, they're still going to take your purchase data and sell it. Supermarkets are especially sneaky, they'll charge you extra if you refuse to be tracked via a "loyalty" card, and then they market that extortion as savings!
- ross-life 11y ago
  Ha, I don't use loyalty cards because of the tracking but I never thought of the price difference that way, it makes sense.
- pravda 11y ago
  Supermarkets will give you as many 'loyalty' cards as you want. And you can put false info on them. I assume they can still track your purchases by credit card #.
  - spdustin 11y ago
    Handy tip: in the US, enter (nnn) 867-5309 (where nnn is the local area code). There's a real good chance someone already has a card assigned to that number.
  - newman314 11y ago
    Actually, BevMo is now being much more brazen about it. They scan your driver's license under the guise of age verification. I'm several decades past 21. There's no way in hell that I'm an underage drinker. When pressed the last time I was there, I was told that age verification up to the age of 50 is store policy.
    - hodwik 11y ago

textaural 11y ago

I have found https://donottrack-doc.com https://donottrack-doc.com [made by the NFB] to be helpful in thinking about data collection. If I have time at the end of this semester I'll be taking my first-year students through the first episode.

mirimir 11y ago

Just disappear. Gradually, a little at a time. Diversify identities. Take your friends with you. Make it a game. Many games. What's left public is just the boring stuff. Make it as normal and unremarkable as possible.

bgar 11y ago
I've toyed with the idea of vanishing completely, but nowadays employers often want to see a web presence.... if it weren't for that, and the convenience of Facebook Messenger, I might have disappeared already. I was just thinking about how since one usually uses the same identity for many different accounts and interests online, it would be very easy to find someone by looking for people that match those specific sets of interests. You would have to make separate identities for every sort of interest, and never mention the other from a different profile.
- mirimir 11y ago
  Well, most people on Facebook just share links. So it's not hard to look normal. Maybe a little more boring than most, but hey, you work hard and all ;) If you're going to use multiple identities, they shouldn't have overlapping interests, or common friends, accounts, etc, etc. And they shouldn't ever refer to each other. However, it's fine to have a bunch of identities that break all of those rules, just for the fun of it. You have unassociated clouds of associated identities. Dig into Mirimir, if you're interested. There are several more-or-less associated identities. Some of them are obvious. Some would take a little work to find. But then, Mirimir is just a relatively superficial pseudonym.
- deleted 11y ago
  [deleted]

gcb0 11y ago

most other countries listed have laws that completely drop the "we can sell your data" clause on the site. If you can find a presence of the company in such countries (and for most you have to provide some sort of tax id to buy national domain sulfix) then you can enter unique data there, and call a lawyer. profit.