7 ms·

Facebook wants to 'normalize' the mass scraping of personal data

231 points by gbrown_ 6y ago

greyscale_2 6y ago
I'm confused by the conflation of 'scraping' and 'leaking'. Is this FB email talking about the mass scraping of information users put on their public profiles or the illicit acquisition of non-public user data?
- MattGaiser 6y ago
  > the mass scraping of information users put on their public profiles I believe it is this.
  - kuroguro 6y ago
    > the illicit acquisition of non-public user data I thought they meant this, as the phone numbers were 'scraped' trough one of their public facing features (I think it was contacts import this time, before that a lot of phones were leaked trough the search bar / forgot password). I think they're misusing 'scrape' here intentionally as if to say they did nothing wrong.
readflaggedcomm 6y ago
The strategy as worded isn't wrong, assuming I understand their terminology. If an account with no special privileges can access the information at least once, it's essentially public information. People are very worried about what it and isn't public, but these in-between areas where a platform puts up hurdles still aren't private.
- uoaei 6y ago
  Of course, Facebook designs their platform to incentivize sharing publicly whenever possible, and designs dark patterns to dissuade people from understanding the full extent of what they can control with respect to their privacy.
  - joe_the_user 6y ago
    Reality also incentivizes sharing things publicly. But screenshot sharing is huge on Facebook and on the Internet generally. The only thing kind-of-like-privacy that exists on the Internet is "encrypted messages sent to well-vetted actual friends" and anonymously posted things well-scrubbed of identifying information. Everything else is just something to make people feel better. And most people's stuff doesn't come out and create a scandal because most people's stuff is boring and unimportant, that's the main protection the average person has.
    - uoaei 5y ago
      > Reality also incentivizes sharing things publicly. Lol, no it doesn't. This is some strange, un-self-critical shilling for Facebook driven by an impulse of what seems only to be contrarianism.
novok 6y ago
Lemme mass scrape facebook and linked in then ;)
- spaced-out 6y ago
  Courts have already said you can https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- MattGaiser 6y ago
  If you can get away with it, you can.
- jandrese 6y ago
  The only thing stopping you is Facebook's tech team. It's basically an arms race between the scrapers and anti-scraping tech.
  - asdff 6y ago
    It is so frustrating how many brilliant minds on either side are just wasted fighting this bullshit war and countless others like it for someone else's bonus check. There are so many hard engineering problems that need to be solved, and it's depressing how we as a society reward who can sell the most widgets instead of who can help the most people.
    - jtsiskin 6y ago
      If you can figure out how to align incentives, you can save the world
- agogdog 6y ago
  I've tried for various reasons, they apparently have a team to fight scraping so it's a game of rate-limit whack-a-mole. This is a decent example of the imbalance of power too... Facebook could probably scrape whatever they want with little resistance from average sized sites... if they really wanted to they could put a team of skilled engineers on a team to get it. On the other end, if you want to scrape Facebook you're staring down a team of skilled engineers that's actively trying to prevent you from doing much. At this point in technology they're more formidable than most countries.
Ticklee 6y ago
They already have, the internet is a mess. Every website you visit wants more and more of your data. Facebook played a huge role in making this level of data sharing widespread.
- api 6y ago
  It's not just Facebook. Every sales, marketing, or product person in the world basically has an unlimited appetite for data and will push to suck up as much data as possible. There is a logical reason for this: one of the toughest things is knowing what your users actually want and what their actual pain points are. In advertising there's an analogous problem often summarized as: "I know I am wasting 80% of my ad spend, but I don't know which 80%." Every single incentive on the business side incentivizes data grabbing. This will never change unless users vote hard with their wallets or unless there is protective legislation.
  - Ticklee 6y ago
    I wholeheartedly think http/s is irreparably damaged, for example it is impossible to find good information on search engines, even the free ones like Searx. If you are able to find a website you can bet it includes 10MiB of trackers and ads. Hopefully someone writes a better protocol with no third party cookies and heavily restricted javascript.
    - elzbardico 6y ago
      Hate to nitpick but those things are not features of the HTTP protocol from the IETF but of HTML from the W3C
      - Ticklee 6y ago
        Nitpick away, when I am ignorant I'd rather be told than stay ignorant.
    - phailhaus 6y ago
      As the other poster pointed out, those are properties of HTML and not HTTP/S. But what I'd like to point out is that this: > heavily restricted javascript Is basically impossible. Any useful subset of javascript would be turing-complete, and therefore enough to do whatever's necessary to track the user. Literally all you need to be able to do is make an HTTP request and bam, you can track.
IceWreck 6y ago
Web scraping is legal and even if it wasn't there is no way you can prevent it. Don't upload stuff on a public website if you don't want it scraped/harvested.
- Nextgrid 6y ago
  Agreed. The problem here as I understand is that Facebook misled users about how "private" their info actually was.
  - jandrese 6y ago
    This is why I tell people to treat anything the put on the internet as public information. This includes cloud storage. If you have to put it up there then you encrypt it yourself before uploading. It only takes one compromised person/machine in the company to undermine all of that company's promises to you.
kuroguro 6y ago
Starting to feel like my login manager will soon need a 'generate random profile' button next to the 'generate password' one...
- arminiusreturns 6y ago
  What a great idea actually. Tired of manually managing my firefox profiles.
- ajdude 6y ago
  I’m at the point where I am using a new Firefox container for almost every service, and I can’t even log into my bank without enabling JavaScript on their site from some tracker domain
intricatedetail 6y ago
I will vote for any party that will make online tracking illegal. Ad targeting should be limited.
joe_the_user 6y ago
Facebook wants to "normalize" the idea that large scale scraping of user data from social networks like its own is a common occurrence Get people used to the truth? Shock, horror! I mean, certainly Facebook rose to it's position through a sort of opposite claim, that a user could be "public" (visible to a wide circle of friends-of-friends-of-etc) but not public (visible to Russian hackers, Brazilian botmasters or whoever). This claim is kind of a fairy tale, something that no only isn't true but couldn't be true. "This information is public to anyone who creates an account but not public en masse to the world". Still, the claim made an average FB user feel safer (and lot of people "got on the Internet" in a big way through FB circa ~2010). And it's got a lot of traction now. But since the situation is fundamentally porous, now that FB is large, it seems it's in their legal interest to drop the bullshit and just say "if it's public, it's public, what the hell else do you expect". And yeah, the exploitation of public data arguably lead to all sorts of bad effects and it would have been and would be nice to head this off in some fashion. But imagining you can this off by maintain a "quote-public versus totally-public" distinction isn't one of those ways.
freshpots 6y ago
Instagram is a prime example of it. They started the platform by making it default that you can't delete old comments. You can't even easily view them. That makes it easy to change a users behavior as 'that is how it always has been'. It is scary how much control these platforms have and how they are increasingly preventing users from removing/viewing past content.
kwdc 6y ago
How different the world would be if companies that hold data about you suddenly have to pay rent unless they have specific explicit permission, eg direct association. Put a stinger that means all permission granted requires a complete chain of custody for the data. So no data brokers lurking in the shadows. And a cost for non-compliance. This might get people thinking twice about building databases "just because". If the database has value then perhaps it should have a regular cost? Who knows what data is out there? My experience with just my credit reports was that the files about me were full of errors. At least I was able to correct them. I also discovered a bunch of linkedin-scraped data about me that was posted on various contact sites. Multiple errors.
- joe_the_user 6y ago
  Your plan would put Facebook, which gets data from people, in a better position, since they collect data on people with those people's permission, in exchange for services. They just have to assign a dollar value to their services and they would have fulfilled your requirements. Where yeah, it would nice if credit companies had to get permission too. And nearly every website already warns me they're going to collect data. With you're step, the next thing is signing away that rent. Or, if your plan involved rent that can't signed away, well, no one would host anyone for anything since they wouldn't want to pay that.
  - labawi 5y ago
    Facebook doesn't have permission to hold my data. Especially not scraping data, and if implemented, where would they scrape it from? Certain regulations only consider meaningful freely given consent (all your popups mean nothing), and you still need to implement the option of no unnecessary data. If they put monetary value on data, then they must have an option of paying out any extra, no? Of course they will try to weasel their way through cracks in laws, but they can be fixed, if (and only if) there is support.
aviraldg 6y ago
A better headline would be "Facebook wants to inform users about the fact that scraping publicly available information is easy (and there's nothing wrong about that.)" Of course, that wouldn't get Vice as many clicks. Don't publish information you don't want to be part of some database publicly on the internet. I wish schools had some sort of tech literacy class where they explained this stuff to people…
- jwalton 6y ago
  The information that was scraped here, though, are phone numbers that Facebook requires you to add to your account and that they promised would not be used for anything other than authentication purposes. Most of these numbers should not have been publicly available in the first place.
  - perl4ever 6y ago
    >Most of these numbers should not have been publicly available in the first place Which are "these" numbers? I thought that in fact, the numbers that were leaked in this batch were set public by the users at some point, and the ones which were not, were not leaked. https://haveibeenpwned.com/ https://haveibeenpwned.com/ indicates my phone has not been leaked, nor my @facebook address.
    - twobitshifter 6y ago
      No, it’s the people you may know feature leaking phone numbers. Facebook eats your contact list and matches it with users based on their phone numbers. This allowed you to see any random phone numbers profile. Facebook assumed that since you have the number you must know the person.
throwawayfeaxcz 6y ago
I am kind of with Facebook on this one, if you didn't want your phone scraped you shouldn't have plastered it on the internet next to your name.
1vuio0pswjnm7 6y ago
This short article is somewhat misleading on Facebook's position. They are against scraping. Not on behalf of users but on behalf of mass user data collecters, what it calls "the industry", like itself. That is why they engage is "anti-scraping". That is also why LinkedIn has tried to sue others for scraping LinkedIn public data. Facebook does not want the public, outside of "the industry", to have the same public data that Facebook has collected. If everyone can potentially have the same data Facebook has, data collection potentially becomes democratised and the world does not need Facebook nor "the industry" anymore. These advertising services companies no longer have any special value. The problem with Facebook, and "the industry", is data collection, not lack of "anti-scraping" competence. Once the sensitive data is collected by private industry on a massive scale, then liability is created. The data is not any safer than if a government had collected it. In some jurisdictions it is less safe, because there are restrictions on this type of activity by government that do not apply to companies. This liability is why some people take the position that the data collection Google or Facebook does to further its "business" is neither harmless nor "acceptable". Facebook is framing this liability problem as one of "scraping", not collection. It is not trying to further the interests of users but instead to further its own interests. Facebook wants the courts and regulatory authorities to see mass quantities of public data about internet users as Facebook's semi-exclusive asset, to be protected as if it was "private" data. Facebook is arguing mass public data "leaks" are not acceptable and that's why "the industry" must step up its "anti-scraping" measures. However "scraping" the internet for public data is not the problem, it is only a symptom. Massive data collection initiated by these companies about internet users, for the purpose of selling advertising services, is the problem.
jwalton 6y ago
Yes, it’s a broad industry issue that companies like Facebook will ask us for our phone numbers and promise they will only be used for authentication [1], and then leak them by publishing them on the Internet and letting third parties scrape them... I’m not sure it’s a problem we should normalize though. [1] https://nakedsecurity.sophos.com/2019/03/05/facebook-criticised-for-misuse-of-phone-numbers-provided-for-security/ https://nakedsecurity.sophos.com/2019/03/05/facebook-critici...
Havoc 6y ago
I mean the PR mindset of trying to actively control the narrative and normalize what’s wrong is quite slimy but they’re not fundamentally wrong.
xyzzy21 6y ago
I've been on the internet since it was called Arpanet and the one conclusion I've made: social media and to some extent the internet itself is/are the biggest mistake humanity has made since the atomic bomb.
- YAThrowaway2876 6y ago
  Assuming what you're saying about being on the internet for that long is true, I'm glad you feel that way because it gives some validity to my matching feelings. I'm much younger however, and am/was unsure if that's just me being an edgy teen ¯\_(ツ)_/¯
anigbrowl 6y ago
That's funny considering how difficult FB makes it to scrape data. Just copying and pasting stuff form their pages (for legal, journalistic, or analytical reasons) is an exercise in frustration. They even obfuscate the date and time on public posts.
soperj 6y ago
Lately with Gmail they've actually taken an account that I didn't have a phone number inputted for, but accessed on my android device, and have instituted 2-factor authentication based on that phone.

231 points by gbrown_ 6y ago

I'm confused by the conflation of 'scraping' and 'leaking'. Is this FB email talking about the mass scraping of information users put on their public profiles or the illicit acquisition of non-public user data?

MattGaiser 6y ago
> the mass scraping of information users put on their public profiles I believe it is this.
- kuroguro 6y ago
  > the illicit acquisition of non-public user data I thought they meant this, as the phone numbers were 'scraped' trough one of their public facing features (I think it was contacts import this time, before that a lot of phones were leaked trough the search bar / forgot password). I think they're misusing 'scrape' here intentionally as if to say they did nothing wrong.

readflaggedcomm 6y ago

The strategy as worded isn't wrong, assuming I understand their terminology. If an account with no special privileges can access the information at least once, it's essentially public information. People are very worried about what it and isn't public, but these in-between areas where a platform puts up hurdles still aren't private.

uoaei 6y ago
Of course, Facebook designs their platform to incentivize sharing publicly whenever possible, and designs dark patterns to dissuade people from understanding the full extent of what they can control with respect to their privacy.
- joe_the_user 6y ago
  Reality also incentivizes sharing things publicly. But screenshot sharing is huge on Facebook and on the Internet generally. The only thing kind-of-like-privacy that exists on the Internet is "encrypted messages sent to well-vetted actual friends" and anonymously posted things well-scrubbed of identifying information. Everything else is just something to make people feel better. And most people's stuff doesn't come out and create a scandal because most people's stuff is boring and unimportant, that's the main protection the average person has.
  - uoaei 5y ago
    > Reality also incentivizes sharing things publicly. Lol, no it doesn't. This is some strange, un-self-critical shilling for Facebook driven by an impulse of what seems only to be contrarianism.

novok 6y ago

Lemme mass scrape facebook and linked in then ;)

spaced-out 6y ago
Courts have already said you can https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
MattGaiser 6y ago
If you can get away with it, you can.
jandrese 6y ago
The only thing stopping you is Facebook's tech team. It's basically an arms race between the scrapers and anti-scraping tech.
- asdff 6y ago
  It is so frustrating how many brilliant minds on either side are just wasted fighting this bullshit war and countless others like it for someone else's bonus check. There are so many hard engineering problems that need to be solved, and it's depressing how we as a society reward who can sell the most widgets instead of who can help the most people.
  - jtsiskin 6y ago
    If you can figure out how to align incentives, you can save the world
agogdog 6y ago
I've tried for various reasons, they apparently have a team to fight scraping so it's a game of rate-limit whack-a-mole. This is a decent example of the imbalance of power too... Facebook could probably scrape whatever they want with little resistance from average sized sites... if they really wanted to they could put a team of skilled engineers on a team to get it. On the other end, if you want to scrape Facebook you're staring down a team of skilled engineers that's actively trying to prevent you from doing much. At this point in technology they're more formidable than most countries.

Ticklee 6y ago

They already have, the internet is a mess. Every website you visit wants more and more of your data. Facebook played a huge role in making this level of data sharing widespread.

api 6y ago
It's not just Facebook. Every sales, marketing, or product person in the world basically has an unlimited appetite for data and will push to suck up as much data as possible. There is a logical reason for this: one of the toughest things is knowing what your users actually want and what their actual pain points are. In advertising there's an analogous problem often summarized as: "I know I am wasting 80% of my ad spend, but I don't know which 80%." Every single incentive on the business side incentivizes data grabbing. This will never change unless users vote hard with their wallets or unless there is protective legislation.
- Ticklee 6y ago
  I wholeheartedly think http/s is irreparably damaged, for example it is impossible to find good information on search engines, even the free ones like Searx. If you are able to find a website you can bet it includes 10MiB of trackers and ads. Hopefully someone writes a better protocol with no third party cookies and heavily restricted javascript.
  - elzbardico 6y ago
    Hate to nitpick but those things are not features of the HTTP protocol from the IETF but of HTML from the W3C
    - Ticklee 6y ago
      Nitpick away, when I am ignorant I'd rather be told than stay ignorant.
  - phailhaus 6y ago
    As the other poster pointed out, those are properties of HTML and not HTTP/S. But what I'd like to point out is that this: > heavily restricted javascript Is basically impossible. Any useful subset of javascript would be turing-complete, and therefore enough to do whatever's necessary to track the user. Literally all you need to be able to do is make an HTTP request and bam, you can track.

IceWreck 6y ago

Web scraping is legal and even if it wasn't there is no way you can prevent it. Don't upload stuff on a public website if you don't want it scraped/harvested.

Nextgrid 6y ago
Agreed. The problem here as I understand is that Facebook misled users about how "private" their info actually was.
- jandrese 6y ago
  This is why I tell people to treat anything the put on the internet as public information. This includes cloud storage. If you have to put it up there then you encrypt it yourself before uploading. It only takes one compromised person/machine in the company to undermine all of that company's promises to you.

kuroguro 6y ago

Starting to feel like my login manager will soon need a 'generate random profile' button next to the 'generate password' one...

arminiusreturns 6y ago
What a great idea actually. Tired of manually managing my firefox profiles.
ajdude 6y ago
I’m at the point where I am using a new Firefox container for almost every service, and I can’t even log into my bank without enabling JavaScript on their site from some tracker domain

intricatedetail 6y ago

I will vote for any party that will make online tracking illegal. Ad targeting should be limited.

joe_the_user 6y ago

Facebook wants to "normalize" the idea that large scale scraping of user data from social networks like its own is a common occurrence Get people used to the truth? Shock, horror! I mean, certainly Facebook rose to it's position through a sort of opposite claim, that a user could be "public" (visible to a wide circle of friends-of-friends-of-etc) but not public (visible to Russian hackers, Brazilian botmasters or whoever). This claim is kind of a fairy tale, something that no only isn't true but couldn't be true. "This information is public to anyone who creates an account but not public en masse to the world". Still, the claim made an average FB user feel safer (and lot of people "got on the Internet" in a big way through FB circa ~2010). And it's got a lot of traction now. But since the situation is fundamentally porous, now that FB is large, it seems it's in their legal interest to drop the bullshit and just say "if it's public, it's public, what the hell else do you expect". And yeah, the exploitation of public data arguably lead to all sorts of bad effects and it would have been and would be nice to head this off in some fashion. But imagining you can this off by maintain a "quote-public versus totally-public" distinction isn't one of those ways.

freshpots 6y ago

Instagram is a prime example of it. They started the platform by making it default that you can't delete old comments. You can't even easily view them. That makes it easy to change a users behavior as 'that is how it always has been'. It is scary how much control these platforms have and how they are increasingly preventing users from removing/viewing past content.

kwdc 6y ago

How different the world would be if companies that hold data about you suddenly have to pay rent unless they have specific explicit permission, eg direct association. Put a stinger that means all permission granted requires a complete chain of custody for the data. So no data brokers lurking in the shadows. And a cost for non-compliance. This might get people thinking twice about building databases "just because". If the database has value then perhaps it should have a regular cost? Who knows what data is out there? My experience with just my credit reports was that the files about me were full of errors. At least I was able to correct them. I also discovered a bunch of linkedin-scraped data about me that was posted on various contact sites. Multiple errors.

joe_the_user 6y ago
Your plan would put Facebook, which gets data from people, in a better position, since they collect data on people with those people's permission, in exchange for services. They just have to assign a dollar value to their services and they would have fulfilled your requirements. Where yeah, it would nice if credit companies had to get permission too. And nearly every website already warns me they're going to collect data. With you're step, the next thing is signing away that rent. Or, if your plan involved rent that can't signed away, well, no one would host anyone for anything since they wouldn't want to pay that.
- labawi 5y ago
  Facebook doesn't have permission to hold my data. Especially not scraping data, and if implemented, where would they scrape it from? Certain regulations only consider meaningful freely given consent (all your popups mean nothing), and you still need to implement the option of no unnecessary data. If they put monetary value on data, then they must have an option of paying out any extra, no? Of course they will try to weasel their way through cracks in laws, but they can be fixed, if (and only if) there is support.

aviraldg 6y ago

A better headline would be "Facebook wants to inform users about the fact that scraping publicly available information is easy (and there's nothing wrong about that.)" Of course, that wouldn't get Vice as many clicks. Don't publish information you don't want to be part of some database publicly on the internet. I wish schools had some sort of tech literacy class where they explained this stuff to people…

jwalton 6y ago
The information that was scraped here, though, are phone numbers that Facebook requires you to add to your account and that they promised would not be used for anything other than authentication purposes. Most of these numbers should not have been publicly available in the first place.
- perl4ever 6y ago
  >Most of these numbers should not have been publicly available in the first place Which are "these" numbers? I thought that in fact, the numbers that were leaked in this batch were set public by the users at some point, and the ones which were not, were not leaked. https://haveibeenpwned.com/ https://haveibeenpwned.com/ indicates my phone has not been leaked, nor my @facebook address.
  - twobitshifter 6y ago
    No, it’s the people you may know feature leaking phone numbers. Facebook eats your contact list and matches it with users based on their phone numbers. This allowed you to see any random phone numbers profile. Facebook assumed that since you have the number you must know the person.

throwawayfeaxcz 6y ago

I am kind of with Facebook on this one, if you didn't want your phone scraped you shouldn't have plastered it on the internet next to your name.

1vuio0pswjnm7 6y ago

This short article is somewhat misleading on Facebook's position. They are against scraping. Not on behalf of users but on behalf of mass user data collecters, what it calls "the industry", like itself. That is why they engage is "anti-scraping". That is also why LinkedIn has tried to sue others for scraping LinkedIn public data. Facebook does not want the public, outside of "the industry", to have the same public data that Facebook has collected. If everyone can potentially have the same data Facebook has, data collection potentially becomes democratised and the world does not need Facebook nor "the industry" anymore. These advertising services companies no longer have any special value. The problem with Facebook, and "the industry", is data collection, not lack of "anti-scraping" competence. Once the sensitive data is collected by private industry on a massive scale, then liability is created. The data is not any safer than if a government had collected it. In some jurisdictions it is less safe, because there are restrictions on this type of activity by government that do not apply to companies. This liability is why some people take the position that the data collection Google or Facebook does to further its "business" is neither harmless nor "acceptable". Facebook is framing this liability problem as one of "scraping", not collection. It is not trying to further the interests of users but instead to further its own interests. Facebook wants the courts and regulatory authorities to see mass quantities of public data about internet users as Facebook's semi-exclusive asset, to be protected as if it was "private" data. Facebook is arguing mass public data "leaks" are not acceptable and that's why "the industry" must step up its "anti-scraping" measures. However "scraping" the internet for public data is not the problem, it is only a symptom. Massive data collection initiated by these companies about internet users, for the purpose of selling advertising services, is the problem.

jwalton 6y ago

Yes, it’s a broad industry issue that companies like Facebook will ask us for our phone numbers and promise they will only be used for authentication [1], and then leak them by publishing them on the Internet and letting third parties scrape them... I’m not sure it’s a problem we should normalize though. [1] https://nakedsecurity.sophos.com/2019/03/05/facebook-criticised-for-misuse-of-phone-numbers-provided-for-security/ https://nakedsecurity.sophos.com/2019/03/05/facebook-critici...

Havoc 6y ago

I mean the PR mindset of trying to actively control the narrative and normalize what’s wrong is quite slimy but they’re not fundamentally wrong.

xyzzy21 6y ago

I've been on the internet since it was called Arpanet and the one conclusion I've made: social media and to some extent the internet itself is/are the biggest mistake humanity has made since the atomic bomb.

YAThrowaway2876 6y ago
Assuming what you're saying about being on the internet for that long is true, I'm glad you feel that way because it gives some validity to my matching feelings. I'm much younger however, and am/was unsure if that's just me being an edgy teen ¯\_(ツ)_/¯

anigbrowl 6y ago

That's funny considering how difficult FB makes it to scrape data. Just copying and pasting stuff form their pages (for legal, journalistic, or analytical reasons) is an exercise in frustration. They even obfuscate the date and time on public posts.

soperj 6y ago

Lately with Gmail they've actually taken an account that I didn't have a phone number inputted for, but accessed on my android device, and have instituted 2-factor authentication based on that phone.