11 ms·

Facebook was used as a proxy by web scraping bots

285 points by avastel 6y ago

AznHisoka 6y ago
How is DataDome different from Cloudflare? The latter offers bot protection for free if you are already a Cloudflare customer
- iampims 6y ago
  They’re a French/EU company, that alone can be an advantage for some businesses.
  - kevin1985 6y ago
    Pretty sure they have an office in New York as well. Site shows POPs all over the map: datadome.co
- ericcholis 6y ago
  My understanding is that it's more "advanced". Take that at face value, I've not used the service.
martimarkov 6y ago
The website doesn’t open for me.
- llacb47 6y ago
  Check your DNS, if you are running a pihole or something similar you will need to disable or allow this site.
  - inetknght 6y ago
    Enable spamware? Big red flag nokthx
cblconfederate 6y ago
You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking <title> tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.cloudflare.com/t/attention-required-message-when-sharing-link/88999 https://community.cloudflare.com/t/attention-required-messag...
- neilparikh 6y ago
  > Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.
  - qwerty456127 6y ago
    There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.
    - cblconfederate 6y ago
      > Blocking crawling should be outlawed That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.
    - sokoloff 6y ago
      Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.
tyingq 6y ago
That's pretty interesting, Facebook as a "web scale / hundreds of pages per second" batch web page summarizer. I imagine you could build a pretty decent general purpose search engine that way...free crawler.
- daniellarusso 6y ago
  As long is they are using opengraph meta tags.
MDinBk 6y ago
interesting!
jedberg 6y ago
So this was a very long web page to say: Facebook forgot to rate limit their web scraper on a per user basis, but we told them and they fixed it.
- ada1981 6y ago
  In the same way your comment is a very long way to say: something happened.
- dmix 6y ago
  Also it’s not really a scraper. Just a json response with the website preview stuff like meta tags, <title/>, and other basic information which could be useful for some bots. But it does not give you the whole HTML. Or anything close.
- lenitabinol 6y ago
  Exactly, they literally didn't do anything but file a Facebook bug report.
sschueller 6y ago
I've used Yahoo's YQL. While I would hit rate limits and other crap when trying to scape data of some sites directly. YQL would provide me nicely structured data without these stupid limits as many don't see yahoo's bot as a scraper.
jtsiskin 6y ago
Why can’t you just make your web crawler look like fb or googlebot (via user agent)? Do website owners actually check the ip?
- colinmhayes 6y ago
  Sounds like this company is checking that the ip is from facebook. That would probably work on less secure sites though.
ddorian43 6y ago
At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?
- echelon 6y ago
  This is so fucked. We've encrusted ourselves into this walled fiefdom, and there's no way to break free. It's only going to get worse. Chrome will be the only browser. AMP the only delivery mechanism. Video will require DRM. Eventually, text content will too. Binary blobs with no ad blocking.
  - einpoklum 6y ago
    Well, I think you're being overly alarmist, at least in the short term: DRM on Video has not really caught on, at least on-line; Non-Chrome browsers continue to have a significant share (mostly on Desktops); and ad blocking remains rather effective. In the long run, I'm definitely worried: Capitalist economies tend to see a concentration of capital, generally and in most sectors individually. And this seems to be a real danger with computing technology. Coupled with mass surveillance and the pushing of people to have their personal information held by those large tech companies, a dystopia is not inconceivable. PS - By AMP, do you mean Amazon Prime?
    - AlexandrB 6y ago
      > DRM on Video has not really caught on, at least on-line This seems like a weird statement. All of the paid streaming services use DRM on Video, so all major browsers include the requisite black-box DRM modules. I'm actually surprised YouTube has not added Widevine DRM for all videos yet, but I'm sure it'll happen if RIAA/MPAA get annoyed enough with youtube-dl and the like. > PS - By AMP, do you mean Amazon Prime? I think he means Google AMP[1], which is slowly infecting more of the top search results on Google. [1] https://developers.google.com/amp/ https://developers.google.com/amp/
intricatedetail 6y ago
Someone who would create a scraping API that site owners could embed in their projects and get paid for feeding crawlers with their data, could make billions.
Ombudsman 6y ago
I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.
- lamby 6y ago
  May I ask which country?
  - anuila 6y ago
    There are a lot of countries where Facebook is “free”, for example India and Philippines.
    - octoberfranklin 6y ago
      I'm guessing that in these countries the cost of Internet traffic is dominated by the undersea cables at their borders. Facebook has been paying to have new undersea cables laid. This is done as part of a consortium, but those cables only have 6-12 strands in them (the repeaters are bulky) so owning even just one whole strand of fiber in an undersea cable is still an obscene amount of bandwidth for a single company that isn't in the business of reselling bandwidth.
      - apostacy 6y ago
        In The Philippines, my understanding is that they have ample bandwidth via Korea and other countries in the region. But the reason they have such expensive terrible internet is because of a lack of net neutrality and deregulation. The cellphone duopoly sells "YouTube passes", that entitle you to get unthrottled YouTube for brief periods of time.
        Nextgrid 6y ago
deleted 6y ago
[deleted]
AmericanChopper 6y ago
Would just like to give an honorable mention to Google Translate, the most accessible http proxy of all time. It’s especially good for bypassing corporate access controls. I’ve used it many times for accessing solution threads on technical subreddits at work.
notRobot 6y ago
datadome.co is blocked for me: > datadome.co is being blocked by AdGuard DNS filter, AdGuard Tracking Protection filter, EasyPrivacy, Goodbye Ads and oisd. Dunno what they do, but it can't be good.

285 points by avastel 6y ago

How is DataDome different from Cloudflare? The latter offers bot protection for free if you are already a Cloudflare customer

iampims 6y ago
They’re a French/EU company, that alone can be an advantage for some businesses.
- kevin1985 6y ago
  Pretty sure they have an office in New York as well. Site shows POPs all over the map: datadome.co
ericcholis 6y ago
My understanding is that it's more "advanced". Take that at face value, I've not used the service.

martimarkov 6y ago

The website doesn’t open for me.

llacb47 6y ago
Check your DNS, if you are running a pihole or something similar you will need to disable or allow this site.
- inetknght 6y ago
  Enable spamware? Big red flag nokthx

cblconfederate 6y ago

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking <title> tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.cloudflare.com/t/attention-required-message-when-sharing-link/88999 https://community.cloudflare.com/t/attention-required-messag...

neilparikh 6y ago
> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.
- qwerty456127 6y ago
  There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.
  - cblconfederate 6y ago
    > Blocking crawling should be outlawed That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.
  - sokoloff 6y ago
    Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.

tyingq 6y ago

That's pretty interesting, Facebook as a "web scale / hundreds of pages per second" batch web page summarizer. I imagine you could build a pretty decent general purpose search engine that way...free crawler.

daniellarusso 6y ago
As long is they are using opengraph meta tags.

MDinBk 6y ago

interesting!

jedberg 6y ago

So this was a very long web page to say: Facebook forgot to rate limit their web scraper on a per user basis, but we told them and they fixed it.

ada1981 6y ago
In the same way your comment is a very long way to say: something happened.
dmix 6y ago
Also it’s not really a scraper. Just a json response with the website preview stuff like meta tags, <title/>, and other basic information which could be useful for some bots. But it does not give you the whole HTML. Or anything close.
lenitabinol 6y ago
Exactly, they literally didn't do anything but file a Facebook bug report.

sschueller 6y ago

I've used Yahoo's YQL. While I would hit rate limits and other crap when trying to scape data of some sites directly. YQL would provide me nicely structured data without these stupid limits as many don't see yahoo's bot as a scraper.

jtsiskin 6y ago

Why can’t you just make your web crawler look like fb or googlebot (via user agent)? Do website owners actually check the ip?

colinmhayes 6y ago
Sounds like this company is checking that the ip is from facebook. That would probably work on less secure sites though.

ddorian43 6y ago

At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?

echelon 6y ago
This is so fucked. We've encrusted ourselves into this walled fiefdom, and there's no way to break free. It's only going to get worse. Chrome will be the only browser. AMP the only delivery mechanism. Video will require DRM. Eventually, text content will too. Binary blobs with no ad blocking.
- einpoklum 6y ago
  Well, I think you're being overly alarmist, at least in the short term: DRM on Video has not really caught on, at least on-line; Non-Chrome browsers continue to have a significant share (mostly on Desktops); and ad blocking remains rather effective. In the long run, I'm definitely worried: Capitalist economies tend to see a concentration of capital, generally and in most sectors individually. And this seems to be a real danger with computing technology. Coupled with mass surveillance and the pushing of people to have their personal information held by those large tech companies, a dystopia is not inconceivable. PS - By AMP, do you mean Amazon Prime?
  - AlexandrB 6y ago
    > DRM on Video has not really caught on, at least on-line This seems like a weird statement. All of the paid streaming services use DRM on Video, so all major browsers include the requisite black-box DRM modules. I'm actually surprised YouTube has not added Widevine DRM for all videos yet, but I'm sure it'll happen if RIAA/MPAA get annoyed enough with youtube-dl and the like. > PS - By AMP, do you mean Amazon Prime? I think he means Google AMP[1], which is slowly infecting more of the top search results on Google. [1] https://developers.google.com/amp/ https://developers.google.com/amp/

intricatedetail 6y ago

Someone who would create a scraping API that site owners could embed in their projects and get paid for feeding crawlers with their data, could make billions.

Ombudsman 6y ago

I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.

lamby 6y ago
May I ask which country?
- anuila 6y ago
  There are a lot of countries where Facebook is “free”, for example India and Philippines.
  - octoberfranklin 6y ago
    I'm guessing that in these countries the cost of Internet traffic is dominated by the undersea cables at their borders. Facebook has been paying to have new undersea cables laid. This is done as part of a consortium, but those cables only have 6-12 strands in them (the repeaters are bulky) so owning even just one whole strand of fiber in an undersea cable is still an obscene amount of bandwidth for a single company that isn't in the business of reselling bandwidth.
    - apostacy 6y ago
      In The Philippines, my understanding is that they have ample bandwidth via Korea and other countries in the region. But the reason they have such expensive terrible internet is because of a lack of net neutrality and deregulation. The cellphone duopoly sells "YouTube passes", that entitle you to get unthrottled YouTube for brief periods of time.
      - Nextgrid 6y ago

deleted 6y ago

[deleted]

AmericanChopper 6y ago

Would just like to give an honorable mention to Google Translate, the most accessible http proxy of all time. It’s especially good for bypassing corporate access controls. I’ve used it many times for accessing solution threads on technical subreddits at work.

notRobot 6y ago

datadome.co is blocked for me: > datadome.co is being blocked by AdGuard DNS filter, AdGuard Tracking Protection filter, EasyPrivacy, Goodbye Ads and oisd. Dunno what they do, but it can't be good.