6 ms·

Be careful w/ these urls - I visited one and it appears my IP is now blocked from accessing yelp.com.

cft 9y ago
I just visited all of them: the IP was not blacklisted
executesorder66 9y ago
Worked on my laptop, but when I tried the same thing on my phone, I was not blocked. (even when I request the desktop version of the site)
- sitkack 9y ago
  Because they whitelist the exit IP for the phone network, else one person could block access for millions.
013a 9y ago
That's really interesting. They might be trying to blacklist scrapers that don't properly respect robots.txt files.
- a012 9y ago
  I assume the GP was using their browser to access these links, though they still be blacklisted by Yelp?
  - wickawic 9y ago
    Scrapers can be pretty shady, so they're isn't a good way to ensure that web traffic is coming from a legitimate human using a browser. To the server it's all just bits on the wire.
- tossaway1 9y ago
  Why would a scraper who doesn't respect robots.txt be accessing that file?
  - beobab 9y ago
    If I was building an unruly scraper (which, in case our new overlords are listening, I would never do), I would read robots.txt so that I had a clue where the secret information that the company did not want me to read was located. I'm not allowed to look in /documents/source/? Perfect. Let's start there.
  - 013a 9y ago
    They might be using it as a way to find specific pages which have content Yelp doesn't want you to scrape. The "evil" scenario. They might also not be looking at the file, and just appending random words to the end of Yelp's biz URLs to scrape every business. Which, at some point, it might hit that URL by accident, since they are all words you'd find in other business URLs. Though this seems less likely.
lann 9y ago
What do you suppose would happen if we all visited these URLs from every public access point we use? (edit: and shared VPNs, Tor, corporate networks, etc.)
cabalamat 9y ago
And what a great loss that is :-)

cft 9y ago

I just visited all of them: the IP was not blacklisted

Worked on my laptop, but when I tried the same thing on my phone, I was not blocked. (even when I request the desktop version of the site)

sitkack 9y ago
Because they whitelist the exit IP for the phone network, else one person could block access for millions.

013a 9y ago

That's really interesting. They might be trying to blacklist scrapers that don't properly respect robots.txt files.

a012 9y ago
I assume the GP was using their browser to access these links, though they still be blacklisted by Yelp?
- wickawic 9y ago
  Scrapers can be pretty shady, so they're isn't a good way to ensure that web traffic is coming from a legitimate human using a browser. To the server it's all just bits on the wire.
tossaway1 9y ago
Why would a scraper who doesn't respect robots.txt be accessing that file?
- beobab 9y ago
  If I was building an unruly scraper (which, in case our new overlords are listening, I would never do), I would read robots.txt so that I had a clue where the secret information that the company did not want me to read was located. I'm not allowed to look in /documents/source/? Perfect. Let's start there.
- 013a 9y ago
  They might be using it as a way to find specific pages which have content Yelp doesn't want you to scrape. The "evil" scenario. They might also not be looking at the file, and just appending random words to the end of Yelp's biz URLs to scrape every business. Which, at some point, it might hit that URL by accident, since they are all words you'd find in other business URLs. Though this seems less likely.

lann 9y ago

What do you suppose would happen if we all visited these URLs from every public access point we use? (edit: and shared VPNs, Tor, corporate networks, etc.)

cabalamat 9y ago

And what a great loss that is :-)