3 ms·

How did he get the data? It's pretty hard to pull the reviews and the data from yelp. I tried to do that to do some querying, but their search isn't so great an

by monksy 9y ago

How did he get the data? It's pretty hard to pull the reviews and the data from yelp. I tried to do that to do some querying, but their search isn't so great and they pull a lot of stunts to prevent you from scraping. --- Oh, I see he's using the kaggle data. That's not guaranteed to be reliable.

Tostino 9y ago
Eh it's possible, the reviews are harder though. I wrote a scraper which pulled address info / phone number / star rating / review count for pretty much every restaurant in the US. It was "easy" because all of that data is available within the search page, and you just need to correctly parse it out. The hardest part was getting around their really crazy rate limiting and IP blocking. I managed to get myself IP banned from yelp prior to ever trying to scrape by just doing a bunch of searches manually pretty quickly over like 20 min, next thing I knew I could no longer access anything on Yelp.
- monksy 9y ago
  That's not suprising. You can get yourself IP blocked just by opening things in other tabs to queue them up to read. (If you notice yourself getting random 404s .. that's when you're being watched)
foreigner 9y ago
Yelp publishes reviews in JSON-LD format so the parsing is trivial.
- monksy 9y ago
  They don't publish all of their reviews. They're limited to 3 reviews.
  - foreigner 9y ago
    Their API only exposes three, but in their HTML they embed JSON-LD metadata for all of them (but sadly not the "not recommended" reviews).
aplorbust 9y ago
FWIW, I just tried gathering some current1 user-generated Yelp Data from Internet Archive and it was very easy to gather a list of all restaurants for a city2 and then all reviews for each restaurant. 1 https://web.archive.org/save/[url] https://web.archive.org/save/[url] and most recent crawl: http://web.archive.org/web/20180109124942/https://yelp.com/ http://web.archive.org/web/20180109124942/https://yelp.com/ 2 By incremental searches that each return under 1000 results.
- malifier 9y ago
  That’s a great tip, thanks!
Zephyr314 9y ago
This [1] also contains a superset of the data used in this post and is direct from Yelp. [1]: https://www.yelp.com/dataset/challenge https://www.yelp.com/dataset/challenge
- malifier 9y ago
  This dataset contains sparse data from businesses in different cities around the world. It has a very small overlap with the dataset used in the study. Focusing on a particular city helps to understand the underlying trends better.
malifier 9y ago
The Kaggle data was matched with recent information about the restaurants which was pulled from the Yelp API.

by monksy 9y ago

Tostino 9y ago

Eh it's possible, the reviews are harder though. I wrote a scraper which pulled address info / phone number / star rating / review count for pretty much every restaurant in the US. It was "easy" because all of that data is available within the search page, and you just need to correctly parse it out. The hardest part was getting around their really crazy rate limiting and IP blocking. I managed to get myself IP banned from yelp prior to ever trying to scrape by just doing a bunch of searches manually pretty quickly over like 20 min, next thing I knew I could no longer access anything on Yelp.

monksy 9y ago
That's not suprising. You can get yourself IP blocked just by opening things in other tabs to queue them up to read. (If you notice yourself getting random 404s .. that's when you're being watched)

foreigner 9y ago

Yelp publishes reviews in JSON-LD format so the parsing is trivial.

monksy 9y ago
They don't publish all of their reviews. They're limited to 3 reviews.
- foreigner 9y ago
  Their API only exposes three, but in their HTML they embed JSON-LD metadata for all of them (but sadly not the "not recommended" reviews).

aplorbust 9y ago

FWIW, I just tried gathering some current1 user-generated Yelp Data from Internet Archive and it was very easy to gather a list of all restaurants for a city2 and then all reviews for each restaurant. 1 https://web.archive.org/save/[url] https://web.archive.org/save/[url] and most recent crawl: http://web.archive.org/web/20180109124942/https://yelp.com/ http://web.archive.org/web/20180109124942/https://yelp.com/ 2 By incremental searches that each return under 1000 results.

malifier 9y ago
That’s a great tip, thanks!

Zephyr314 9y ago

This [1] also contains a superset of the data used in this post and is direct from Yelp. [1]: https://www.yelp.com/dataset/challenge https://www.yelp.com/dataset/challenge

malifier 9y ago
This dataset contains sparse data from businesses in different cities around the world. It has a very small overlap with the dataset used in the study. Focusing on a particular city helps to understand the underlying trends better.

malifier 9y ago

The Kaggle data was matched with recent information about the restaurants which was pulled from the Yelp API.