4 ms·

I teach software development in a data science master's degree. We learn about web scraping, it is an important skill for a future data scientist IMHO, as the w

I teach software development in a data science master's degree. We learn about web scraping, it is an important skill for a future data scientist IMHO, as the web is the largest and most important dataset in the world. Google is massively scraping the web and is building products on top of the data, e.g. flight/hotel search. Why shouldn't we be allowed to do the same? As others pointed out that one should take care about ToS.

malshe 7y ago
Thanks. Do you have any set guidelines for students when they should not scrape a website?
cookiecaper 7y ago
> Google is massively scraping the web and is building products on top of the data, e.g. flight/hotel search. Why shouldn't we be allowed to do the same? The short answer is because they have the muscle and you don't. This was litigated in Perfect 10 v. Amazon, and the only way out was for the panel deciding the case was to claim Google's use is "fair" due to its unprecedented and "transformative" nature, which basically should be read as "we don't want to face the public scorn of being the judges responsible for shutting down Google Images". Such advantage is unlikely to be a factor in less-prominent cases. Even if you believe you can convince a panel of judges that your project specifically meets the four-prong test for fair use, it takes millions of dollars to litigate a case that far, which is well outside the realm of possibility for most independent projects. Flight and hotel search is a good example. Why can't you find Southwest fares on any aggregator? People try to scrape them all the time, and as soon as they come to Southwest's attention, they get C&D'd and shut down. This is a common and well-established practice and hundreds of companies die by it every year.
- la_fayette 7y ago
  Ok. As i am not a lawyer i cannot say anything about the us. In germany there are many meta search or aggregator services for various things. All of them use web-scraping extensively. E.g. popular price comparison: https://geizhals.de/ https://geizhals.de/. The argumentation about "tranformative" nature of something making it exceptional and above the law sound not intuitive to me.
toast0 7y ago
Google's flight and hotel search doesn't come from scraping the web, it comes from ITA, the Google subsidiary that operates a Global Distributuon System. Building flight and hotel search by scraping is possible, but will make whoever you scrape very angry very quickly, because they're paying their technology vendors per search, so they watch their look to book ratios very closely.
- la_fayette 7y ago
  So and where does ITA gets the data from? German competitors like swoodoo getting the data 100% from web-scraping.
  - toast0 7y ago
    ITA is in the same business as SABRE, providing inventory management for airlines and some hotel chains. Many airline's own site will run searches with ITA or similar.

malshe 7y ago

Thanks. Do you have any set guidelines for students when they should not scrape a website?

cookiecaper 7y ago

> Google is massively scraping the web and is building products on top of the data, e.g. flight/hotel search. Why shouldn't we be allowed to do the same? The short answer is because they have the muscle and you don't. This was litigated in Perfect 10 v. Amazon, and the only way out was for the panel deciding the case was to claim Google's use is "fair" due to its unprecedented and "transformative" nature, which basically should be read as "we don't want to face the public scorn of being the judges responsible for shutting down Google Images". Such advantage is unlikely to be a factor in less-prominent cases. Even if you believe you can convince a panel of judges that your project specifically meets the four-prong test for fair use, it takes millions of dollars to litigate a case that far, which is well outside the realm of possibility for most independent projects. Flight and hotel search is a good example. Why can't you find Southwest fares on any aggregator? People try to scrape them all the time, and as soon as they come to Southwest's attention, they get C&D'd and shut down. This is a common and well-established practice and hundreds of companies die by it every year.

la_fayette 7y ago
Ok. As i am not a lawyer i cannot say anything about the us. In germany there are many meta search or aggregator services for various things. All of them use web-scraping extensively. E.g. popular price comparison: https://geizhals.de/ https://geizhals.de/. The argumentation about "tranformative" nature of something making it exceptional and above the law sound not intuitive to me.

toast0 7y ago

Google's flight and hotel search doesn't come from scraping the web, it comes from ITA, the Google subsidiary that operates a Global Distributuon System. Building flight and hotel search by scraping is possible, but will make whoever you scrape very angry very quickly, because they're paying their technology vendors per search, so they watch their look to book ratios very closely.

la_fayette 7y ago
So and where does ITA gets the data from? German competitors like swoodoo getting the data 100% from web-scraping.
- toast0 7y ago
  ITA is in the same business as SABRE, providing inventory management for airlines and some hotel chains. Many airline's own site will run searches with ITA or similar.