3 ms·

Is web scraping legal?

martin-adams 13y ago
It can be, I've just launched https://myshopdata.com https://myshopdata.com where online retailers can scrape their own content for synchronising with third party marketplaces.
- darrhiggs 13y ago
  Your app looks really good. What tech did you use if you don't mind answering?
  - martin-adams 13y ago
    Thanks! My Shop Data is all PHP & MySQL, with Slim, Twig & Bootstrap. The web scraping aspect is another product of mine (forgive the clunky homepage, I'm going to turn this into an API platform) - https://grabnotify.com https://grabnotify.com GrabNotify is Node.js, Mongo, PHP, Bootstrap and PhantomJS. The undocumented API allows you to create a web crawler but define a JavaScript algorithm to extract the data off the page. Some retailers have dropdowns which update stock, images, etc, so this crawler can simulate mouse events, etc. My Shop Data will supply a custom crawler algorithm for each e-commerce web site through the API. And finally, I've written a HTML to Markdown translator to extract page descriptions but keep some formatting while being transferable to other systems that don't support HTML. The whole legality issue of web scraping is an interesting one. I'm planning to position GrabNotify as a web crawler, page monitor and HTML -> data tool, but only if you own or have permission to scrape the original content but need a simple way to grab and monitor the HTML into data. I'm not really interested in building a business of scraping other people's content without their permission.
adamnemecek 13y ago
If anything, it might be against TOS but ASAIK, breaking TOS is not illegal.
- lsiebert 13y ago
  Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.
  - adamnemecek 13y ago
    Are you thinking of weev? http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach
- lsiebert 13y ago
  Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.
- tlrobinson 13y ago
  Is there's no signup/login process is the TOS even enforceable?

It can be, I've just launched https://myshopdata.com https://myshopdata.com where online retailers can scrape their own content for synchronising with third party marketplaces.

darrhiggs 13y ago
Your app looks really good. What tech did you use if you don't mind answering?
- martin-adams 13y ago
  Thanks! My Shop Data is all PHP & MySQL, with Slim, Twig & Bootstrap. The web scraping aspect is another product of mine (forgive the clunky homepage, I'm going to turn this into an API platform) - https://grabnotify.com https://grabnotify.com GrabNotify is Node.js, Mongo, PHP, Bootstrap and PhantomJS. The undocumented API allows you to create a web crawler but define a JavaScript algorithm to extract the data off the page. Some retailers have dropdowns which update stock, images, etc, so this crawler can simulate mouse events, etc. My Shop Data will supply a custom crawler algorithm for each e-commerce web site through the API. And finally, I've written a HTML to Markdown translator to extract page descriptions but keep some formatting while being transferable to other systems that don't support HTML. The whole legality issue of web scraping is an interesting one. I'm planning to position GrabNotify as a web crawler, page monitor and HTML -> data tool, but only if you own or have permission to scrape the original content but need a simple way to grab and monitor the HTML into data. I'm not really interested in building a business of scraping other people's content without their permission.

adamnemecek 13y ago

If anything, it might be against TOS but ASAIK, breaking TOS is not illegal.

lsiebert 13y ago
Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.
- adamnemecek 13y ago
  Are you thinking of weev? http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach
lsiebert 13y ago
Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.
tlrobinson 13y ago
Is there's no signup/login process is the TOS even enforceable?