4 ms·

I recently joined a brilliant web scraping API company called ScrapFly, who provided me with the resources to create lots of open knowledge on web scraping that

I recently joined a brilliant web scraping API company called ScrapFly, who provided me with the resources to create lots of open knowledge on web scraping that I always wanted to create! So, to add to this list here are my top 3 favorite article that could expand OP's document: 0 - central guide to avoiding blocking - this one a tough one because there's so much information: request headers, http versions, TLS fingerprinting, javascript fingerprinting etc. I spent almost a month working on these and it was an amazing research experience that I could never afford myself before. 1, 2 - xpath and css selector introduction articles where I built a widget into our article that allows to test all css and xpath selectors right there in the learning material. 3 - introduction to reverse engineering - quick introduction to using browser devtools for web scraping, how to inspect the network and replicate it in your program. This is where I point all beginners as understanding the browser really helps to understand web scraping! 0 - https://scrapfly.io/blog/parsing-html-with-css/ https://scrapfly.io/blog/parsing-html-with-css/ 1 - https://scrapfly.io/blog/parsing-html-with-xpath/ https://scrapfly.io/blog/parsing-html-with-xpath/ 2 - https://scrapfly.io/blog/how-to-scrape-without-getting-blocked-tutorial/ https://scrapfly.io/blog/how-to-scrape-without-getting-block... 3 - https://scrapecrow.com/reverse-engineering-intro.html https://scrapecrow.com/reverse-engineering-intro.html

throwaway290 4y ago
Your first link is broken, it is not about blocking.
- wraptile 4y ago
  sorry, I reordered the links and forgot to update the footnotes - you can probably guess which one is which one by the link text itself though :)
karterk 4y ago
Do you follow robots.txt or do you allow your customers to bypass restrictions placed by the website? How do you feel about bypassing that? I know it's not illegal but it's certainly not ethical either.
- gdcbe 4y ago
  The robots.txt time makes it at times easier to scrape a target by the info that a website can reveal in it (e.g. allow a specific bot to scrape all). Their sitemaps are another gem.
- wraptile 4y ago
  Could you expand on why you think it's ethically required to follow robots.txt instructions? Primary argument _in favor_ of automation (e.g. web scraping) is that it would be unethical to hire hudnreds people to do meanial, unfufilling tasks like mindlessly clicking around the website and saving the pages when it could be done by a program which is countless times more efficient for everyone involved (the website included) and safer.
  - stingraycharles 4y ago
    There is no clear “right” or “wrong” side here, although not honoring robots.txt is a bit frowned upon, comparable to someone telling you certain areas are off-limits, but you decide to go there anyway. Having said that, if you’re in the web scraping business dealing with anti-scrape shields and whatnot, ignoring robots.txt is the least nefarious of them all.
    - wraptile 4y ago
      My personal point of view (my opinion, not my company's) is that I think it's unethical to take advantage of all the benefits public data provides which lets be honest are absolutely massive - like search engine indexing, brand building, content previews etc. while at the same time avoid paying the costs of having data publically - like someone automating their browsing experience. I mean, we had a solution to web scrapers since the inception of web authentication, but the value of having the data publically clearly outweighs the costs of having your data scraped to the point where big corporations would rather take web scrapers all the way to Ninth Circuit (Linkedin case) than shut down the public access. That being said our understanding of information philosophy is still in complete infancy so it's hard to discuss ethics here. Generally, I'm in favor of hackers, individuals and decentralization over big corporations and access to web scraping empowers the former and weakens latter - so, I'm rooting for the healthier, better version of the internet above all!
PigiVinci83 4y ago
Thank you, much appreciated your comment

throwaway290 4y ago

Your first link is broken, it is not about blocking.

wraptile 4y ago
sorry, I reordered the links and forgot to update the footnotes - you can probably guess which one is which one by the link text itself though :)

karterk 4y ago

Do you follow robots.txt or do you allow your customers to bypass restrictions placed by the website? How do you feel about bypassing that? I know it's not illegal but it's certainly not ethical either.

gdcbe 4y ago
The robots.txt time makes it at times easier to scrape a target by the info that a website can reveal in it (e.g. allow a specific bot to scrape all). Their sitemaps are another gem.
wraptile 4y ago
Could you expand on why you think it's ethically required to follow robots.txt instructions? Primary argument _in favor_ of automation (e.g. web scraping) is that it would be unethical to hire hudnreds people to do meanial, unfufilling tasks like mindlessly clicking around the website and saving the pages when it could be done by a program which is countless times more efficient for everyone involved (the website included) and safer.
- stingraycharles 4y ago
  There is no clear “right” or “wrong” side here, although not honoring robots.txt is a bit frowned upon, comparable to someone telling you certain areas are off-limits, but you decide to go there anyway. Having said that, if you’re in the web scraping business dealing with anti-scrape shields and whatnot, ignoring robots.txt is the least nefarious of them all.
  - wraptile 4y ago
    My personal point of view (my opinion, not my company's) is that I think it's unethical to take advantage of all the benefits public data provides which lets be honest are absolutely massive - like search engine indexing, brand building, content previews etc. while at the same time avoid paying the costs of having data publically - like someone automating their browsing experience. I mean, we had a solution to web scrapers since the inception of web authentication, but the value of having the data publically clearly outweighs the costs of having your data scraped to the point where big corporations would rather take web scrapers all the way to Ninth Circuit (Linkedin case) than shut down the public access. That being said our understanding of information philosophy is still in complete infancy so it's hard to discuss ethics here. Generally, I'm in favor of hackers, individuals and decentralization over big corporations and access to web scraping empowers the former and weakens latter - so, I'm rooting for the healthier, better version of the internet above all!

PigiVinci83 4y ago

Thank you, much appreciated your comment