9 ms·

Web scraping with Python open knowledge

219 points by PigiVinci83 4y ago

deleted 4y ago
[deleted]
PigiVinci83 4y ago
A work in progress guide about web scraping in python, anti bot softwares and techniques and so on. Please feel free to share and contribute with your own experience too.
- alexchamberlain 4y ago
  The tab formatting seems like an odd (and rather unPythonic) addition. What's the intention there?
  - hrbf 4y ago
    Same here. It feels out of place, unnecessary and its rationalization unconvincing. Considering Python, an outright weird suggestion.
    - wheelerof4te 4y ago
      Python allows indenting using tabs, so I don't understand why it's a weird decision. In fact, they even stated their reasoning in the document. I don't see why anyone has to blindly follow PEP8 nor do I get why 4 spaces indent has to be considered a standard.
      - mekoka 4y ago
        > Python allows indenting using tabs, so I don't understand why it's a weird decision. A standard is not a set of rules already enforced in a language, otherwise it would not be needed. It's rather a set of practical guidelines that a group of people agrees upon, with the purpose of making each other's lives easier. That's why indenting with tabs is weird. > I don't see why anyone has to blindly follow PEP8 In the <tabs> vs <space> debate there's really no reasons not to follow PEP8. The number of people (and editors, and tooling, etc) that abide by it is quite large and it seems to work well enough for most. The only reason that someone would even mention spaces as an inconvenience, is that they can somehow perceive a difference when editing code, which may point to badly configured or outdated tooling, rather than faulty standard. Most people who code Python have their editor set to feed 4 spaces when the <tab> key is pressed and delete 4 consecutive spaces on <Backspace>. If instead you repeatedly press (4 times) the <space> bar or the <Backspace> to insert or remove code indentations, you're doing it wrong.
        GekkePrutser 4y ago
        I think tabs are handier exactly because they don't have a fixed width. You can adjust code easily to your readability preference without actually having to change the code. Some people like deep indents, others like them more shallow. Just modify the tab width to your personal preference. Also, it avoids the issue of only accepting 2 or 4 spaces, leaving the 1,3,5 spaces as incorrect combinations leading to issues. With tabs there are no combinations which are invalid, though you can still have too many or few indents than intended of course. But the hunting for that extra space you copied in is gone. I just don't subscribe to the 'tabs are evil' narrative. I like that python supports them but I think it's really annoying that YAML doesn't. The argument of "each editor does things differently" is also not really valid when you're going to need a special editor that can convert tabs to spaces and delete spaces in bunches to really work with it comfortably. It would have been much easier to just use an editor that handles tabs in the way that was needed. Either way you're going to want specific editor features.
  - datalopers 4y ago
    - PigiVinci83 4y ago
      Because it's not a tutorial on web scraping but a mix of what we suggest internally to do and what we've learnt from our experience in this field in these years. For our codebase we prefer tabs instead of spaces, but i understand it's a subject for debates that last decades :) But thanks for the point, I'll rephrase the topic in the guide
      - datalopers 4y ago
        It's odd to me that your apparent revenue stream is from scraping difficult-to-scrape sites and you're broadcasting the exact tactics you use to bypass anti-scraping systems. You're making your own life difficult by giving Cloudflare/PerimeterX/etc the information necessary to improve their tooling. You also seem to advertise many of the sites/datasets you're scraping, which opens you up to litigation. Especially if they're employing anti-scraping tooling and you're brazenly bypassing those. It doesn't matter that it's legal in most jurisdictions of the world, you'll still have to handle cease and desists or potential lawsuits, which is a major cost and distraction.
        punnerud 4y ago
        « You also seem to advertise many of the sites/datasets you're scraping, which opens you up to litigation.» Is that a done deal now after the “LinkedIn vs HiQ” case public information only hold copyright, but you can use the by product as it’s fit you for new business?
        datalopers 4y ago
        The only clear outcome from the LinkedIn case, afaik, is that scraping publicly of accessible data is not a federal crime under CFAA [1]. There are still plenty of other civil ways that someone can sue you to stop scraping their site: breach of contract, trespass to chattels, trademark infringement, etc. And they can do so over and over again til you're broke. OP is based in Italy anyway so I have absolutely no clue what does and doesn't apply. I'd like to point out that, while HiQ Labs "won" the case, that company is basically dead. The CEO and CTO are both working for other companies now. So I think the bigger takeaway is: don't get yourself sued while you're a tiny little startup. [1] https://www.natlawreview.com/article/hiq-labs-v-linkedin https://www.natlawreview.com/article/hiq-labs-v-linkedin
- etskinner 4y ago
  On the page about canvas fingerprinting[0], it only mentions Cloudflare. From what I can tell, reCaptcha v3 also uses canvas fingerprinting [1] [0] https://github.com/reanalytics-databoutique/webscraping-open-project/blob/main/Pages/Antibot/Canvasfingerprint.md https://github.com/reanalytics-databoutique/webscraping-open... [1] https://brianwjoe.com/2019/02/06/how-does-recaptcha-v3-work/ https://brianwjoe.com/2019/02/06/how-does-recaptcha-v3-work/
  - PigiVinci83 4y ago
    Thanks for sharing, i'll update soon the page.
- jamestimmins 4y ago
  I appreciate the inclusion of anti-bot software. As someone who builds plugins for enterprise apps (currently Airtable), I really want to build automated tests for my apps with Selenium, but keep getting foiled by anti-bot measures. Can anyone recommend other resources for understanding anti-bot tech and their workarounds?
  - judge2020 4y ago
    Anyone with a stake in bypassing anti-bot measures isn't going to share their tactics, since sharing it will lead to such workaround being patched or mitigated, requiring them to research for more bot detection workarounds. Projects like cloudscraper[0] are often linked to point and say "look! they broke Cloudflare!" but CF and the rest of the industry has detections for tools like this, and instead of rolling out blocks for these tools, they give website owners tools like bot score[1] to manage their own risk level on a per-page basis. 0: https://github.com/VeNoMouS/cloudscraper https://github.com/VeNoMouS/cloudscraper 1: https://developers.cloudflare.com/bots/concepts/bot-score/ https://developers.cloudflare.com/bots/concepts/bot-score/
    - jamestimmins 4y ago
      Yeah that's a shame but it makes sense. Probably need to find an in with bot builders, if that's really a goal I have.
captn3m0 4y ago
Good list, confused about the “tabs weighing less” bit. Isn’t that a preference left for the end-devs? Another tip I’ve found is to check if the data is accessible on a mobile app and proxy it to see if there is a JSON API available.
- PigiVinci83 4y ago
  Thanks for your reply, mobile data it's a thing i need to add soon. Usually we check using Fiddler if there's an API inside, but only for really problematic website.
deleted 4y ago
[deleted]
Xeoncross 4y ago
Plug for https://commoncrawl.org/ https://commoncrawl.org/ if you need billions of pages but don't want to deal with scraping the web yourself.
- deleted 4y ago
  [deleted]
- squiggy22 4y ago
  Is there subsets of common crawl anywhere for individual sites. E.g. YouTube for example?
  - magundu 4y ago
    You can query subset of specific sites from common crawl itself.
- mdaniel 4y ago
  It would thrill me if common crawl were updated with such frequency that it would allow new search engines to enter the market I haven't dug into it enough to know if there's some technical reason it's not currently the case, or just lack of (interest|willpower)
  - input_sh 4y ago
    I'd argue that one broad crawl every 2-3 months in addition to their updated-daily news crawl[0] should be good enough to make a rudimentary search engine. [0] https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html
    - mdaniel 4y ago
      You're right about the "rudimentary" part, because I don't know how they do it but the major players have some not-kidding-around freshness: https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%22common%20crawl%22%20site%3Anews.ycombinator.com&tbs=qdr%3Ad https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%2... https://www.bing.com/search?q=%22thrill+me%22+%22common+crawl%22+site%3Anews.ycombinator.com https://www.bing.com/search?q=%22thrill+me%22+%22common+craw... (and DDG similarly, because bing) ed: I was curious if maybe HN publishes a sitemap, and it seems no. Then again, hnreplies knows about the HN API so maybe it's special-cased by the big crawlers https://github.com/ggerganov/hnreplies#hnreplies https://github.com/ggerganov/hnreplies#hnreplies
jonatron 4y ago
As the second sentence says, it's a cat and mouse game, so there's no incentive on either side of bot vs anti-bot to share information.
- PigiVinci83 4y ago
  I'm sure no one will add here its secret sauce :)
  - gdcbe 4y ago
    Indeed. Personally I would want to give away more, but for now mostly getting the red light here, for obvious reasons. What we could however do better as a community is share generic tooling.
account-5 4y ago
I was reading another thread about webscraping, someone mentioned CSS selectors being way quicker than xpath. I'm easy either way but apart from a more powerful syntax what other benefits are there?
- byteface 4y ago
  I'm not sure about quicker. Doesn't scrapy use elementpath?. which converts a css query to an xpath under the hood as there is no complete CSSOM available for python. Likely as there is no modern standards based python dom to operate on so doing it on lxml tree is probably the best option. I find the main difference is xpath can return an attribute value where as css returns the node. You can use either from the terminal in my lib... https://github.com/byteface/domonic https://github.com/byteface/domonic (as it uses elementpath like scrapy)
  - byteface 4y ago
    sorry i meant cssselect... https://pypi.org/project/cssselect/ https://pypi.org/project/cssselect/ which converts to xpath.
- mdaniel 4y ago
  In my experience, it's not that CSS selectors are "more powerful," but rather "more legible." XPath is for sure more powerful, but also usually lower signal to noise ratio response.css("#the-id") # vs response.xpath("//*[@id='the-id']") Thankfully, Scrapy (well, pedantically "parsel") allows mixing and matching, using the one which makes the most sense response.css(".someClass").xpath(".//*[starts-with(text(), 'Price')]")
- showerst 4y ago
  CSS is nice because it's more readable than XPATH for longer queries, and is friendlier to newer programmers who didn't come up when XML was big. XPATH is generally more powerful for really gnarly things and for backtracking. "Show me the 3rd paragraph that's a sibling of the fourth div id="subhed" and contains the text "starting".
afandian 4y ago
If you're the kind of person who wants "open data" (read as broadly as you like) and could get it in snapshots direct from the source without having to scrape, what would your ideal format be? I know it's a very open ended question.
- mdaniel 4y ago
  Alternatively, think how much traffic a site could save itself if it allowed subscribing to webhooks on new content, or having a functional sitemap.xml to say nothing of the end-user experience of having fresh content I would speculate (based on anecdata) that a lot of the actual load placed upon sites is from the discovery phase -- what pages are there, and have any of them changed -- not so much "hit this one endpoint and unpack its data"
  - tasn 4y ago
    It's an interesting idea to have a sort of a "what's up?" endpoint that is just basic text and tells you what's fresh. At least for static websites, it's quite easy to do.
- dwrodri 4y ago
  The way that PushShift offers huge amounts of reddit data in compressed JSON is very convenient. Checkout files.pushshift.io
- jshen 4y ago
  Probably RDF serialized as hextuples https://github.com/ontola/hextuples https://github.com/ontola/hextuples
  - afandian 4y ago
    Looks interesting. From that page I couldn't see what 'graph' field relates to. Is it the identifier for a distinct named graph? It was blank in the examples. Do you use it? What for?
    - 4y ago
sgtquack 4y ago
As someone who recently dealt with scraping sites behind cloudflare...I never want to scrape again
- gdcbe 4y ago
  Honestly cloudflare isn't really a big deal in terms of scraping.
loudthing 4y ago
It can be frustrating learning web scraping with Python when so many sites actively block scraping.
- 1vuio0pswjnm7 4y ago
  Any chance you could name a few of those sites. LinkedIn is the only example I know of.
  - loudthing 4y ago
    The main perpetrator I've come across (in, I'll admit, my limited experience with scraping) was Google services. It seemed somewhat obvious to me initially while playing around with Beautiful Soup, the most valuable resource I could use for scraping was Google search, followed by Gmail and Google Finance. I had no luck using BS with any of these services. Largely this taught me to be creative with my data sources. For example, I built a virtual weather vane powered by a Raspberry Pi that would scrape my local airport's website to get wind direction data, then turn the vane via a servo to the correct direction. So my takeaway from this project was scraping isn't as straight forward as one would thing, there's more of an art to it in order to figuring out where to get the information you want.
    - 1vuio0pswjnm7 4y ago
      Agreed. Most public websites are not trying to force visitors to use a certain web browser by selectively denying access. No public website should be the sole, exclusive source of its data because if the data is public then by definition the data can be copied by any other website. As such, chances are the data can be found in multiple locations and at least one will not be trying to force the use of a certian web browser. Am I correct that the examples listed here are (a) www.google.com, (b) mail.google.com and (c) www.google.com/finance/. I have no trouble extracting data from these examples.[FN1] I do not use a graphical web browser to make HTTP requests nor do I use Python or BeatifulSoup. A cookie is required for mail.google.com, in lieu of a password, but the cookie can be saved and will work for years. 1. Of course, Google Web Search is crippled. Using a basic HTTP client, e.g., no cookies, Javascript, FLoC, etc., one cannot retrieve more than 250-300 results total. Searching "too fast" will draw a temporary IP block. This "search engine" is designed for advertising not discovery. Advertisers compete for space at the top of the first page of results. Popular websites are prioritised, potentially making them even more popular. Websites that "rank"[FN2] too low in a search are not discoverable as they have no value for advertising. An index of public websites and public data is treated as properietary and secret. Google actively tries to prevent anyone from copying even a small portion of it. 2. Google makes it impossible to sort results by URL, date, or even number of keyword/string hits in the page. Results are ordered according to secret algorithm, designed for advertising.
holografix 4y ago
For websites that require auth via Google Auth this is a non starter. There’s no way to bypass its bot detection
- gdcbe 4y ago
  Perhaps you should anyhow not burn your hands on such targets even if technically possible, you start to get in murky water quickly. Requiring Google Auth meaning that you need to login, makes it debatable whether or not that data is still public data.
lapser 4y ago
Is there an FLOSS project that combines Scrapy scrapers and just makes the results publicly available?
- mdaniel 4y ago
  This is the one I know about: https://morph.io/ https://morph.io/ and https://github.com/openaustralia/morph#readme https://github.com/openaustralia/morph#readme (AGPLv3) -- they used to be at the intersection of "heroku for scrapers" and DoltHub (e.g. https://www.dolthub.com/repositories/dolthub/us-businesses/data/master/businesses https://www.dolthub.com/repositories/dolthub/us-businesses/d...) since the scrapers would run but then make their data available as CSV or sqlite or whatever. But, when I just tried to load one of the morph.io scrapers, the page just said "creating new template" so I'm guessing they've gone the way of the ScraperWiki.com that preceded them: turns out, hosted compute for free isn't free
wraptile 4y ago
I recently joined a brilliant web scraping API company called ScrapFly, who provided me with the resources to create lots of open knowledge on web scraping that I always wanted to create! So, to add to this list here are my top 3 favorite article that could expand OP's document: 0 - central guide to avoiding blocking - this one a tough one because there's so much information: request headers, http versions, TLS fingerprinting, javascript fingerprinting etc. I spent almost a month working on these and it was an amazing research experience that I could never afford myself before. 1, 2 - xpath and css selector introduction articles where I built a widget into our article that allows to test all css and xpath selectors right there in the learning material. 3 - introduction to reverse engineering - quick introduction to using browser devtools for web scraping, how to inspect the network and replicate it in your program. This is where I point all beginners as understanding the browser really helps to understand web scraping! 0 - https://scrapfly.io/blog/parsing-html-with-css/ https://scrapfly.io/blog/parsing-html-with-css/ 1 - https://scrapfly.io/blog/parsing-html-with-xpath/ https://scrapfly.io/blog/parsing-html-with-xpath/ 2 - https://scrapfly.io/blog/how-to-scrape-without-getting-blocked-tutorial/ https://scrapfly.io/blog/how-to-scrape-without-getting-block... 3 - https://scrapecrow.com/reverse-engineering-intro.html https://scrapecrow.com/reverse-engineering-intro.html
- throwaway290 4y ago
  Your first link is broken, it is not about blocking.
  - wraptile 4y ago
    sorry, I reordered the links and forgot to update the footnotes - you can probably guess which one is which one by the link text itself though :)
- karterk 4y ago
  Do you follow robots.txt or do you allow your customers to bypass restrictions placed by the website? How do you feel about bypassing that? I know it's not illegal but it's certainly not ethical either.
tatoalo 4y ago
I recently transitioned one of my scraping projects away from selenium to playwright and I must say that the developer experience is way better, in my opinion. I also implemented to receive a telegram message with the debug trace in case of errors in my pipeline, so that I could have the entire scraping flow to analyze. That’s pretty neat.
rustdeveloper 4y ago
I think that if you don’t want to invest a lot of time into learning web scraping and money to get a pool of residential, or even better mobile, proxies it’s easy to quickly get good results with web scraping API like https://scrapingfish.com https://scrapingfish.com They have good blogposts, for example, for how to scrape public Instagram profiles: https://scrapingfish.com/blog/scraping-instagram https://scrapingfish.com/blog/scraping-instagram

219 points by PigiVinci83 4y ago

deleted 4y ago

[deleted]

A work in progress guide about web scraping in python, anti bot softwares and techniques and so on. Please feel free to share and contribute with your own experience too.

alexchamberlain 4y ago
The tab formatting seems like an odd (and rather unPythonic) addition. What's the intention there?
- hrbf 4y ago
  Same here. It feels out of place, unnecessary and its rationalization unconvincing. Considering Python, an outright weird suggestion.
  - wheelerof4te 4y ago
    Python allows indenting using tabs, so I don't understand why it's a weird decision. In fact, they even stated their reasoning in the document. I don't see why anyone has to blindly follow PEP8 nor do I get why 4 spaces indent has to be considered a standard.
    - mekoka 4y ago
      > Python allows indenting using tabs, so I don't understand why it's a weird decision. A standard is not a set of rules already enforced in a language, otherwise it would not be needed. It's rather a set of practical guidelines that a group of people agrees upon, with the purpose of making each other's lives easier. That's why indenting with tabs is weird. > I don't see why anyone has to blindly follow PEP8 In the <tabs> vs <space> debate there's really no reasons not to follow PEP8. The number of people (and editors, and tooling, etc) that abide by it is quite large and it seems to work well enough for most. The only reason that someone would even mention spaces as an inconvenience, is that they can somehow perceive a difference when editing code, which may point to badly configured or outdated tooling, rather than faulty standard. Most people who code Python have their editor set to feed 4 spaces when the <tab> key is pressed and delete 4 consecutive spaces on <Backspace>. If instead you repeatedly press (4 times) the <space> bar or the <Backspace> to insert or remove code indentations, you're doing it wrong.
      - GekkePrutser 4y ago
        I think tabs are handier exactly because they don't have a fixed width. You can adjust code easily to your readability preference without actually having to change the code. Some people like deep indents, others like them more shallow. Just modify the tab width to your personal preference. Also, it avoids the issue of only accepting 2 or 4 spaces, leaving the 1,3,5 spaces as incorrect combinations leading to issues. With tabs there are no combinations which are invalid, though you can still have too many or few indents than intended of course. But the hunting for that extra space you copied in is gone. I just don't subscribe to the 'tabs are evil' narrative. I like that python supports them but I think it's really annoying that YAML doesn't. The argument of "each editor does things differently" is also not really valid when you're going to need a special editor that can convert tabs to spaces and delete spaces in bunches to really work with it comfortably. It would have been much easier to just use an editor that handles tabs in the way that was needed. Either way you're going to want specific editor features.
- datalopers 4y ago
  - PigiVinci83 4y ago
    Because it's not a tutorial on web scraping but a mix of what we suggest internally to do and what we've learnt from our experience in this field in these years. For our codebase we prefer tabs instead of spaces, but i understand it's a subject for debates that last decades :) But thanks for the point, I'll rephrase the topic in the guide
    - datalopers 4y ago
      It's odd to me that your apparent revenue stream is from scraping difficult-to-scrape sites and you're broadcasting the exact tactics you use to bypass anti-scraping systems. You're making your own life difficult by giving Cloudflare/PerimeterX/etc the information necessary to improve their tooling. You also seem to advertise many of the sites/datasets you're scraping, which opens you up to litigation. Especially if they're employing anti-scraping tooling and you're brazenly bypassing those. It doesn't matter that it's legal in most jurisdictions of the world, you'll still have to handle cease and desists or potential lawsuits, which is a major cost and distraction.
      - punnerud 4y ago
        « You also seem to advertise many of the sites/datasets you're scraping, which opens you up to litigation.» Is that a done deal now after the “LinkedIn vs HiQ” case public information only hold copyright, but you can use the by product as it’s fit you for new business?
        datalopers 4y ago
        The only clear outcome from the LinkedIn case, afaik, is that scraping publicly of accessible data is not a federal crime under CFAA [1]. There are still plenty of other civil ways that someone can sue you to stop scraping their site: breach of contract, trespass to chattels, trademark infringement, etc. And they can do so over and over again til you're broke. OP is based in Italy anyway so I have absolutely no clue what does and doesn't apply. I'd like to point out that, while HiQ Labs "won" the case, that company is basically dead. The CEO and CTO are both working for other companies now. So I think the bigger takeaway is: don't get yourself sued while you're a tiny little startup. [1] https://www.natlawreview.com/article/hiq-labs-v-linkedin https://www.natlawreview.com/article/hiq-labs-v-linkedin
etskinner 4y ago
On the page about canvas fingerprinting[0], it only mentions Cloudflare. From what I can tell, reCaptcha v3 also uses canvas fingerprinting [1] [0] https://github.com/reanalytics-databoutique/webscraping-open-project/blob/main/Pages/Antibot/Canvasfingerprint.md https://github.com/reanalytics-databoutique/webscraping-open... [1] https://brianwjoe.com/2019/02/06/how-does-recaptcha-v3-work/ https://brianwjoe.com/2019/02/06/how-does-recaptcha-v3-work/
- PigiVinci83 4y ago
  Thanks for sharing, i'll update soon the page.
jamestimmins 4y ago
I appreciate the inclusion of anti-bot software. As someone who builds plugins for enterprise apps (currently Airtable), I really want to build automated tests for my apps with Selenium, but keep getting foiled by anti-bot measures. Can anyone recommend other resources for understanding anti-bot tech and their workarounds?
- judge2020 4y ago
  Anyone with a stake in bypassing anti-bot measures isn't going to share their tactics, since sharing it will lead to such workaround being patched or mitigated, requiring them to research for more bot detection workarounds. Projects like cloudscraper[0] are often linked to point and say "look! they broke Cloudflare!" but CF and the rest of the industry has detections for tools like this, and instead of rolling out blocks for these tools, they give website owners tools like bot score[1] to manage their own risk level on a per-page basis. 0: https://github.com/VeNoMouS/cloudscraper https://github.com/VeNoMouS/cloudscraper 1: https://developers.cloudflare.com/bots/concepts/bot-score/ https://developers.cloudflare.com/bots/concepts/bot-score/
  - jamestimmins 4y ago
    Yeah that's a shame but it makes sense. Probably need to find an in with bot builders, if that's really a goal I have.

captn3m0 4y ago

Good list, confused about the “tabs weighing less” bit. Isn’t that a preference left for the end-devs? Another tip I’ve found is to check if the data is accessible on a mobile app and proxy it to see if there is a JSON API available.

PigiVinci83 4y ago
Thanks for your reply, mobile data it's a thing i need to add soon. Usually we check using Fiddler if there's an API inside, but only for really problematic website.

deleted 4y ago

[deleted]

Xeoncross 4y ago

Plug for https://commoncrawl.org/ https://commoncrawl.org/ if you need billions of pages but don't want to deal with scraping the web yourself.

deleted 4y ago
[deleted]
squiggy22 4y ago
Is there subsets of common crawl anywhere for individual sites. E.g. YouTube for example?
- magundu 4y ago
  You can query subset of specific sites from common crawl itself.
mdaniel 4y ago
It would thrill me if common crawl were updated with such frequency that it would allow new search engines to enter the market I haven't dug into it enough to know if there's some technical reason it's not currently the case, or just lack of (interest|willpower)
- input_sh 4y ago
  I'd argue that one broad crawl every 2-3 months in addition to their updated-daily news crawl[0] should be good enough to make a rudimentary search engine. [0] https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html
  - mdaniel 4y ago
    You're right about the "rudimentary" part, because I don't know how they do it but the major players have some not-kidding-around freshness: https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%22common%20crawl%22%20site%3Anews.ycombinator.com&tbs=qdr%3Ad https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%2... https://www.bing.com/search?q=%22thrill+me%22+%22common+crawl%22+site%3Anews.ycombinator.com https://www.bing.com/search?q=%22thrill+me%22+%22common+craw... (and DDG similarly, because bing) ed: I was curious if maybe HN publishes a sitemap, and it seems no. Then again, hnreplies knows about the HN API so maybe it's special-cased by the big crawlers https://github.com/ggerganov/hnreplies#hnreplies https://github.com/ggerganov/hnreplies#hnreplies

jonatron 4y ago

As the second sentence says, it's a cat and mouse game, so there's no incentive on either side of bot vs anti-bot to share information.

PigiVinci83 4y ago
I'm sure no one will add here its secret sauce :)
- gdcbe 4y ago
  Indeed. Personally I would want to give away more, but for now mostly getting the red light here, for obvious reasons. What we could however do better as a community is share generic tooling.

account-5 4y ago

I was reading another thread about webscraping, someone mentioned CSS selectors being way quicker than xpath. I'm easy either way but apart from a more powerful syntax what other benefits are there?

byteface 4y ago
I'm not sure about quicker. Doesn't scrapy use elementpath?. which converts a css query to an xpath under the hood as there is no complete CSSOM available for python. Likely as there is no modern standards based python dom to operate on so doing it on lxml tree is probably the best option. I find the main difference is xpath can return an attribute value where as css returns the node. You can use either from the terminal in my lib... https://github.com/byteface/domonic https://github.com/byteface/domonic (as it uses elementpath like scrapy)
- byteface 4y ago
  sorry i meant cssselect... https://pypi.org/project/cssselect/ https://pypi.org/project/cssselect/ which converts to xpath.
mdaniel 4y ago
In my experience, it's not that CSS selectors are "more powerful," but rather "more legible." XPath is for sure more powerful, but also usually lower signal to noise ratio response.css("#the-id") # vs response.xpath("//*[@id='the-id']") Thankfully, Scrapy (well, pedantically "parsel") allows mixing and matching, using the one which makes the most sense response.css(".someClass").xpath(".//*[starts-with(text(), 'Price')]")
showerst 4y ago
CSS is nice because it's more readable than XPATH for longer queries, and is friendlier to newer programmers who didn't come up when XML was big. XPATH is generally more powerful for really gnarly things and for backtracking. "Show me the 3rd paragraph that's a sibling of the fourth div id="subhed" and contains the text "starting".

afandian 4y ago

If you're the kind of person who wants "open data" (read as broadly as you like) and could get it in snapshots direct from the source without having to scrape, what would your ideal format be? I know it's a very open ended question.

mdaniel 4y ago
Alternatively, think how much traffic a site could save itself if it allowed subscribing to webhooks on new content, or having a functional sitemap.xml to say nothing of the end-user experience of having fresh content I would speculate (based on anecdata) that a lot of the actual load placed upon sites is from the discovery phase -- what pages are there, and have any of them changed -- not so much "hit this one endpoint and unpack its data"
- tasn 4y ago
  It's an interesting idea to have a sort of a "what's up?" endpoint that is just basic text and tells you what's fresh. At least for static websites, it's quite easy to do.
dwrodri 4y ago
The way that PushShift offers huge amounts of reddit data in compressed JSON is very convenient. Checkout files.pushshift.io
jshen 4y ago
Probably RDF serialized as hextuples https://github.com/ontola/hextuples https://github.com/ontola/hextuples
- afandian 4y ago
  Looks interesting. From that page I couldn't see what 'graph' field relates to. Is it the identifier for a distinct named graph? It was blank in the examples. Do you use it? What for?
  - 4y ago

sgtquack 4y ago

As someone who recently dealt with scraping sites behind cloudflare...I never want to scrape again

gdcbe 4y ago
Honestly cloudflare isn't really a big deal in terms of scraping.

loudthing 4y ago

It can be frustrating learning web scraping with Python when so many sites actively block scraping.

1vuio0pswjnm7 4y ago
Any chance you could name a few of those sites. LinkedIn is the only example I know of.
- loudthing 4y ago
  The main perpetrator I've come across (in, I'll admit, my limited experience with scraping) was Google services. It seemed somewhat obvious to me initially while playing around with Beautiful Soup, the most valuable resource I could use for scraping was Google search, followed by Gmail and Google Finance. I had no luck using BS with any of these services. Largely this taught me to be creative with my data sources. For example, I built a virtual weather vane powered by a Raspberry Pi that would scrape my local airport's website to get wind direction data, then turn the vane via a servo to the correct direction. So my takeaway from this project was scraping isn't as straight forward as one would thing, there's more of an art to it in order to figuring out where to get the information you want.
  - 1vuio0pswjnm7 4y ago
    Agreed. Most public websites are not trying to force visitors to use a certain web browser by selectively denying access. No public website should be the sole, exclusive source of its data because if the data is public then by definition the data can be copied by any other website. As such, chances are the data can be found in multiple locations and at least one will not be trying to force the use of a certian web browser. Am I correct that the examples listed here are (a) www.google.com, (b) mail.google.com and (c) www.google.com/finance/. I have no trouble extracting data from these examples.[FN1] I do not use a graphical web browser to make HTTP requests nor do I use Python or BeatifulSoup. A cookie is required for mail.google.com, in lieu of a password, but the cookie can be saved and will work for years. 1. Of course, Google Web Search is crippled. Using a basic HTTP client, e.g., no cookies, Javascript, FLoC, etc., one cannot retrieve more than 250-300 results total. Searching "too fast" will draw a temporary IP block. This "search engine" is designed for advertising not discovery. Advertisers compete for space at the top of the first page of results. Popular websites are prioritised, potentially making them even more popular. Websites that "rank"[FN2] too low in a search are not discoverable as they have no value for advertising. An index of public websites and public data is treated as properietary and secret. Google actively tries to prevent anyone from copying even a small portion of it. 2. Google makes it impossible to sort results by URL, date, or even number of keyword/string hits in the page. Results are ordered according to secret algorithm, designed for advertising.

holografix 4y ago

For websites that require auth via Google Auth this is a non starter. There’s no way to bypass its bot detection

gdcbe 4y ago
Perhaps you should anyhow not burn your hands on such targets even if technically possible, you start to get in murky water quickly. Requiring Google Auth meaning that you need to login, makes it debatable whether or not that data is still public data.

lapser 4y ago

Is there an FLOSS project that combines Scrapy scrapers and just makes the results publicly available?

mdaniel 4y ago
This is the one I know about: https://morph.io/ https://morph.io/ and https://github.com/openaustralia/morph#readme https://github.com/openaustralia/morph#readme (AGPLv3) -- they used to be at the intersection of "heroku for scrapers" and DoltHub (e.g. https://www.dolthub.com/repositories/dolthub/us-businesses/data/master/businesses https://www.dolthub.com/repositories/dolthub/us-businesses/d...) since the scrapers would run but then make their data available as CSV or sqlite or whatever. But, when I just tried to load one of the morph.io scrapers, the page just said "creating new template" so I'm guessing they've gone the way of the ScraperWiki.com that preceded them: turns out, hosted compute for free isn't free

wraptile 4y ago

I recently joined a brilliant web scraping API company called ScrapFly, who provided me with the resources to create lots of open knowledge on web scraping that I always wanted to create! So, to add to this list here are my top 3 favorite article that could expand OP's document: 0 - central guide to avoiding blocking - this one a tough one because there's so much information: request headers, http versions, TLS fingerprinting, javascript fingerprinting etc. I spent almost a month working on these and it was an amazing research experience that I could never afford myself before. 1, 2 - xpath and css selector introduction articles where I built a widget into our article that allows to test all css and xpath selectors right there in the learning material. 3 - introduction to reverse engineering - quick introduction to using browser devtools for web scraping, how to inspect the network and replicate it in your program. This is where I point all beginners as understanding the browser really helps to understand web scraping! 0 - https://scrapfly.io/blog/parsing-html-with-css/ https://scrapfly.io/blog/parsing-html-with-css/ 1 - https://scrapfly.io/blog/parsing-html-with-xpath/ https://scrapfly.io/blog/parsing-html-with-xpath/ 2 - https://scrapfly.io/blog/how-to-scrape-without-getting-blocked-tutorial/ https://scrapfly.io/blog/how-to-scrape-without-getting-block... 3 - https://scrapecrow.com/reverse-engineering-intro.html https://scrapecrow.com/reverse-engineering-intro.html

throwaway290 4y ago
Your first link is broken, it is not about blocking.
- wraptile 4y ago
  sorry, I reordered the links and forgot to update the footnotes - you can probably guess which one is which one by the link text itself though :)
karterk 4y ago
Do you follow robots.txt or do you allow your customers to bypass restrictions placed by the website? How do you feel about bypassing that? I know it's not illegal but it's certainly not ethical either.

tatoalo 4y ago

I recently transitioned one of my scraping projects away from selenium to playwright and I must say that the developer experience is way better, in my opinion. I also implemented to receive a telegram message with the debug trace in case of errors in my pipeline, so that I could have the entire scraping flow to analyze. That’s pretty neat.

rustdeveloper 4y ago

I think that if you don’t want to invest a lot of time into learning web scraping and money to get a pool of residential, or even better mobile, proxies it’s easy to quickly get good results with web scraping API like https://scrapingfish.com https://scrapingfish.com They have good blogposts, for example, for how to scrape public Instagram profiles: https://scrapingfish.com/blog/scraping-instagram https://scrapingfish.com/blog/scraping-instagram