18 ms·

How we learnt to stop worrying and love web scraping

300 points by danso 6y ago

mherrmann 6y ago
Shameless plug for my open source library Helium [1]. It extends Selenium's API and makes it easier to use. The name comes from Helium also being a chemical element, but lighter than Selenium. 1: https://github.com/mherrmann/selenium-python-helium https://github.com/mherrmann/selenium-python-helium
- isaacimagine 6y ago
  I'm not affiliated, but have used helium and have found it to be intuitive and work quite well.
- nickthemagicman 6y ago
  Just looked. Very cool! Thanks for linking. Also, cool idea with the name. The lightest of the noble gasses.
- worldmerge 6y ago
  I just finished up an app that uses Selenium. Wish I discovered this before building it out. This looks cool!
- dotancohen 6y ago
  I've never heard of Helium, but this code snippet from the docs sold me: name = Text(to_right_of='Name:', below=Image(alt='Profile picture')).value Thank you!
whatever1 6y ago
I am a proponent of web scraping. As long as the overall volume does not overload the available resources I don't see why automated retrieval of publicly available info is an issue. If the data in my website could help someone else's research, a simple reference to the source would make me super happy.
- Aperocky 6y ago
  I like APIs better, but people gate their API behind paywalls.
DoingIsLearning 6y ago
> We strongly feel that more researchers should be developing code to help conduct their research, and then sharing it with the community. If manual data collection has been an issue for your project, a web scraper could be the perfect solution and a great beginner coding project. I totally agree with the general idea of sharing code and tools, in an open research community but incentivising someone from a field completely unrelated to CS to 'just code' sounds like pretty bad advice. This is very likely to lead to a lot of time spent, frustration, and underwhelming results. Why not incentivise building a professional Software team at inter-departmental University level and have that team act as multi-department shared resource? edit: missed words
- rytis 6y ago
  I guess it depends on the complexity of the task. If it's something I could knock up in a few hours time, and experiment with the results myself, why would I engage in endless inter-departmental politics, of who's project is more important, etc. On the other hand, there's a threshold where project is significantly complicated and requires professional assistance.
- wjnc 6y ago
  That would require something that is mostly missing in universities: strategy. Cooperation between departments and colleagues without direct sight on short term benefits is very slim. I'm quite close to my local university (father worked there, alumnus, wife works there, friends who are professor; it's a quite good Euro-business and law school). The thing I've learned: a university is more akin to a village than to a business even though the administrators try to run it as a business (but have terms a lot shorter than most professors). There is no clear line of control. Professors are tenured and some have super-star tendencies. Budgets are fragmented between departments, groups, and in our country are built upon subsidies for specific research tasks. There is a lot of cooperation between individual researchers in fields. But that is a response to it satisficing the demands of subsidies. Ýou could call that strategy, but I'd call that 'external motivation'. Strategy as in internal motivation: "How can we get greater research results from all our research staff by leveraging technology" is a question hardly asked.
- 6y ago
b6z 6y ago
I admit being a bit disappointed that a well-known disadvantage of web scraping was not mentioned: Web scraping is fragile! Web sites change, web frameworks evolve, and just some subtle reordering of some <divs> or renaming of CSS classes, and your perfect scraping code from yesterday will break tomorrow -- maybe not leaving you empty-handed, but probably missing some data or delivering the wrong one. If there is an API you can use, use it. If your budget allows to pay for API access, buy it. APIs tend to be more stable than scraping, and the data provider will probably inform you if it changes. Contacting them might even get you more interesting data, as not every column they have in their database might become published on the web site.
- danso 6y ago
  For the purposes of their research and study, web scraping is stable enough, given that many websites (especially government sites), aren’t overhauled frequently. And many government sites, like those run by coroners, aren’t likely to have APIs. Contacting the agency for the data is always a good step, but even if they are responsive, they may not be willing to email you on a daily basis with data updates.
  - anilakar 6y ago
    We used to despise web scraping and tried to dump data from an industrial system by listening to raw Modbus RS-485 traffic. Turned out that there was no way to get the register maps from the vendor nor the firmware. We ended up writing a scraper for the main unit. Once they have been installed and configured, they're likely not going to be updated unless absolutely necessary (such as when adding new, unsupported controllers to the bus).
    - deleted 6y ago
      [deleted]
  - b6z 6y ago
    That is a good argument, and I should have mentioned it, yes. For a one-off job, web scraping will probably be the best choice, and maybe even the fastest to implement. I have done my own share of web scraping for personal projects (and thus know about the fragility), but I didn't care much about broken results in the long run. But in the article they mentioned re-running their program to update their data, so it could be a long-term effort. And anyone planning a long-term project reading that article and taking their advice should at least be warned about this possible problem.
andrewnc 6y ago
During my studies, the web scraping projects were among my favorites. I adore the challenge and building a robust scraper. I recently sold a 'business' that does web scraping. Does anyone have insight into which industries need more web scraping 'experts'?
- ttttodayjunior 6y ago
  Hedge funds - check out the field of "alternative data"
  - andrewnc 6y ago
    oh, THAT is fascinating. Thanks for taking the time to respond. That definitely feels like just what I'm looking for.
ricardo81 6y ago
It all makes sense if you believe in an information is universal idea where it should all be freely available. Though, you you get cases like celebritynetworth.com (no affiliation to me) that was featured on HN where Google wanted to APIize the data and when that was not available they decided to scrape it. It effectively killed the business. I think if a company decides to offer an API or other structured data then fine, they've decided that their data can be shared in a way that makes it OK to collect en masse. I wouldn't say we should "learn to love" scraping when there are sites that put a lot of man hours into the data that's seen on public html pages only for someone else to spend a few hours scraping it and repurpose it. Now, if the major traffic drivers like search engines and social networks were able to differentiate who the true authoritative source is, that'd be great. But they don't.
- cik2e 6y ago
  Link to the celebritynetworth.com discussion: https://news.ycombinator.com/item?id=24105465 https://news.ycombinator.com/item?id=24105465
- wolco 6y ago
  The big problem is once google killed it no new information will come and google can't scrape what doesn't exist
- pjc50 6y ago
  The EU sui generis copyright in databases: https://europa.eu/youreurope/business/running-business/intellectual-property/database-protection/index_en.htm https://europa.eu/youreurope/business/running-business/intel... Not part of the Berne convention, and not applicable in the US.
- wzdd 6y ago
  There are a significant number of missing steps, in terms of moral justifiability, between "scraping information from the web for academic research purposes" and "using the scraped information to add value to our primary generator of advertisement-derived revenue".
  - 6y ago
sys_64738 6y ago
Won't switching to WASM do away with the ability to web scrape?
- detaro 6y ago
  a) it is unlikely the majority of the web will "switch to WASM" b) what does "switching to WASM" mean for you? If it still generates a DOM or access a private API, many scraping techniques don't care.
- chundicus 6y ago
  why would it?
- dotancohen 6y ago
  Much scraping is done with automated control of a (possibly headless) web browser. It runs all the same scripts and has all the same features as a standard web browser for sighted users.
compscistd 6y ago
One thing I tend to see a lot (90% of requests) is from people who are stuck with mediocre CRMs and want better analytics and email reporting than they’re given. The only way to accomplish anything is to spoof login and scrape because the only api is through “partners” who either pay a premium or are giant. I turn these requests down because it’s a ticking time bomb when you add in the element of login (password resets, 2FA, another point of change). On the other hand, I wonder how different this is from what Plaid does...
mmerlin 6y ago
I've written scrapers and web automation bots going as far back as 1998 (using VB5 and VB6 automating IE) Nowadays mostly I write custom scrapers use PHP (Guzzle, Curl, etc) and Python (I was introduced to Beautiful Soup, by the "Python for Secret Agents" book) I've tried several commercial scraper tools and services over the years but few stuck for various reasons. UbotStudio had great potential but in the end was buggy and painful, almost abandonware. Scrapinghub.com is decent enough but bit expensive for my projects. 80legs.com is cool for massive scale but was overly robots.txt restrictive at the time I tried using it (for what I was scraping) and I don't like the syntax. A scraper colleague likes using Winautomation however it's no longer for sale separately, because Microsoft acquired the company and rolled it into their Power Automate SaaS (RPA/RDA focused) There is a new tool called RTILA that I used for the very first time a 9 days ago, which actually is the easiest way to create and run scrapers I have found in 20 years. The RTILA software currently has minimal documentation (apparently is being worked on now), however the new features are being developed fast, all releases are here on GitHub (see the frequency of releases) https://github.com/IKAJIAN/rtila-releases/releases https://github.com/IKAJIAN/rtila-releases/releases Another user of the software has produced several video tutorials here showing how it works: https://www.youtube.com/channel/UCH6ov8LnB8-4ZF0yraxjw8Q/videos https://www.youtube.com/channel/UCH6ov8LnB8-4ZF0yraxjw8Q/vid... You download it from GitHub but also still need to buy a license key from here https://codecanyon.net/user/ikajian https://codecanyon.net/user/ikajian The RTILA home page is here https://rtila.ikajian.com/ https://rtila.ikajian.com/ I am a genuine customer and have no connection in any way with the solo founder developing it, (except for a few emails and support forum messages). Genuine recommendation for writing a bot more easily.
- Aperocky 6y ago
  Having written a few scrapers, including the browser kind. I found the trick was to specialize. Each website respond differently, some can just use requests, others would need selenium with pre-existing profile. To develop a common utility is almost futile. I do have useful building blocks, but for each individual things I want to scrape I scale out using project specific code. It's never too slow either - the time it would take to fill in all of the required bits in a do it all tool would have been similar.
uhtred 6y ago
Seems like a lot of interesting sites recognise that I am scraping and block access with a capture. I've no idea how to get around that.
- gmac 6y ago
  How are you doing it? Automating an actual browser using the developer tools is probably the most under-the-radar way, and also quite nice to work with. See: https://github.com/jawj/web-scraping-for-researchers https://github.com/jawj/web-scraping-for-researchers
  - ivoecpereira 6y ago
    What out for "Devtools detect" :) https://github.com/sindresorhus/devtools-detect https://github.com/sindresorhus/devtools-detect
    - gmac 6y ago
      This is a good point, and I've noticed this phenomenon on some video sites lately. Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter. [1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."
- umaar 6y ago
  I've put an example here, for doing it in Node.js: https://github.com/umaar/learn-browser-testing/tree/master/2-scraping/10-captcha https://github.com/umaar/learn-browser-testing/tree/master/2...
deleted 6y ago
[deleted]
umaar 6y ago
I'm working on a video course for all things browser automation, whether it's testing, scraping, auditing, deploying etc. I'm keeping the codebase open on GitHub: https://github.com/umaar/learn-browser-testing/ https://github.com/umaar/learn-browser-testing/ so anyone who wants to follow along can do so for free. In the 2-scraping folder, there's a bunch of scraping examples such as bypassing captchas, building an Amazon price checker, blocking CSS and JavaScript resources during scraping to make the process much quicker, and a few others. Hope it's useful.
gabrielsroka 6y ago
The article mentions Python, Requests and BeautifulSoup. Here's a short example that scrapes HN Favorites. [0] #!/usr/bin/env python3 import requests from bs4 import BeautifulSoup username = input('username: ') # session uses connection pooling, often resulting in faster execution. session = requests.Session() base = 'https://news.ycombinator.com/' path = f'favorites?id={username}' while path: r = session.get(base + path) s = BeautifulSoup(r.text, 'html.parser') for a in s.select('a.storylink'): print(a.text, a['href']) more = s.select_one('a.morelink') path = more['href'] if more else None [0] Python: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/master/getHNFavorites.py https://github.com/gabrielsroka/gabrielsroka.github.io/blob/... [1] JavaScript, runs in browser with a UI: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/master/getHNFavorites.js https://github.com/gabrielsroka/gabrielsroka.github.io/blob/...
- lixtra 6y ago
  Use this an example, but for real stuff prefer using the official API https://github.com/HackerNews/API https://github.com/HackerNews/API
  - Dangeranger 6y ago
    To be fair, the code above is for scraping favorites which I do not believe is included in the official API. There was a very popular story[0] a few days ago that used scraped data to compile a list of the most favorite posts, which many users didn't even know was a feature until reading the story. Dang even joined in. [0] https://news.ycombinator.com/item?id=24351073 https://news.ycombinator.com/item?id=24351073
    - gabrielsroka 6y ago
      Correct, the official API doesn't support favorites.
- pj_mukh 6y ago
  People love BeautifulSoup, but I've had better luck with fully emulating a browser with Selenium. While more complicated it seems to run into less issues. Is there a rule of thumb as to when to use one over the other?
Dowwie 6y ago
This seems like the tip of the iceberg for their work. The authors automated document collection but are left with the task of searching through the docs and crawling onward to linked documents. What kind of libraries are available today for crawling pdf documents?
extremeMath 6y ago
I failed a webscraping project due to strong anti bot detection. They checked for bots through useragent/screen size, maybe mouse movements, trends in searches (same area code), etc... (Can they really detect me through my internet connection headers, despite proxies?) It was impossible for me to scrape, they won.
- billconan 6y ago
  same here. there are 2 approaches they use that make developing bots very difficult. 1. they detect device input. if there is no mouse movement, while the website is being loaded, they will consider it's a bot. 2. they detect the order of page visiting. A human visitor will not enumerate all paths, instead, they follow certain patterns. This is detectable with their machine learning model. I really don't have a solution for #2
  - forgotmypw17 6y ago
    I think the solution is "hybrid" scraping with a human driving the clicks and the scraper passively collecting the data. If you record, you can probably teach AI to emulate.
    - extremeMath 6y ago
      I love this. I might try it. It doesn't scale, but that's okay for my project.
- r_singh 6y ago
  Which website are you talking about here? Even the hardest to scrape sites can be scraped with proxies and headless browsers.
  - extremeMath 6y ago
    Theoretically. Think about websites that have every reason to stop you from scraping. It's not reasonable to disect their huge obfuscated js code. So headless doesn't really work.
welanes 6y ago
The beautiful thing about web scraping is that it automates so much tedious work that the benefits in saved time offsets the brittleness inherent in scraping data. And with Puppeteer (also Playwright) it's never been easier. Recaptcha solving, Ad blocking etc. in just a few lines of code[1]. I've built a business on the back of Puppeteer - https://simplescraper.io https://simplescraper.io. Ten months in and we've just passed 100 customers so there's mucho opportunities in solving these kind of problems. [1] https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra https://github.com/berstend/puppeteer-extra/tree/master/pack...
- davidbarker 6y ago
  This looks useful! Thanks for sharing. It reminds me of KimonoLabs, which I loved before it was acquired by Palantir and shut down.
dtjones 6y ago
Learnt -> learned
- edmundsauto 6y ago
  Learnt is the preferred correct word in non-US English speaking countries. https://www.grammarly.com/blog/learned-learnt/ https://www.grammarly.com/blog/learned-learnt/
superdeeda 6y ago
I’d be curious to know why they “worried” in the first place though.
- mrzool 6y ago
  Genuinely thought this was an editorial by nature.com explaining how they stopped worrying about researchers scraping their website for data and instead embraced them, and the rationale behind this decision. Turned out to be a just a 101 about web scraping in general. Disappointed.
- jonh1016 6y ago
  I too was wondering if this was going to be explained. Had to re-read the article just to make sure I didn't miss something.
wombatmobile 6y ago
Corey Schafer is an articulate teacher of the craft of web scraping. Python Tutorial: Web Scraping with BeautifulSoup and Requests https://youtu.be/ng2o98k983k https://youtu.be/ng2o98k983k
dannyfraser 6y ago
My favourite kind of web scraping is when you have to pick apart some undocumented API that serves up data to an SPA, then figure out how to tidy up the response(s) into a single pandas dataframe. Always a satisfying feeling to solve one of those little puzzles.
intrasight 6y ago
I've not done it recently, but have in the past used Power BI for web scraping. It's a pretty slick solution. A quick Google found this article. https://spencerbaucke.com/2020/04/29/web-scraping-in-power-bi/ https://spencerbaucke.com/2020/04/29/web-scraping-in-power-b...
dastx 6y ago
Surprised no one has mentioned weboob. If your interest isn't specifically along the lines of "go to all pages, and get all of the data" and more along the lines of "go to my bank, log in using username and password, get the transactions" etc, weboob is ideal. It seems to be extremely well thought out and easily extendable.it has plugins for a whole bunch of websites already, including many, many banks, some of the biggest websites such as reddit, Google, and many more. The only downside I've had so far is that it will fall flat on it's arse if the website has to load must do javascripty things (it can handle APIs and html easily though).
tracker1 6y ago
I really like the options of node + cheerio, or node + puppeteer for these kinds of jobs. Language of the browser, easy enough selection/usage, cheerio if it's plain html, puppeteer if it's dynamic/spa content. In the end, not hammering a site is key... also, on your own end, possibly hashing the main content, so you aren't creating duplicate entries on your own side are important. As to fragility, that happens... in general, you need to update to match site updates, but most sites won't be dramatically updated more than a couple times a year if they aren't in active development.
spicyramen 6y ago
Used to do web scrapping and took very long as most websites change formatting and tags quite often. This required engineering just to maintain few sites all the time.
junilop 6y ago
found this lib in the comments, tried it and looks nice. can be good for finding the xpath automatically and save the manual traversing overhead. https://github.com/alirezamika/autoscraper https://github.com/alirezamika/autoscraper

300 points by danso 6y ago

Shameless plug for my open source library Helium [1]. It extends Selenium's API and makes it easier to use. The name comes from Helium also being a chemical element, but lighter than Selenium. 1: https://github.com/mherrmann/selenium-python-helium https://github.com/mherrmann/selenium-python-helium

isaacimagine 6y ago
I'm not affiliated, but have used helium and have found it to be intuitive and work quite well.
nickthemagicman 6y ago
Just looked. Very cool! Thanks for linking. Also, cool idea with the name. The lightest of the noble gasses.
worldmerge 6y ago
I just finished up an app that uses Selenium. Wish I discovered this before building it out. This looks cool!
dotancohen 6y ago
I've never heard of Helium, but this code snippet from the docs sold me: name = Text(to_right_of='Name:', below=Image(alt='Profile picture')).value Thank you!

whatever1 6y ago

I am a proponent of web scraping. As long as the overall volume does not overload the available resources I don't see why automated retrieval of publicly available info is an issue. If the data in my website could help someone else's research, a simple reference to the source would make me super happy.

Aperocky 6y ago
I like APIs better, but people gate their API behind paywalls.

DoingIsLearning 6y ago

> We strongly feel that more researchers should be developing code to help conduct their research, and then sharing it with the community. If manual data collection has been an issue for your project, a web scraper could be the perfect solution and a great beginner coding project. I totally agree with the general idea of sharing code and tools, in an open research community but incentivising someone from a field completely unrelated to CS to 'just code' sounds like pretty bad advice. This is very likely to lead to a lot of time spent, frustration, and underwhelming results. Why not incentivise building a professional Software team at inter-departmental University level and have that team act as multi-department shared resource? edit: missed words

rytis 6y ago
I guess it depends on the complexity of the task. If it's something I could knock up in a few hours time, and experiment with the results myself, why would I engage in endless inter-departmental politics, of who's project is more important, etc. On the other hand, there's a threshold where project is significantly complicated and requires professional assistance.
wjnc 6y ago
That would require something that is mostly missing in universities: strategy. Cooperation between departments and colleagues without direct sight on short term benefits is very slim. I'm quite close to my local university (father worked there, alumnus, wife works there, friends who are professor; it's a quite good Euro-business and law school). The thing I've learned: a university is more akin to a village than to a business even though the administrators try to run it as a business (but have terms a lot shorter than most professors). There is no clear line of control. Professors are tenured and some have super-star tendencies. Budgets are fragmented between departments, groups, and in our country are built upon subsidies for specific research tasks. There is a lot of cooperation between individual researchers in fields. But that is a response to it satisficing the demands of subsidies. Ýou could call that strategy, but I'd call that 'external motivation'. Strategy as in internal motivation: "How can we get greater research results from all our research staff by leveraging technology" is a question hardly asked.
6y ago

b6z 6y ago

I admit being a bit disappointed that a well-known disadvantage of web scraping was not mentioned: Web scraping is fragile! Web sites change, web frameworks evolve, and just some subtle reordering of some <divs> or renaming of CSS classes, and your perfect scraping code from yesterday will break tomorrow -- maybe not leaving you empty-handed, but probably missing some data or delivering the wrong one. If there is an API you can use, use it. If your budget allows to pay for API access, buy it. APIs tend to be more stable than scraping, and the data provider will probably inform you if it changes. Contacting them might even get you more interesting data, as not every column they have in their database might become published on the web site.

danso 6y ago
For the purposes of their research and study, web scraping is stable enough, given that many websites (especially government sites), aren’t overhauled frequently. And many government sites, like those run by coroners, aren’t likely to have APIs. Contacting the agency for the data is always a good step, but even if they are responsive, they may not be willing to email you on a daily basis with data updates.
- anilakar 6y ago
  We used to despise web scraping and tried to dump data from an industrial system by listening to raw Modbus RS-485 traffic. Turned out that there was no way to get the register maps from the vendor nor the firmware. We ended up writing a scraper for the main unit. Once they have been installed and configured, they're likely not going to be updated unless absolutely necessary (such as when adding new, unsupported controllers to the bus).
  - deleted 6y ago
    [deleted]
- b6z 6y ago
  That is a good argument, and I should have mentioned it, yes. For a one-off job, web scraping will probably be the best choice, and maybe even the fastest to implement. I have done my own share of web scraping for personal projects (and thus know about the fragility), but I didn't care much about broken results in the long run. But in the article they mentioned re-running their program to update their data, so it could be a long-term effort. And anyone planning a long-term project reading that article and taking their advice should at least be warned about this possible problem.

andrewnc 6y ago

During my studies, the web scraping projects were among my favorites. I adore the challenge and building a robust scraper. I recently sold a 'business' that does web scraping. Does anyone have insight into which industries need more web scraping 'experts'?

ttttodayjunior 6y ago
Hedge funds - check out the field of "alternative data"
- andrewnc 6y ago
  oh, THAT is fascinating. Thanks for taking the time to respond. That definitely feels like just what I'm looking for.

ricardo81 6y ago

It all makes sense if you believe in an information is universal idea where it should all be freely available. Though, you you get cases like celebritynetworth.com (no affiliation to me) that was featured on HN where Google wanted to APIize the data and when that was not available they decided to scrape it. It effectively killed the business. I think if a company decides to offer an API or other structured data then fine, they've decided that their data can be shared in a way that makes it OK to collect en masse. I wouldn't say we should "learn to love" scraping when there are sites that put a lot of man hours into the data that's seen on public html pages only for someone else to spend a few hours scraping it and repurpose it. Now, if the major traffic drivers like search engines and social networks were able to differentiate who the true authoritative source is, that'd be great. But they don't.

cik2e 6y ago
Link to the celebritynetworth.com discussion: https://news.ycombinator.com/item?id=24105465 https://news.ycombinator.com/item?id=24105465
wolco 6y ago
The big problem is once google killed it no new information will come and google can't scrape what doesn't exist
pjc50 6y ago
The EU sui generis copyright in databases: https://europa.eu/youreurope/business/running-business/intellectual-property/database-protection/index_en.htm https://europa.eu/youreurope/business/running-business/intel... Not part of the Berne convention, and not applicable in the US.
wzdd 6y ago
There are a significant number of missing steps, in terms of moral justifiability, between "scraping information from the web for academic research purposes" and "using the scraped information to add value to our primary generator of advertisement-derived revenue".
- 6y ago

sys_64738 6y ago

Won't switching to WASM do away with the ability to web scrape?

detaro 6y ago
a) it is unlikely the majority of the web will "switch to WASM" b) what does "switching to WASM" mean for you? If it still generates a DOM or access a private API, many scraping techniques don't care.
chundicus 6y ago
why would it?
dotancohen 6y ago
Much scraping is done with automated control of a (possibly headless) web browser. It runs all the same scripts and has all the same features as a standard web browser for sighted users.

compscistd 6y ago

One thing I tend to see a lot (90% of requests) is from people who are stuck with mediocre CRMs and want better analytics and email reporting than they’re given. The only way to accomplish anything is to spoof login and scrape because the only api is through “partners” who either pay a premium or are giant. I turn these requests down because it’s a ticking time bomb when you add in the element of login (password resets, 2FA, another point of change). On the other hand, I wonder how different this is from what Plaid does...

mmerlin 6y ago

I've written scrapers and web automation bots going as far back as 1998 (using VB5 and VB6 automating IE) Nowadays mostly I write custom scrapers use PHP (Guzzle, Curl, etc) and Python (I was introduced to Beautiful Soup, by the "Python for Secret Agents" book) I've tried several commercial scraper tools and services over the years but few stuck for various reasons. UbotStudio had great potential but in the end was buggy and painful, almost abandonware. Scrapinghub.com is decent enough but bit expensive for my projects. 80legs.com is cool for massive scale but was overly robots.txt restrictive at the time I tried using it (for what I was scraping) and I don't like the syntax. A scraper colleague likes using Winautomation however it's no longer for sale separately, because Microsoft acquired the company and rolled it into their Power Automate SaaS (RPA/RDA focused) There is a new tool called RTILA that I used for the very first time a 9 days ago, which actually is the easiest way to create and run scrapers I have found in 20 years. The RTILA software currently has minimal documentation (apparently is being worked on now), however the new features are being developed fast, all releases are here on GitHub (see the frequency of releases) https://github.com/IKAJIAN/rtila-releases/releases https://github.com/IKAJIAN/rtila-releases/releases Another user of the software has produced several video tutorials here showing how it works: https://www.youtube.com/channel/UCH6ov8LnB8-4ZF0yraxjw8Q/videos https://www.youtube.com/channel/UCH6ov8LnB8-4ZF0yraxjw8Q/vid... You download it from GitHub but also still need to buy a license key from here https://codecanyon.net/user/ikajian https://codecanyon.net/user/ikajian The RTILA home page is here https://rtila.ikajian.com/ https://rtila.ikajian.com/ I am a genuine customer and have no connection in any way with the solo founder developing it, (except for a few emails and support forum messages). Genuine recommendation for writing a bot more easily.

Aperocky 6y ago
Having written a few scrapers, including the browser kind. I found the trick was to specialize. Each website respond differently, some can just use requests, others would need selenium with pre-existing profile. To develop a common utility is almost futile. I do have useful building blocks, but for each individual things I want to scrape I scale out using project specific code. It's never too slow either - the time it would take to fill in all of the required bits in a do it all tool would have been similar.

uhtred 6y ago

Seems like a lot of interesting sites recognise that I am scraping and block access with a capture. I've no idea how to get around that.

gmac 6y ago
How are you doing it? Automating an actual browser using the developer tools is probably the most under-the-radar way, and also quite nice to work with. See: https://github.com/jawj/web-scraping-for-researchers https://github.com/jawj/web-scraping-for-researchers
- ivoecpereira 6y ago
  What out for "Devtools detect" :) https://github.com/sindresorhus/devtools-detect https://github.com/sindresorhus/devtools-detect
  - gmac 6y ago
    This is a good point, and I've noticed this phenomenon on some video sites lately. Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter. [1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."
umaar 6y ago
I've put an example here, for doing it in Node.js: https://github.com/umaar/learn-browser-testing/tree/master/2-scraping/10-captcha https://github.com/umaar/learn-browser-testing/tree/master/2...

deleted 6y ago

[deleted]

umaar 6y ago

I'm working on a video course for all things browser automation, whether it's testing, scraping, auditing, deploying etc. I'm keeping the codebase open on GitHub: https://github.com/umaar/learn-browser-testing/ https://github.com/umaar/learn-browser-testing/ so anyone who wants to follow along can do so for free. In the 2-scraping folder, there's a bunch of scraping examples such as bypassing captchas, building an Amazon price checker, blocking CSS and JavaScript resources during scraping to make the process much quicker, and a few others. Hope it's useful.

gabrielsroka 6y ago

The article mentions Python, Requests and BeautifulSoup. Here's a short example that scrapes HN Favorites. [0] #!/usr/bin/env python3 import requests from bs4 import BeautifulSoup username = input('username: ') # session uses connection pooling, often resulting in faster execution. session = requests.Session() base = 'https://news.ycombinator.com/' path = f'favorites?id={username}' while path: r = session.get(base + path) s = BeautifulSoup(r.text, 'html.parser') for a in s.select('a.storylink'): print(a.text, a['href']) more = s.select_one('a.morelink') path = more['href'] if more else None [0] Python: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/master/getHNFavorites.py https://github.com/gabrielsroka/gabrielsroka.github.io/blob/... [1] JavaScript, runs in browser with a UI: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/master/getHNFavorites.js https://github.com/gabrielsroka/gabrielsroka.github.io/blob/...

lixtra 6y ago
Use this an example, but for real stuff prefer using the official API https://github.com/HackerNews/API https://github.com/HackerNews/API
- Dangeranger 6y ago
  To be fair, the code above is for scraping favorites which I do not believe is included in the official API. There was a very popular story[0] a few days ago that used scraped data to compile a list of the most favorite posts, which many users didn't even know was a feature until reading the story. Dang even joined in. [0] https://news.ycombinator.com/item?id=24351073 https://news.ycombinator.com/item?id=24351073
  - gabrielsroka 6y ago
    Correct, the official API doesn't support favorites.
pj_mukh 6y ago
People love BeautifulSoup, but I've had better luck with fully emulating a browser with Selenium. While more complicated it seems to run into less issues. Is there a rule of thumb as to when to use one over the other?

Dowwie 6y ago

This seems like the tip of the iceberg for their work. The authors automated document collection but are left with the task of searching through the docs and crawling onward to linked documents. What kind of libraries are available today for crawling pdf documents?

extremeMath 6y ago

I failed a webscraping project due to strong anti bot detection. They checked for bots through useragent/screen size, maybe mouse movements, trends in searches (same area code), etc... (Can they really detect me through my internet connection headers, despite proxies?) It was impossible for me to scrape, they won.

billconan 6y ago
same here. there are 2 approaches they use that make developing bots very difficult. 1. they detect device input. if there is no mouse movement, while the website is being loaded, they will consider it's a bot. 2. they detect the order of page visiting. A human visitor will not enumerate all paths, instead, they follow certain patterns. This is detectable with their machine learning model. I really don't have a solution for #2
- forgotmypw17 6y ago
  I think the solution is "hybrid" scraping with a human driving the clicks and the scraper passively collecting the data. If you record, you can probably teach AI to emulate.
  - extremeMath 6y ago
    I love this. I might try it. It doesn't scale, but that's okay for my project.
r_singh 6y ago
Which website are you talking about here? Even the hardest to scrape sites can be scraped with proxies and headless browsers.
- extremeMath 6y ago
  Theoretically. Think about websites that have every reason to stop you from scraping. It's not reasonable to disect their huge obfuscated js code. So headless doesn't really work.

welanes 6y ago

The beautiful thing about web scraping is that it automates so much tedious work that the benefits in saved time offsets the brittleness inherent in scraping data. And with Puppeteer (also Playwright) it's never been easier. Recaptcha solving, Ad blocking etc. in just a few lines of code[1]. I've built a business on the back of Puppeteer - https://simplescraper.io https://simplescraper.io. Ten months in and we've just passed 100 customers so there's mucho opportunities in solving these kind of problems. [1] https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra https://github.com/berstend/puppeteer-extra/tree/master/pack...

davidbarker 6y ago
This looks useful! Thanks for sharing. It reminds me of KimonoLabs, which I loved before it was acquired by Palantir and shut down.

dtjones 6y ago

Learnt -> learned

edmundsauto 6y ago
Learnt is the preferred correct word in non-US English speaking countries. https://www.grammarly.com/blog/learned-learnt/ https://www.grammarly.com/blog/learned-learnt/

superdeeda 6y ago

I’d be curious to know why they “worried” in the first place though.

mrzool 6y ago
Genuinely thought this was an editorial by nature.com explaining how they stopped worrying about researchers scraping their website for data and instead embraced them, and the rationale behind this decision. Turned out to be a just a 101 about web scraping in general. Disappointed.
jonh1016 6y ago
I too was wondering if this was going to be explained. Had to re-read the article just to make sure I didn't miss something.

wombatmobile 6y ago

Corey Schafer is an articulate teacher of the craft of web scraping. Python Tutorial: Web Scraping with BeautifulSoup and Requests https://youtu.be/ng2o98k983k https://youtu.be/ng2o98k983k

dannyfraser 6y ago

My favourite kind of web scraping is when you have to pick apart some undocumented API that serves up data to an SPA, then figure out how to tidy up the response(s) into a single pandas dataframe. Always a satisfying feeling to solve one of those little puzzles.

intrasight 6y ago

I've not done it recently, but have in the past used Power BI for web scraping. It's a pretty slick solution. A quick Google found this article. https://spencerbaucke.com/2020/04/29/web-scraping-in-power-bi/ https://spencerbaucke.com/2020/04/29/web-scraping-in-power-b...

dastx 6y ago

Surprised no one has mentioned weboob. If your interest isn't specifically along the lines of "go to all pages, and get all of the data" and more along the lines of "go to my bank, log in using username and password, get the transactions" etc, weboob is ideal. It seems to be extremely well thought out and easily extendable.it has plugins for a whole bunch of websites already, including many, many banks, some of the biggest websites such as reddit, Google, and many more. The only downside I've had so far is that it will fall flat on it's arse if the website has to load must do javascripty things (it can handle APIs and html easily though).

tracker1 6y ago

I really like the options of node + cheerio, or node + puppeteer for these kinds of jobs. Language of the browser, easy enough selection/usage, cheerio if it's plain html, puppeteer if it's dynamic/spa content. In the end, not hammering a site is key... also, on your own end, possibly hashing the main content, so you aren't creating duplicate entries on your own side are important. As to fragility, that happens... in general, you need to update to match site updates, but most sites won't be dramatically updated more than a couple times a year if they aren't in active development.

spicyramen 6y ago

Used to do web scrapping and took very long as most websites change formatting and tags quite often. This required engineering just to maintain few sites all the time.

junilop 6y ago

found this lib in the comments, tried it and looks nice. can be good for finding the xpath automatically and save the manual traversing overhead. https://github.com/alirezamika/autoscraper https://github.com/alirezamika/autoscraper