4 ms·

I think the overall software architecture approach of this post is fundamentally backwards. Given how much of the web is rendered client-side these days you nee

I think the overall software architecture approach of this post is fundamentally backwards. Given how much of the web is rendered client-side these days you need to start out with a headless option. Headless means that you fire up a true browser and then automate the actions that you need to perform. It's indistinguishable from a real person using a browser. If you try to use urllib3 on a webpage that does heavy client-side rendering then you're going to get incomplete HTML returned from the server (because the website intends to use JavaScript to complete the rendering of the page). On the more rare occasions when you are dealing with static HTML (i.e. there is no rendering on the client; the HTML returned from the server is the complete content) then you can use something like urllib3. Re: which headless library to use this post mentions Selenium which was one of the first headless libs but from what I've heard probably not the best (in terms of developer experience or reliability or robustness) but that's only hearsay... I've never used Selenium myself. Playwright seems like the best option in town if you want to use Python. Built by the former Chrome DevTools team (meaning those people really know how browser internals work). https://playwright.dev/python/docs/intro https://playwright.dev/python/docs/intro

datalopers 4y ago
Headless scraping quickly becomes a very expensive approach when you try and scale the effort. I only employ it when absolutely necessary. And it’s most definitely distinguishable by any modern (incapsula, perimeterx, cloudflare) WAF.
- apienx 4y ago
  The Apify library tries to address most of these issues (I'd say quite successfully). Apify.com provides a platform you can deploy the scrapers on. And yes, at scale. There's also a software marketplace where you can order custom scrapers. 98% of the projects ran thru it have a 5-star rating (Disclaimer: I moderate that marketplace). Pro-tip: submit your project with a Gmail address to skip sales and reach me directly.
  - pocket_cheese 4y ago
    A federated marketplace for scrapers is an idea I have thought considerably about. If you have time, I would love to chat to a) discuss being a paying user and b) to talk about the industry and see how we could submit some scrapers. Let me know how I can reach out to you!
lapser 4y ago
Or you could use the API that the web app has to inevitably use.
- chrsig 4y ago
  this is assuming that it's documented...otherwise you're just hand evaluating javascript to figure out what it would call...and then you get to thinking that you should just embed a javascript interpreter and evaluate it...and at that point, you've gone down the path of implementing a headless browser.
  - lapser 4y ago
    Not really. The API these days tends to be JSON so you can just figure out how to it works and what represents what. For example I've been able to reimplement xmltv scrapers for several sources in less than a 100 lines with Scrapy. It's not hard, just requires a little discretion.
    - chrsig 4y ago
      The difficulty isn't in making a scraper for a single site, but rather in the general case. That is, making a scraper that can be pointed at an arbitrary site not known at the time of development.
      - vertere 4y ago
        I assumed most people were talking about dealing with single sites. From your previous comment about API documentation and "hand evaluating" Javascript I gathered that you were too. How would those things help one solve the general case?
  - hombre_fatal 4y ago
    Almost everything uses simple JSON APIs which are far more trivial than html scraping. You also don't need to evaluate the Javascript to figure out what it's doing (something I can't even imagine doing, do you really do this? and where have you done it?), just browse the website normally with your network tab open and look at the endpoints and you're basically done. Obfuscated APIs like Pokemon GO and Netflix are in the tiny minority.
daolf 4y ago
We've decided to go this route because based on billions of web-scraped page, Headless-based scraping is still a minority. And, it's way harder and more expensive to do at scale.
elorant 4y ago
From my experience news sites are the one category that requires a headless browser the most. With e-commerce sites, or anything else, it's like 80% of the cases will work with a normal http request.
samwillis 4y ago
From my experience headless browser scraping is in the order of 100x slower due to increased bandwidth, cpu and memory. I would seriously suggest the other, start with traditional scraping and if you can’t make it work then go headless. The difference in speed is to the point that going a longer route (maybe having to scrape 10x more pages for example) to the content that enables you to not use headless will probably still end up quicker. On the memory side, headless you end up with far more memory leaks, having to manage stopping and starting new browser instances while maintaining scraping state. The devops overhead is probably 10x more with headless.
- taosx 4y ago
  Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects. Scraping becomes boring really fast if you don't use the data in meaningful ways.
- MonaroVXR 4y ago
  What is traditional?
- elorant 4y ago
  If you don't load images, and you disable 3rd party JavaScript you can save a lot of loading time with a headless. I'm using a preconfigured profile in Firefox with uMatrix installed and sites load at least 50% faster. Add another 20-30% from not loading images, and suddenly using a headless becomes affordable.
1vuio0pswjnm7 4y ago
"Given how much of the web is rendered client-side these days you need to start out with a headless option." What does "rendered client-side" mean. Assuming that "rendered client-side" means interpretation and execution of Javascript is necessary to read a site's textual content, then how much of the web is rendered client-side. If the focus is on textual content, e.g.,, someone is primarily "scraping" text as opposed to images and video, I would guess that only a minority of the web is "rendered client-side". How would we prove otherwise. This guess I am making would not be an uneducated one. I have been accessing the web without using Javascript for over 30 years. Today, I still use a text-only browser to render HTML as text/hypertext. This allows me to read the site's textual content, quickly and easily. I initiate most HTTP requests with TCP clients, not the browser. All requests, whether from TCP client, browser, or otherwise, are made through a localhost forward proxy. If most websites were truly dependent on Javascript, it stands to reason I would not be able to read much of the web. In other words, another web user who reads the web with a Javascript-enabled browser should be able to read websites that I could not read. This has not been the case. In fact, I often see commenters on HN complaining that they cannot read a site that I am having no trouble reading. The culprit is often Javascript. The truth is that I rarely encounter a site that cannot be read with the text-only browser. For example, I can read the content of almost every site submitted to HN. A very small minority of sites I find are, more or less, empty shells with links to some Javascripts but no textual content for the visitor to read. These "landing pages" expect a Javascript-enabled browser that automatically follows links in the page (e.g., to remote Javascript files), and that retrieves, interprets and executes Javascript automatically and indiscriminately.[FN1] In what some might see as a Rube Goldberg design pattern, the scripts then make HTTP requests to the "real" site. In such cases it generally only takes me a few minutes to find the "real" site, often what some refer to as a "JSON endpoint".[FN2] However this process has not lead me to rely on a "headless" browser to read websites. Honestly, if a majority of sites adopted the "JSON endpoint" approach to serving textual content it would make reading websites even easier for me. I could just retrieve JSON and reformat it to a uniform brand of simple HTML that I prefer, as I already do for some sites. I could make the format of all websites 100% identical. IME, a web of uniformly-formatted content is much easier and faster to digest. I would imagine it would easier for machines to digest as well. The text-only browser I use currently makes the format of all sites look almost the same, since it only uses a single font and so many websites use similar designs. Because it does not automatically follow links or execute Javascript, it also tends to make the "load" time of all sites very similar. For me, this uniformity speeds up the ability to digest web content as compared to using a graphical browser for the same purpose. FN1. Today we see "modern" browsers incorporating an ever-changing array of "features" and options to try to mitigate the risks of this behaviour. FN2. Generally, IME, these "endpoints" serve the textual content with minimal markup or sometimes no mark up at all. Thus, the end user is free to format the text into whatever design suits their personal tastes. As a website visitor, this is relatively more efficient IMO than trying to read an infinite number of possible "web designs" which is the approach we currently see on today's www. It is more predictable. With the later approach, visiting a new website with a Javascript-enabled, graphical browser is always a "surprise". It might be easy to read or it might not. Visiting "endpoints" generally does not suffer from this problem.

datalopers 4y ago

Headless scraping quickly becomes a very expensive approach when you try and scale the effort. I only employ it when absolutely necessary. And it’s most definitely distinguishable by any modern (incapsula, perimeterx, cloudflare) WAF.

apienx 4y ago
The Apify library tries to address most of these issues (I'd say quite successfully). Apify.com provides a platform you can deploy the scrapers on. And yes, at scale. There's also a software marketplace where you can order custom scrapers. 98% of the projects ran thru it have a 5-star rating (Disclaimer: I moderate that marketplace). Pro-tip: submit your project with a Gmail address to skip sales and reach me directly.
- pocket_cheese 4y ago
  A federated marketplace for scrapers is an idea I have thought considerably about. If you have time, I would love to chat to a) discuss being a paying user and b) to talk about the industry and see how we could submit some scrapers. Let me know how I can reach out to you!

lapser 4y ago

Or you could use the API that the web app has to inevitably use.

chrsig 4y ago
this is assuming that it's documented...otherwise you're just hand evaluating javascript to figure out what it would call...and then you get to thinking that you should just embed a javascript interpreter and evaluate it...and at that point, you've gone down the path of implementing a headless browser.
- lapser 4y ago
  Not really. The API these days tends to be JSON so you can just figure out how to it works and what represents what. For example I've been able to reimplement xmltv scrapers for several sources in less than a 100 lines with Scrapy. It's not hard, just requires a little discretion.
  - chrsig 4y ago
    The difficulty isn't in making a scraper for a single site, but rather in the general case. That is, making a scraper that can be pointed at an arbitrary site not known at the time of development.
    - vertere 4y ago
      I assumed most people were talking about dealing with single sites. From your previous comment about API documentation and "hand evaluating" Javascript I gathered that you were too. How would those things help one solve the general case?
- hombre_fatal 4y ago
  Almost everything uses simple JSON APIs which are far more trivial than html scraping. You also don't need to evaluate the Javascript to figure out what it's doing (something I can't even imagine doing, do you really do this? and where have you done it?), just browse the website normally with your network tab open and look at the endpoints and you're basically done. Obfuscated APIs like Pokemon GO and Netflix are in the tiny minority.

daolf 4y ago

We've decided to go this route because based on billions of web-scraped page, Headless-based scraping is still a minority. And, it's way harder and more expensive to do at scale.

elorant 4y ago

From my experience news sites are the one category that requires a headless browser the most. With e-commerce sites, or anything else, it's like 80% of the cases will work with a normal http request.

samwillis 4y ago

From my experience headless browser scraping is in the order of 100x slower due to increased bandwidth, cpu and memory. I would seriously suggest the other, start with traditional scraping and if you can’t make it work then go headless. The difference in speed is to the point that going a longer route (maybe having to scrape 10x more pages for example) to the content that enables you to not use headless will probably still end up quicker. On the memory side, headless you end up with far more memory leaks, having to manage stopping and starting new browser instances while maintaining scraping state. The devops overhead is probably 10x more with headless.

taosx 4y ago
Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects. Scraping becomes boring really fast if you don't use the data in meaningful ways.
MonaroVXR 4y ago
What is traditional?
elorant 4y ago
If you don't load images, and you disable 3rd party JavaScript you can save a lot of loading time with a headless. I'm using a preconfigured profile in Firefox with uMatrix installed and sites load at least 50% faster. Add another 20-30% from not loading images, and suddenly using a headless becomes affordable.

1vuio0pswjnm7 4y ago

"Given how much of the web is rendered client-side these days you need to start out with a headless option." What does "rendered client-side" mean. Assuming that "rendered client-side" means interpretation and execution of Javascript is necessary to read a site's textual content, then how much of the web is rendered client-side. If the focus is on textual content, e.g.,, someone is primarily "scraping" text as opposed to images and video, I would guess that only a minority of the web is "rendered client-side". How would we prove otherwise. This guess I am making would not be an uneducated one. I have been accessing the web without using Javascript for over 30 years. Today, I still use a text-only browser to render HTML as text/hypertext. This allows me to read the site's textual content, quickly and easily. I initiate most HTTP requests with TCP clients, not the browser. All requests, whether from TCP client, browser, or otherwise, are made through a localhost forward proxy. If most websites were truly dependent on Javascript, it stands to reason I would not be able to read much of the web. In other words, another web user who reads the web with a Javascript-enabled browser should be able to read websites that I could not read. This has not been the case. In fact, I often see commenters on HN complaining that they cannot read a site that I am having no trouble reading. The culprit is often Javascript. The truth is that I rarely encounter a site that cannot be read with the text-only browser. For example, I can read the content of almost every site submitted to HN. A very small minority of sites I find are, more or less, empty shells with links to some Javascripts but no textual content for the visitor to read. These "landing pages" expect a Javascript-enabled browser that automatically follows links in the page (e.g., to remote Javascript files), and that retrieves, interprets and executes Javascript automatically and indiscriminately.[FN1] In what some might see as a Rube Goldberg design pattern, the scripts then make HTTP requests to the "real" site. In such cases it generally only takes me a few minutes to find the "real" site, often what some refer to as a "JSON endpoint".[FN2] However this process has not lead me to rely on a "headless" browser to read websites. Honestly, if a majority of sites adopted the "JSON endpoint" approach to serving textual content it would make reading websites even easier for me. I could just retrieve JSON and reformat it to a uniform brand of simple HTML that I prefer, as I already do for some sites. I could make the format of all websites 100% identical. IME, a web of uniformly-formatted content is much easier and faster to digest. I would imagine it would easier for machines to digest as well. The text-only browser I use currently makes the format of all sites look almost the same, since it only uses a single font and so many websites use similar designs. Because it does not automatically follow links or execute Javascript, it also tends to make the "load" time of all sites very similar. For me, this uniformity speeds up the ability to digest web content as compared to using a graphical browser for the same purpose. FN1. Today we see "modern" browsers incorporating an ever-changing array of "features" and options to try to mitigate the risks of this behaviour. FN2. Generally, IME, these "endpoints" serve the textual content with minimal markup or sometimes no mark up at all. Thus, the end user is free to format the text into whatever design suits their personal tastes. As a website visitor, this is relatively more efficient IMO than trying to read an infinite number of possible "web designs" which is the approach we currently see on today's www. It is more predictable. With the later approach, visiting a new website with a Javascript-enabled, graphical browser is always a "surprise". It might be easy to read or it might not. Visiting "endpoints" generally does not suffer from this problem.