5 ms·

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear r

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots. Using a "real browser" is often advised as a way around these issues. However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found: - Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources. - Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want. - Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.

londons_explore 3y ago
Headless browsers are expensive, but if you needed them, I suspect there are easy wins to be had on the performance front. For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.
marginalia_nu 3y ago
Depends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views. I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.
- abhibeckert 3y ago
  The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the content of the page. And they execute the click event handlers on every link/button and when we use "history.pushState()" to change the URL Google considers that a new page. You're just going to get a loading spinner with no content if you do a dumb crawl (I disagree with that and think we should be running a headless browser server side to execute javascript and generate the initial page content for all our pages... but so far management hasn't prioritised that change... instead they just keep telling us to make our client side javascript run faster... imagine if there was no javascript to execute at all? At least none before first contentfull paint)
  - marginalia_nu 3y ago
    > The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which. I don't personally attempt to crawl SPAs because it's not the sort of content I want to index.
    - r3trohack3r 3y ago
      I have a pet theory that there are two forms of the web: the document web and the application web. SPAs have some very attractive properties for the application web but complicate/break the document web. That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Content can be user submitted in-browser. Content can be “engaged with” in browser. Some handle this blurring better than others. HN is an example IMO of getting it right where nearly everything that should be addressable (like comments) can be linked to. Others not so much. (As an aside, I love marginalia!)
ricardo81 3y ago
The issues you list can probably be split into 2 main issues - The host has a tendency to block unknown user agents, or user agents that claim to be a browser but are not - Anything that requires client side rendering I'd suppose both problems are more pertinent in 2023 than they were in 2012. At web scale, the issue appears on at what point would you be required to use a headless browser when not using one in the first place e.g. if React is included/referenced. Perhaps some simple fingerprinting of JS files would do, IMO in reality the line is very blurry, so you either do or don't.
jaymzcampbell 3y ago
On the topic of JSON and whatnot that reminded me of this awesome post[1] about looking within heap snapshots for entire data structures where they may have a lot of nicely structured data within it not readily available from more direct URL calls. [1] https://www.adriancooney.ie/blog/web-scraping-via-javascript-heap-snapshots https://www.adriancooney.ie/blog/web-scraping-via-javascript... (Previous comments about it from 2022 here: https://news.ycombinator.com/item?id=31205139 https://news.ycombinator.com/item?id=31205139)
arek_nawo 3y ago
Which way to go will depend on your use-case and what websites you want to scrape. For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need. For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.
bluedino 3y ago
> Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources. Instead of setting up some kind of partnership with our vendors, where they just send us information or provide an API, we scrape their websites. The old version ran in a hour, using one thread on one machine. Downloaded PDF's and extracted the values. The new version is Selenium based, uses 20 cores, 300GB of memory, and takes all night to run. It does the same thing, but from inside of a browser. As a bonus, the 'web scrapers' were blamed for every performance issue we had for a long time.
tempest_ 3y ago
Site owners adding browser fingerprinting explicitly is vanishingly rare however site owners who sit behind cloud flare who very likely fingerprint browsers is very common.
moneywoes 3y ago
Is there like a custom performance based headless browser How are they scaled?
cacoos 3y ago
I've always wondered why not to use request interceptors, get the html/json/xml/whatever url and then just call them. If you need cookies/headers, you can always open the browser, log in and then make the same requests in the console, instead of waiting for the browser to load and scrape the UI (by xpath, etc.) sounds weird going in circles: - SPA call some URL - SPA use the response data to populate the UI - You scrape the UI instead of just calling the url inside the browser? am i missing something?
rocho 3y ago
It also depends on what the goal is. If you need to extract some data from a specific site (as opposed to obtaining faithful representations of pages from arbitrary sites like a search engine might need to), then SPAs might be the easiest targets of all. Just do some inspection beforehand to find out where the SPA is loading the data from, very often the easiest thing is to query the API directly. Sometimes the API will need some form of session tokens found in HTTP meta tags, form inputs, or cookies. Sometimes they have some CORS checks in place that are very easy to bypass from a server-side environment by spoofing the Origin header.
Nakroma 3y ago
I do a lot of scraping at my day job, and I agree the times we need to use a headless browser are very rare. Vast majority of the time you can either find an API or JSON endpoint, or scrape the JS contained in the returned SSR document to get what you want.

londons_explore 3y ago

Headless browsers are expensive, but if you needed them, I suspect there are easy wins to be had on the performance front. For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.

marginalia_nu 3y ago

Depends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views. I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.

abhibeckert 3y ago
The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the content of the page. And they execute the click event handlers on every link/button and when we use "history.pushState()" to change the URL Google considers that a new page. You're just going to get a loading spinner with no content if you do a dumb crawl (I disagree with that and think we should be running a headless browser server side to execute javascript and generate the initial page content for all our pages... but so far management hasn't prioritised that change... instead they just keep telling us to make our client side javascript run faster... imagine if there was no javascript to execute at all? At least none before first contentfull paint)
- marginalia_nu 3y ago
  > The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which. I don't personally attempt to crawl SPAs because it's not the sort of content I want to index.
  - r3trohack3r 3y ago
    I have a pet theory that there are two forms of the web: the document web and the application web. SPAs have some very attractive properties for the application web but complicate/break the document web. That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Content can be user submitted in-browser. Content can be “engaged with” in browser. Some handle this blurring better than others. HN is an example IMO of getting it right where nearly everything that should be addressable (like comments) can be linked to. Others not so much. (As an aside, I love marginalia!)

ricardo81 3y ago

The issues you list can probably be split into 2 main issues - The host has a tendency to block unknown user agents, or user agents that claim to be a browser but are not - Anything that requires client side rendering I'd suppose both problems are more pertinent in 2023 than they were in 2012. At web scale, the issue appears on at what point would you be required to use a headless browser when not using one in the first place e.g. if React is included/referenced. Perhaps some simple fingerprinting of JS files would do, IMO in reality the line is very blurry, so you either do or don't.

jaymzcampbell 3y ago

On the topic of JSON and whatnot that reminded me of this awesome post[1] about looking within heap snapshots for entire data structures where they may have a lot of nicely structured data within it not readily available from more direct URL calls. [1] https://www.adriancooney.ie/blog/web-scraping-via-javascript-heap-snapshots https://www.adriancooney.ie/blog/web-scraping-via-javascript... (Previous comments about it from 2022 here: https://news.ycombinator.com/item?id=31205139 https://news.ycombinator.com/item?id=31205139)

arek_nawo 3y ago

Which way to go will depend on your use-case and what websites you want to scrape. For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need. For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.

bluedino 3y ago

> Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources. Instead of setting up some kind of partnership with our vendors, where they just send us information or provide an API, we scrape their websites. The old version ran in a hour, using one thread on one machine. Downloaded PDF's and extracted the values. The new version is Selenium based, uses 20 cores, 300GB of memory, and takes all night to run. It does the same thing, but from inside of a browser. As a bonus, the 'web scrapers' were blamed for every performance issue we had for a long time.

tempest_ 3y ago

Site owners adding browser fingerprinting explicitly is vanishingly rare however site owners who sit behind cloud flare who very likely fingerprint browsers is very common.

moneywoes 3y ago

Is there like a custom performance based headless browser How are they scaled?

cacoos 3y ago

I've always wondered why not to use request interceptors, get the html/json/xml/whatever url and then just call them. If you need cookies/headers, you can always open the browser, log in and then make the same requests in the console, instead of waiting for the browser to load and scrape the UI (by xpath, etc.) sounds weird going in circles: - SPA call some URL - SPA use the response data to populate the UI - You scrape the UI instead of just calling the url inside the browser? am i missing something?

rocho 3y ago

It also depends on what the goal is. If you need to extract some data from a specific site (as opposed to obtaining faithful representations of pages from arbitrary sites like a search engine might need to), then SPAs might be the easiest targets of all. Just do some inspection beforehand to find out where the SPA is loading the data from, very often the easiest thing is to query the API directly. Sometimes the API will need some form of session tokens found in HTTP meta tags, form inputs, or cookies. Sometimes they have some CORS checks in place that are very easy to bypass from a server-side environment by spoofing the Origin header.

Nakroma 3y ago

I do a lot of scraping at my day job, and I agree the times we need to use a headless browser are very rare. Vast majority of the time you can either find an API or JSON endpoint, or scrape the JS contained in the returned SSR document to get what you want.