5 ms·

How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage w

How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)

werds 4y ago
its not useful in those cases, but usually for those js rendered sites you can replicate the ajax requests which happen and get nicely formed json documents to parse through instead.
- squaresmile 4y ago
  Or the data is stored in js objects within script tags in the html and can be extracted programmatically. It's getting common with SSG sites using SPA frameworks. For example, the new Google Play Store website stores the data in AF_initDataCallback calls and can be extracted with re.findall(r"<script nonce=\"\S+\">AF_initDataCallback\((.*?)\);", html_string).
- phone8675309 4y ago
  I used to do that when I was responsible for a set of web crawlers to extract public records data, but the problem is that changes happen and these sorts of things become out of date fairly quickly. Getting this working in a headless browser driven by Selenium would probably be easier for maintainability.
- 867-5309 4y ago
  nowadays you usually have to submit http headers and cookies too, that's always a fun process of elimination
marginalia_nu 4y ago
Easiest in that case is probably to use something like headless chrome. But that is also significantly more demanding in terms of resources.
radus 4y ago
In those cases you might want to check out SeleniumBase: https://seleniumbase.io/ https://seleniumbase.io/
deleted 4y ago
[deleted]
sergiotapia 4y ago
If you're lucky those sites have the raw data a server side generated JSON payload right in the site source code markup. For example Target is clientside, but has all the data in a `window.FOOBAR = json` variable you can fetch and parse with some substring magic. Much easier than spinning up chromedriver and some package.

werds 4y ago

its not useful in those cases, but usually for those js rendered sites you can replicate the ajax requests which happen and get nicely formed json documents to parse through instead.

squaresmile 4y ago
Or the data is stored in js objects within script tags in the html and can be extracted programmatically. It's getting common with SSG sites using SPA frameworks. For example, the new Google Play Store website stores the data in AF_initDataCallback calls and can be extracted with re.findall(r"<script nonce=\"\S+\">AF_initDataCallback\((.*?)\);", html_string).
phone8675309 4y ago
I used to do that when I was responsible for a set of web crawlers to extract public records data, but the problem is that changes happen and these sorts of things become out of date fairly quickly. Getting this working in a headless browser driven by Selenium would probably be easier for maintainability.
867-5309 4y ago
nowadays you usually have to submit http headers and cookies too, that's always a fun process of elimination

marginalia_nu 4y ago

Easiest in that case is probably to use something like headless chrome. But that is also significantly more demanding in terms of resources.

radus 4y ago

In those cases you might want to check out SeleniumBase: https://seleniumbase.io/ https://seleniumbase.io/

deleted 4y ago

[deleted]

sergiotapia 4y ago

If you're lucky those sites have the raw data a server side generated JSON payload right in the site source code markup. For example Target is clientside, but has all the data in a `window.FOOBAR = json` variable you can fetch and parse with some substring magic. Much easier than spinning up chromedriver and some package.