6 ms·

I have done a lot of scraping in the past. Cookies are a pain, this is a really elegant solution. Of course the biggest problem is that everything interesting i

I have done a lot of scraping in the past. Cookies are a pain, this is a really elegant solution. Of course the biggest problem is that everything interesting is hidden away behind JavaScript these days and then you have to resort to Selenium and the whole thing just spirals out of control. But I'm looking forward to giving this a shot for non-JavaScript content in the future. edit: JavaScript not Java

mkl 4y ago
Do you mean JavaScript? I have never run into content hidden by Java, but many pages load content dynamically using JavaScript. I have found it's quite easy to snoop on those JavaScript API requests using the Network tab of Chrome Devtools, then copy the network request as a curl command for bash scripts or as JavaScript for browser extensions.
- ghqst 4y ago
  Yeah, you can sometimes find the API or find data sent in JavaScript but not in prerendered HTML, which can save you the pain of headless scraping.
- juujian 4y ago
  I do mean JavaScript. Not sure how many times I have made that mistake... And great advice, that sounds like a neat approach.
- tomashubelbauer 4y ago
  > I have never run into content hidden by Java Tongue in cheek: You'd never know - servers running Java code generating HTML pages have probably conditionally not-rendered many pieces of HTML that you've never come across in your browsing :)
totetsu 4y ago
There are python libraries you can use that import cookies directly from wherever your browsers stores them to use in selenium projects.
berkle4455 4y ago
Javascript is delivered as text and sends text-based HTTP calls to the server to fetch more data. Why do you need selenium?
- rhd 4y ago
  I've once used Selenium to run javascript in the webpage to steal a few dynamic tokens required by the sites API to reuse in my more well-trodden python-requests workflow.
- LelouBil 4y ago
  if you don't want to reverse engineer the javascript
  - KomoD 4y ago
    Most of the time you don't need to, just open up devtools, look at the network tab, locate the right request(s).
deleted 4y ago
[deleted]
bdcravens 4y ago
If you'd be standing up CDP to grab the cookies, you'd probably use Puppeteer or Playwright instead of Selenium.
- juujian 4y ago
  Appreciate the recommendation, I just used whatever python had to offer, Puppeteer looks promising though!
  - bdcravens 4y ago
    Using the tools at hand is often the best approach. That said, I've spent most of the last 13 years of my career automating browsers. For years, I used Selenium with a variety of libraries. After switching to Puppeteer/Playwright, I have zero interest in going back lol. Playwright actually has first party Python support. (Puppeteer has a port called Pyppeteer, but it's no longer maintained and the author recommends using Playwright) https://playwright.dev/python/ https://playwright.dev/python/
    - rgrieselhuber 4y ago
      I second Playwright, it's amazing.
      - robertlagrant 4y ago
        Third.
1vuio0pswjnm7 4y ago
The term "everything interesting" is of course subjective. What is interesting to person A might not be interesting to person B. I never use Selenium and I generally have no problem acessing "everything interesting". The simplest example is reading and submitting HN comments. Presumably we all find this interesting enough. Javascript is neither required to read, vote nor submit to HN. What if the phrase "everything interesting" was replaced with specific examples and questions. Something like, "I cannot access X without Javascript. How do I access X without using Javascript."
- RockRobotRock 4y ago
  HN is in the minority of websites in that it works completely without JS. Surely you're aware of this, right?
  - 1vuio0pswjnm7 4y ago
    1. Define "works". 2. Provide examples of sites that do not "work". It's possible that people might disagree on the definition of "works". For example, perhaps web developers might be biased toward a definition that puts them in control instead of the user. If I can retrieve information from a server with HTTP requests then the website "works" for me. As a user, I certainly do not need to use Javascript to make HTTP requests. Nor do I need to use a particular client. One could argue that even HN does not "work" completely without Javascript. For example, the script at https://news.ycombinator.com/hn.js https://news.ycombinator.com/hn.js will not run.

mkl 4y ago

Do you mean JavaScript? I have never run into content hidden by Java, but many pages load content dynamically using JavaScript. I have found it's quite easy to snoop on those JavaScript API requests using the Network tab of Chrome Devtools, then copy the network request as a curl command for bash scripts or as JavaScript for browser extensions.

ghqst 4y ago
Yeah, you can sometimes find the API or find data sent in JavaScript but not in prerendered HTML, which can save you the pain of headless scraping.
juujian 4y ago
I do mean JavaScript. Not sure how many times I have made that mistake... And great advice, that sounds like a neat approach.
tomashubelbauer 4y ago
> I have never run into content hidden by Java Tongue in cheek: You'd never know - servers running Java code generating HTML pages have probably conditionally not-rendered many pieces of HTML that you've never come across in your browsing :)

totetsu 4y ago

There are python libraries you can use that import cookies directly from wherever your browsers stores them to use in selenium projects.

berkle4455 4y ago

Javascript is delivered as text and sends text-based HTTP calls to the server to fetch more data. Why do you need selenium?

rhd 4y ago
I've once used Selenium to run javascript in the webpage to steal a few dynamic tokens required by the sites API to reuse in my more well-trodden python-requests workflow.
LelouBil 4y ago
if you don't want to reverse engineer the javascript
- KomoD 4y ago
  Most of the time you don't need to, just open up devtools, look at the network tab, locate the right request(s).

deleted 4y ago

[deleted]

bdcravens 4y ago

If you'd be standing up CDP to grab the cookies, you'd probably use Puppeteer or Playwright instead of Selenium.

juujian 4y ago
Appreciate the recommendation, I just used whatever python had to offer, Puppeteer looks promising though!
- bdcravens 4y ago
  Using the tools at hand is often the best approach. That said, I've spent most of the last 13 years of my career automating browsers. For years, I used Selenium with a variety of libraries. After switching to Puppeteer/Playwright, I have zero interest in going back lol. Playwright actually has first party Python support. (Puppeteer has a port called Pyppeteer, but it's no longer maintained and the author recommends using Playwright) https://playwright.dev/python/ https://playwright.dev/python/
  - rgrieselhuber 4y ago
    I second Playwright, it's amazing.
    - robertlagrant 4y ago
      Third.

1vuio0pswjnm7 4y ago

The term "everything interesting" is of course subjective. What is interesting to person A might not be interesting to person B. I never use Selenium and I generally have no problem acessing "everything interesting". The simplest example is reading and submitting HN comments. Presumably we all find this interesting enough. Javascript is neither required to read, vote nor submit to HN. What if the phrase "everything interesting" was replaced with specific examples and questions. Something like, "I cannot access X without Javascript. How do I access X without using Javascript."

RockRobotRock 4y ago
HN is in the minority of websites in that it works completely without JS. Surely you're aware of this, right?
- 1vuio0pswjnm7 4y ago
  1. Define "works". 2. Provide examples of sites that do not "work". It's possible that people might disagree on the definition of "works". For example, perhaps web developers might be biased toward a definition that puts them in control instead of the user. If I can retrieve information from a server with HTTP requests then the website "works" for me. As a user, I certainly do not need to use Javascript to make HTTP requests. Nor do I need to use a particular client. One could argue that even HN does not "work" completely without Javascript. For example, the script at https://news.ycombinator.com/hn.js https://news.ycombinator.com/hn.js will not run.