3 ms·

Why no mention of selenium? Is it not cool anymore? I have never heard of mechanicalsoup: is it selenium replacement?

by gcatalfamo 5y ago

nicoburns 5y ago
Selenium is famously unreliable, so a lot of people have been replacing it with headless chrome where they can.
- melomal 5y ago
  Interesting. I was about to start on some web automation and so far I've had hammered into my head that Selenium is the 'language of the internet' or something along those lines. What would be a better solution, if you have any to recommend?
  - xzel 5y ago
    I'd suggest Puppeteer / Playwright. Both are great. Iirc the puppeteer team largely moved to playwright.
    - kjkjadksj 5y ago
      Puppeteer is frustrating to me. When I tried to use it I couldn’t get it to click buttons, but I did get it to hover on the button so I know I had the correct element in my code. Their click function just did nothing at all. I resorted to tabbing a certain amount of times and hitting enter.
    - melomal 5y ago
      Thank you for the suggestions! I will check them out.
- gcatalfamo 5y ago
  I have been using selenium with chromedriver, I mistakenly thought those were basically the same thing. Can you tell me more?
IceWreck 5y ago
> is it selenium replacement No completely different use case. Selenium is browser automation. Mechanical soup/Mechanize/Robobrowser are not actually web browsers, they have no javascript support either. They're python libraries that can simulate a web browser but doing GET requests, storing cookies across requests, filling http POST forms, etc. The downside is that they don't work with websites which rely on JavaScript to load content. But if you're scraping a website like that, then it might be easier and way way faster to analyze web requests using dev tools or mitmproxy, then automating those API calls instead of automating a browser.
- gcatalfamo 5y ago
  I started reverse engineering web apis years ago as a more efficient way to scrape. Unfortunately there are new website builders (like salesforce lightning) that make really hard to recreate an http request due to the complexity of the parameters
duckmysick 5y ago
I moved from selenium to playwright. It has a pleasant API and things just works our of box. I ran into odd problems with selenium before, especially when waiting for a specific element. Selenium didn't register it, but I could see it load. It was uncharacteristic of me, because I tend to use boring, older technologies. But this gamble paid off for me. https://playwright.dev/ https://playwright.dev/

Why no mention of selenium? Is it not cool anymore? I have never heard of mechanicalsoup: is it selenium replacement?

nicoburns 5y ago

Selenium is famously unreliable, so a lot of people have been replacing it with headless chrome where they can.

melomal 5y ago
Interesting. I was about to start on some web automation and so far I've had hammered into my head that Selenium is the 'language of the internet' or something along those lines. What would be a better solution, if you have any to recommend?
- xzel 5y ago
  I'd suggest Puppeteer / Playwright. Both are great. Iirc the puppeteer team largely moved to playwright.
  - kjkjadksj 5y ago
    Puppeteer is frustrating to me. When I tried to use it I couldn’t get it to click buttons, but I did get it to hover on the button so I know I had the correct element in my code. Their click function just did nothing at all. I resorted to tabbing a certain amount of times and hitting enter.
  - melomal 5y ago
    Thank you for the suggestions! I will check them out.
gcatalfamo 5y ago
I have been using selenium with chromedriver, I mistakenly thought those were basically the same thing. Can you tell me more?

IceWreck 5y ago

> is it selenium replacement No completely different use case. Selenium is browser automation. Mechanical soup/Mechanize/Robobrowser are not actually web browsers, they have no javascript support either. They're python libraries that can simulate a web browser but doing GET requests, storing cookies across requests, filling http POST forms, etc. The downside is that they don't work with websites which rely on JavaScript to load content. But if you're scraping a website like that, then it might be easier and way way faster to analyze web requests using dev tools or mitmproxy, then automating those API calls instead of automating a browser.

gcatalfamo 5y ago
I started reverse engineering web apis years ago as a more efficient way to scrape. Unfortunately there are new website builders (like salesforce lightning) that make really hard to recreate an http request due to the complexity of the parameters

duckmysick 5y ago

I moved from selenium to playwright. It has a pleasant API and things just works our of box. I ran into odd problems with selenium before, especially when waiting for a specific element. Selenium didn't register it, but I could see it load. It was uncharacteristic of me, because I tend to use boring, older technologies. But this gamble paid off for me. https://playwright.dev/ https://playwright.dev/