4 ms·
> is it selenium replacement No completely different use case. Selenium is browser automation. Mechanical soup/Mechanize/Robobrowser are not actually web brows
by IceWreck 5y ago
> is it selenium replacement
No completely different use case. Selenium is browser automation. Mechanical soup/Mechanize/Robobrowser are not actually web browsers, they have no javascript support either. They're python libraries that can simulate a web browser but doing GET requests, storing cookies across requests, filling http POST forms, etc.
The downside is that they don't work with websites which rely on JavaScript to load content. But if you're scraping a website like that, then it might be easier and way way faster to analyze web requests using dev tools or mitmproxy, then automating those API calls instead of automating a browser.
- gcatalfamo 5y agoI started reverse engineering web apis years ago as a more efficient way to scrape. Unfortunately there are new website builders (like salesforce lightning) that make really hard to recreate an http request due to the complexity of the parameters