3 ms·
Yeah, I ran into a site a while back that was clever enough to spot that I wasn't using a browser. So I did, Selenium allows you to automate a real browser and
by noir_lord 11y ago
Yeah, I ran into a site a while back that was clever enough to spot that I wasn't using a browser.
So I did, Selenium allows you to automate a real browser and capture the responses, hell if need be you can create a fake profile with Chrome and make it completely indistinguishable.
EDIT: explanation, it was a video tutorial site for one of the technologies I use, each video had a download link but it had no way to batch downloads for offline use, scraping it with normal tools didn't work since it was doing UA and other sniffing so I whipped up a python script to control chrome to authenticate, sign in get the cookie and then pulled the unique to that session download link for each video, since I'm not a dirtbag I set the time between downloads to 30 minutes (average video is 15 minutes) and left it running for 24 hours to get the ones I wanted.
- WalterGR 11y agoSelenium allows you to automate a real browser and capture the responses, hell if need be you can create a fake profile with Chrome and make it completely indistinguishable. Are there no headless browsers that let you accomplish the same thing? Honest question... I've used Selenium, but I just assumed that headless browsers were exactly: real browsers minus the UI.
- noir_lord 11y agoQuite possibly but I use selenium extensively during dev to automate boring tasks (filling in forms over and over is boring) as well as for integration and functional testing. So I had a hammer and the problem looked like a nail. ;)
- user_0001 11y agoheadless browsers are easy to identify. The best you have to not be ID'd and "kinda headless" is slimer or selenium to XVFB, which aren't really headless but sort of [edit] slimer [AND] selenium to [OR]