25 ms·
I very briefly looked into extensions for some things I wanted to do, but got kind of lost in the documentation. The browsers seem to expose a very large API an
by tzs 4y ago
I very briefly looked into extensions for some things I wanted to do, but got kind of lost in the documentation. The browsers seem to expose a very large API and I failed to figure out if it supports what I wanted.
Essentially what I wanted was to be able to implement a simplified version of the "wget" command. I wanted to be able to invoke the extension on a page and have it save the current page contents to a file, then navigate to each link on the page that doesn't go offsite and save those pages, and so on to a specified depth.
(Why not just use wget itself from my terminal? Because the pages I was interested in have JavaScript that modifies the DOM, and it is that modified DOM I want).
- geraldwhen 4y agoThis will take you less than 30 minutes with playwright and or puppeteer. A browser extension probably won’t work and would take at least 10x as long to write.
- tzs 4y agoI've tried something similar using Selenium. It worked for a while but then started failing at a site that was using Cloudflare's CAPTCHA. If my browser was launched by Selenium it would end up in a seemingly endless CAPTCHA loop. It failed using both Chromium and Firefox. I was able to get it working again for a while with some experimental option settings in the Chromium driver which apparently made it harder for a site to figure out that some sort of browser automation was in use, but maybe a couple months later that broke too. I found a Chromium driver that was specifically made to not be detectable. That didn't help. I just gave Puppeteer a try, with Chromium and Firefox, and they both got similar CAPTCHA loops. With Playwright same thing with Chromium and Webkit. With Firefox, to my surprise, it actually worked. In fact I wasn't even shown the CAPTCHA. The checking for a human passed without requiring interaction. So...looks like Playwright with Firefox might do the trick, but I'm a bit concerned about how reliable that would be long term. That's one of the reasons I was thinking about an extension--most people using a site are going to have extensions so they aren't going to be able to use the existence of extensions as a sign that a visitor is a bot. A lot fewer normal users of a site will be using web testing automation frameworks, and so if the site can detect those it might trigger anti-bot measures. I wonder if Cloudflare not being bothered by Playwright in Firefox is due to something Playwright is specifically doing to be undetectable, or just something Cloudflare hasn't gotten around to acting on yet? (I assume it is nothing Firefox specifically did, because of Firefox getting stuck in the CAPTCHA loop under Puppeteer and Selenium).
- pwdisswordfish9 4y agoYou don't need an extension to do that. You can get by with a bookmarklet, which would be wise, considering how much contempt browser makers have for extension authors.
- tzs 4y agoWait...a bookmarklet can do stuff on multiple pages? I thought that if I had a bookmarklet and invoked it on a page, and the bookmarklet navigated to a different page the running instance of that bookmarklet would go away.
- pwdisswordfish9 4y agoA bookmarklet is more or less just a JS expression that executes on command (the user's, that is). The execution model is approximately the same as for content-delivered scripts, and it's subject to the same constraints. So, yes, if you just naively write a bookmarklet that navigates to a new page with e.g. assignment to window.location and then expect any result other than the next line of code not executing, then you're going to be disappointed. You solve this the same way you'd solve it if you were writing an ordinary Web app--implemented in JS delivered by the server with script elements on your own page. Two stupid easy solutions that immediately come to mind: use XHR/fetch instead of actual page navigation; alternatively, have the bookmarklet open up a ~postage stamp-sized window with window.open that you can use both to output visible diagnostics and to keep the crawler resident (by doing all the work in the diagnostic window's context, which uses window.opener to control the initial tab as its puppet)... etc.