5 ms·
I think some crawlers run JS, because a lot of the web simply won't work without JS to initialise the page state these days. You can use captcha or similar, on
by hnick 3y ago
I think some crawlers run JS, because a lot of the web simply won't work without JS to initialise the page state these days.
You can use captcha or similar, one workaround I've seen has a submit that is hidden so never clicked by real people then a visible submit that sets a hidden input and clicks the other one which requires the hidden input... not foolproof but avoids some accidents.
- 8organicbits 3y agoA crawler that follows links found in emails and sends POST requests / submits forms will cause so much havoc. It could buy things, validate account sign-ups, delete data, etc. I have a hard time believing the answer isn't to ask users to switch to a normal email provider.
- hnick 3y agoI know web search at least runs JS to get better results. Not sure about email pre-fetch but I assume they do. I don't think crawlers click buttons though unless they are malicious so it's probably fine for unique email links.