4 ms·
With the custom headers, you can actually trick a lot of sites with bot protection to let you load their sites (even big sites like youtube, which I have found
by jpyles 1y ago
With the custom headers, you can actually trick a lot of sites with bot protection to let you load their sites (even big sites like youtube, which I have found success in)
- dotancohen 1y agoHow do you work around pop-ups for newsletters and such? Look at the BBC for a good example.
- anxman 1y agoPack ad blockers into your containers. They can be loaded into Chrome and help immensely in suppressing popovers while crawling.
- dotancohen 1y agoThank you, I'll experiment with that. Tips and advice welcome!
- tough 1y agoAnother cool trick is to deny all the content types you don't care about in your playwright. so if you only want text why bother allowing requests for fonts, css, svgs, images, videos, etc Just request the html and cap down all the other stuff PS: I also think this has the nice side-effect of you consuming less resources (that you didnt care about/need anyways) from the server, so win win
- dotancohen 1y agoThat is a great tip, thank you!