3 ms·
I’ve been using this package and like it a lot. One problem I’d like to find a solution for is how to get past cookie pop ups when scraping a website. I’ve no
by d4rkp4ttern 3y ago
I’ve been using this package and like it a lot.
One problem I’d like to find a solution for is how to get past cookie pop ups when scraping a website.
I’ve not found a satisfactory packaged solution for this. Clearly a tough problem in general but wondered if people have found good libs to help with this. I’ve heard of solutions involving playwright etc.
- adbarba 3y agoThanks! Here is what I put together in the docs, you could basically preprocess/render/filter the webpages with the software of your choice and then pass the result to trafilatura: https://trafilatura.readthedocs.io/en/latest/troubleshooting.html#beyond-raw-html https://trafilatura.readthedocs.io/en/latest/troubleshooting...