Good to know! Yes, trafilatura is great, sure it breaks sometimes, but everything breaks on some website - the real questions are how often and what is the exte

Good to know! Yes, trafilatura is great, sure it breaks sometimes, but everything breaks on some website - the real questions are how often and what is the extent of breakage. For general info., the library was published about here [1], where in Table 1 they provide some benchmarks. I also forgot to mention another interesting scraper that's an LLM based service. A quick search here tells me it was mentioned once by simonw, but I think it should be better known just for the convenience! Prepend "r.jina.ai" to any URL to extract text. For ex., check out [2] or [3]. [1] https://aclanthology.org/2021.acl-demo.15.pdf https://aclanthology.org/2021.acl-demo.15.pdf [2] https://r.jina.ai/news.ycombinator.com/ https://r.jina.ai/news.ycombinator.com/ [3] (this discussion) https://r.jina.ai/news.ycombinator.com/item?id=41428274 https://r.jina.ai/news.ycombinator.com/item?id=41428274