Beside of having more and more ads in the search results (which I think mostly driven by the pandemic because of significant revenue loss in 2020 Q2?), I think

Beside of having more and more ads in the search results (which I think mostly driven by the pandemic because of significant revenue loss in 2020 Q2?), I think the web itself is the most significant offender of this problem. Not only those annoying SEO (which makes the problem exponentially harder), now more and more quality contents are going beyond unsearchable walled gardens. Basically almost all web sites from mid-sized tech companies now require some sort of sign-in by default and if you try to crawl it at scale, you'll get banned. Perhaps Twitter is the only significant exception to this trend.