I'd argue that paywalled sites, Captcha-guarded pages, things like that are second-class content and should be treated as such. If somebody wants to dig deeper

I'd argue that paywalled sites, Captcha-guarded pages, things like that are second-class content and should be treated as such. If somebody wants to dig deeper and see such results, that should be an option, but as you said, this is a tough problem to solve satisfactorily and the truth lies somewhere inbetween. An obstacle to this happening right now is that Google doesn't want to shake the advertising tree too hard. Their business model is spammy so they tolerate spamminess. That's the hardest problem of all, and one I don't think anybody's close to solving yet: How do you monetize web search without automatically wrapping results in ads? But some bright person will come up with an answer and hopefully defeat this huge conflict of interest.