Precisely this. Website operators who wish to establish effective paywalls with publicly indexed content need a way to cryptographically authenticate web crawle

Precisely this. Website operators who wish to establish effective paywalls with publicly indexed content need a way to cryptographically authenticate web crawlers. That means working with search engines to establish a standard for registering and authenticating authorized crawlers. If Google isn't interested in such a standard (because it reduces the quality of open search results) then that's the publishers' problem, not Google's or users'.