There are not that many publishers and not all crawlers need access. My guess would be the publishers gave Google their own license with which a specific crawle

There are not that many publishers and not all crawlers need access. My guess would be the publishers gave Google their own license with which a specific crawler could identify itself.