I think they are referring to the data in its rawest format before they have indexed and ranked the information themselves. They will all crawl the information

I think they are referring to the data in its rawest format before they have indexed and ranked the information themselves. They will all crawl the information in exactly the same way. They will just take the plain text and store it. I don't think any bot would actually do anything else with the data on the fly. If you think about it, it does make sense in a lot of respects. I have dealt with a lot of companies that sell data, the only difference is this data is freely available to everyone so everyone thinks they should crawl the information themselves. The only people who lose out are the people paying the bandwidth bills. The internet would actually be slower due to the amount of information passing around when it is not needed. This idea makes more sense the more we discuss it