Limewire had an interesting protocol for exactly this back in the day, over its Gnutella network. I don't know if it scales to search engine sizes (most things

Limewire had an interesting protocol for exactly this back in the day, over its Gnutella network. I don't know if it scales to search engine sizes (most things don't, especially distributed) but there's at least some prior production grade work in this space. Edit: having just double checked it looks like Gnutella predates limewire; that's just the avenue through which I came across it.