Give each document a unique identifier linked to a namespace (the domain or website-url) and let the crawlers do their work. A bit more optimizations here and

Give each document a unique identifier linked to a namespace (the domain or website-url) and let the crawlers do their work. A bit more optimizations here and there might be benefitial, like special support from browsers, a versionized linking-descriptopn of your namespace to allow local building of the real ressource-endpoint and such things.