3 ms·
Git was already mentioned as an example, but there is a more generic answer to your question. Look up https://en.wikipedia.org/wiki/Content-addressable_storage
by bostik 8y ago
Git was already mentioned as an example, but there is a more generic answer to your question. Look up https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage ; the design is basically a DHT. Key collisions are standard feature in any hash table.
In any hash table, collisions are unavoidable. In a distributed KV store, the collision resolution might be done at a higher level, so any lookup reads all the values for a given key. The reason is simple: opportunistic caching. Sometimes you get lucky and are asked for two or more items that hash to the same key within cache expiration window. If you can avoid a read just 0.1% of the time, that's an improvement on tail latency AND it just left capacity available for other clients to do their reads.
We are all aware Spotify stores a lot of media. I know nothing about their internal architecture, but they might use hash of the track metadata as its content key. If assumptions (or "happy optimisations") about their size distribution don't hold, some extra tracks could end up getting through.