4 ms·
Funny, to me this sounds like a hash collision. (For a map key, not the cryptographic kind.) Knowing only that Spotify is a large distributed system, and seein
by bostik 8y ago
Funny, to me this sounds like a hash collision. (For a map key, not the cryptographic kind.)
Knowing only that Spotify is a large distributed system, and seeing that the mystery tracks are short in title and length, as well as plentiful, it feels like a these tracks are ending up as collisions in a distributed hash table.
From there it'd be a matter of bubbling up top due to sorting order properties.
- Illniyar 8y agoAre you suggesting that spotify is somewherre using a hash-table without a collision resolution method (like a linked list). Is that something that is done in some industries? Hash collisions to me has always been a performance problem not a correctness one.
- nkozyra 8y agoI could be wrong but the suggestion seems to be that plays are hashed before storage and the tracks exploit the hash algorithm in use. That, to me, seems far less likely than simple account compromise. Hash collisions have always been a security threat vector.
- bostik 8y agoGoodness me, no. They could be fetching all values for a given key (for performance reasons, since the seek for a read is surprisingly expensive). Depending on how they then use that data, these short and plentiful tracks can end up being included instead of dropped.
- Illniyar 8y agoSorry if I'm being obtuse but why would they be using a hash as a key instead of a uuid? I've never heard of any distributed system (or any system) that use hash as a key. Or are you saying there is somekind of auxiliary hash table in use for performance somewhere?
- kingosticks 8y agoIs it not fair to say that git is a distributed system using a hash as a key?
- Illniyar 8y agoI suppose it is. Though it's not the sort of systems I meant in my reply. Never even crossed my mind. P2P networks (like torrents) could probably also count as ones.
- bostik 8y agoGit was already mentioned as an example, but there is a more generic answer to your question. Look up https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage ; the design is basically a DHT. Key collisions are standard feature in any hash table. In any hash table, collisions are unavoidable. In a distributed KV store, the collision resolution might be done at a higher level, so any lookup reads all the values for a given key. The reason is simple: opportunistic caching. Sometimes you get lucky and are asked for two or more items that hash to the same key within cache expiration window. If you can avoid a read just 0.1% of the time, that's an improvement on tail latency AND it just left capacity available for other clients to do their reads. We are all aware Spotify stores a lot of media. I know nothing about their internal architecture, but they might use hash of the track metadata as its content key. If assumptions (or "happy optimisations") about their size distribution don't hold, some extra tracks could end up getting through.
- flurdy 8y agoDefinite hash collision, or similar id mismatch. And being included in more aggregated cohort data of suggested music by being overplayed. I recently tried to play a track and it kept playing a completely different artist and track instead every time I clicked on it. I moaned on Twitter [1] and Spotify replied that the Linux client is not supported... Which is true, just not very helpful, when all I tried was to help them. [1] https://twitter.com/flurdy/status/1072134634828390400 https://twitter.com/flurdy/status/1072134634828390400
- rv-de 8y agoIf it sounds to like that then listen closer as you will hear nothing but silence for those songs do not exist.
- self 8y agoIt can't be a hash collision if the mysterious tracks/artists are no longer on Spotify. If Spotify deleted the artists or tracks instead of giving them new ids, that suggests something else is going on.