4 ms·
I've worked on many eventually consistent systems, up to exascale at one of the companies you mention, and that explanation just seems extremely unlikely. In a
by notacoward 3y ago
I've worked on many eventually consistent systems, up to exascale at one of the companies you mention, and that explanation just seems extremely unlikely. In a system using tombstones, those should (and generally will) go the same place as the originals so "missing shard" doesn't work. Over that time scale there should then have been multiple rounds of compaction, including those forced by disk or server replacements, which would have also purged that deleted/overwritten data. Nobody at that scale can afford to let uncompacted garbage accumulate forever. Therefore, if not tombstones, they must be deliberately keeping old versions, but in that case the issue just moves to the index and doesn't really change much. The only way I can see this happening is if there was an unrecoverable data loss that required restoring from backup. Also, let's not forget that true deletion is sometimes required under various regulatory systems. I know that at least two of the companies you mention have systems specifically dedicated to satisfying those requirements.
It's still possible that this is innocent(ish), but at low enough probability that malice really does seem like a compelling alternative explanation. After all, spez has been caught doing something very similar before. There's proof of malicious nature, not just an assumption. When a component in a system has a certain known behavior or limit, you don't just discard that knowledge due to some handy rule of thumb.
- nebula8804 3y agoMaybe I have this wrong but looking at academic torrents, the entire pushshift archive from 2005 to 2022 is 2TB compressed. Are they really operating at a massive scale that you are describing? Pushshift is supposed to have ingested all of Reddit's comment data no? (Of course this is assuming pushsift has gotten the majority of comments)
- notacoward 3y agoWell, let me put it this way: if they're operating at such a small scale, they have no reason to be using any eventually consistent data store, so explanations about missing shards etc. still don't work. There are still other possibilities, as I and others have outlined, but not enough to dismiss the possibility that spez is just up to his old tricks.