3 ms·
Maybe I have this wrong but looking at academic torrents, the entire pushshift archive from 2005 to 2022 is 2TB compressed. Are they really operating at a massi
by nebula8804 3y ago
Maybe I have this wrong but looking at academic torrents, the entire pushshift archive from 2005 to 2022 is 2TB compressed. Are they really operating at a massive scale that you are describing? Pushshift is supposed to have ingested all of Reddit's comment data no?
(Of course this is assuming pushsift has gotten the majority of comments)
- notacoward 3y agoWell, let me put it this way: if they're operating at such a small scale, they have no reason to be using any eventually consistent data store, so explanations about missing shards etc. still don't work. There are still other possibilities, as I and others have outlined, but not enough to dismiss the possibility that spez is just up to his old tricks.