4 ms·

Hanlon's razor. Anyone who has used Reddit for any period of time (or basically any similar site, which includes FB, Twitter, and even HN), has seen data consi

Hanlon's razor. Anyone who has used Reddit for any period of time (or basically any similar site, which includes FB, Twitter, and even HN), has seen data consistency issues during high load (like periods of activism). Shards die or fall out of quorum, work queues get purged, etc. That's the reality of systems with a lot of very low value data. Eventually consistent, but also perpetually inconsistent.

goykasi 3y ago
Yes, I have a strong love/hate relationship with multi-DC Cassandra.
willis936 3y ago
Does Hanlon's razor apply when the actor has openly stated malicious intent?
MichaelZuo 3y ago
Which is a bit surprising since nowadays an entire reddit instance could fit on a single server rack.
seanalltogether 3y ago
Yeah people are going into overdrive right now with speculating all the bad things reddit is doing. Someone claimed that reddit admins had taken over their local city's subreddit to install new mods, as if that was a priority for them right now.
hospitalJail 3y ago
Its not a shock to anyone who knows how cheap data is, but it just proves there is no true 'delete'. I never deleted my facebook profile/pictures because I didn't want to shine a spotlight on my account. I care more about what a 'deleted' post says, than any other post in a thread. I imagine there is a good reason for that.
- 4BynHmBFUxBD 3y ago
  Same here. I also thought if I deleted my FB account, they would just maintain a “shadow profile” for me.
pontiacbandit8 3y ago
Data consistency on comments n years old?
- llm_nerd 3y ago
  Unless someone has invented a data system that includes time travel as a feature, the original date of the comments is irrelevant.
jacksnipe 3y ago
Unlike every other data operation, deletion is heavily regulated by law.
- llm_nerd 3y ago
  Sure. I can't wait to see all the lawsuits and fines overwhelming Reddit, and any other service that have occasional data consistency issues. This is heavily regulated by law, of course. Of course this isn't "heavily regulated". No law demands 100% consistency when you run some third party script during an outage. Clicking "delete" on a comment, or having a script do it, is not some GPDR enforced action.
  - jacksnipe 3y ago
    You are incredibly wrong about this. Consent for data processing is the meat & potatoes of both GDPR and the largest enforcement actions that have happened through it. The delete button revoked consent.
aprentic 3y ago
That would be a reasonable hypothesis if our information was limited to a few isolated inconsistency issues. The more posts we see deleted the less likely it is that it's an accident. I just checked and found out that posts from an account that I scrubbed and deleted 5 years ago are popping back up. That stretches "eventual" pretty thin. We also have a large amount of external information that suggests that reddit is not acting in good faith, much of it from Spez himself. Between the two of those, malice is more likely than stupidity to be the motivating factor.
crazygringo 3y ago
That would easily explain comments coming back that were hours old or even days old, but it's extremely unlikely to explain comments coming back that were months or years old, as others in this thread have found. No database shard would ever have been left that far out of date due to high load. The only Hanlon's razor I can imagine here is somebody wrote a query to try to undelete something small (intending a single user or something) and forgot the WHERE clause and started to undelete everything. And didn't realize until the query had been executing for several minutes. Yikes. (I do actually think this is the most likely, not because of spez's motivations, but because there's zero benefit for Reddit here. It might make sense for them to force subreddits back to public, but undeleting old comments has no strategic value right now.)
- fho 3y ago
  > (I do actually think this is the most likely, because there's zero benefit for Reddit here.) The obvious benefit would be to artificially keep interaction rates high by "grave robbing" old comments and accounts. "Nobody is ever going to notice, those accounts are long dead!"
  - crazygringo 3y ago
    What interaction rates? For what ends? The metrics you use to attract investment are things like monthly active users, and whether a comment was deleted after it was made doesn't affect that at all.
    - arendjr 3y ago
      From what I gather, a lot of Reddit usage comes from Google referrals, similar to how StackOverflow gets a lot of its traffic. If people leaving Reddit delete or scramble their comments (which some are definitely doing), they also screw up Reddit’s referral value and Google ranking, so Reddit has quite some motivation for trying to restore those comments.
    - fho 3y ago
      I was thinking pure "comments per second" numbers, ie how much users interact with reddit. No idea what's the right word for that.
notacoward 3y ago
I've worked on many eventually consistent systems, up to exascale at one of the companies you mention, and that explanation just seems extremely unlikely. In a system using tombstones, those should (and generally will) go the same place as the originals so "missing shard" doesn't work. Over that time scale there should then have been multiple rounds of compaction, including those forced by disk or server replacements, which would have also purged that deleted/overwritten data. Nobody at that scale can afford to let uncompacted garbage accumulate forever. Therefore, if not tombstones, they must be deliberately keeping old versions, but in that case the issue just moves to the index and doesn't really change much. The only way I can see this happening is if there was an unrecoverable data loss that required restoring from backup. Also, let's not forget that true deletion is sometimes required under various regulatory systems. I know that at least two of the companies you mention have systems specifically dedicated to satisfying those requirements. It's still possible that this is innocent(ish), but at low enough probability that malice really does seem like a compelling alternative explanation. After all, spez has been caught doing something very similar before. There's proof of malicious nature, not just an assumption. When a component in a system has a certain known behavior or limit, you don't just discard that knowledge due to some handy rule of thumb.
- nebula8804 3y ago
  Maybe I have this wrong but looking at academic torrents, the entire pushshift archive from 2005 to 2022 is 2TB compressed. Are they really operating at a massive scale that you are describing? Pushshift is supposed to have ingested all of Reddit's comment data no? (Of course this is assuming pushsift has gotten the majority of comments)
  - notacoward 3y ago
    Well, let me put it this way: if they're operating at such a small scale, they have no reason to be using any eventually consistent data store, so explanations about missing shards etc. still don't work. There are still other possibilities, as I and others have outlined, but not enough to dismiss the possibility that spez is just up to his old tricks.

goykasi 3y ago

Yes, I have a strong love/hate relationship with multi-DC Cassandra.

willis936 3y ago

Does Hanlon's razor apply when the actor has openly stated malicious intent?

MichaelZuo 3y ago

Which is a bit surprising since nowadays an entire reddit instance could fit on a single server rack.

seanalltogether 3y ago

Yeah people are going into overdrive right now with speculating all the bad things reddit is doing. Someone claimed that reddit admins had taken over their local city's subreddit to install new mods, as if that was a priority for them right now.

hospitalJail 3y ago

Its not a shock to anyone who knows how cheap data is, but it just proves there is no true 'delete'. I never deleted my facebook profile/pictures because I didn't want to shine a spotlight on my account. I care more about what a 'deleted' post says, than any other post in a thread. I imagine there is a good reason for that.

4BynHmBFUxBD 3y ago
Same here. I also thought if I deleted my FB account, they would just maintain a “shadow profile” for me.

pontiacbandit8 3y ago

Data consistency on comments n years old?

llm_nerd 3y ago
Unless someone has invented a data system that includes time travel as a feature, the original date of the comments is irrelevant.

jacksnipe 3y ago

Unlike every other data operation, deletion is heavily regulated by law.

llm_nerd 3y ago
Sure. I can't wait to see all the lawsuits and fines overwhelming Reddit, and any other service that have occasional data consistency issues. This is heavily regulated by law, of course. Of course this isn't "heavily regulated". No law demands 100% consistency when you run some third party script during an outage. Clicking "delete" on a comment, or having a script do it, is not some GPDR enforced action.
- jacksnipe 3y ago
  You are incredibly wrong about this. Consent for data processing is the meat & potatoes of both GDPR and the largest enforcement actions that have happened through it. The delete button revoked consent.

aprentic 3y ago

That would be a reasonable hypothesis if our information was limited to a few isolated inconsistency issues. The more posts we see deleted the less likely it is that it's an accident. I just checked and found out that posts from an account that I scrubbed and deleted 5 years ago are popping back up. That stretches "eventual" pretty thin. We also have a large amount of external information that suggests that reddit is not acting in good faith, much of it from Spez himself. Between the two of those, malice is more likely than stupidity to be the motivating factor.

crazygringo 3y ago

That would easily explain comments coming back that were hours old or even days old, but it's extremely unlikely to explain comments coming back that were months or years old, as others in this thread have found. No database shard would ever have been left that far out of date due to high load. The only Hanlon's razor I can imagine here is somebody wrote a query to try to undelete something small (intending a single user or something) and forgot the WHERE clause and started to undelete everything. And didn't realize until the query had been executing for several minutes. Yikes. (I do actually think this is the most likely, not because of spez's motivations, but because there's zero benefit for Reddit here. It might make sense for them to force subreddits back to public, but undeleting old comments has no strategic value right now.)

fho 3y ago
> (I do actually think this is the most likely, because there's zero benefit for Reddit here.) The obvious benefit would be to artificially keep interaction rates high by "grave robbing" old comments and accounts. "Nobody is ever going to notice, those accounts are long dead!"
- crazygringo 3y ago
  What interaction rates? For what ends? The metrics you use to attract investment are things like monthly active users, and whether a comment was deleted after it was made doesn't affect that at all.
  - arendjr 3y ago
    From what I gather, a lot of Reddit usage comes from Google referrals, similar to how StackOverflow gets a lot of its traffic. If people leaving Reddit delete or scramble their comments (which some are definitely doing), they also screw up Reddit’s referral value and Google ranking, so Reddit has quite some motivation for trying to restore those comments.
  - fho 3y ago
    I was thinking pure "comments per second" numbers, ie how much users interact with reddit. No idea what's the right word for that.

notacoward 3y ago

I've worked on many eventually consistent systems, up to exascale at one of the companies you mention, and that explanation just seems extremely unlikely. In a system using tombstones, those should (and generally will) go the same place as the originals so "missing shard" doesn't work. Over that time scale there should then have been multiple rounds of compaction, including those forced by disk or server replacements, which would have also purged that deleted/overwritten data. Nobody at that scale can afford to let uncompacted garbage accumulate forever. Therefore, if not tombstones, they must be deliberately keeping old versions, but in that case the issue just moves to the index and doesn't really change much. The only way I can see this happening is if there was an unrecoverable data loss that required restoring from backup. Also, let's not forget that true deletion is sometimes required under various regulatory systems. I know that at least two of the companies you mention have systems specifically dedicated to satisfying those requirements. It's still possible that this is innocent(ish), but at low enough probability that malice really does seem like a compelling alternative explanation. After all, spez has been caught doing something very similar before. There's proof of malicious nature, not just an assumption. When a component in a system has a certain known behavior or limit, you don't just discard that knowledge due to some handy rule of thumb.

nebula8804 3y ago
Maybe I have this wrong but looking at academic torrents, the entire pushshift archive from 2005 to 2022 is 2TB compressed. Are they really operating at a massive scale that you are describing? Pushshift is supposed to have ingested all of Reddit's comment data no? (Of course this is assuming pushsift has gotten the majority of comments)
- notacoward 3y ago
  Well, let me put it this way: if they're operating at such a small scale, they have no reason to be using any eventually consistent data store, so explanations about missing shards etc. still don't work. There are still other possibilities, as I and others have outlined, but not enough to dismiss the possibility that spez is just up to his old tricks.