3 ms·

Neutral party here: not the case, on your end, the mind-reading distracts from hearing the voice of Chesterton talking about his fence. c.f. sibling comment wr

Neutral party here: not the case, on your end, the mind-reading distracts from hearing the voice of Chesterton talking about his fence. c.f. sibling comment written 10 minutes before yours, several trivial things. More concretely from a downvoter: 1. I want _nothing_ to do with user data. Nothing. Toxic nuclear waste. The idea of keeping the waste on hand needs strong justification. 2. The idea of "just leave the rows but mark is_deleted" has a fundamental distate to a profession where O(n) is a constant worry. 3. The breezy way it dismisses this as an obvious solution that ~eveyone agrees suggets a small breadth of experience. That's not bad! But then we're at Chesterton's fence. In summary, you can get downvoted for ruling out someone else's concrete lived experience, not just because your dangerous ideas threaten their paycheck (I don't even have a job!)

hinkley 3y ago
The lived experience you talk about is that this is a difficult problem without a solution, not a different solution that I dismissed. What I dismissed is the problem. How do we solve X? You don't solve X, you solve Y. The XY problem, not Chesterton's Fence. Not sure I follow you entirely on user data. There's user data that absolutely requires an audit trail, like subscriptions or orders. There's user data that might start out as null and eventually transition to an inane value, like avatar or self-description. You don't need complex merge rules for that data. It possibly only happens when their account is being hacked and that's not the data you need in that particular situation, yeah?
vlovich123 3y ago
Couldn’t you just mark a deletion timestamp and then lazily delete later as part of a GC operation?
- chrismarlow9 3y ago
  In larger systems you're pretty much forced to do it this way or face an ugly realization once you scale. Delete from s3 is an example. Under the hood most db's do this as well (vacuum, repair database, etc). At this point I've learned this lesson the hard way enough that I'd need to hear some really good reasons to NOT do it this way.
  - hinkley 3y ago
    I’m not sure this is quite true for S3, but for the general case, once there’s consensus that a record is eligible to be deleted, you don’t require consensus to delete them. You can do it any time after the drop dead date. Once a month for instance. One of the consensus algorithms Google was bragging about a few years back was built on very high precision hardware clocks that set a ridiculously short timeframe to achieve consensus on new data. After a few hundred milliseconds you could be certain a record had settled and make business decisions based off of it. It’s the same basic idea, but three orders of magnitude faster.
person3 3y ago
> 1. I want _nothing_ to do with user data. Nothing. Toxic nuclear waste. The idea of keeping the waste on hand needs strong justification. Additionally, with regulations like CCPA in some jurisdictions, this isn't even optional anymore. At some point you will need to hard delete user data.
- hinkley 3y ago
  I can forget that you were my customer, but I can’t forget that there was a customer. That quickly turns into tax evasion, for one thing. Much of the user information we acquire is the result of greed, nosiness, or laziness. If deleting users is difficult for you, that’s an architectural problem that has next to nothing to do with my comment. The world is absolutely full of rules that have exactly one exception. If they have two we apply the Rule of Three and either fix it or change it back to two. I have absolutely no qualms about treating user data as the exception here. If you’re Amazon, you don’t even need much of the PII until checkout time. Collecting or looking up that data up early is a security risk. Checkouts are going to be orders of magnitude fewer operations than your browsing traffic. When the order of magnitude changes, the solutions often change. And lastly, checkouts are when you make money. Expensive operations, like inserting into a table with fragmentation problems, are much easier to justify when they are attached to revenue events. An ad campaign that falls flat can bankrupt you. A fire or earthquake can bankrupt you. A fancy and unusable site relaunch can bankrupt you. Spending a little money at the point of sale cannot.

hinkley 3y ago

The lived experience you talk about is that this is a difficult problem without a solution, not a different solution that I dismissed. What I dismissed is the problem. How do we solve X? You don't solve X, you solve Y. The XY problem, not Chesterton's Fence. Not sure I follow you entirely on user data. There's user data that absolutely requires an audit trail, like subscriptions or orders. There's user data that might start out as null and eventually transition to an inane value, like avatar or self-description. You don't need complex merge rules for that data. It possibly only happens when their account is being hacked and that's not the data you need in that particular situation, yeah?

vlovich123 3y ago

Couldn’t you just mark a deletion timestamp and then lazily delete later as part of a GC operation?

chrismarlow9 3y ago
In larger systems you're pretty much forced to do it this way or face an ugly realization once you scale. Delete from s3 is an example. Under the hood most db's do this as well (vacuum, repair database, etc). At this point I've learned this lesson the hard way enough that I'd need to hear some really good reasons to NOT do it this way.
- hinkley 3y ago
  I’m not sure this is quite true for S3, but for the general case, once there’s consensus that a record is eligible to be deleted, you don’t require consensus to delete them. You can do it any time after the drop dead date. Once a month for instance. One of the consensus algorithms Google was bragging about a few years back was built on very high precision hardware clocks that set a ridiculously short timeframe to achieve consensus on new data. After a few hundred milliseconds you could be certain a record had settled and make business decisions based off of it. It’s the same basic idea, but three orders of magnitude faster.

person3 3y ago

> 1. I want _nothing_ to do with user data. Nothing. Toxic nuclear waste. The idea of keeping the waste on hand needs strong justification. Additionally, with regulations like CCPA in some jurisdictions, this isn't even optional anymore. At some point you will need to hard delete user data.

hinkley 3y ago
I can forget that you were my customer, but I can’t forget that there was a customer. That quickly turns into tax evasion, for one thing. Much of the user information we acquire is the result of greed, nosiness, or laziness. If deleting users is difficult for you, that’s an architectural problem that has next to nothing to do with my comment. The world is absolutely full of rules that have exactly one exception. If they have two we apply the Rule of Three and either fix it or change it back to two. I have absolutely no qualms about treating user data as the exception here. If you’re Amazon, you don’t even need much of the PII until checkout time. Collecting or looking up that data up early is a security risk. Checkouts are going to be orders of magnitude fewer operations than your browsing traffic. When the order of magnitude changes, the solutions often change. And lastly, checkouts are when you make money. Expensive operations, like inserting into a table with fragmentation problems, are much easier to justify when they are attached to revenue events. An ad campaign that falls flat can bankrupt you. A fire or earthquake can bankrupt you. A fancy and unusable site relaunch can bankrupt you. Spending a little money at the point of sale cannot.