3 ms·

> I’d love for data to be seen as a liability - something audited and discarded often to minimise risk. Perhaps it's just the industries I've worked in, which

> I’d love for data to be seen as a liability - something audited and discarded often to minimise risk. Perhaps it's just the industries I've worked in, which tend to be highly regulated, but I have such a hard time seeing why this idea isn't more intuitive to people. Every single scrap of sensitive information you have laying around is something that has the potential to fuel a very expensive, possibly even existential, crisis for your company. Even for the non-sensitive stuff, keeping it around "just in case" increases the risk that somebody who has taken a couple classes in how to use R or Pandas and is still a bit starry-eyed about Big Data starts making questionable business decisions based on unskeptical interpretations of biased results drawn from convenience-sampled data of uncertain provenance. The default shouldn't be collecting and warehousing everything, the default should be, "when in doubt, throw it out."

tomlockwood 7y ago
Author here. Yeah, and to expand on the idea of liability, I also additionally mean the possibility of a financial liability. If we were to say, charge per month for a retained customer date of birth, you'd see business analysts internally constantly asking why the data needed to be stored so long, or whether just age could be stored, or something. If the price was high enough, the problem would sort of just... go away. Obviously much smarter people than me would need to figure out those regulations though.
- afiori 7y ago
  On the liability side the GDPR was already good progress
diogofranco 7y ago
Surely there are many reasons to not hoard everything, but I find this argument strange. If you're in a business that is making bad decisions due to wrong analysis of data, it seems like there should be more nuanced and pertinent ways to tackle this then just throwing it out, which also prevents good analysis.
- bunderbunder 7y ago
  Well, some of the issue there is that it's really hard to get a good analysis out of bad or unsuitable data. Garbage in, garbage out. Generally it's better to put the horse in front of the cart: Figure out what kinds of questions you want to be answering, and then design a way to collect the data you need to answer those questions. This isn't far off from the lesson that medical science somewhat recently had to learn the hard way, about how just dragnet collecting heaps data and then figuring out what to do with it after the fact will yield far more incorrect conclusions than correct ones.
deleted 7y ago
[deleted]
Balgair 7y ago
I mean, I'd like for that to be true, but look at Equifax. They leaked nearly every critical detail about nearly every single working American. Not only are they still alive, their stock is up ~130% since December and is getting back to where they were before the leak was exposed (~$140)[0] Leaking data has little consequences. [0] https://finance.yahoo.com/quote/EFX/ https://finance.yahoo.com/quote/EFX/
eximius 7y ago
> potential to fuel a very expensive, possibly even existential, crisis for your company. In terms of security breaches, this maybe SHOULD be true, but isn't. In terms of bad feature development, it's a little melodramatic. I'm not saying you can't misinterpret something so bad it causes something catastrophic, just that it's more likely to be a couple percent drop in some subset of traffic for some cohort. Not the worst thing. > The default shouldn't be collecting and warehousing everything, the default should be, "when in doubt, throw it out." We have a lot of things where I work that I wish we were collecting now, not because we could use it now, but because we could use it later. And you can't go back in time and get it (usually). Throwing data away is easy. I wish I could have the foresight, the prescience to only store what I know I'll need.

tomlockwood 7y ago

Author here. Yeah, and to expand on the idea of liability, I also additionally mean the possibility of a financial liability. If we were to say, charge per month for a retained customer date of birth, you'd see business analysts internally constantly asking why the data needed to be stored so long, or whether just age could be stored, or something. If the price was high enough, the problem would sort of just... go away. Obviously much smarter people than me would need to figure out those regulations though.

afiori 7y ago
On the liability side the GDPR was already good progress

diogofranco 7y ago

Surely there are many reasons to not hoard everything, but I find this argument strange. If you're in a business that is making bad decisions due to wrong analysis of data, it seems like there should be more nuanced and pertinent ways to tackle this then just throwing it out, which also prevents good analysis.

bunderbunder 7y ago
Well, some of the issue there is that it's really hard to get a good analysis out of bad or unsuitable data. Garbage in, garbage out. Generally it's better to put the horse in front of the cart: Figure out what kinds of questions you want to be answering, and then design a way to collect the data you need to answer those questions. This isn't far off from the lesson that medical science somewhat recently had to learn the hard way, about how just dragnet collecting heaps data and then figuring out what to do with it after the fact will yield far more incorrect conclusions than correct ones.

deleted 7y ago

[deleted]

Balgair 7y ago

I mean, I'd like for that to be true, but look at Equifax. They leaked nearly every critical detail about nearly every single working American. Not only are they still alive, their stock is up ~130% since December and is getting back to where they were before the leak was exposed (~$140)[0] Leaking data has little consequences. [0] https://finance.yahoo.com/quote/EFX/ https://finance.yahoo.com/quote/EFX/

eximius 7y ago

> potential to fuel a very expensive, possibly even existential, crisis for your company. In terms of security breaches, this maybe SHOULD be true, but isn't. In terms of bad feature development, it's a little melodramatic. I'm not saying you can't misinterpret something so bad it causes something catastrophic, just that it's more likely to be a couple percent drop in some subset of traffic for some cohort. Not the worst thing. > The default shouldn't be collecting and warehousing everything, the default should be, "when in doubt, throw it out." We have a lot of things where I work that I wish we were collecting now, not because we could use it now, but because we could use it later. And you can't go back in time and get it (usually). Throwing data away is easy. I wish I could have the foresight, the prescience to only store what I know I'll need.