5 ms·

The reason to collect everything (or rather, more things than you think you'll need to answer the questions you think you'll need to answer) is that…you could b

by zeveb 11y ago

The reason to collect everything (or rather, more things than you think you'll need to answer the questions you think you'll need to answer) is that…you could be wrong about what data are required to answer the questions you've identified, and that you could be wrong about which questions you will care about. And historical data can be extremely useful, e.g. when looking at seasonal or cyclical trends (woe betide the grocer who doesn't stock up on turkeys in mid November, which means woe betide the grocer who doesn't order turkeys in the summer). Yes, he's absolutely right that data management and privacy impose a cost: the cost of storing a GB of data is more than S3's 10¢. It takes a human being to make a judgement call about whether this data is likely to be worth its overall cost and risk. That's why managers and other decision-makers get paid what they get paid. 'Data is a liability' is a nice soundbite, but it doesn't capture the full reality; one can't manage to soundbites.

donlzx 11y ago
Collecting everything would never be the optimum strategy and apparently the least meaningful one economically. For the unexpected extra values from history data, the possibility that the return value is greater than the associated total cost doesn't necessarily increases with the dataset size. If we look at the trends for history things, only those with scant availability and high quality will be of real values. I agree with the author that a better strategy would be to start with some unique and useful questions, store data for a purpose (insight). Asset or liability? It is not a black or white thing, it is a scale depends on the information value and fidelity. In information theories, the information quantity (entropy or bits) is inverse proportion to the available possibility.
oconnore 11y ago
This is a strange critique. His recommendation isn't a soundbite. It is: ask questions, figure out what you need to know to answer those questions, and then collect that data. That (asking questions and making informed decisions) sounds like exactly what those "managers and decision makers" ought to be doing in the first place. To collect everything because you can't decide on a strategy is exactly the opposite. (Re: turkeys and fairly complex retail sales cycles. A year or three of aggregated purchase history is likely to yield actionable insight. It's unlikely that a slight upward trend in turkey purchases since 2006 is very actionable.)
- bhntr3 11y ago
  Old data is possibly uninteresting. That's debatable. But all uncollected data is freshly uncollected data. I strongly believe in collecting as much granular data as possible. It's totally possible to throw it away after it has outlived its likely usefulness. But if I want to calculate a new machine learning signal today , I can't wait six months to accumulate enough data. I want that data to exist. And the only solution is to overcollect. And to collect at a granular level so I can aggregate and transform later. The liability and security concerns are real. The fact that most companies are stupid about investing in unnecessary big data infrastructure is real too. But a recommendation to spool to cheap offline or nearline storage immediately is interesting. A recommendation to throw data away seems like folly to me.
  - oconnore 11y ago
    If your operation is the sort that capitalizes on machine learning on subtle signals, you capture the sorts of data that you know might be beneficial 6 months down the road. Although I suspect that you overstate the sort of "surprise questions" that might come up (and also the granularity necessary to answer them), that is a correct response to a business model that demands those insights at that granularity. That doesn't mean that "collect everything at max granularity" is good advice, because as you said: > The fact that most companies are stupid about investing in unnecessary big data infrastructure is real
    - yummyfajitas 11y ago
      The fact of the matter is that I've never been unhappy about overcollecting data. Worst case, step 1 of my pipeline is 10x or 50x slower than it needs to be due to filtering out a bunch of junk. The added latency to my workflow might be a few minutes. Every time I've undercollected I've been unhappy, and this was hardly a rare occurrence. I need to build the collector, deploy it, and wait for data to flow in. Added latency = 1 week, minimum. You can always throw useless stale data away. You can never retroactively collect data you needed.
- rcthompson 11y ago
Spooky23 11y ago
The reason to collect everything is laziness. It's easier to ask for everything and store everything than tonight's out what you actually need. My employer's HR collects my date of birth and SSN 4-5 times a year. Why? So some clerk doesn't have to look it up.
- jsprogrammer 11y ago
  Instead of looking the values up in a database, the clerk contacts you to look up the value...and then readds it to some database?
  - ams6110 11y ago
    Same is true of my kids schools. Every year, we have to fill out (on paper) the same extensive forms to collect parent names and addresses, phone, email, whether you are homeless, what your racial makeup is, on and on. No option for "everything is the same as last year" and they completely flip out if you don't return the forms.

by zeveb 11y ago

donlzx 11y ago

Collecting everything would never be the optimum strategy and apparently the least meaningful one economically. For the unexpected extra values from history data, the possibility that the return value is greater than the associated total cost doesn't necessarily increases with the dataset size. If we look at the trends for history things, only those with scant availability and high quality will be of real values. I agree with the author that a better strategy would be to start with some unique and useful questions, store data for a purpose (insight). Asset or liability? It is not a black or white thing, it is a scale depends on the information value and fidelity. In information theories, the information quantity (entropy or bits) is inverse proportion to the available possibility.

oconnore 11y ago

This is a strange critique. His recommendation isn't a soundbite. It is: ask questions, figure out what you need to know to answer those questions, and then collect that data. That (asking questions and making informed decisions) sounds like exactly what those "managers and decision makers" ought to be doing in the first place. To collect everything because you can't decide on a strategy is exactly the opposite. (Re: turkeys and fairly complex retail sales cycles. A year or three of aggregated purchase history is likely to yield actionable insight. It's unlikely that a slight upward trend in turkey purchases since 2006 is very actionable.)

bhntr3 11y ago
Old data is possibly uninteresting. That's debatable. But all uncollected data is freshly uncollected data. I strongly believe in collecting as much granular data as possible. It's totally possible to throw it away after it has outlived its likely usefulness. But if I want to calculate a new machine learning signal today , I can't wait six months to accumulate enough data. I want that data to exist. And the only solution is to overcollect. And to collect at a granular level so I can aggregate and transform later. The liability and security concerns are real. The fact that most companies are stupid about investing in unnecessary big data infrastructure is real too. But a recommendation to spool to cheap offline or nearline storage immediately is interesting. A recommendation to throw data away seems like folly to me.
- oconnore 11y ago
  If your operation is the sort that capitalizes on machine learning on subtle signals, you capture the sorts of data that you know might be beneficial 6 months down the road. Although I suspect that you overstate the sort of "surprise questions" that might come up (and also the granularity necessary to answer them), that is a correct response to a business model that demands those insights at that granularity. That doesn't mean that "collect everything at max granularity" is good advice, because as you said: > The fact that most companies are stupid about investing in unnecessary big data infrastructure is real
  - yummyfajitas 11y ago
    The fact of the matter is that I've never been unhappy about overcollecting data. Worst case, step 1 of my pipeline is 10x or 50x slower than it needs to be due to filtering out a bunch of junk. The added latency to my workflow might be a few minutes. Every time I've undercollected I've been unhappy, and this was hardly a rare occurrence. I need to build the collector, deploy it, and wait for data to flow in. Added latency = 1 week, minimum. You can always throw useless stale data away. You can never retroactively collect data you needed.
rcthompson 11y ago

Spooky23 11y ago

The reason to collect everything is laziness. It's easier to ask for everything and store everything than tonight's out what you actually need. My employer's HR collects my date of birth and SSN 4-5 times a year. Why? So some clerk doesn't have to look it up.

jsprogrammer 11y ago
Instead of looking the values up in a database, the clerk contacts you to look up the value...and then readds it to some database?
- ams6110 11y ago
  Same is true of my kids schools. Every year, we have to fill out (on paper) the same extensive forms to collect parent names and addresses, phone, email, whether you are homeless, what your racial makeup is, on and on. No option for "everything is the same as last year" and they completely flip out if you don't return the forms.