4 ms·

Old data is possibly uninteresting. That's debatable. But all uncollected data is freshly uncollected data. I strongly believe in collecting as much granular d

by bhntr3 11y ago

Old data is possibly uninteresting. That's debatable. But all uncollected data is freshly uncollected data. I strongly believe in collecting as much granular data as possible. It's totally possible to throw it away after it has outlived its likely usefulness. But if I want to calculate a new machine learning signal today , I can't wait six months to accumulate enough data. I want that data to exist. And the only solution is to overcollect. And to collect at a granular level so I can aggregate and transform later. The liability and security concerns are real. The fact that most companies are stupid about investing in unnecessary big data infrastructure is real too. But a recommendation to spool to cheap offline or nearline storage immediately is interesting. A recommendation to throw data away seems like folly to me.

oconnore 11y ago
If your operation is the sort that capitalizes on machine learning on subtle signals, you capture the sorts of data that you know might be beneficial 6 months down the road. Although I suspect that you overstate the sort of "surprise questions" that might come up (and also the granularity necessary to answer them), that is a correct response to a business model that demands those insights at that granularity. That doesn't mean that "collect everything at max granularity" is good advice, because as you said: > The fact that most companies are stupid about investing in unnecessary big data infrastructure is real
- yummyfajitas 11y ago
  The fact of the matter is that I've never been unhappy about overcollecting data. Worst case, step 1 of my pipeline is 10x or 50x slower than it needs to be due to filtering out a bunch of junk. The added latency to my workflow might be a few minutes. Every time I've undercollected I've been unhappy, and this was hardly a rare occurrence. I need to build the collector, deploy it, and wait for data to flow in. Added latency = 1 week, minimum. You can always throw useless stale data away. You can never retroactively collect data you needed.

by bhntr3 11y ago

oconnore 11y ago

If your operation is the sort that capitalizes on machine learning on subtle signals, you capture the sorts of data that you know might be beneficial 6 months down the road. Although I suspect that you overstate the sort of "surprise questions" that might come up (and also the granularity necessary to answer them), that is a correct response to a business model that demands those insights at that granularity. That doesn't mean that "collect everything at max granularity" is good advice, because as you said: > The fact that most companies are stupid about investing in unnecessary big data infrastructure is real

yummyfajitas 11y ago
The fact of the matter is that I've never been unhappy about overcollecting data. Worst case, step 1 of my pipeline is 10x or 50x slower than it needs to be due to filtering out a bunch of junk. The added latency to my workflow might be a few minutes. Every time I've undercollected I've been unhappy, and this was hardly a rare occurrence. I need to build the collector, deploy it, and wait for data to flow in. Added latency = 1 week, minimum. You can always throw useless stale data away. You can never retroactively collect data you needed.