4 ms·

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the da

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally. As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is marginally useful to determining future trends and analyzing new consumer behavior regimes?

gregw2 2y ago
If you have seasonal demand patterns, you generally need three years history to do good predictive analytics. I do tend to agree data from five years ago is rarely relevant BUT our business is still using for some BI purposes data from the fiscal year before COVID as a comparison baseline for certain analytics/business processes which have been slow to reach pre-COVID levels of performance. So that means we are now using data 6 years old, comparing this year to that pre-COVID year for certain analytics!
- physicsguy 2y ago
  Yeah 100%. I worked in wind energy for a while and the DS team would be pulling as much data as they could get to establish a baseline for normality due to seasonal trends in the wind. This also varied enormously around the world - for e.g. the UK is fairly windy all year, but India typically gets 2/3 of it's generated wind energy in the monsoon season which is about 3 months.
smfjaw 2y ago
I work in finance and it's great having big historical datasets, even if the figures are far lower in previous years it's good to see system 'shocks' and these can be used at a different magnitude/scaled for future forecasting
- refset 2y ago
  Yep backtesting is critical step in the deployment of new trading algorithms. Related, I rather enjoyed reading this other thread from June: "Ask HN: Is KDB a sane choice for a datalake in 2024?" https://news.ycombinator.com/item?id=40625800 https://news.ycombinator.com/item?id=40625800
mr_toad 2y ago
Customer retention, infrequent purchases, and time series forecasts all benefit from having at least several years of data.
jedberg 2y ago
Training a foundation model that understands shopping queries. 25 years of shopping data goes a long way.

gregw2 2y ago

If you have seasonal demand patterns, you generally need three years history to do good predictive analytics. I do tend to agree data from five years ago is rarely relevant BUT our business is still using for some BI purposes data from the fiscal year before COVID as a comparison baseline for certain analytics/business processes which have been slow to reach pre-COVID levels of performance. So that means we are now using data 6 years old, comparing this year to that pre-COVID year for certain analytics!

physicsguy 2y ago
Yeah 100%. I worked in wind energy for a while and the DS team would be pulling as much data as they could get to establish a baseline for normality due to seasonal trends in the wind. This also varied enormously around the world - for e.g. the UK is fairly windy all year, but India typically gets 2/3 of it's generated wind energy in the monsoon season which is about 3 months.

smfjaw 2y ago

I work in finance and it's great having big historical datasets, even if the figures are far lower in previous years it's good to see system 'shocks' and these can be used at a different magnitude/scaled for future forecasting

refset 2y ago
Yep backtesting is critical step in the deployment of new trading algorithms. Related, I rather enjoyed reading this other thread from June: "Ask HN: Is KDB a sane choice for a datalake in 2024?" https://news.ycombinator.com/item?id=40625800 https://news.ycombinator.com/item?id=40625800

mr_toad 2y ago

Customer retention, infrequent purchases, and time series forecasts all benefit from having at least several years of data.

jedberg 2y ago

Training a foundation model that understands shopping queries. 25 years of shopping data goes a long way.