4 ms·
There is a split in the comments below as I write this. Some people are coming from the Kaggle / Text book/ Statistics perspective - the data set is ready for
by sgt101 6y ago
There is a split in the comments below as I write this.
Some people are coming from the Kaggle / Text book/ Statistics perspective - the data set is ready for analysis so I think we should use analytical tool x or y to give us some information about the conclusions that we are likely to be trying to demonstrate from it - or the models that we will build.
Other people are looking from a Data Science pov. The data is made of bits and is likely to have pathological features due to it's history and the biases of the different collection and assembly methods. The task is to find these and mitigate them - at that point we have a decent data set and can move to analysis and modelling.
- heresie-dabord 6y agoFor any small data set, the inefficiencies and Library Hell of $tooling don't matter... much. For any non-trivial data set, the efficiency and internal consistency of $tooling is critical. Throwing stupendous hardware resources at a problem because the $tooling is unstable rubbish is the wrong approach. For any non-trivial data set, the data must have a cohesive reason to exist. In other words... Big Data Garbage In, Big Data Garbage Out. Telemetry is often BDGI.