4 ms·
I'm not sure the author fully understands what big data is and where his point actually lies. Big data is about processing large amounts of data in a fashion t
by monksy 8y ago
I'm not sure the author fully understands what big data is and where his point actually lies.
Big data is about processing large amounts of data in a fashion that can get your results. This is typically where distributed/clustered systems lies and often deals with system optimization.
If your data requires fast enough results and you need answers quickly sampling is done. Now he does have merit in his argument that considerations about sampling or approaches to sampling needs to be considered. Big data can get his accurate results on the population of the data. (I worked with a company with 35 PB of images, they did get analyzed via ML and image processing algorithms. There was very little "sampling" involved.)
My bone to pick with "big data" is that it tends to be ambiguously thrown around to have cookie cutter systems such as Hadoop/spark or a large db (i.e. redshift) that don't work for the needs of the business. (Hadoop is horrible for everything except for 1-time batch ops, or image stitching)