3 ms·
> Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it. IIRC the term "big data" was coined to refer to data volumes that are
by geezerjay 8y ago
> Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.
IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.
- gaius 8y agolarge enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data What about too large for Excel, is that “big data” too? (Neither are)
- samuel_ms 8y agoThat definition might be a bit loose. If you have an existing application designed to run on a Raspberry Pi, it'll hit that limit pretty quickly. This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.