4 ms·
Not for data analysis, it's a pointless constraint unless you're having issues. Most data analysis isn't against datasets larger than memory, and I'd rather ha
by tomnipotent 6y ago
Not for data analysis, it's a pointless constraint unless you're having issues.
Most data analysis isn't against datasets larger than memory, and I'd rather have more data than I need to spend time waiting to bring it all local again because I forgot a few columns that turn out to be useful later on.
- disgruntledphd2 6y agoTrue, but in that kind of exploratory environment, you'd normally reduce data load and speed up iteration by using sampling, which is super easy in SQL. > Not for data analysis, it's a pointless constraint unless you're having issues. This will always happen, as time spent on the project scales up (unless you use some kind of autoscaling magic I suppose).
- EForEndeavour 6y agoI totally get what you're saying because most of my datasets also used to be smaller than my laptop's memory. Used to. I'm in the process of moving more and more processing "upstream" from local pandas to the DBMS and it's already proving to be a bit of a superpower. I regret not learning it earlier, but the second-best time is now.