3 ms·
If it all fits in memory, isn't it simpler to just load it first? Sure, you're kinda painting yourself into a corner, but it can be a big corner. e.g. I know
by oofbey 3y ago
If it all fits in memory, isn't it simpler to just load it first? Sure, you're kinda painting yourself into a corner, but it can be a big corner. e.g. I know that until recently Amazon still ran a bunch of key recommendation algorithms on a single big machine each night, because it's just simpler to load everything into memory and crank on it.
- hn72774 3y agoWhen reading from the dataframe, depending on the columns, filters, and aggregates you want returned, most query engines can skip data that is irrelevant, making the query run much faster. In your amazon example the data is probably optimized for fast lookups on a small number of records, like an in-memory cache.
- theLiminator 3y agoSimpler, but you can avoid huge amounts of work just simply by not loading what you need. That's the beauty of the scan_* apis in polars. Most query engines have some level of support for io skipping.