3 ms·
Modern query engines are designed to read data in parallel because it's so much faster. The data could be stored in 50 different one-gig files that were read i
by MrPowers 3y ago
Modern query engines are designed to read data in parallel because it's so much faster. The data could be stored in 50 different one-gig files that were read in parallel.
- jandrewrogers 3y agoNone of this requires multiple files, either in theory or in practice. It may be a limitation of a particular implementation but that is a design choice. There are functional advantages to not splitting up files unnecessarily when processing queries.
- theLiminator 3y agoSpec compliant CSV parsing cannot be done in parallel iirc. Most CSV parsers are not spec compliant, but splitting up CSVs is one way to read them in parallel while ensuring you can still use a spec compliant parser.