I've played with (part of) this dataset too, or at least one like it. The CSV structure looks familiar. Note that while the actual CSV files are ~500GB, a lot o

I've played with (part of) this dataset too, or at least one like it. The CSV structure looks familiar. Note that while the actual CSV files are ~500GB, a lot of that is wasted bytes on stuff like addresses in which a lot of information is repeated as well as long dates/identifiers/numbers/coordinates being written out as strings. If you write some custom software for it that uses binary representations and maps repetitive information to shorter IDs, you should be able to fit the entire dataset into 64GB of RAM (less than 60 bytes / row). At that point running most analytical queries on it becomes really quick even if you do not care about indexing the main data and just rely on tools like MapReduce. You could probably dump that representation into an mmap'ed file on a SSD and get similar results even with less RAM. Should you still use a DB? Probably. But if your DB is a bigger hassle and significantly slower than one machine running some wonky code you wrote in an afternoon, it probably sucks. Now after reading the article I'm still not 100% sure whether the initial issue with DuckDB was performance or data related. If it's the former, then any DB running on modern hardware that starts choking handling a million anything doesn't deserve to be called a database in this day and age - at that point Microsoft Excel is your main competitor and offers a better user experience.