In this order, I recommend: 1. Bash. Can you get the answer you need with awk, sed, sort, etc.? Linux commands behave nicely with CPU and RAM, and "Command-lin

In this order, I recommend: 1. Bash. Can you get the answer you need with awk, sed, sort, etc.? Linux commands behave nicely with CPU and RAM, and "Command-line Tools can be 235x Faster than your Hadoop Cluster" (https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...). 2. SQLite. The maximum database size is 140 terabytes, and SQLite can join different database files together. First I would recommend using bash to cull your data down before import, if possible (https://www.sqlite.org/limits.html https://www.sqlite.org/limits.html). 3. PostgreSQL. Again, I would recommend using bash to cull your data down before import, if possible (https://www.postgresql.org/about/ https://www.postgresql.org/about/). Maximum Database Size Unlimited Maximum Table Size 32 TB Maximum Row Size 1.6 TB Maximum Field Size 1 GB Maximum Rows per Table Unlimited Maximum Columns per Table 250 - 1600 depending on column types Maximum Indexes per Table Unlimited More: "Bane's Law (Going Deep)" (https://news.ycombinator.com/item?id=8902739 https://news.ycombinator.com/item?id=8902739)