2 ms·
Yes! If your data is too big to comfortably manipulate in memory with R/Python/your language of choice, and the data is (or can be made to be) tabular, the next
by civilized 5y ago
Yes! If your data is too big to comfortably manipulate in memory with R/Python/your language of choice, and the data is (or can be made to be) tabular, the next best place for it is a relational database. If you don't have an existing server that's well-suited, use SQLite or DuckDB, the "SQLite of analytics". Definitely use a columnar database if you're going to be doing complex queries and joins.
This is battle-worn experience and I will absolutely die on this hill. Relational databases JUST WORK.
The only exception is when you need to do complex ML on big data, in which case, take your pick from Apache Spark, Dask, or some similar framework. Hopefully these will be increasingly supplemented by in-database ML going forward, so that there is no reason to ever leave the database.