3 ms·

Anyone have any recommendations for a SQL engine that works on in-memory data and has a simple/monolithic architecture? Our data is about 50-100gb (uncompressed

Anyone have any recommendations for a SQL engine that works on in-memory data and has a simple/monolithic architecture? Our data is about 50-100gb (uncompressed) and thus easily fits into memory. I am sure we could do our processing using something like polars or pandas in memory quite quickly but we prefer a SQL interface. Using postgres is still quite slow even when it has more than enough memory available compared to something like duckdb. Duckdb has other limitations however. I’ve been eying MemSQL but that also seems to be targeted more towards multi machine deployments.

mritchie712 4y ago
what limit are you hitting with duckdb?
giraffe_lady 4y ago
sqlite?
chaxor 4y ago
SQLite is almost always the answer
specialist 4y ago
H2 is trivial to embed. Written in Java. It's one of my favorite things. http://www.h2database.com/html/features.html#in_memory_databases http://www.h2database.com/html/features.html#in_memory_datab...
whichdan 4y ago
MySQL has a MEMORY[0] storage engine which might be of interest? [0] https://dev.mysql.com/doc/refman/8.0/en/memory-storage-engine.html#memory-storage-engine-performance-characteristics https://dev.mysql.com/doc/refman/8.0/en/memory-storage-engin...