4 ms·
Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres g
by arnon 7y ago
Not all relational DBs break at scale.
Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes.
Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight.
Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.
- misterman0 7y agoWhich of those indexes/databases should I be using if I want to query over big data using natural language?
- kthejoker2 7y agoThis question makes no sense. First, NLP based search can be executed on top of any engine (APIs are very handy), relational, kv, graph, filesystem .. so that part is totally irrelevant. Assuming "big data" in this context is still relational data, then any of those systems would suffice, within their own particular tradeoffs and features.
- arnon 7y agoThere is no direct relation between the query language and the underlying infrastructure. If you're talking about taking some questions and getting graphs from them, ThoughtSpot does a good job.
- fmajid 7y agoThere's also CitusDB, which performs much better than Hadoop because it doesn't have the bloat of the JVM or the inefficiency of brain-dead HDFS. You can write your queries in a few lines of SQL rather than reams of convoluted Java code.