4 ms·
I like the idea of that, though Im curious if an EL approach is the right way to go especially at scale where queries might get all blocked run out of mem in Oc
by yevpats 4y ago
I like the idea of that, though Im curious if an EL approach is the right way to go especially at scale where queries might get all blocked run out of mem in OctoSQL.
- cube2222 4y agoCould you please explain in more detail what you mean (I'm not sure what you mean with "EL approach")? OctoSQL isn't really meant to be used "at scale", as it's not distributed. It's not trying to compete with Spark, Flink, or BigQuery. You should be good to use it with data up to the order of Gigabytes / Tens of Gigabytes, locally. OctoSQL never loads the whole table into memory if it doesn't need to. Processing happens record by record, and if predicates allow it, only part of the underlying table will ever be read (by pushing down predicates to a SQL DB you're reading from for example).