3 ms·
> Manipulating the data? Sure, use SQL, as long as you're operating on data from one database. Trying to collate data from multiple databases may have unpredict
by onlyrealcuzzo 4y ago
> Manipulating the data? Sure, use SQL, as long as you're operating on data from one database. Trying to collate data from multiple databases may have unpredictable performance issues even in the engines that support it.
Depending on how big your data & org is - orgs solve this by writing their own query engines that can query many different data formats.
It is much more practical to optimize an engine, than to expect every data engineer to know how to optimally manipulate data from many different formats.
- crest 4y agoAnd why shouldn't this engine accept SQL queries as well if it's accessing existing SQL databases (probably just with different weights, retries, and batching/spooling)?
- onlyrealcuzzo 4y agoIt usually does! If you want to use the same logic / language to query some exotic dataset for specific use cases - it can often be worth it to write a custom engine that can do it - rather than expect all end-users to learn the ins & outs of the other databases & datasets (not to mention, to learn something beside SQL). Instead, a single team (the query engine owners) can optimize the query engine - rather than individual users trying to optimize every script individually. Your users can become masters of their engine / language (SQL) - because it can be used for the vast majority of cases. For many reasons - you might want to store data in a format that MySQL / Postgres does not support natively (see Google Spanner). But, ideally, you'd still be able to leverage the fact that almost every programmer in the world can write SQL.