3 ms·
For a question that ought to be one SQL query.
by Animats 4y ago
For a question that ought to be one SQL query.
- avmich 4y agoWell, reality often works differently, and the bulk of the work is to prepare the environment where that SQL query may run. You have various data sources - and which one exactly, that SQL query won't tell you, you'll usually have to go and find out with some other means. The data sources have various interfaces and formats - and sometimes protocols too. You might decide to collect the data together, to that proverbial data lake - then to run the SQL query you'll have to design that data lake, tables, columns and all, and convert the data into it. Or - I was wondering if this applies to the described problem - for one-off calculations you may choose to work with streams of data, skipping building data lakes - this can work, but how to correctly map SQL to streaming data could be an interesting problem - what if some data you need several times for several different intermediate functions, will you have to read streams many times or do something else?.. So, yes, the SQL query. Perhaps one of the small and easy parts of the problem.
- nborwankar 4y agoYes but to get all the raw data into a form that can be queried by SQL or getting a tool that can query JSON, CSV … via SQL is the non trivial part. Not to mention reconciling date formats etc from multiple sources into one format. 80% of the job is data cleaning.
- shrikant 4y ago> Them: Can you just quickly pull this data for me? > Me: Sure, let me just: > SELECT * FROM some_ideal_clean_and_pristine.table_that_you_think_exists Credit: https://twitter.com/sethrosen/status/1252291581320757249 https://twitter.com/sethrosen/status/1252291581320757249