3 ms·
You can easily add more components to start thinking about models/assumptions/etc, like adding interfaces to Data Catalogs (i.e. Glue Data Catalog). As part of
by mrbungie 4y ago
You can easily add more components to start thinking about models/assumptions/etc, like adding interfaces to Data Catalogs (i.e. Glue Data Catalog).
As part of a POC I made, I built a similar bot without recursion for debugging and iterative query building though. It does the following:
- It predicts most probable entities from the question.
- Searches AWS Glue Data Catalog for the most probable/useful tables.
- It builds an Athena SQL Query from N most useful tables.
It obviously get it catastrophically wrong sometimes, but hell, it was a 3 hour POC. If you can make better indices that map entity->table relationships it should get better at searching tables. Add this kind of recursive/iterative debugging of queries, and you get at least something near a junior-level SQL Analyst.
These kind of bots are analogous to Stable Diffusion, they DO need a good prompter/puppeteer/solution-verifier. Most non-senior Data Analysts also need one anyways.
- arrosenberg 4y agoThats not really fair to the human. The time to learn basic SQL may be longer, but the algorithm will never develop intuition or independence, while many junior analysts will eventually operate effectively on their own. It’s a neat tool for analysts as a query generator - I would use it in situations where I’m not familiar with the schema, but it would become less useful as I learn.
- mrbungie 4y agoYeah, maybe I should've said "first-weeks/months" level junior data analyst. But hell, as an analyst I would've paid a lot for a tool that searched intelligently through giant datawarehouses (or whatever consultants call them now) and at least gave you probable matches. Now that same thing exists and you can even finetune its "DSL" towards your own organization.