3 ms·
AI enhanced data analysis is nice, but this is not how I would have implemented it. Passing the data frame as part of the query limits the quantity of data you
by parsimo2010 3y ago
AI enhanced data analysis is nice, but this is not how I would have implemented it. Passing the data frame as part of the query limits the quantity of data you can handle, and you’re at risk of the quiet adding undesired operations on your data.
I expected that this would be a Copilot like approach that was tuned for pandas specifically, and you pass it your data frame schema/column names or something, and it passing back some code to evaluate locally.
I’m definitely not excited about passing the data frame itself and letting the llm handle it. Maybe with time I’ll get there, but I think this is a step too far for anyone that needs reproducible/verifiable results.
- boredemployee 3y agoI understand what you mean and kinda agree. I think that you entered in a "niche" discussion now. Because I see that people are building tools with LLM in the data analysis space trying to go more in the business direction, where product managers (and other stakeholders, mainly in startups) don't need to write a single line of code and get the data visualization/analysis that they want.
- findalex 3y agoThere is a package for querying dataframes with SQL: https://github.com/yhat/pandasql/ https://github.com/yhat/pandasql/. I assumed this was going to be LLM -> SQL -> Pandas.
- ibestvina 3y agoThat (LLM -> SQL -> Pandas using PandaSQL) was my approach with Datasloth [1] some time ago. It then grew into Qluent [2] which I now work on full time, along the lines of what the sibling comment wrote about managers being able to analyze their data without coding. [1] https://github.com/ibestvina/datasloth https://github.com/ibestvina/datasloth [2] https://qluent.com/ https://qluent.com/