2 ms·
The example with "interesting insight" works well with a small dataset (CSV) that fits into the AI context memory, but how do you deal with a large dataset (e.g
by XCSme 3y ago
The example with "interesting insight" works well with a small dataset (CSV) that fits into the AI context memory, but how do you deal with a large dataset (e.g. a database of tens of GBs), the AI can't read/process all that data? Does it just do some "random" queries to generate some facts?
- zahmed1 3y agoHey, yes, this gets more complicated as datasets get larger. Our early customers have simple datasets (using spreadsheets, csvs, and relatively small postgres or mysql dbs), so it works very well for them. As we continue to move to larger customers, data warehouses, and more complex use cases, we'll build semantic layers (aka. provide a lot of context on the db, tables, business metrics, sample queries, and more), use RAG alongside our SQL agent, introduce more robust auditing / RLHF, etc. All solvable as our customer needs require it. Also, context window limits continue to increase, and since we're plugging into the best models to solve customer problems (not building our own), tbd on how much of what I've listed above (e.g. RAG) we'll need to do when the time comes.