3 ms·
How are you measuring the accuracy? Are you running this against any benchmarks? I see this covers a file based approach, was there ever a consideration for a
by modus-tollens 4mo ago
How are you measuring the accuracy? Are you running this against any benchmarks?
I see this covers a file based approach, was there ever a consideration for a graph based approach?
For business context, how do you handle context that evolves over time?
- deleted 4mo ago[deleted]
- deleted 4mo ago[deleted]
- deleted 4mo ago[deleted]
- deleted 4mo ago[deleted]
- deleted 4mo ago[deleted]
- andreybavt 4mo agoGreat questions! we're currently working on Spider 2 submission, hope to have first results soon. It's true that we took file-first approach and not a graph DB. Main reason is that the ktx wiki and semantic layer entities while being written in plain text files (md or yaml) still contain links to each other. This allows an agent to find the right entry point (with the help of lexical and semantic searches merged with RRF) and then traverse these links to collect enough context. As for the business context evolution - that's exactly the reason we have ingestion reconciliation and git versioning. The idea is to give ingestion agent a way to deduplicate/consolidate knowledge during the ingestion and leave complex conflicts to humans to resolve
- lucamrtl 4mo agoAnd development was done around link detection and text to sql benchmarks to measure/compare different approaches
- andreybavt 4mo agoyep, happy to share more details with you @modus-tollens if you're interested
- hannune 4mo ago[dead]