3 ms·
Jan, can you explain briefly how the deduplicator checks if the new answer is significantly different? Is there code in the repository we can take a look at?
by anupsurendran 3y ago
Jan, can you explain briefly how the deduplicator checks if the new answer is significantly different? Is there code in the repository we can take a look at?
- janchorowski 3y agoSure: when a new response is produced because some source documents have changed we ask an LLM to compare the responses and tell if they are significantly different. Even a simplistic prompt, like the one used in the example would do: Are the two following responses deviating? Answer with Yes or No. First response: "{old}" Second response: "{new}" (used in https://github.com/pathwaycom/llm-app/blob/69709a2cf58cdf6eaefee1d6baaa9a3d4532e5f9/examples/pipelines/drive_alert/app.py#L119 https://github.com/pathwaycom/llm-app/blob/69709a2cf58cdf6ea...)
- pstorm 3y agoCouldn't you just compare the similarity of the embeddings? I imagine that would work in the vast majority of cases and save a lot of LLM calls.
- janchorowski 3y agoThat's a good idea, the deduplication criterion is easy to change, using an llm is faster to get started, but after a while a corpus of decisions is created and can be used to either select another mechanism, or e.g. train one on top of bert embeddings.