3 ms·
Sure: when a new response is produced because some source documents have changed we ask an LLM to compare the responses and tell if they are significantly diffe
by janchorowski 3y ago
Sure: when a new response is produced because some source documents have changed we ask an LLM to compare the responses and tell if they are significantly different. Even a simplistic prompt, like the one used in the example would do:
Are the two following responses deviating?
Answer with Yes or No.
First response: "{old}"
Second response: "{new}"
(used in https://github.com/pathwaycom/llm-app/blob/69709a2cf58cdf6eaefee1d6baaa9a3d4532e5f9/examples/pipelines/drive_alert/app.py#L119 https://github.com/pathwaycom/llm-app/blob/69709a2cf58cdf6ea...)
- pstorm 3y agoCouldn't you just compare the similarity of the embeddings? I imagine that would work in the vast majority of cases and save a lot of LLM calls.
- janchorowski 3y agoThat's a good idea, the deduplication criterion is easy to change, using an llm is faster to get started, but after a while a corpus of decisions is created and can be used to either select another mechanism, or e.g. train one on top of bert embeddings.