You should be able to use embeddings for this (sort by the cosine similarity). eg OpenAI has an off the shelf offering: https://beta.openai.com/docs/guides/embe

You should be able to use embeddings for this (sort by the cosine similarity). eg OpenAI has an off the shelf offering: https://beta.openai.com/docs/guides/embeddings/what-are-embeddings https://beta.openai.com/docs/guides/embeddings/what-are-embe... We used something similar to build a “similar articles” feature & it gave us de-duplication essentially for free.