4 ms·

The simplest way of building an MLP on top of the embeddings is simply to concatenate the embeddings and put some dense layers on top. However, if you use a "t

The simplest way of building an MLP on top of the embeddings is simply to concatenate the embeddings and put some dense layers on top. However, if you use a "two towers" approach and have several additional MLP layers on top of each embedding separately, and then a dense MLP on the concatenations of each tower, the individual tower MLP layers are an embedding transformation, and will improve retrieval. Like this: Final MLP / \ QP MLP DP MLP | | QE DE Now, you can apply the document preprocessor ("DP MLP") to your document embeddings ("DE") before storing them in the vector database, and apply the query preprocessor ("QP MLP") to your query embeddings ("QE") before querying your vector database. This should improve the precision and recall of your vector retrieval step beyond using e.g. raw LLM embeddings. Even better is for the final layer to just be cosine similarity, maximum inner product, or L2 distance, rather than having an MLP so you can just use a raw threshold (it's at least worth trying).

anon373839 3y ago
Interesting. This sounds like it would be similar to fine-tuning the embeddings, but with added benefit of learning different representations for the query and document. If you keep a distance/similarity measure as the final layer, then I'm assuming this isn't going to work with binary labels?
- jmalicki 3y ago
  If you have e.g. cosine distance as the final layer, if the label is 1 you reward the cosine distance being close to 1, if the label is 0 you reward the cosine distance being close to 0. The finetuning here is specific to optimizing for retrieval, which may be different than just matching documents, which can be an advantage. You may want to force the query and document finetunings to be the same, which makes a lot of sense, but the advantage in them differing can be that query strings are often rather short, and in a different sort of language structure than documents, so the differing query and document tunings can in some sense "normalize" queries and documents to be in the same space, when it works well.
  - anon373839 3y ago
    This has got me curious. I don't really understand how binary labels could give us embeddings that are well-ordered by distance. For a training pair where (Q, D) are highly similar and a pair where (Q, D) are just barely related, the model is being trained that they're the same distance apart. Is there something I'm not seeing here?
    - jmalicki 3y ago
      See contrastive losses and siamese networks, like here (where they use L2 distance): https://lilianweng.github.io/posts/2021-05-31-contrastive/ https://lilianweng.github.io/posts/2021-05-31-contrastive/ If documents are similar, you want the two embeddings to be close to each other, if they are dissimilar, you want them to be far apart. Binary relevance judgements of course don't necessarily produce an ordering, but usually over a large enough set of training examples some will be better matches than others. "Learning to Rank" gets you into all kinds of labels and losses if you want to go down that rabbit hoel.

anon373839 3y ago

Interesting. This sounds like it would be similar to fine-tuning the embeddings, but with added benefit of learning different representations for the query and document. If you keep a distance/similarity measure as the final layer, then I'm assuming this isn't going to work with binary labels?

jmalicki 3y ago
If you have e.g. cosine distance as the final layer, if the label is 1 you reward the cosine distance being close to 1, if the label is 0 you reward the cosine distance being close to 0. The finetuning here is specific to optimizing for retrieval, which may be different than just matching documents, which can be an advantage. You may want to force the query and document finetunings to be the same, which makes a lot of sense, but the advantage in them differing can be that query strings are often rather short, and in a different sort of language structure than documents, so the differing query and document tunings can in some sense "normalize" queries and documents to be in the same space, when it works well.
- anon373839 3y ago
  This has got me curious. I don't really understand how binary labels could give us embeddings that are well-ordered by distance. For a training pair where (Q, D) are highly similar and a pair where (Q, D) are just barely related, the model is being trained that they're the same distance apart. Is there something I'm not seeing here?
  - jmalicki 3y ago
    See contrastive losses and siamese networks, like here (where they use L2 distance): https://lilianweng.github.io/posts/2021-05-31-contrastive/ https://lilianweng.github.io/posts/2021-05-31-contrastive/ If documents are similar, you want the two embeddings to be close to each other, if they are dissimilar, you want them to be far apart. Binary relevance judgements of course don't necessarily produce an ordering, but usually over a large enough set of training examples some will be better matches than others. "Learning to Rank" gets you into all kinds of labels and losses if you want to go down that rabbit hoel.