3 ms·

It is maybe counter intuitive, but I think modifying the embedding vectors themselves is not a good idea. Tunable cross-encoders are much more interesting imo

• lildaemon 3y ago
  That's an interesting idea! Do you say that because you expect a tunable cross-encoder to have better performance? I think depending on how you implement it, the embedding model that I am building is like a cross-encoder that is essentially one complicated attention layer. But outside of this special case, a cross-encoder is not computationally efficient. That is, if you implement the cross encoder as a feedforward NN with linear transformations with activations in between, that computes a similarity score, then in order to do semantic search over a document database, you'd have to compute the similarity between a query and each document, which is computationally expensive. In contrast, with the model I'm building, you apply the model to the document embedding to modify it, and store it in a vector database. You don't need to apply it queries, just when you are saving the document. To retrieve a document with high similarity, you just need to do the cosine similarity, and if you use a vector database, they have efficient algorithms that only need a cosine similarity of a subset of the documents, which is computationally more efficient. In the special case that the cross-encoder model takes the form of a single complicated attention computation, i.e. y \cdot f(x, \theta) (where f is a neural network, x is the document embedding, and \theta are the parameters of the NN) then this would be a type of cross-encoder, and you could just compute f(x, \theta) for each document, and use cosine similarity to retrieve it.