3 ms·

You’re correct. There are lots of ways to project a string of embeddings back onto a single embedding - lots of times however these simpler methods do something

by pizza 3y ago

You’re correct. There are lots of ways to project a string of embeddings back onto a single embedding - lots of times however these simpler methods do something like pooling the sum if the word embeddings by averaging and hoping that the answer embedding has a small angle with that projection (cosine similarity). You could try adding a way to focus on individual tokens, pairwise interactions, and more sophisticated ways of pooling and projecting embeddings, and then boom you’ve basically just created transformers.

naraga 3y ago
oh boy, this looks like an alchemy to me. on one hand one cant deny success of llm's on another we are shifting the responsibiliteis to non-deterministic fuzzy duck-taped functions.
- weinzierl 3y ago
  My impression is similar and that's why I asked the question. Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition? Asked more abstractly: In a vector space like SBERT's, can I expect questions and answers about the same topic to lie near each other? Especially will the correct answers lie near their question?
  - famouswaffles 3y ago
    >Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition? This not how 99% of embedding models work(though you can train for specific tasks) but as it turns out such a thing is possible and is beneficial. Instruct tuned embedding model here - https://instructor-embedding.github.io/ https://instructor-embedding.github.io/
    - pizza 3y ago
      This kind of thing is going to seem so obvious in hindsight in six months once all the sota methods start converge upon similar improvements lol. It’s still so early, I keep telling myself.

by pizza 3y ago

naraga 3y ago

oh boy, this looks like an alchemy to me. on one hand one cant deny success of llm's on another we are shifting the responsibiliteis to non-deterministic fuzzy duck-taped functions.

weinzierl 3y ago
My impression is similar and that's why I asked the question. Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition? Asked more abstractly: In a vector space like SBERT's, can I expect questions and answers about the same topic to lie near each other? Especially will the correct answers lie near their question?
- famouswaffles 3y ago
  >Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition? This not how 99% of embedding models work(though you can train for specific tasks) but as it turns out such a thing is possible and is beneficial. Instruct tuned embedding model here - https://instructor-embedding.github.io/ https://instructor-embedding.github.io/
  - pizza 3y ago
    This kind of thing is going to seem so obvious in hindsight in six months once all the sota methods start converge upon similar improvements lol. It’s still so early, I keep telling myself.