5 ms·

I wonder how much better is this, compared to taking the average ( or some other aggregation) of embeddings with a smaller context length. Has anyone done a sim

by dcastm 3y ago

I wonder how much better is this, compared to taking the average ( or some other aggregation) of embeddings with a smaller context length. Has anyone done a similar comparison?

pietro72ohboy 3y ago
The issue with averaging is that over large inputs, it drowns out small signal. For example, there is a chance that it completely loses a reference to something made only in a single sentence somewhere in a large document.