3 ms·
That math is for random projections? Note that JL lemma is a worst case guarantee and in practice, there's a lot more distortion tolerance than the given bounds
by Vetch 1y ago
That math is for random projections? Note that JL lemma is a worst case guarantee and in practice, there's a lot more distortion tolerance than the given bounds would suggest. Concepts tend to live in a space of much lower intrinsic dimensionality than the data's and we often care more about neighbor and rank information than precise pair-wise distances.
Also, JL is only a part of the story for the transformers.
- yorwba 1y agoJohnson-Lindenstrauss is an example of a probabilistic existence argument: the probability of a random projection having low error is nonzero, therefore a low-error projection must exist. That doesn't mean any given random projection can be expected to have low error, although if you keep rerolling often enough, you'll eventually find one. The existence argument does only provide a lower bound on the number of dimensions that can be represented with low error, but there's not necessarily much room for improvement left.