4 ms·
That might be true for one-hot vectors but it's not true for learned embedding through the lens of attention. That said, I only made to page 3/9 of the paper be
by superkuh 3y ago
That might be true for one-hot vectors but it's not true for learned embedding through the lens of attention. That said, I only made to page 3/9 of the paper before the mark-up for the math went over my head.
- _t89y 3y agoIt is true. And if you want to say anything about meaning this isn't even the right math.
- latency-guy2 3y agoIf we're talking about adding dimensionality and relying on the kernel, sure, but I only get so much incorrect by going from 1 to M dims. I can't of course be certain in all cases, but dimensions are typically (past experience, and using knowledge from word2vec experiments from years ago) derivative of higher dimensions. The kernel still operates on the same concept by applying a norm along with whatever weightings to each dim. Semantic understanding is still not there in my opinion, we might feign it by increasing specificity, but only so much. Largest contributor will likely still the determining factor rather than the series of smaller, more specific dimensions. I tested this using similar sentences as my original comment and failing in more scenarios than passing. I of course am biased since it may be given I did not select the right dimensions or measures.