3 ms·

Candidly, what the hell am I reading? The fundamental thing a network operates on is a (often massive) multidimensional tensor. GPT3+ has per-token dimensionali

Candidly, what the hell am I reading? The fundamental thing a network operates on is a (often massive) multidimensional tensor. GPT3+ has per-token dimensionality in the thousands. Not only do we already operate on incredibly high dimensional objects, it’s the only thing modern neural networks do.

ekleraki 4y ago
If we are being pedantic, transformers operate on integer vectors, and in the process turn them into high dimensional matrices.
bckr 4y ago
I actually had to check the publish date on this. The first few paragraphs are absurd.
mdp2021 4y ago
But they are not transparently symbolic.
- AbrahamParangi 4y ago
  They are not but there’s a lot of lifting being done by assuming a “hypervector” (an embedding, to literally everyone else) for a concept. Does such a separable embedding exist and do you have a good way to find it? Seems like one of those things where, oops, it’s just worse than gradient descent on massive functions.
bob1029 4y ago
I had the same reaction. The whole time I was expecting a paragraph to explain why this is fundamentally different from binary classification techniques like SVM, which explicitly uses a hyperplane in high-dimensional space to divide the classes.
yazzku 4y ago
Hah, same here. How is [colour, shape] or [blue, red, square, circle] not a vector? How does a NN work if not by operating on vectors/matrices/tensors? This Quanta Magazine is absolute horseshit. Might as well just remove all that text and link to the author's paper. But hey, it makes for a nice article with a progress bar that animates as you scroll up and down. The papers: https://neuroscience.berkeley.edu/wp-content/uploads/2016/05/Pentti_Kanerva-NICE2016-slides.pdf https://neuroscience.berkeley.edu/wp-content/uploads/2016/05... https://redwood.berkeley.edu/wp-content/uploads/2022/05/kanerva2022hdmss.pdf https://redwood.berkeley.edu/wp-content/uploads/2022/05/kane...
- HybridCurve 4y ago
  If they linked the papers, it would be too obvious how poorly they described what they were about.

ekleraki 4y ago

If we are being pedantic, transformers operate on integer vectors, and in the process turn them into high dimensional matrices.

bckr 4y ago

I actually had to check the publish date on this. The first few paragraphs are absurd.

mdp2021 4y ago

But they are not transparently symbolic.

AbrahamParangi 4y ago
They are not but there’s a lot of lifting being done by assuming a “hypervector” (an embedding, to literally everyone else) for a concept. Does such a separable embedding exist and do you have a good way to find it? Seems like one of those things where, oops, it’s just worse than gradient descent on massive functions.

bob1029 4y ago

I had the same reaction. The whole time I was expecting a paragraph to explain why this is fundamentally different from binary classification techniques like SVM, which explicitly uses a hyperplane in high-dimensional space to divide the classes.

yazzku 4y ago

Hah, same here. How is [colour, shape] or [blue, red, square, circle] not a vector? How does a NN work if not by operating on vectors/matrices/tensors? This Quanta Magazine is absolute horseshit. Might as well just remove all that text and link to the author's paper. But hey, it makes for a nice article with a progress bar that animates as you scroll up and down. The papers: https://neuroscience.berkeley.edu/wp-content/uploads/2016/05/Pentti_Kanerva-NICE2016-slides.pdf https://neuroscience.berkeley.edu/wp-content/uploads/2016/05... https://redwood.berkeley.edu/wp-content/uploads/2022/05/kanerva2022hdmss.pdf https://redwood.berkeley.edu/wp-content/uploads/2022/05/kane...

HybridCurve 4y ago
If they linked the papers, it would be too obvious how poorly they described what they were about.