5 ms·

Before reading the paper's obliquely mentioned in the article: How is this different from embeddings? After reading the papers: How is this different from embe

Before reading the paper's obliquely mentioned in the article: How is this different from embeddings? After reading the papers: How is this different from embeddings?

timmb 4y ago
My understanding is limited, but I'm guessing it's to do with the algebraic properties of the embedding vectors. I'm familiar with embeddings that you can add and subtract, which may reveal concepts existing as linear directions within the embedding space. Here, they're talking about multiplying, dividing and permuting vectors. Multiplying combines concepts, adding creates a superposition. They also mention randomly selecting embeddings for the concepts in mind. So my guess is that instead of one-hot encoding the classifier, they instead use random encodings on the output, and are working to give those encodings desirable properties. I would also hazard a guess that the random vectors they choose are close to zero in most components.
- randcraw 4y ago
  Or a high-dimensional word-to-vec that can represent compound / complex objects?
- chombier 4y ago
  This is my understanding as well. There a somewhat accessible introductory video I found useful [0]. The algebra makes it possible to encode sets, key/value associations and sequences to build a knowledge base, and the dot product provides a similarity measure for querying the base. IIUC the key is that for large space dimensions, any two random vectors (say with uniform distribution over {-1, +1}^d) are almost guaranteed to be near- orthogonal. This makes it easy to add new items to the base (by sampling a new random vector and updating the base using algebraic operations with other items), yet the amount of noise introduced by near-orthogonality remains controlled and can be filtered out to keep the algebraic structure working as the base grows. Honestly it seems a bit too good to be true, I'd be very interested to see what are the tradeoffs in practice. [0] https://www.youtube.com/watch?v=oB_mHCurNCI https://www.youtube.com/watch?v=oB_mHCurNCI
chaxor 4y ago
[dead]
abecedarius 4y ago
All three of the papers linked to are closed access. (Did I miss one?) If anyone can link a favorite review paper on this approach, you'd be doing us a favor. Edit: one of them does link to a preprint version: https://arxiv.org/abs/2203.04571 https://arxiv.org/abs/2203.04571
- mdp2021 4y ago
  Great, and the implementation (Neuro-vector-symbolic architectures): https://github.com/IBM/neuro-vector-symbolic-architectures https://github.com/IBM/neuro-vector-symbolic-architectures
  - westurner 4y ago
    "A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matrices" (2023) https://arxiv.org/abs/2203.04571 https://arxiv.org/abs/2203.04571 : > […] our proposed neuro-vector-symbolic architecture (NVSA) [implements] powerful operators on high-dimensional distributed representations that serve as a common language between neural networks and symbolic AI. The efficacy of NVSA is demonstrated by solving the Raven's progressive matrices datasets. Compared to state-of-the-art deep neural network and neuro-symbolic approaches, end-to-end training of NVSA achieves a new record of 87.7% average accuracy in RAVEN, and 88.1% in I-RAVEN datasets. Moreover, compared to the symbolic reasoning within the neuro-symbolic approaches, the probabilistic reasoning of NVSA with less expensive operations on the distributed representations is two orders of magnitude faster.

Before reading the paper's obliquely mentioned in the article: How is this different from embeddings? After reading the papers: How is this different from embeddings?

timmb 4y ago

My understanding is limited, but I'm guessing it's to do with the algebraic properties of the embedding vectors. I'm familiar with embeddings that you can add and subtract, which may reveal concepts existing as linear directions within the embedding space. Here, they're talking about multiplying, dividing and permuting vectors. Multiplying combines concepts, adding creates a superposition. They also mention randomly selecting embeddings for the concepts in mind. So my guess is that instead of one-hot encoding the classifier, they instead use random encodings on the output, and are working to give those encodings desirable properties. I would also hazard a guess that the random vectors they choose are close to zero in most components.

randcraw 4y ago
Or a high-dimensional word-to-vec that can represent compound / complex objects?
chombier 4y ago
This is my understanding as well. There a somewhat accessible introductory video I found useful [0]. The algebra makes it possible to encode sets, key/value associations and sequences to build a knowledge base, and the dot product provides a similarity measure for querying the base. IIUC the key is that for large space dimensions, any two random vectors (say with uniform distribution over {-1, +1}^d) are almost guaranteed to be near- orthogonal. This makes it easy to add new items to the base (by sampling a new random vector and updating the base using algebraic operations with other items), yet the amount of noise introduced by near-orthogonality remains controlled and can be filtered out to keep the algebraic structure working as the base grows. Honestly it seems a bit too good to be true, I'd be very interested to see what are the tradeoffs in practice. [0] https://www.youtube.com/watch?v=oB_mHCurNCI https://www.youtube.com/watch?v=oB_mHCurNCI

chaxor 4y ago

[dead]

abecedarius 4y ago

All three of the papers linked to are closed access. (Did I miss one?) If anyone can link a favorite review paper on this approach, you'd be doing us a favor. Edit: one of them does link to a preprint version: https://arxiv.org/abs/2203.04571 https://arxiv.org/abs/2203.04571

mdp2021 4y ago
Great, and the implementation (Neuro-vector-symbolic architectures): https://github.com/IBM/neuro-vector-symbolic-architectures https://github.com/IBM/neuro-vector-symbolic-architectures
- westurner 4y ago
  "A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matrices" (2023) https://arxiv.org/abs/2203.04571 https://arxiv.org/abs/2203.04571 : > […] our proposed neuro-vector-symbolic architecture (NVSA) [implements] powerful operators on high-dimensional distributed representations that serve as a common language between neural networks and symbolic AI. The efficacy of NVSA is demonstrated by solving the Raven's progressive matrices datasets. Compared to state-of-the-art deep neural network and neuro-symbolic approaches, end-to-end training of NVSA achieves a new record of 87.7% average accuracy in RAVEN, and 88.1% in I-RAVEN datasets. Moreover, compared to the symbolic reasoning within the neuro-symbolic approaches, the probabilistic reasoning of NVSA with less expensive operations on the distributed representations is two orders of magnitude faster.