6 ms·

Quaternion Knowledge Graph Embeddings (2019)

100 points by teleforce 3y ago

palad1n 3y ago
Is anyone using this? It's from 2019.
- cess11 3y ago
  If someone is it's probably pretty niche. RDF is a bit simpler and still not commonly used due to its perceived complexity compared to RDBMS tables or JSON.
  - nl 3y ago
    This completely misunderstands what this is. In this context "knowledge graph" means a representation of knowledge independent of it's serialization format (ie, RDF vs whatever). It's usable wherever you have a key value (key:value of relationship) or a triple (key:relationship:something). This is probably 85% of everything that is stored in a database, with the exception being paths that vary enough that you can't use the path as a key or relationship. In particular, most trees are suitable if you use collapse the path down to a single relationship. I've done a bunch of work on graph embedding. They are very effective for use in anything that can be thought of as a recommendation system ("this person would like these books") or similarity ("this person is similar to these people"). Back when I was working on them I found Starspace wonderfully easy and effective: https://github.com/facebookresearch/StarSpace https://github.com/facebookresearch/StarSpace
    - cess11 3y ago
      I didn't compare RDF directly to the technique in TFA.
    - DrDroop 3y ago
      It is wrong the call this a "complete misunderstanding" as the paper clearly deals with triplets and formulates a different encoding scheme for entities and relations. Ok, so it is not RDF or even hyper graphs but it is dealing with a graph structure.
      - nl 3y ago
        Sure, but as I explain almost everything can fit that structure. The misunderstanding is the " probably pretty niche" part since related techniques (graph embedding generally) are so widely used.
- VHRanger 3y ago
  These exotic (non linear algebra based) embedding representations are often slow to take off unless they have an obvious use case. The other one that I've always been curious of is Poincarré Embeddings [1] - where the embedding also has a hierarchical representation of the space. There's issues with these becoming popular: 1. Querying the embeddings requires more math knowledge than just "lol cosine similarity". This also requires to write code for the query 2. You can often easily match the performance with regular embeddings by just adding dimensions and training more. So the advantage of exotic embeddings has to be with the information in the more complex mathematical abstraction. So they need a killer usecase to become popular, it's hard to move the needle. [1]: https://arxiv.org/abs/1705.08039 https://arxiv.org/abs/1705.08039
  - nico 3y ago
    Super interesting Do you think it makes sense to have a group of models, each with more ad-hoc embeddings, and coordinate them to respond according to the domain of the input? Do multi-modal models use the same embedding type/structure for an image, sound, text?
    - VHRanger 3y ago
      I think your first question is open for people to explore. The answer to the second is yes - it's all vector embeddings, and they're aligned to each other by finding a dataset that matches pairs (eg. images with captions) The real use for exotic embeddings will have to be in analyzing the embeddings themselves I think, otherwise it's easier to shove normal vectors downstream into other models.
- adultSwim 3y ago
  The paper was cited over 500 times, https://scholar.google.com/scholar?cites=15051838971302171507&as_sdt=5,33&sciodt=0,33&hl=en https://scholar.google.com/scholar?cites=1505183897130217150...
- adultSwim 3y ago
  QuatE is implemented in PyKEEN, a library for KnowEdge graph EmbeddiNgs, https://github.com/pykeen/pykeen https://github.com/pykeen/pykeen
ziofill 3y ago
I always wondered why bother with quaternions and abstruse math when at the end of the day they behave like some matrices of floats.
- hi-v-rocknroll 3y ago
  For 3D graphics, it makes rotation and translation simpler by adding another dimension. Maybe you had a less-than-stellar teacher for this topic. If so, here's a master lecturer elucidating it: https://youtu.be/mHVwd8gYLnI https://youtu.be/mHVwd8gYLnI https://en.wikipedia.org/wiki/Transformation_matrix#Examples_in_3D_computer_graphics https://en.wikipedia.org/wiki/Transformation_matrix#Examples...
  - klyrs 3y ago
    To be fair, some of us learned about quaternions over finite fields before anything about 3D graphics. The first time I heard about this application of them, I was gobsmacked. Saying this because my professor for that course was quite excellent.
    - hi-v-rocknroll 3y ago
      Nice! For undergrad, I was 2 courses shy of a math major || CS major, but then I saw neither was ABET accredited. Wat? Not to worry, because the CS & Eng major was accredited but required physics and chemistry for engineers series, interfacing, computer architecture, and a load and store (or was it compare and swap?) more electives approximating an EE major. It was really an EE/CS program but they couldn't call it that for historical and intercampus political reasons. I think most of the difference in mastery of a given topic in an academic setting comes down to the skill and interactivity with a competent and expert lecturer who also has expertise in the additional domains of public speaking and teaching. Ken Joy and Sean Davis were the virtuosos of teaching at UC Davis: massive and accessible brains. Intellectual curiosity and an semi-extroverted personality help too.
    - lanstin 3y ago
      Or over the reals. I have studied quaternions a number of times but never studied 3d graphics. They are interesting algebraically as part of the sequence of divison algebras reals, complex numbers, quaternions, octonions, nothing. https://en.m.wikipedia.org/wiki/Hurwitz%27s_theorem_(composition_algebras) https://en.m.wikipedia.org/wiki/Hurwitz%27s_theorem_(composi...
vslira 3y ago
I'm not nearly smart enough to state this confidently, but doesn't [1] imply that exotic embeddings can always be replaced by larger Euclidean embeddings? [1] https://en.wikipedia.org/wiki/Nash_embedding_theorems https://en.wikipedia.org/wiki/Nash_embedding_theorems
- hansvm 3y ago
  No. Consider a square (with graph shortest path distances) for example. All euclidean embeddings have a minimum error of 20-40% or so. If you try to embed any graph containing that subgraph (embedding friends, advertisers, words, ...), you'll similarly have guaranteed error. Relaxing the square to a squircle, you'll see that error even for a really simple manifold. Riemannian manifolds are a bit more special.

100 points by teleforce 3y ago

palad1n 3y ago

Is anyone using this? It's from 2019.

cess11 3y ago
If someone is it's probably pretty niche. RDF is a bit simpler and still not commonly used due to its perceived complexity compared to RDBMS tables or JSON.
- nl 3y ago
  This completely misunderstands what this is. In this context "knowledge graph" means a representation of knowledge independent of it's serialization format (ie, RDF vs whatever). It's usable wherever you have a key value (key:value of relationship) or a triple (key:relationship:something). This is probably 85% of everything that is stored in a database, with the exception being paths that vary enough that you can't use the path as a key or relationship. In particular, most trees are suitable if you use collapse the path down to a single relationship. I've done a bunch of work on graph embedding. They are very effective for use in anything that can be thought of as a recommendation system ("this person would like these books") or similarity ("this person is similar to these people"). Back when I was working on them I found Starspace wonderfully easy and effective: https://github.com/facebookresearch/StarSpace https://github.com/facebookresearch/StarSpace
  - cess11 3y ago
    I didn't compare RDF directly to the technique in TFA.
  - DrDroop 3y ago
    It is wrong the call this a "complete misunderstanding" as the paper clearly deals with triplets and formulates a different encoding scheme for entities and relations. Ok, so it is not RDF or even hyper graphs but it is dealing with a graph structure.
    - nl 3y ago
      Sure, but as I explain almost everything can fit that structure. The misunderstanding is the " probably pretty niche" part since related techniques (graph embedding generally) are so widely used.
VHRanger 3y ago
These exotic (non linear algebra based) embedding representations are often slow to take off unless they have an obvious use case. The other one that I've always been curious of is Poincarré Embeddings [1] - where the embedding also has a hierarchical representation of the space. There's issues with these becoming popular: 1. Querying the embeddings requires more math knowledge than just "lol cosine similarity". This also requires to write code for the query 2. You can often easily match the performance with regular embeddings by just adding dimensions and training more. So the advantage of exotic embeddings has to be with the information in the more complex mathematical abstraction. So they need a killer usecase to become popular, it's hard to move the needle. [1]: https://arxiv.org/abs/1705.08039 https://arxiv.org/abs/1705.08039
- nico 3y ago
  Super interesting Do you think it makes sense to have a group of models, each with more ad-hoc embeddings, and coordinate them to respond according to the domain of the input? Do multi-modal models use the same embedding type/structure for an image, sound, text?
  - VHRanger 3y ago
    I think your first question is open for people to explore. The answer to the second is yes - it's all vector embeddings, and they're aligned to each other by finding a dataset that matches pairs (eg. images with captions) The real use for exotic embeddings will have to be in analyzing the embeddings themselves I think, otherwise it's easier to shove normal vectors downstream into other models.
adultSwim 3y ago
The paper was cited over 500 times, https://scholar.google.com/scholar?cites=15051838971302171507&as_sdt=5,33&sciodt=0,33&hl=en https://scholar.google.com/scholar?cites=1505183897130217150...
adultSwim 3y ago
QuatE is implemented in PyKEEN, a library for KnowEdge graph EmbeddiNgs, https://github.com/pykeen/pykeen https://github.com/pykeen/pykeen

ziofill 3y ago

I always wondered why bother with quaternions and abstruse math when at the end of the day they behave like some matrices of floats.

hi-v-rocknroll 3y ago
For 3D graphics, it makes rotation and translation simpler by adding another dimension. Maybe you had a less-than-stellar teacher for this topic. If so, here's a master lecturer elucidating it: https://youtu.be/mHVwd8gYLnI https://youtu.be/mHVwd8gYLnI https://en.wikipedia.org/wiki/Transformation_matrix#Examples_in_3D_computer_graphics https://en.wikipedia.org/wiki/Transformation_matrix#Examples...
- klyrs 3y ago
  To be fair, some of us learned about quaternions over finite fields before anything about 3D graphics. The first time I heard about this application of them, I was gobsmacked. Saying this because my professor for that course was quite excellent.
  - hi-v-rocknroll 3y ago
    Nice! For undergrad, I was 2 courses shy of a math major || CS major, but then I saw neither was ABET accredited. Wat? Not to worry, because the CS & Eng major was accredited but required physics and chemistry for engineers series, interfacing, computer architecture, and a load and store (or was it compare and swap?) more electives approximating an EE major. It was really an EE/CS program but they couldn't call it that for historical and intercampus political reasons. I think most of the difference in mastery of a given topic in an academic setting comes down to the skill and interactivity with a competent and expert lecturer who also has expertise in the additional domains of public speaking and teaching. Ken Joy and Sean Davis were the virtuosos of teaching at UC Davis: massive and accessible brains. Intellectual curiosity and an semi-extroverted personality help too.
  - lanstin 3y ago
    Or over the reals. I have studied quaternions a number of times but never studied 3d graphics. They are interesting algebraically as part of the sequence of divison algebras reals, complex numbers, quaternions, octonions, nothing. https://en.m.wikipedia.org/wiki/Hurwitz%27s_theorem_(composition_algebras) https://en.m.wikipedia.org/wiki/Hurwitz%27s_theorem_(composi...

vslira 3y ago

I'm not nearly smart enough to state this confidently, but doesn't [1] imply that exotic embeddings can always be replaced by larger Euclidean embeddings? [1] https://en.wikipedia.org/wiki/Nash_embedding_theorems https://en.wikipedia.org/wiki/Nash_embedding_theorems

hansvm 3y ago
No. Consider a square (with graph shortest path distances) for example. All euclidean embeddings have a minimum error of 20-40% or so. If you try to embed any graph containing that subgraph (embedding friends, advertisers, words, ...), you'll similarly have guaranteed error. Relaxing the square to a squircle, you'll see that error even for a really simple manifold. Riemannian manifolds are a bit more special.