10 ms·
Deep Quaternion Networks (2017) [pdf]
- adamnemecek 9y agoThere's a lot of space to be explored at the intersection of ml and hypercomplex numbers. There's a Clifford svm that unlike a regular sam which learns a hyperplane, learns any manifold.
- dbranes 9y agoA stupid person would respond to this comment with "Look into work by Bayro-Corrochano. I can't find a public link."
- adamnemecek 9y agoLook into work by Bayro-Corrochano. I can't find a public link.
- mike_n 9y agohttps://pdfs.semanticscholar.org/c3b3/9a378e6f805d6cae5b05f59cbcd734b95b60.pdf https://pdfs.semanticscholar.org/c3b3/9a378e6f805d6cae5b05f5...
- MrQuincle 9y agoThat's also what I thought. This is a dissertation on multilayer perceptrons using backprop over Clifford type neurons by Sven Buchholz: https://www.informatik.uni-kiel.de/inf/Sommer/doc/Dissertationen/Sven_Buchholz/diss.pdf https://www.informatik.uni-kiel.de/inf/Sommer/doc/Dissertati...
- mike_n 9y agoif quaternions, why not octonions?
- danharaj 9y agoThe lack of associativity might suck.
- mike_n 9y agoperhaps, but i would think that if you threw another tool (a different way to relate some inputs) at a neural net, it might figure out some way to exploit it, even if it's not clear to you or me. just like a RL agent sometimes finds + exploits bugs in the environment.
- wolfgke 9y agoThe lack of commutativity for quaternion multiplication might suck, too.
- danharaj 9y agoIt doesn't because neural networks are already (matrices + nonlinearity) over the ground field and therefore all but the simplest possible neural networks are non-commutative anyway. Non-associativity is much more troublesome because it sabotages the most important property of neural networks for deep learning, that you can compose them by feeding outputs of one as inputs into another. Take out the associativity and you've taken out the "deep" in "deep learning". Edit: On further reflection, the non-commutativity of neural networks is also a crucial component of machine learning. Without it, a neural network can't make a decision at one layer that depends on its decisions at a previous level!
- repsilat 9y agoI'm not sure I buy this. If - your network structure is fixed, and - you always evaluate the matrix in a given order, and - you are careful/smart about how you train the weights (Ok, I haven't thought through the ramifications here...) then I'm not sure you care much about either commutativity or associativity. Maybe a lack of associativity makes backprop impossible, and maybe commutativity makes "Google deep dream" impossible (no idea), but I don't quite agree with the "composability" objection to a lack of associativity and I don't understand the objection to a lack of commutativity sorry.
- doyoulikeworms 9y agoI’m able to follow neither the article nor the discussion. What would I have to learn in order to be able to? Even if it was, like, years of studying. I’m just curious how deep this rabbit hole is.
- theoh 9y agoQuaternions are an extension of the idea of complex numbers. Complex numbers have a real and an imaginary part, while quaternions have a real part and multiple imaginary parts (3). So the basic idea is that these richer types of number, when used to build a network (instead of plain real numbers) have benefits. So to get started with reading this paper you just need to learn about deep learning, and then also the very basics of quaternions, which would be taught in, for example, a first course on abstract algebra.
- smadge 9y agoI don’t think quaternions would be taught in a typical first course of abstract algebra. Do you know of a textbook where they are featured prominently?
- lasagnaphil 9y agoProbably some of the introductory level Computer Graphics books. The field uses a lot of quaternions, so they might have a good explanation about it.
- dsacco 9y agoI don't know of any book which features them "prominently", but I also don't think you'd really need one. They are taught in various abstract algebra books, they're just taught in the fashion of, "Here's an exercise that introduces a peripheral topic it's useful to know about." For example, groups and rings of quaternions show up in MacLane & Birkhoff's Algebra (62, 426; 282) and Lang's Algebra (9, 545, 723, 758). Edit: In an effort to find more applied information I put down my math books and picked up the information theoretic ones. You can find more information about the use of quaternions in the two volume Handbook of Digital Signal Processing and Salomon's Data Compression. More generally, when quaternions aren't explicitly referred to it's helpful to look up the coverage of complex rotations, especially with respect to the Discrete Fourier Transform. For a discussion of rotations with quaternions in the context of animation, this is a reasonably short paper: http://www.cs.cmu.edu/~kiranb/animation/p245-shoemake.pdf http://www.cs.cmu.edu/~kiranb/animation/p245-shoemake.pdf.
- loxias 9y agoDoes use of complex numbers really provide improvement? How does this work? (other than cramming 2 numbers into 1... which itself is suspect... the complex plane has the same cardnality as the reals...)
- moultano 9y agoIt isn't just the size of the space that matters, but how smooth and connected it is.
- norrius 9y agoAre quaternions more smooth and connected than complex numbers? My understanding was that higher-dimensional hypercomplex numbers tend to lose useful structure. I'm also curious what being connected in this context means.
- loxias 9y agoI'm still having trouble wrapping my head around how complex numbers are an improvement over the reals. >.<
- wespiser_2018 9y agoYou can use complex numbers number to represent higher dimensional objects using only primitive operations, scalar values, and an imaginary number for each dimension. However, computing with these values is significantly more challenging that real vectors. This book on 'Geometric Algebra' starts to explain: http://www2.montgomerycollege.edu/departments/planet/planet/Numerical_Relativity/bookGA.pdf http://www2.montgomerycollege.edu/departments/planet/planet/...
- yorwba 9y agoComplex numbers are a particular 2d slice of 2x2 matrices that happen to capture rotation and other periodic phenomena very well. If you are trying to solve some problem that you suspect to involve periodicity, focusing on complex numbers helps you get there faster.
- deleted 9y ago[deleted]
- Jeff_Brown 9y agoI skipped to the table at the end. The gains don't seem enormous. Is there a kind of problem where we would expect quaternions to perform dramatically better than other kinds of numbers?
- highd 9y agoNot to mention that it appears they're comparing against networks of the same architecture. If you build your quaternion components with with same number types as your reals you effectively have 4 times the number of parameters, which could be most of the benefit. They should also benchmark against similar architectures with equivalent parameter counts.
- deleted 9y ago[deleted]
- gaudetcj 9y agoHi I'm one of the authors of this paper. Sorry if it unclear, but we reduce the number of filters per layer to account for this. The quaternion networks actually have fewer parameters.
- gaudetcj 9y agoThis paper was mostly to lay out the framework and give out the Keras layers for others to use. We expect that the biggest improvements will come from segmentation where the gains may come from treating each color channel as an imaginary axis. And from architectures like PointNet https://arxiv.org/abs/1612.00593 https://arxiv.org/abs/1612.00593.
- kuwze 9y agoI remember being introduced to quaternions recently by this post[0] which recommended this book[1]. [0]: https://www.haroldserrano.com/blog/best-books-to-develop-a-game-engine https://www.haroldserrano.com/blog/best-books-to-develop-a-g... [1]: https://www.amazon.com/Quaternions-Computer-Graphics-John-Vince/dp/0857297597/ https://www.amazon.com/Quaternions-Computer-Graphics-John-Vi...
- dschuetz 9y agoYou had me on "The field of deep learning...". Sounds seriously scientific to me. What's the next big flashy field? "Deep thought"? Oh, nope, Douglas Adams already covered that one.
- godelmachine 9y agoMay I ask its applications?
- eleitl 9y agoThis assumes numerics is free at very large scale, which is not reasonable if you want to create efficient biologically inspired AI.
- freethemullet 9y agoTurns out we can approximate many-body wavefunctions using networks with complex weights: https://arxiv.org/abs/1606.02318 https://arxiv.org/abs/1606.02318.
- thesz 9y agoThe fine article generalizes complex numbers to quaternions. Okay. But quaternions are themselves generalized by Geometric Algebra. And there is a plenty of information about use of GA in the field of neural computing: https://arxiv.org/pdf/1305.5663.pdf https://arxiv.org/pdf/1305.5663.pdf (page 3). For example, universal approximation theorem for GA is presented at https://www.informatik.uni-kiel.de/inf/Sommer/doc/Dissertationen/Sven_Buchholz/diss.pdf https://www.informatik.uni-kiel.de/inf/Sommer/doc/Dissertati... I think that fine article is a step back.
- enriquto 9y agogeometric algebra is the Rust of mathematics. You guys cannot simply let people do their work without trying to evangelize ?
- thesz 9y agoIn my opinion, GA is Haskell of mathematics. Just like Haskell subsumes Rust as a library (OCaml does [1], thus Haskell does too), GA subsumes quaternions (which are Rust here). Using subpar tool is not a good practice if better tool is readily available. And it is better tooling what I am trying to advertise here. [1] https://news.ycombinator.com/item?id=16597329 https://news.ycombinator.com/item?id=16597329
- enriquto 9y agoless general does not necessarily mean subpar. On the contrary.
- cgearhart 9y agoThanks for sharing this. There's a lot to digest in there, but there were a few highlights that stood out as possibly relevant to the OP paper. > Theorem 6.4 ([2]) Complex FCMLPs having (6.9) as activation function are only universal approximators in L∞ for the class of analytic functions, but not for the class of complex continuous functions. > ... the complex numbers (C0,1) are a subalgebra of the quaternions (C0,2). Hence the quaternionic logistic function is also unbounded. Neither could it give rise to universal approximation (w.r.t. L∞) since this does not hold for the complex case. One may argue that such things become more and more less important when proceeding to higher dimensional algebras since less and less components are affected. This is somehow true, but it hardly justify the efforts. > ... Summarising all of the above the case of Complex FCMLPs looks settled down in a negative way. ... Hence Complex FCMLPs remain not very promising. Unless I'm misreading, it seems already known that you _can_ use complex numbers (or quaternions) in neural networks...but you don't really gain anything from doing it.
- naveen99 9y agoMathoma videos on geometric algebra: https://youtu.be/ERpcSJzX448 https://youtu.be/ERpcSJzX448