11 ms·

Category Theory ∩ Machine Learning

118 points by bgavran 4y ago

rmdamiao 4y ago
Is this simply a consequence of exponential growth in CS publications driven by machine learning or is there something really going on here?
- adamnemecek 4y ago
  The field needs better foundations. CT is pretty good.
  - KRAKRISMOTT 4y ago
    No. It won't make a significant (if any at all) difference to effectiveness. Rewriting Pytorch in Haskell won't magically get you AGI.
    - adamnemecek 4y ago
      It's not about rewriting things in Haskell but about using CT to reason about architectures.
    - solomonb 4y ago
      No one was suggesting rewriting anything in Haskell afaict..
    - bawolff 4y ago
      You wouldn't expect improved foundations to increase effectiveness in the short term. And AGI is totally irrelavent here.
  - hgsgm 4y ago
    Why? How? The OP GitHub site doesn't promote any material that introduces the concepts at all. The "survey" paper at the top is nigh-impenetrable. I'm sure the category theorists are having fun modelling machine learning, but it doesn't show how machine learning benefits from the category theory.
    - adamnemecek 4y ago
      Residual connection serves as a feedback/trace a la trace in traced monoidal categories. That is one insight I have gleaned from CT.
      - resource0x 4y ago
        You beat me to this. Indeed, of all categories the monoidal ones are the most potent. Look how nicely they fit in crypto ledgers: https://www.cl.cam.ac.uk/events/syco/3/slides/Nester.pdf https://www.cl.cam.ac.uk/events/syco/3/slides/Nester.pdf \s
    - Yoric 4y ago
      Category Theory (just as all mathematical models for programming or subsets thereof) are building blocks for reasoning on what we build. Past applications of such mathematical models include: - programming languages with semantics that are better adapted to specific problems (e.g. Rust's ownership); - better compilers (see e.g. Haskell's supercompiler, which puts to shame `constexpr`-style features); - better static analyzers (e.g. better type systems, abstract interpretation, model checkers). In the case of Machine Learning, it might some day help us create Machine Learning that we can understand and trust better. Or it might fail. Or it might help us invent something different entirely, in 30 years.
      - riku_iki 4y ago
        why those approaches never picked up outside of some academia projects?..
        Yoric 4y ago
        These days, Microsoft requires model-checking proofs before accepting new device drivers. That's their secret weapon that finally got (mostly) rid of the BSOD. I suspect that Apple is also using model-checking at various layers, but I have no proof :)
        4y ago
- bgavran 4y ago
  OP here. The exponential growth in CS publication is much faster. This repository is simply a testament that CT is slowly ramping up. It's meant to show what kind of expressive power and breadth current CT models have, which to my knowledge isn't something that's well-known outside of our niche community.
  - bgavran 4y ago
    It's also meant to suggest where things are going (the kind of a chart I have in mind is this one https://twitter.com/bgavran3/status/1422206118688956420 https://twitter.com/bgavran3/status/1422206118688956420 ), though I understand this is something that deserves a much more substantial proof.
  - moralestapia 4y ago
    >The exponential growth in CS publication is much faster. So ... yes?
adamnemecek 4y ago
I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834 https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence between Hopf algebra and cyclical linear logic and Hopf algebras are related to zonotopes, which are polyhedra that have been used in verified numerical computation. I'm strongly convinced the LL connection can provide proofs over zonotopes which paves the way towards interpretable AI and will be central for XAI. I know this sounds too good to be true but Persi Diaconis has also written a paper that shows how useful Hopf algebras are in the context of Markov chains https://arxiv.org/abs/1206.3620 https://arxiv.org/abs/1206.3620 I'm working on a next gen Hopf algebra based machine learning framework. Join my discord if you want to discuss this further https://discord.cofunctional.ai https://discord.cofunctional.ai. ==== My account is currently rate limited so I will use this comment to respond to comments below. red_trumped: What about Hopf algebras do I not understand? gaze: Haha, it's been a while since I have commented about QC. What do I not understand about it? And what comment are you referring to?
- gexaha 4y ago
  could you advertise your research a bit less often, please? i see your post like literally almost every other day here
  - deleted 4y ago
    [deleted]
  - hgsgm 4y ago
    Or at least explain it in more accessible way. Every time Adam posts about the paper, it gets confused comments and no engagement on the content, because it's pretty deep graduate level pure math, which is occasionally seen but rare on HN.
    - red_trumpet 4y ago
donnowhy 4y ago
category theory is 'native 2-dimensional' math. i.e. category theory explains everything in terms of graphs, where a graph is made from two different sorts of 'entities', nodes and vertices i.e. categories and morphisms this being math, I wonder to which extent can category theory be re-expressed in terms of sets. perhaps a better question is if category theory can be re-expressed (or founded on) functions? lastly, I wonder if category theory can be expressed in terms of functions (i think maybe it can, without sets?) why shouldn't it be expressible in terms of sets (for some reason I don't think just sets are sufficient, may have to define functions (which possible in terms of sets) before 'expressing' categories starting with set theory)?
- voxl 4y ago
  Set theory is fine, see the (Stack Project)[https://stacks.math.columbia.edu/browse https://stacks.math.columbia.edu/browse] which develops a ton of modern Category Theory on ZFC (Zermelo-Fraenkel Set Theory with the Axiom of Choice) alone. Alternative foundations of mathematics (Set Theory, Category Theory, Type Theory, and all their variations) can all mutually interpret the other by just postulating sufficiently large universes. You don't pick or advocate one based off its ability to encode mathematics, but instead based on its ability to express your intention and ideas. Really its no different from programming language preference in my book.
- Koshkin 4y ago
  > i.e. categories and morphisms Objects, not categories.
- mydogcanpurr 4y ago
  > for some reason I don't think just sets are sufficient The reason you're looking for is that the category of sets is not a set.
  - vishal0123 4y ago
    While category of sets technically could not be expressed as a ZFC set, the idea behind the set theory is enough. Also you could add an axiom[0] in ZFC to make category of set a set. [0]: https://en.wikipedia.org/wiki/Grothendieck_universe https://en.wikipedia.org/wiki/Grothendieck_universe
umutisik 4y ago
It is tempting to believe that category theory will shed new light on and simplify machine learning, just like it did in algebraic geometry, algebraic topology and other mathematical things. This is wishful thinking. Folks who care about doing something useful should stay away from this content.
- lgas 4y ago
  I'd suggest providing some justification for your declaration if you want anyone to listen to you.
- epgui 4y ago
  That's a supremely anti-intellectual take.
AlexCoventry 4y ago
> Category Theory has been finding increasing applications in machine learning What's the most compelling application so far?
- bawolff 4y ago
  Application in the sense they are using it is probably different than the sense you are using it. Although its still probably a fair question regardless.
  - AlexCoventry 4y ago
    Any application where Category Theory is making it substantially easier to express the software or reason about it is fair game, from my perspective.
    - lgas 4y ago
      How about https://arxiv.org/abs/1803.05316 https://arxiv.org/abs/1803.05316?
eigenform 4y ago
I'm not experienced/well-read in either ML or CT, but awhile ago I remember hearing Tai-Danae Bradley equate "knowing a word by the company it keeps" to the Yoneda lemma, and I always thought that was kind of interesting (although I guess I'm not qualified enough to know whether that statement is useful or vacuous)
- haskellandchill 4y ago
  > knowing a word by the company it keeps I'm still shocked no one has developed language learning software along these lines. I had a prototype in the works for thai years ago but never got time to get it off the ground. using statistical models trained on web corpus for a language learning app seems like a no brainer. think of it like navigating a word as a point in a graph connected to every example context it is in, with associated words being clickable into similar context bundles. then make it differential between host and target language given a translation so you can see which contexts the translation fails and succeeds in.

118 points by bgavran 4y ago

Is this simply a consequence of exponential growth in CS publications driven by machine learning or is there something really going on here?

adamnemecek 4y ago
The field needs better foundations. CT is pretty good.
- KRAKRISMOTT 4y ago
  No. It won't make a significant (if any at all) difference to effectiveness. Rewriting Pytorch in Haskell won't magically get you AGI.
  - adamnemecek 4y ago
    It's not about rewriting things in Haskell but about using CT to reason about architectures.
  - solomonb 4y ago
    No one was suggesting rewriting anything in Haskell afaict..
  - bawolff 4y ago
    You wouldn't expect improved foundations to increase effectiveness in the short term. And AGI is totally irrelavent here.
- hgsgm 4y ago
  Why? How? The OP GitHub site doesn't promote any material that introduces the concepts at all. The "survey" paper at the top is nigh-impenetrable. I'm sure the category theorists are having fun modelling machine learning, but it doesn't show how machine learning benefits from the category theory.
  - adamnemecek 4y ago
    Residual connection serves as a feedback/trace a la trace in traced monoidal categories. That is one insight I have gleaned from CT.
    - resource0x 4y ago
      You beat me to this. Indeed, of all categories the monoidal ones are the most potent. Look how nicely they fit in crypto ledgers: https://www.cl.cam.ac.uk/events/syco/3/slides/Nester.pdf https://www.cl.cam.ac.uk/events/syco/3/slides/Nester.pdf \s
  - Yoric 4y ago
    Category Theory (just as all mathematical models for programming or subsets thereof) are building blocks for reasoning on what we build. Past applications of such mathematical models include: - programming languages with semantics that are better adapted to specific problems (e.g. Rust's ownership); - better compilers (see e.g. Haskell's supercompiler, which puts to shame `constexpr`-style features); - better static analyzers (e.g. better type systems, abstract interpretation, model checkers). In the case of Machine Learning, it might some day help us create Machine Learning that we can understand and trust better. Or it might fail. Or it might help us invent something different entirely, in 30 years.
    - riku_iki 4y ago
      why those approaches never picked up outside of some academia projects?..
      - Yoric 4y ago
        These days, Microsoft requires model-checking proofs before accepting new device drivers. That's their secret weapon that finally got (mostly) rid of the BSOD. I suspect that Apple is also using model-checking at various layers, but I have no proof :)
        4y ago
bgavran 4y ago
OP here. The exponential growth in CS publication is much faster. This repository is simply a testament that CT is slowly ramping up. It's meant to show what kind of expressive power and breadth current CT models have, which to my knowledge isn't something that's well-known outside of our niche community.
- bgavran 4y ago
  It's also meant to suggest where things are going (the kind of a chart I have in mind is this one https://twitter.com/bgavran3/status/1422206118688956420 https://twitter.com/bgavran3/status/1422206118688956420 ), though I understand this is something that deserves a much more substantial proof.
- moralestapia 4y ago
  >The exponential growth in CS publication is much faster. So ... yes?

adamnemecek 4y ago

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834 https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence between Hopf algebra and cyclical linear logic and Hopf algebras are related to zonotopes, which are polyhedra that have been used in verified numerical computation. I'm strongly convinced the LL connection can provide proofs over zonotopes which paves the way towards interpretable AI and will be central for XAI. I know this sounds too good to be true but Persi Diaconis has also written a paper that shows how useful Hopf algebras are in the context of Markov chains https://arxiv.org/abs/1206.3620 https://arxiv.org/abs/1206.3620 I'm working on a next gen Hopf algebra based machine learning framework. Join my discord if you want to discuss this further https://discord.cofunctional.ai https://discord.cofunctional.ai. ==== My account is currently rate limited so I will use this comment to respond to comments below. red_trumped: What about Hopf algebras do I not understand? gaze: Haha, it's been a while since I have commented about QC. What do I not understand about it? And what comment are you referring to?

gexaha 4y ago
could you advertise your research a bit less often, please? i see your post like literally almost every other day here
- deleted 4y ago
  [deleted]
- hgsgm 4y ago
  Or at least explain it in more accessible way. Every time Adam posts about the paper, it gets confused comments and no engagement on the content, because it's pretty deep graduate level pure math, which is occasionally seen but rare on HN.
  - red_trumpet 4y ago

donnowhy 4y ago

category theory is 'native 2-dimensional' math. i.e. category theory explains everything in terms of graphs, where a graph is made from two different sorts of 'entities', nodes and vertices i.e. categories and morphisms this being math, I wonder to which extent can category theory be re-expressed in terms of sets. perhaps a better question is if category theory can be re-expressed (or founded on) functions? lastly, I wonder if category theory can be expressed in terms of functions (i think maybe it can, without sets?) why shouldn't it be expressible in terms of sets (for some reason I don't think just sets are sufficient, may have to define functions (which possible in terms of sets) before 'expressing' categories starting with set theory)?

voxl 4y ago
Set theory is fine, see the (Stack Project)[https://stacks.math.columbia.edu/browse https://stacks.math.columbia.edu/browse] which develops a ton of modern Category Theory on ZFC (Zermelo-Fraenkel Set Theory with the Axiom of Choice) alone. Alternative foundations of mathematics (Set Theory, Category Theory, Type Theory, and all their variations) can all mutually interpret the other by just postulating sufficiently large universes. You don't pick or advocate one based off its ability to encode mathematics, but instead based on its ability to express your intention and ideas. Really its no different from programming language preference in my book.
Koshkin 4y ago
> i.e. categories and morphisms Objects, not categories.
mydogcanpurr 4y ago
> for some reason I don't think just sets are sufficient The reason you're looking for is that the category of sets is not a set.
- vishal0123 4y ago
  While category of sets technically could not be expressed as a ZFC set, the idea behind the set theory is enough. Also you could add an axiom[0] in ZFC to make category of set a set. [0]: https://en.wikipedia.org/wiki/Grothendieck_universe https://en.wikipedia.org/wiki/Grothendieck_universe

umutisik 4y ago

It is tempting to believe that category theory will shed new light on and simplify machine learning, just like it did in algebraic geometry, algebraic topology and other mathematical things. This is wishful thinking. Folks who care about doing something useful should stay away from this content.

lgas 4y ago
I'd suggest providing some justification for your declaration if you want anyone to listen to you.
epgui 4y ago
That's a supremely anti-intellectual take.

AlexCoventry 4y ago

> Category Theory has been finding increasing applications in machine learning What's the most compelling application so far?

bawolff 4y ago
Application in the sense they are using it is probably different than the sense you are using it. Although its still probably a fair question regardless.
- AlexCoventry 4y ago
  Any application where Category Theory is making it substantially easier to express the software or reason about it is fair game, from my perspective.
  - lgas 4y ago
    How about https://arxiv.org/abs/1803.05316 https://arxiv.org/abs/1803.05316?

eigenform 4y ago

I'm not experienced/well-read in either ML or CT, but awhile ago I remember hearing Tai-Danae Bradley equate "knowing a word by the company it keeps" to the Yoneda lemma, and I always thought that was kind of interesting (although I guess I'm not qualified enough to know whether that statement is useful or vacuous)

haskellandchill 4y ago
> knowing a word by the company it keeps I'm still shocked no one has developed language learning software along these lines. I had a prototype in the works for thai years ago but never got time to get it off the ground. using statistical models trained on web corpus for a language learning app seems like a no brainer. think of it like navigating a word as a point in a graph connected to every example context it is in, with associated words being clickable into similar context bundles. then make it differential between host and target language given a translation so you can see which contexts the translation fails and succeeds in.