11 ms·

Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

cs702 5y ago
Very cool. I have only one question: Has anyone successfully applied DEQs to larger-scale cognitive tasks or benchmarks, as opposed to MNIST, which is a tiny trivial task by today's standards? Think ImageNet-1000, COCO, LVIS, WMT language translation, ..., SuperGLUE. There's a long list of datasets and benchmarks that regular boring fixed-depth NNs tackle with remarkable ease these days. Has anyone anywhere applied DEQs to any of those datasets / benchmarks?
- avikpal1410 5y ago
  MDEQ work applies DEQ to some of the large scale benchmarks you mention: https://arxiv.org/pdf/2006.08656.pdf https://arxiv.org/pdf/2006.08656.pdf
  - cs702 5y ago
    Thank you! The results don't look that great (e.g., EfficientNet models achieve greater accuracy on ImageNet-1000 with ~5x fewer parameters), but the works looks interesting and worthwhile. I'll take a look.
rpmuller 5y ago
One of the best things about Julia is that people like Chris Rackauckas are developing great packages for it.
- lytefm 5y ago
  Definetely. I've just been working with Stan and Pyro + Python so far for modelling, but post like this encourage me to finally pick up Julia and get seroius with neural ODEs.
tomtomftomtom 5y ago
Fauci funded COVID-19: https://www.zerohedge.com/covid-19/nih-admits-funding-gain-function-covid-experiments-gives-ecohealth-five-days-report https://www.zerohedge.com/covid-19/nih-admits-funding-gain-f...
grzff 5y ago
Fauci funded COVID-19: https://www.zerohedge.com/covid-19/nih-admits-funding-gain-function-covid-experiments-gives-ecohealth-five-days-report https://www.zerohedge.com/covid-19/nih-admits-funding-gain-f... Everyone involved should face the firing squad.
adgjlsfhk1 5y ago
This is a really cool idea, especially because it is a type of NN that can take more time for harder inputs. This makes it relatively unique since most types of NN have O(1) runtime, which is often nice, but puts limits on the types of problems they can solve.
bomz24 5y ago
NIH Admits Fauci lied to congress and funded covid-19: https://theintercept.com/2021/10/21/virus-mers-wuhan-experiments/ https://theintercept.com/2021/10/21/virus-mers-wuhan-experim...
jstx1 5y ago
What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new approaches to solve ML problems? How legit is the whole thing? I just don't know.)
- UncleOxidant 5y ago
  There's some info from the Pyhon/Pytorch camp: https://towardsdatascience.com/neural-odes-with-pytorch-lightning-and-torchdyn-87ca4a7c6ffd https://towardsdatascience.com/neural-odes-with-pytorch-ligh... I suspect neural ODE work was done in Julia earlier because it was easier given some language features and libraries. But there does seem to be some work on neural ODEs in Python/Pytorch.
- adgjlsfhk1 5y ago
  What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).
  - jstx1 5y ago
    More bluntly my question is if SciML is that good, why aren’t more people doing it yet? Why is it limited to a small group of Julia developers and packages? (There are good possible explanations - it could be very new, have only niche applications, Julia is somehow uniquely suited for it etc. I don’t know)
    - krastanov 5y ago
      Some minor clarifications: NeuralODEs are not a Julia invention. I am pretty sure the first papers on the topic were using a python package implementing a rather crude ODE solver in torch or tensorflow. Julia just happens to be light years ahead of any other tool when it comes to solving ODEs, while having many high-quality autodifferentiation packages as well, so it feels natural to use it for these problems. But more importantly, SciML is not just for your typical Machine Learning tasks: being able to solve ODEs and have autodiff over them is incredibly empowering for boring old science and engineering, and SciML has become one of the most popular set of libraries when it comes to unwieldy ODEs.
fauscist1984 5y ago
NIH admits Fauci lied about funding Wuhan gain-of-function experiments https://www.washingtonexaminer.com/opinion/nih-admits-fauci-lied-about-funding-wuhan-gain-of-function-experiments https://www.washingtonexaminer.com/opinion/nih-admits-fauci-...
Tarrosion 5y ago
I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months there's a new blog post, usually with some of these folks as authors, that describes...something...to do with ODEs, machine learning, neural ODEs, GPUs, adjoints, scientific machine learning, ... I have never once been able to follow one of these blog posts. Seems like universally these posts have terrible curse of knowledge [1]. To be fair, I know only a light-to-moderate amount about machine learning and very little about differential equations, but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. Maybe I just need to...try harder? I wonder how many people would be excited but are totally lost by these posts. To that point, if anybody has a recommendation for a gentle introduction to these topics (preferably Julia), I'd be most appreciative. [1] https://en.wikipedia.org/wiki/Curse_of_knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge
- jimsimmons 5y ago
  I consider myself to be reasonably good at ML and work for one of the cool labs and this stuff is impenetrable to me. Stuff like this are deceptively written and a cynic might say that they are primarily to show off rather than help.
- agumonkey 5y ago
  What's funny is how most of them are revolving around a similar culture of concepts (ODE, automated differentiation, analysis) which is very unlike mainstream computing. ps: I need to read those adjoints articles.
- uoaei 5y ago
  > but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. I don't understand this part. Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? I would expect anyone with a mathy PhD to understand ODEs and PDEs, and neural ODEs are commonly understood (by those who read the papers, where I think it is an appropriate assumption that people who want to understand this stuff would do) to be effectively infinite-depth neural networks where every layer represents the same function.
gugagore 5y ago
> . For example, when we apply convolution filters on images the network consists of repetitive blocks of convolutional layers, and one linear output layer at the very end. It's essentially f(f(f(...f(x))...)) where f is the neural network, and we call this "deep" because of the layers of composition. But what if we make this composition go to infinity? This really does not jive with my understanding. Each layer of, e.g. VGG 16 [1] does not implement the same function. Each layer has its own weights. There are certain architectures that tie weights across layers, but not all of them. [1] https://neurohive.io/en/popular-networks/vgg16/ https://neurohive.io/en/popular-networks/vgg16/
- taeric 5y ago
  I think this works if you consider subscripted fs. That is, it is always the same arity, and the output is the same. So, would be better it they said f_n, where n is the layer if they network. (I mean this as a question, but don't see an obvious place for a question mark...)
  - gugagore 5y ago
    well, I think it has to be a fixed f for the interpretation of a (time-invariant) dynamical system x_{n+1} = f(x_n) to work.
    - taeric 5y ago
      But deeper layers of the network don't get the original input, do they?
- ChrisRackauckas 5y ago
  Correct, not all models have equivalent weights at each step. But it is an abstraction that needs to be done to send it to infinity.
savant_penguin 5y ago
15000x speed up compared to what?
- adgjlsfhk1 5y ago
  the previous implantation used by Nasa in I believe simulink. See https://www.youtube.com/watch?v=tQpqsmwlfY0 https://www.youtube.com/watch?v=tQpqsmwlfY0 for more details.
- snicker7 5y ago
  Simulink -- probably the most popular modeling toolbox.

Very cool. I have only one question: Has anyone successfully applied DEQs to larger-scale cognitive tasks or benchmarks, as opposed to MNIST, which is a tiny trivial task by today's standards? Think ImageNet-1000, COCO, LVIS, WMT language translation, ..., SuperGLUE. There's a long list of datasets and benchmarks that regular boring fixed-depth NNs tackle with remarkable ease these days. Has anyone anywhere applied DEQs to any of those datasets / benchmarks?

avikpal1410 5y ago
MDEQ work applies DEQ to some of the large scale benchmarks you mention: https://arxiv.org/pdf/2006.08656.pdf https://arxiv.org/pdf/2006.08656.pdf
- cs702 5y ago
  Thank you! The results don't look that great (e.g., EfficientNet models achieve greater accuracy on ImageNet-1000 with ~5x fewer parameters), but the works looks interesting and worthwhile. I'll take a look.

rpmuller 5y ago

One of the best things about Julia is that people like Chris Rackauckas are developing great packages for it.

lytefm 5y ago
Definetely. I've just been working with Stan and Pyro + Python so far for modelling, but post like this encourage me to finally pick up Julia and get seroius with neural ODEs.

tomtomftomtom 5y ago

Fauci funded COVID-19: https://www.zerohedge.com/covid-19/nih-admits-funding-gain-function-covid-experiments-gives-ecohealth-five-days-report https://www.zerohedge.com/covid-19/nih-admits-funding-gain-f...

grzff 5y ago

adgjlsfhk1 5y ago

This is a really cool idea, especially because it is a type of NN that can take more time for harder inputs. This makes it relatively unique since most types of NN have O(1) runtime, which is often nice, but puts limits on the types of problems they can solve.

bomz24 5y ago

NIH Admits Fauci lied to congress and funded covid-19: https://theintercept.com/2021/10/21/virus-mers-wuhan-experiments/ https://theintercept.com/2021/10/21/virus-mers-wuhan-experim...

jstx1 5y ago

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new approaches to solve ML problems? How legit is the whole thing? I just don't know.)

UncleOxidant 5y ago
There's some info from the Pyhon/Pytorch camp: https://towardsdatascience.com/neural-odes-with-pytorch-lightning-and-torchdyn-87ca4a7c6ffd https://towardsdatascience.com/neural-odes-with-pytorch-ligh... I suspect neural ODE work was done in Julia earlier because it was easier given some language features and libraries. But there does seem to be some work on neural ODEs in Python/Pytorch.
adgjlsfhk1 5y ago
What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).
- jstx1 5y ago
  More bluntly my question is if SciML is that good, why aren’t more people doing it yet? Why is it limited to a small group of Julia developers and packages? (There are good possible explanations - it could be very new, have only niche applications, Julia is somehow uniquely suited for it etc. I don’t know)
  - krastanov 5y ago
    Some minor clarifications: NeuralODEs are not a Julia invention. I am pretty sure the first papers on the topic were using a python package implementing a rather crude ODE solver in torch or tensorflow. Julia just happens to be light years ahead of any other tool when it comes to solving ODEs, while having many high-quality autodifferentiation packages as well, so it feels natural to use it for these problems. But more importantly, SciML is not just for your typical Machine Learning tasks: being able to solve ODEs and have autodiff over them is incredibly empowering for boring old science and engineering, and SciML has become one of the most popular set of libraries when it comes to unwieldy ODEs.

fauscist1984 5y ago

NIH admits Fauci lied about funding Wuhan gain-of-function experiments https://www.washingtonexaminer.com/opinion/nih-admits-fauci-lied-about-funding-wuhan-gain-of-function-experiments https://www.washingtonexaminer.com/opinion/nih-admits-fauci-...

Tarrosion 5y ago

I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months there's a new blog post, usually with some of these folks as authors, that describes...something...to do with ODEs, machine learning, neural ODEs, GPUs, adjoints, scientific machine learning, ... I have never once been able to follow one of these blog posts. Seems like universally these posts have terrible curse of knowledge [1]. To be fair, I know only a light-to-moderate amount about machine learning and very little about differential equations, but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. Maybe I just need to...try harder? I wonder how many people would be excited but are totally lost by these posts. To that point, if anybody has a recommendation for a gentle introduction to these topics (preferably Julia), I'd be most appreciative. [1] https://en.wikipedia.org/wiki/Curse_of_knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge

jimsimmons 5y ago
I consider myself to be reasonably good at ML and work for one of the cool labs and this stuff is impenetrable to me. Stuff like this are deceptively written and a cynic might say that they are primarily to show off rather than help.
agumonkey 5y ago
What's funny is how most of them are revolving around a similar culture of concepts (ODE, automated differentiation, analysis) which is very unlike mainstream computing. ps: I need to read those adjoints articles.
uoaei 5y ago
> but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. I don't understand this part. Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? I would expect anyone with a mathy PhD to understand ODEs and PDEs, and neural ODEs are commonly understood (by those who read the papers, where I think it is an appropriate assumption that people who want to understand this stuff would do) to be effectively infinite-depth neural networks where every layer represents the same function.

gugagore 5y ago

> . For example, when we apply convolution filters on images the network consists of repetitive blocks of convolutional layers, and one linear output layer at the very end. It's essentially f(f(f(...f(x))...)) where f is the neural network, and we call this "deep" because of the layers of composition. But what if we make this composition go to infinity? This really does not jive with my understanding. Each layer of, e.g. VGG 16 [1] does not implement the same function. Each layer has its own weights. There are certain architectures that tie weights across layers, but not all of them. [1] https://neurohive.io/en/popular-networks/vgg16/ https://neurohive.io/en/popular-networks/vgg16/

taeric 5y ago
I think this works if you consider subscripted fs. That is, it is always the same arity, and the output is the same. So, would be better it they said f_n, where n is the layer if they network. (I mean this as a question, but don't see an obvious place for a question mark...)
- gugagore 5y ago
  well, I think it has to be a fixed f for the interpretation of a (time-invariant) dynamical system x_{n+1} = f(x_n) to work.
  - taeric 5y ago
    But deeper layers of the network don't get the original input, do they?
ChrisRackauckas 5y ago
Correct, not all models have equivalent weights at each step. But it is an abstraction that needs to be done to send it to infinity.

savant_penguin 5y ago

15000x speed up compared to what?

adgjlsfhk1 5y ago
the previous implantation used by Nasa in I believe simulink. See https://www.youtube.com/watch?v=tQpqsmwlfY0 https://www.youtube.com/watch?v=tQpqsmwlfY0 for more details.
snicker7 5y ago
Simulink -- probably the most popular modeling toolbox.