6 ms·

Julia is the Haskell of numerical computing. It seems like AD is a solved problem, which is not. I can personally think two instances in which I have hand roll

by plafl 5y ago

Julia is the Haskell of numerical computing. It seems like AD is a solved problem, which is not. I can personally think two instances in which I have hand rolled my own gradients: in the context of recommender systems (dense matrices are evil) and right now in the context of real time collision detection (memory allocations are evil). One of the domains where I hope Julia will excel is precisely as you comment on physics + ml (I check frequently if MuJoCo source code is available at last). Anyway, I encourage everyone with any background in scientific computing or DS to have a look at Julia. The ecosystem is nowhere near Python yet, but the language is very good and the tooling is getting better. The performance Julia provides without interfacing C/C++ or Fortran is not just a convenience, it has architectural consequences. It's not about of coding faster, it's about coding "further".

gostsamo 5y ago
I think I have a good news for you. Deep mind acquired and open sourced the project a few months ago. https://github.com/deepmind/mujoco https://github.com/deepmind/mujoco
- plafl 5y ago
  I know, I know, but the source code is not yet available. The github repo is a placeholder for now
ssivark 5y ago
MuJoCo was purchased by Deepmind and open-sourced, a few weeks ago: https://deepmind.com/blog/announcements/mujoco https://deepmind.com/blog/announcements/mujoco
pjmlp 5y ago
I thought Julia is the Dylan of numerical computing. :)
porker 5y ago
> Julia is the Haskell of numerical computing. Is that a compliment or an insult? When Haskell is involved I can never tell.
- plafl 5y ago
  Me neither, and yet I maintain my opinion.
  - vanderZwan 5y ago
    That's because Haskell is like the programming language equivalent of The Gig That Changed The World[0], although I guess Algol60 also has a strong claim to that title. Let me explain by quoting these paragraphs Roger Ebert's review[1] of 24h party people: > As the film opens, Wilson is attending the first, legendary Sex Pistols concert in Manchester, England. (...) Wilson is transfixed by the Pistols as they sing "Anarchy in the U.K." and sneer at British tradition. He tells the camera that everyone in the audience will leave the room transformed and inspired, and then the camera pans to show a total of 42 people, two or three of them half-heartedly dancing in the aisles. Sounds like the average language designer entranced and inspired by their first time grokking Haskell. > Wilson features the Pistols and other bands on his Manchester TV show. Because of a ban by London TV, his show becomes the only venue for punk rock. Turns out he was right about the Pistols. They let loose something that changed rock music. And they did it in the only way that Wilson could respect, by thoroughly screwing up everything they did, and ending in bankruptcy and failure, followed by Sid Vicious' spectacular murder-suicide flameout. The Sex Pistols became successful because they failed; if they had succeeded, they would have sold out, or become diluted or commercial. I saw Johnny Rotten a few years ago at Sundance, still failing, and it made me feel proud of him. I could rephrase that last sentence at "I checked out a contalk by Simon Peyton Jones from a few years ago, still avoiding success at all costs[2], and it made me feel proud of him" and it would be absolutely true. And no, I also did not expect to find a parallel between Haskell and punk music, but there you go. [0] https://openculture.com/2015/06/the-sex-pistols-1976-manchester-gig-that-changed-the-world.html https://openculture.com/2015/06/the-sex-pistols-1976-manches... [1] https://www.rogerebert.com/reviews/24-hour-party-people-2002 https://www.rogerebert.com/reviews/24-hour-party-people-2002 [2] https://www.youtube.com/watch?v=re96UgMk6GQ&t=1372s https://www.youtube.com/watch?v=re96UgMk6GQ&t=1372s
    - Gatsky 5y ago
      This is the best HN comment I have seen for quite some time.
vletal 5y ago
> The performance Julia provides without interfacing C/C++ or Fortran is not just a convenience, it has architectural consequences. These statements set the expectations really high. Yet they omit the ugly truth of increasingly slow compilation times, pure compilation caching, JIT which might affect overall performance due to type instability. Yeah, like Haskell has a nice Quick-sort example prominently displayed at the front page, Julia is sleek for scientific computing, yet it can bite you really bad as soon as you start wrapping the developed model into a real world REST app. Do not get me wrong, I see some of the benefits of Julia. On the other hand, I do not think this uncritical hype about "coding 'further'" does good for the language.
- DNF2 5y ago
  Compilation times are still an issue, but what do you mean with 'increasingly slow' compilation times? As far as I know and can tell, compilation times have been getting faster not slower, and by a significant amount.
  - ChrisRackauckas 5y ago
    Let's even put raw numbers to it. DifferentialEquations.jl usage has seen compile times drop from 22 seconds to 3 seconds over the last few months. https://github.com/SciML/DifferentialEquations.jl/issues/786 https://github.com/SciML/DifferentialEquations.jl/issues/786
    - vletal 5y ago
      Yeah, I should not have used "increasingly", yet what you are pointing out is an example of an improvement in a single module, not the language compiler & llvm. It does not actually prove the original comment wrong.
      - ChrisRackauckas 5y ago
        Almost all of the improvements came from changes to Julia's Base that improved compilation times and changes to tooling which suggest trivial changes to improve compilation times. In v1.5 or so this would've been a heroic feat. In v1.7 this is something any undergrad could do in a weekend to any module. So yes that proves that things have changed quite a bit.
ChrisRackauckas 5y ago
>One of the domains where I hope Julia will excel is precisely as you comment on physics + ml (I check frequently if MuJoCo source code is available at last). Somebody already did a comparison of DifferentialEquations.jl's standard usage against MuJoCo and DiffTaichi and showed that it outperformed them by about 5x-10x. https://arxiv.org/abs/2012.06684 https://arxiv.org/abs/2012.06684 https://homes.cs.washington.edu/~thickstn/ctpg-project-page/ctpg.html https://homes.cs.washington.edu/~thickstn/ctpg-project-page/... That's all showing the raw iteration count to show that it algorithmically is faster, but the time per iteration is also fast for many reasons showcased in the SciMLBenchmarks routinely outperforming C and Fortran solvers (https://github.com/SciML/SciMLBenchmarks.jl https://github.com/SciML/SciMLBenchmarks.jl). So it's excelling pretty well, and things like the automated discovery of black hole dynamics are all done using the universal differential equation framework enabled by the SciML tools (see https://arxiv.org/abs/2102.12695 https://arxiv.org/abs/2102.12695 for that application). What we are missing however is that, right now these simulations are all writing raw differential equations so we do need a better set of modeling tools. That said, MuJoCo and DiffTaichi are not great physical modeling environments for building real systems, instead we would point to Simulink and Modelica as what are really useful for building real-world systems. So it would be cool if there was a modeling language in Julia which extends that universe and directly does optimal code generation for the Julia solvers... and that's what ModelingToolkit.jl is (https://github.com/SciML/ModelingToolkit.jl https://github.com/SciML/ModelingToolkit.jl). That project is still pretty new, but there's already enough to show some large-scale models outperforming Dymola on examples that require symbolic tearing and index reduction, which is far more than what physical simulation environments used for non-scientific purposes (MuJoCo and DiffTaichi) are able to do. See the workshop for details (https://www.youtube.com/watch?v=HEVOgSLBzWA https://www.youtube.com/watch?v=HEVOgSLBzWA). And that's just the top level details, there's a whole Julia Computing product called JuliaSim (https://juliacomputing.com/products/juliasim/ https://juliacomputing.com/products/juliasim/) which is then being built on these pieces to do things like automatically generate ML-accelerated components and add model building GUIs. That said, MuJoCo and DiffTaichi have much better visualizations and animations than MTK. Our focus so far has been on the core routines, making them fast, scalable, stable, and extensive. You'll need to wait for the near future (or build something with Makie) if you want the pretty pictures of the robot to happen automatically. That said, Julia's Makie visualization system has already been shown to be sufficiently powerful for this kind of application (https://nextjournal.com/sdanisch/taking-your-robot-for-a-walk https://nextjournal.com/sdanisch/taking-your-robot-for-a-wal...), so we're excited to see where that will go in the future.
klowrey 5y ago
I manage the MuJoCo.jl wrapper, and having the source code wont (naively) help with AD. Internally, there's a number of iterative algorithms that you wouldn't want to automatically differentiate, and if analytical derivatives were apparent they would be in there (the MuJoCo creator was my advisor). As it stands, finite differencing is the best way to extract derivative/gradient information from the underlying physics model of MuJoCo, and is the suggested method. We had a paper that included MuJoCo (with custom defined adjoints) within Julia's DiffEq framework to learn continuous control policies (https://arxiv.org/abs/2012.06684 https://arxiv.org/abs/2012.06684) which revealed a whole host of other issues, namely that gradients are not all that helpful when your optimization landscape is insanely non-linear. But that's a different problem. Open to chatting about; Julia is definitely the tool for this kind of work.
- ChrisRackauckas 5y ago
  > namely that gradients are not all that helpful when your optimization landscape is insanely non-linear Did you try multiple shooting? https://diffeqflux.sciml.ai/dev/examples/multiple_shooting/ https://diffeqflux.sciml.ai/dev/examples/multiple_shooting/ I want to restructure the docs because I have come to think that any non-trivial fitting problem requires multiple shooting in order to be stable. Otherwise the loss becomes dominated by "loss values calculated beyond where the simulation has already fallen off". So I'd like to dig into some of the non-fitting examples and see if this or some other tricks are the right answer.
- plafl 5y ago
  I'm not thinking just about AD. I would love lots and lots of little simulation modules instead of monolithic solutions. Now some uninteresting story: the reason to have a look at MuJoCo is personal: I studied Mechanics I, II and Analytical Mechanics at college and thought non-continuos mechanics a solved problem. When doing my masters thesis about helicopter simulation I needed at some point to model ground contact and I thought "easy, just add a damped spring". I feel that thesis was nice for an undergrad overall but that part as you can imagine was a complete and utter piece of crap. And now at my forties I want to have vengeance.

by plafl 5y ago

gostsamo 5y ago

I think I have a good news for you. Deep mind acquired and open sourced the project a few months ago. https://github.com/deepmind/mujoco https://github.com/deepmind/mujoco

plafl 5y ago
I know, I know, but the source code is not yet available. The github repo is a placeholder for now

ssivark 5y ago

MuJoCo was purchased by Deepmind and open-sourced, a few weeks ago: https://deepmind.com/blog/announcements/mujoco https://deepmind.com/blog/announcements/mujoco

pjmlp 5y ago

I thought Julia is the Dylan of numerical computing. :)

porker 5y ago

> Julia is the Haskell of numerical computing. Is that a compliment or an insult? When Haskell is involved I can never tell.

plafl 5y ago
Me neither, and yet I maintain my opinion.
- vanderZwan 5y ago
  That's because Haskell is like the programming language equivalent of The Gig That Changed The World[0], although I guess Algol60 also has a strong claim to that title. Let me explain by quoting these paragraphs Roger Ebert's review[1] of 24h party people: > As the film opens, Wilson is attending the first, legendary Sex Pistols concert in Manchester, England. (...) Wilson is transfixed by the Pistols as they sing "Anarchy in the U.K." and sneer at British tradition. He tells the camera that everyone in the audience will leave the room transformed and inspired, and then the camera pans to show a total of 42 people, two or three of them half-heartedly dancing in the aisles. Sounds like the average language designer entranced and inspired by their first time grokking Haskell. > Wilson features the Pistols and other bands on his Manchester TV show. Because of a ban by London TV, his show becomes the only venue for punk rock. Turns out he was right about the Pistols. They let loose something that changed rock music. And they did it in the only way that Wilson could respect, by thoroughly screwing up everything they did, and ending in bankruptcy and failure, followed by Sid Vicious' spectacular murder-suicide flameout. The Sex Pistols became successful because they failed; if they had succeeded, they would have sold out, or become diluted or commercial. I saw Johnny Rotten a few years ago at Sundance, still failing, and it made me feel proud of him. I could rephrase that last sentence at "I checked out a contalk by Simon Peyton Jones from a few years ago, still avoiding success at all costs[2], and it made me feel proud of him" and it would be absolutely true. And no, I also did not expect to find a parallel between Haskell and punk music, but there you go. [0] https://openculture.com/2015/06/the-sex-pistols-1976-manchester-gig-that-changed-the-world.html https://openculture.com/2015/06/the-sex-pistols-1976-manches... [1] https://www.rogerebert.com/reviews/24-hour-party-people-2002 https://www.rogerebert.com/reviews/24-hour-party-people-2002 [2] https://www.youtube.com/watch?v=re96UgMk6GQ&t=1372s https://www.youtube.com/watch?v=re96UgMk6GQ&t=1372s
  - Gatsky 5y ago
    This is the best HN comment I have seen for quite some time.

vletal 5y ago

> The performance Julia provides without interfacing C/C++ or Fortran is not just a convenience, it has architectural consequences. These statements set the expectations really high. Yet they omit the ugly truth of increasingly slow compilation times, pure compilation caching, JIT which might affect overall performance due to type instability. Yeah, like Haskell has a nice Quick-sort example prominently displayed at the front page, Julia is sleek for scientific computing, yet it can bite you really bad as soon as you start wrapping the developed model into a real world REST app. Do not get me wrong, I see some of the benefits of Julia. On the other hand, I do not think this uncritical hype about "coding 'further'" does good for the language.

DNF2 5y ago
Compilation times are still an issue, but what do you mean with 'increasingly slow' compilation times? As far as I know and can tell, compilation times have been getting faster not slower, and by a significant amount.
- ChrisRackauckas 5y ago
  Let's even put raw numbers to it. DifferentialEquations.jl usage has seen compile times drop from 22 seconds to 3 seconds over the last few months. https://github.com/SciML/DifferentialEquations.jl/issues/786 https://github.com/SciML/DifferentialEquations.jl/issues/786
  - vletal 5y ago
    Yeah, I should not have used "increasingly", yet what you are pointing out is an example of an improvement in a single module, not the language compiler & llvm. It does not actually prove the original comment wrong.
    - ChrisRackauckas 5y ago
      Almost all of the improvements came from changes to Julia's Base that improved compilation times and changes to tooling which suggest trivial changes to improve compilation times. In v1.5 or so this would've been a heroic feat. In v1.7 this is something any undergrad could do in a weekend to any module. So yes that proves that things have changed quite a bit.

ChrisRackauckas 5y ago

>One of the domains where I hope Julia will excel is precisely as you comment on physics + ml (I check frequently if MuJoCo source code is available at last). Somebody already did a comparison of DifferentialEquations.jl's standard usage against MuJoCo and DiffTaichi and showed that it outperformed them by about 5x-10x. https://arxiv.org/abs/2012.06684 https://arxiv.org/abs/2012.06684 https://homes.cs.washington.edu/~thickstn/ctpg-project-page/ctpg.html https://homes.cs.washington.edu/~thickstn/ctpg-project-page/... That's all showing the raw iteration count to show that it algorithmically is faster, but the time per iteration is also fast for many reasons showcased in the SciMLBenchmarks routinely outperforming C and Fortran solvers (https://github.com/SciML/SciMLBenchmarks.jl https://github.com/SciML/SciMLBenchmarks.jl). So it's excelling pretty well, and things like the automated discovery of black hole dynamics are all done using the universal differential equation framework enabled by the SciML tools (see https://arxiv.org/abs/2102.12695 https://arxiv.org/abs/2102.12695 for that application). What we are missing however is that, right now these simulations are all writing raw differential equations so we do need a better set of modeling tools. That said, MuJoCo and DiffTaichi are not great physical modeling environments for building real systems, instead we would point to Simulink and Modelica as what are really useful for building real-world systems. So it would be cool if there was a modeling language in Julia which extends that universe and directly does optimal code generation for the Julia solvers... and that's what ModelingToolkit.jl is (https://github.com/SciML/ModelingToolkit.jl https://github.com/SciML/ModelingToolkit.jl). That project is still pretty new, but there's already enough to show some large-scale models outperforming Dymola on examples that require symbolic tearing and index reduction, which is far more than what physical simulation environments used for non-scientific purposes (MuJoCo and DiffTaichi) are able to do. See the workshop for details (https://www.youtube.com/watch?v=HEVOgSLBzWA https://www.youtube.com/watch?v=HEVOgSLBzWA). And that's just the top level details, there's a whole Julia Computing product called JuliaSim (https://juliacomputing.com/products/juliasim/ https://juliacomputing.com/products/juliasim/) which is then being built on these pieces to do things like automatically generate ML-accelerated components and add model building GUIs. That said, MuJoCo and DiffTaichi have much better visualizations and animations than MTK. Our focus so far has been on the core routines, making them fast, scalable, stable, and extensive. You'll need to wait for the near future (or build something with Makie) if you want the pretty pictures of the robot to happen automatically. That said, Julia's Makie visualization system has already been shown to be sufficiently powerful for this kind of application (https://nextjournal.com/sdanisch/taking-your-robot-for-a-walk https://nextjournal.com/sdanisch/taking-your-robot-for-a-wal...), so we're excited to see where that will go in the future.

klowrey 5y ago

I manage the MuJoCo.jl wrapper, and having the source code wont (naively) help with AD. Internally, there's a number of iterative algorithms that you wouldn't want to automatically differentiate, and if analytical derivatives were apparent they would be in there (the MuJoCo creator was my advisor). As it stands, finite differencing is the best way to extract derivative/gradient information from the underlying physics model of MuJoCo, and is the suggested method. We had a paper that included MuJoCo (with custom defined adjoints) within Julia's DiffEq framework to learn continuous control policies (https://arxiv.org/abs/2012.06684 https://arxiv.org/abs/2012.06684) which revealed a whole host of other issues, namely that gradients are not all that helpful when your optimization landscape is insanely non-linear. But that's a different problem. Open to chatting about; Julia is definitely the tool for this kind of work.

ChrisRackauckas 5y ago
> namely that gradients are not all that helpful when your optimization landscape is insanely non-linear Did you try multiple shooting? https://diffeqflux.sciml.ai/dev/examples/multiple_shooting/ https://diffeqflux.sciml.ai/dev/examples/multiple_shooting/ I want to restructure the docs because I have come to think that any non-trivial fitting problem requires multiple shooting in order to be stable. Otherwise the loss becomes dominated by "loss values calculated beyond where the simulation has already fallen off". So I'd like to dig into some of the non-fitting examples and see if this or some other tricks are the right answer.
plafl 5y ago
I'm not thinking just about AD. I would love lots and lots of little simulation modules instead of monolithic solutions. Now some uninteresting story: the reason to have a look at MuJoCo is personal: I studied Mechanics I, II and Analytical Mechanics at college and thought non-continuos mechanics a solved problem. When doing my masters thesis about helicopter simulation I needed at some point to model ground contact and I thought "easy, just add a damped spring". I feel that thesis was nice for an undergrad overall but that part as you can imagine was a complete and utter piece of crap. And now at my forties I want to have vengeance.