5 ms·

We are getting closer and closer to differentiable APL running on GPU. I like this direction :)

by mlajtos 5y ago

cl3misch 5y ago
Not sure if you mean exactly this, but JAX is pretty close to "differentiable APL running on GPU" I'd say.
- mlajtos 5y ago
  Yes, I agree. Only if we could have a thin syntactic sugar on top and start using algebra to compose stuff.
FridgeSeal 5y ago
It’d be a lot cooler if so much of the work wasn’t so inextricably linked to Python.
- mlajtos 5y ago
  Well, Python was at the right place at the right time. I am actually happy that array-oriented programming (via Numpy) got mainstream. A lot of great ideas came from sheer number of people doing tinkering and great engineering in this domain. If only people caught on to Iverson's 'Notation as a Tool of Thought' [0] and Backus' 'Can programming be liberated from the von Neumann style? A functional style and its algebra of programs'. [0] https://dl.acm.org/doi/pdf/10.1145/358896.358899 https://dl.acm.org/doi/pdf/10.1145/358896.358899 [1] https://dl.acm.org/doi/pdf/10.1145/359576.359579 https://dl.acm.org/doi/pdf/10.1145/359576.359579
Certhas 5y ago
Julia is there now. It seems absurd to me to build this on top of Python. You are building everything as a DSL that needs to be carefully crafted to never touch the data structures of the host language. Changing from Python to Julia and suddenly being able to use datastructures other than numpy arrays, and writing things that should be for loops as for loops, was such a relieve. But, I guess as Javascript has shown, with an infinite amount of engineering resources dedicated to it you can build anything in any language.
- bertr4nd 5y ago
  I’m extremely sympathetic to this viewpoint. I spent the last couple years building a JIT compiler for PyTorch, and a constant battle was that people want to write “normal” idiomatic Python, but those idioms work to defeat optimizations, unless you go to really crazy lengths. Some of the recent work in this space (torch.fx in particular) has made things easier by allowing the user to “clean up” their model, but it was definitely an eye-opening experience.
  - mpweiher 5y ago
    This sounds very interesting! Did you ever write something up about the issues encountered?
    - bertr4nd 5y ago
      I have not, but I will certainly consider doing so!
      - mpweiher 5y ago
        That would be awesome!
servytor 5y ago
I think Futhark[0] is close to that, no? Or what about Co-DFNS[1]? [0]: https://futhark-lang.org/ https://futhark-lang.org/ [1]: https://github.com/Co-dfns/Co-dfns https://github.com/Co-dfns/Co-dfns
- rowanG077 5y ago
  Futhark doesn't support AD yet. But it's pretty close afaik.
ByThyGrace 5y ago
Which GPU? On whose drivers? On which systems?
N1H1L 5y ago
But also jax.numpy can do this quite a bit too.

We are getting closer and closer to differentiable APL running on GPU. I like this direction :)

cl3misch 5y ago

Not sure if you mean exactly this, but JAX is pretty close to "differentiable APL running on GPU" I'd say.

mlajtos 5y ago
Yes, I agree. Only if we could have a thin syntactic sugar on top and start using algebra to compose stuff.

FridgeSeal 5y ago

It’d be a lot cooler if so much of the work wasn’t so inextricably linked to Python.

mlajtos 5y ago
Well, Python was at the right place at the right time. I am actually happy that array-oriented programming (via Numpy) got mainstream. A lot of great ideas came from sheer number of people doing tinkering and great engineering in this domain. If only people caught on to Iverson's 'Notation as a Tool of Thought' [0] and Backus' 'Can programming be liberated from the von Neumann style? A functional style and its algebra of programs'. [0] https://dl.acm.org/doi/pdf/10.1145/358896.358899 https://dl.acm.org/doi/pdf/10.1145/358896.358899 [1] https://dl.acm.org/doi/pdf/10.1145/359576.359579 https://dl.acm.org/doi/pdf/10.1145/359576.359579

Certhas 5y ago

Julia is there now. It seems absurd to me to build this on top of Python. You are building everything as a DSL that needs to be carefully crafted to never touch the data structures of the host language. Changing from Python to Julia and suddenly being able to use datastructures other than numpy arrays, and writing things that should be for loops as for loops, was such a relieve. But, I guess as Javascript has shown, with an infinite amount of engineering resources dedicated to it you can build anything in any language.

bertr4nd 5y ago
I’m extremely sympathetic to this viewpoint. I spent the last couple years building a JIT compiler for PyTorch, and a constant battle was that people want to write “normal” idiomatic Python, but those idioms work to defeat optimizations, unless you go to really crazy lengths. Some of the recent work in this space (torch.fx in particular) has made things easier by allowing the user to “clean up” their model, but it was definitely an eye-opening experience.
- mpweiher 5y ago
  This sounds very interesting! Did you ever write something up about the issues encountered?
  - bertr4nd 5y ago
    I have not, but I will certainly consider doing so!
    - mpweiher 5y ago
      That would be awesome!

servytor 5y ago

I think Futhark[0] is close to that, no? Or what about Co-DFNS[1]? [0]: https://futhark-lang.org/ https://futhark-lang.org/ [1]: https://github.com/Co-dfns/Co-dfns https://github.com/Co-dfns/Co-dfns

rowanG077 5y ago
Futhark doesn't support AD yet. But it's pretty close afaik.

ByThyGrace 5y ago

Which GPU? On whose drivers? On which systems?

N1H1L 5y ago

But also jax.numpy can do this quite a bit too.