3 ms·

Very interesting that it's coming from Google. I did my masters in tokamak simulation, so my first question is about performance. Python is very rarely used in

by aqme28 2y ago

Very interesting that it's coming from Google. I did my masters in tokamak simulation, so my first question is about performance. Python is very rarely used in this space just for performance reasons. Even though Python can call out to BLAS or whatever, it's still usually worth it to code in Fortran or C or maybe Julia.

cokernel_hacker 2y ago
This python actually builds a graph under the hood which then gets JIT compiled for CPU/GPU/TPU.
- senseiV 2y ago
  Does a TPU have XLA-graph for GPUs Cuda-graphs? Not sure on TPU theory
  - dekhn 2y ago
    TPUs lower XLA into their own instruction set. https://pytorch.org/xla/release/2.3/index.html https://pytorch.org/xla/release/2.3/index.html
uoaei 2y ago
It's built on JAX, not vanilla Python. The metric being optimized is not just performance, but also the ability to build reasonably performant workflows with arbitrary differentiable (i.e., ML) inputs and outputs.
nestorD 2y ago
I am doing quite a bit of work with JAX (the Python library used here) in a high-performance numerical computing context. On GPU/TPU, it is not going to reach perfect 100% hardware usage, but it is going to get close enough (far above vanilla Python performance) and be significantly more productive than alternatives. That makes it a sweet spot for research (where you will want to tweak things as you go) and extremely complex codes (where you already need to put your full focus on the correctness of the code). I highly recommend it to domain experts who need performance for their research project.
317070 2y ago
> Even though Python can call out to BLAS or whatever, it's still usually worth it to code in Fortran or C or maybe Julia. I write in a mix of C++ and Python, and have also dabbled with tokamaks, and I think this is a (common) misunderstanding. Fundamentally, you are optimizing the speed of project progress. Now, if you don't need your results in real time, e.g. because you are building a simulator which is not in a control loop for instance, you are often better at taking the easy language and ignoring the compute performance of the language. The compute performance of languages is a fixed multiplier, and with numerical code you might see things up to 10x. But having readable code, which is easy to manipulate and change to test ideas, is speeding up the project progress by such a large factor, that it is hard to keep up with other languages. The reason python is omnipresent in machine learning, is not for a lack of trying of other languages. Python is just very good at allowing you to keep up with a fast-moving field.

by aqme28 2y ago

cokernel_hacker 2y ago

This python actually builds a graph under the hood which then gets JIT compiled for CPU/GPU/TPU.

senseiV 2y ago
Does a TPU have XLA-graph for GPUs Cuda-graphs? Not sure on TPU theory
- dekhn 2y ago
  TPUs lower XLA into their own instruction set. https://pytorch.org/xla/release/2.3/index.html https://pytorch.org/xla/release/2.3/index.html

uoaei 2y ago

It's built on JAX, not vanilla Python. The metric being optimized is not just performance, but also the ability to build reasonably performant workflows with arbitrary differentiable (i.e., ML) inputs and outputs.

nestorD 2y ago

I am doing quite a bit of work with JAX (the Python library used here) in a high-performance numerical computing context. On GPU/TPU, it is not going to reach perfect 100% hardware usage, but it is going to get close enough (far above vanilla Python performance) and be significantly more productive than alternatives. That makes it a sweet spot for research (where you will want to tweak things as you go) and extremely complex codes (where you already need to put your full focus on the correctness of the code). I highly recommend it to domain experts who need performance for their research project.

317070 2y ago

> Even though Python can call out to BLAS or whatever, it's still usually worth it to code in Fortran or C or maybe Julia. I write in a mix of C++ and Python, and have also dabbled with tokamaks, and I think this is a (common) misunderstanding. Fundamentally, you are optimizing the speed of project progress. Now, if you don't need your results in real time, e.g. because you are building a simulator which is not in a control loop for instance, you are often better at taking the easy language and ignoring the compute performance of the language. The compute performance of languages is a fixed multiplier, and with numerical code you might see things up to 10x. But having readable code, which is easy to manipulate and change to test ideas, is speeding up the project progress by such a large factor, that it is hard to keep up with other languages. The reason python is omnipresent in machine learning, is not for a lack of trying of other languages. Python is just very good at allowing you to keep up with a fast-moving field.