7 ms·

In TensorFlow one doesn't write Python code and run it on a G/TPU, one writes Python code that uses an API to construct a TensorFlow graph computation which is

In TensorFlow one doesn't write Python code and run it on a G/TPU, one writes Python code that uses an API to construct a TensorFlow graph computation which is compiled and runs on the G/TPU. It's really a completely separate programming language with different semantics and a totally separate runtime hiding behind a Python API. This is why you can't use any of Python's normal libraries, you have to use TensorFlow libraries. "Julia runs on GPUs" actually means that you write Julia code and it is compiled to natively run on GPUs [1]. Similarly, this post is about writing Julia code and compiling it to run natively on TPUs, not calling some predefined TPU library. Yes, of course you _can_ call libraries that are compiled for GPUs and TPUs—as you can in Python or any other language—but in Julia, nearly arbitrary user code can be compiled and run on GPUs and TPUs. The major restriction on GPUs is that you cannot do dynamic allocation, so you need to write non-allocating Julia code, but that's common requirement for high-performance code anyway. [1] https://github.com/JuliaGPU/CUDAnative.jl https://github.com/JuliaGPU/CUDAnative.jl

mlthoughts2018 8y ago
This has been available for Python going waay back to PyCuda, and also numba has support for jit compiling to a GPU target. You can do similar things with Cython as well.
- krastanov 8y ago
  Except for numba (when used on a specific subset of python), none of these utilities let you use a pure-python function with them. You can not write a generic python algorithm unaware of these tools and expect it to work efficiently with any of them without rewriting.
  - mlthoughts2018 8y ago
    Totally false. Cython allows you to choose whatever subset of the CPython API you want to use, to suit whatever performance tradeoffs you’d like to make in exchange for access to convenience, dynamic attribute access, whatever. You can write fully no-python C functions, all the way up to straight CPython dynamically typed code relying on the GIL and reference counting, etc.
    - krastanov 8y ago
      I think you are missing the point. How do I get a pure-python function written by someone else to be "C level" fast when used in the inner loop of your otherwise superfast cython? How do you do that without rewriting it in cython? Sure, cython is great if you decide to rewrite the library you want to use in cython. Same with scipy.optimize for instance: it can be super fast for everything but ordinary python cost functions. Here is an easy way to prove me wrong: Find the minimum of a pure python function quickly, with whatever python tool you want, without having to deal with the enormous python function call overhead, without rewriting the function. Imagine this function is deep down in another library and it uses a bunch of 3rd party libraries.
      - mlthoughts2018 8y ago
        I think you have a very fundamental misunderstanding here. When you ask for a pure Python loop, for example, you are directly saying you want something like the iterator protocol of Python, inclusive of its overhead, because your use case needs the dynanic behaviors, overloading via custom iterators, whatever. You’re directly saying the performance trade-off is worth it for you personally because a low overhead C-style loop won’t give you the extra features of e.g. the iterator protocol that matters more to you. When you say, but my use case doesn’t need the iterator protocol, then you just write that piece of code in Cython or use a nopython numba jit call, etc., because you want to make your trade-off differently in that single case. You’re essentially saying, “how do I make a triangle with 5 sides,” by requesting a pure Python section of code to not be pure Python. On a side note though, you can take pure Python code and compile it directly with Cython, with no modifications, and in many cases it will still be quite faster because it can compile away some types of attribute accesses, overhead of boolean logical checks as function calls, and also reduce function call overhead for many standard library functions or data structures that are already implemented as C extension modules (by bypassing some of the method lookup logic to avoid checks on PyFunction objects in CPython). Usually this is a recommended first step before ever adding a type annotation or doing anything more difficult. Finally, just to be clear, numba (and tools using llvmlite more generally) don’t worry about this, since they perform static type inference and have other rules about enforcing static types when jitting a Python code segment.
marmaduke 8y ago
This is treading a fine line here, since you don’t mention Numba, which applies the same approach as Julia, namely translating the language to LLVM IR for generating PTX. The same applies to those OpenACC pragmas that can offload a butt ugly Fortran loop to GPU: no one says Fortran is running on the GPU, rather the compiler is doing code gen and RT calls to make user life easy. It thus smells like marketing rhetoric.
- StefanKarpinski 8y ago
  Numba compiling Python to PTX is absolutely "Python running on GPUs" and OpenACC is also "Fortran running on GPUs". If user code written in language X is compiled to target hardware Y then that is "X running on Y". This is fairly standard compiler terminology, not marketing speak. You specifically mentioned TensorFlow, which is NOT Python running on G/TPUs since the code that has neither Python's semantics nor its runtime.

mlthoughts2018 8y ago

This has been available for Python going waay back to PyCuda, and also numba has support for jit compiling to a GPU target. You can do similar things with Cython as well.

krastanov 8y ago
Except for numba (when used on a specific subset of python), none of these utilities let you use a pure-python function with them. You can not write a generic python algorithm unaware of these tools and expect it to work efficiently with any of them without rewriting.
- mlthoughts2018 8y ago
  Totally false. Cython allows you to choose whatever subset of the CPython API you want to use, to suit whatever performance tradeoffs you’d like to make in exchange for access to convenience, dynamic attribute access, whatever. You can write fully no-python C functions, all the way up to straight CPython dynamically typed code relying on the GIL and reference counting, etc.
  - krastanov 8y ago
    I think you are missing the point. How do I get a pure-python function written by someone else to be "C level" fast when used in the inner loop of your otherwise superfast cython? How do you do that without rewriting it in cython? Sure, cython is great if you decide to rewrite the library you want to use in cython. Same with scipy.optimize for instance: it can be super fast for everything but ordinary python cost functions. Here is an easy way to prove me wrong: Find the minimum of a pure python function quickly, with whatever python tool you want, without having to deal with the enormous python function call overhead, without rewriting the function. Imagine this function is deep down in another library and it uses a bunch of 3rd party libraries.
    - mlthoughts2018 8y ago
      I think you have a very fundamental misunderstanding here. When you ask for a pure Python loop, for example, you are directly saying you want something like the iterator protocol of Python, inclusive of its overhead, because your use case needs the dynanic behaviors, overloading via custom iterators, whatever. You’re directly saying the performance trade-off is worth it for you personally because a low overhead C-style loop won’t give you the extra features of e.g. the iterator protocol that matters more to you. When you say, but my use case doesn’t need the iterator protocol, then you just write that piece of code in Cython or use a nopython numba jit call, etc., because you want to make your trade-off differently in that single case. You’re essentially saying, “how do I make a triangle with 5 sides,” by requesting a pure Python section of code to not be pure Python. On a side note though, you can take pure Python code and compile it directly with Cython, with no modifications, and in many cases it will still be quite faster because it can compile away some types of attribute accesses, overhead of boolean logical checks as function calls, and also reduce function call overhead for many standard library functions or data structures that are already implemented as C extension modules (by bypassing some of the method lookup logic to avoid checks on PyFunction objects in CPython). Usually this is a recommended first step before ever adding a type annotation or doing anything more difficult. Finally, just to be clear, numba (and tools using llvmlite more generally) don’t worry about this, since they perform static type inference and have other rules about enforcing static types when jitting a Python code segment.

marmaduke 8y ago

This is treading a fine line here, since you don’t mention Numba, which applies the same approach as Julia, namely translating the language to LLVM IR for generating PTX. The same applies to those OpenACC pragmas that can offload a butt ugly Fortran loop to GPU: no one says Fortran is running on the GPU, rather the compiler is doing code gen and RT calls to make user life easy. It thus smells like marketing rhetoric.

StefanKarpinski 8y ago
Numba compiling Python to PTX is absolutely "Python running on GPUs" and OpenACC is also "Fortran running on GPUs". If user code written in language X is compiled to target hardware Y then that is "X running on Y". This is fairly standard compiler terminology, not marketing speak. You specifically mentioned TensorFlow, which is NOT Python running on G/TPUs since the code that has neither Python's semantics nor its runtime.