5 ms·

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches l

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-can-I-connect-Matlab-to-TensorFlow https://www.quora.com/How-can-I-connect-Matlab-to-TensorFlow Does anyone have any ideas for moving towards something more general?

harias 8y ago
CUDA libraries fit your description I guess.
fnbr 8y ago
At work, we have a bunch of vectorized computations that we run in Tensorflow, as it's a convenient way to get GPU-optimized code, so that is still an option (albeit an awkward one). You could also use something like CUDA, or OpenGL to do this; there are some Python libraries to do basic numerical work, such as PyCUDA or gnumpy.
geoalchimista 8y ago
If you are not confined by the MATLAB platform, then there are a few options. For example, Julia [1] is a general-purpose numerical computing language that has more or less similar syntax with MATLAB for vector/matrix computation, and there is a Julia package TensorFlow.jl [2] that allows you to call TensorFlow in Julia. There are also quite a few packages in development to adapt Julia to GPU-based computation. And, to be fair, the NumPy/SciPy stack of Python can also be seen as a general-purpose vector computing platform. My feeling is that MATLAB had its days in the 90s. It's just that the most cutting-edge technologies do not seem to be developed in MATLAB any more. [1] https://github.com/JuliaLang/julia https://github.com/JuliaLang/julia [2] https://github.com/malmaud/TensorFlow.jl https://github.com/malmaud/TensorFlow.jl
albertzeyer 8y ago
Why do you think TensorFlow is a subset of general purpose computing? What do you think what is missing? I think nothing is really missing, only that it's maybe more difficult to perform certain kind of tasks. But compared to Matlab/Octave, I don't really see much lacking (in the platform). I would even say the opposite, that the Matlab/Octave platform seems to me like a subset of what TensorFlow offers.
- zackmorris 8y ago
  I wasn't sure who was first, so see my reply to bmer here: https://news.ycombinator.com/item?id=17420821 https://news.ycombinator.com/item?id=17420821
joe_the_user 8y ago
I don't know about tensor flow in particular but are little-known methods of running "general purpose" parallel programs on GPUs. Specifically, H. Dietz' MOG, "Mimd on GPU". It's a shame the project hasn't gotten more attention imo. http://aggregate.org/MOG/ http://aggregate.org/MOG/ See: https://en.wikipedia.org/wiki/Flynn%27s_taxonomy https://en.wikipedia.org/wiki/Flynn%27s_taxonomy for explanations of terms.
- zackmorris 8y ago
  Sorry for my late reply, I just wanted to thank you because your comment is an exceptionally good example of what I was trying to get at with my longwinded explanations. Compiling MIMD to SIMD is the future of programming, although it seems that companies will try every other course of action before they realize that.
bmer 8y ago
What do you mean by "TensorFlow is a subset of general purpose computing, and thus will always be limited to niches"? It's not clear to me at all what one could mean by this. Doesn't TensorFlow have to use matrix math deep down (just like any other digital computing system)?
- zackmorris 8y ago
  I'm not sure if you or albertzeyer asked first, but what I meant by that is that MATLAB is similar to any other C-like language, except uses the vector as its primitive instead of something like integer or float. That's really all there is to it. Other than a few details about notation, every major concept of MATLAB stems from that and is easily understood and predictable. MATLAB (or non-proprietary analogs like GNU Octave) let you write C-like code and then its runtime deals with architecture and optimization details internally (so there are no limitations on vector size or number of samplers or anything like that). Whereas things like TensorFlow, CUDA, OpenCL, OpenGL etc seem to deal more with DSP processing of buffer(s). They all have their own abstractions and lingo which work extremely well for certain use cases, kind of like domain-specific languages (DSLs). The end result is that it's trivial (at least in theory) to go from a TensorFlow implementation to a MATLAB implementation. But it's very difficult to go the other direction. Another way to think of this is that any solution written in TensorFlow can be run by MATLAB, but the reverse is not necessarily true. Trying to run MATLAB code within TensorFlow might encounter hardware limitations or other restrictions that makes the code run thousands of times slower. Now I could be wrong about this - maybe they truly are equivalent. But until I'm able to transpile MATLAB code directly to TensorFlow or OpenCL or whatever and have it be performant, I'm going to continue working under this assumption.
  - bmer 8y ago
    Yeah, I am pretty certain you're not right in this understanding. I am not the most knowledgeable, but basically all I know is scientific computing, so here are my two cents: Any C-like language does not use the "vector" as its "primitive" (not clear what you mean by primitive, and I am instead interpreting it as "machine type"), since integers, floats, characters, are the "primitives" (machine types), which are the things that the hardware itself "knows" how to store using a consistent system ultimately involving groups of bits. On top of these primitives, single-type arrays (e.g. basic C arrays) are built, for instance a string is an array of characters. Using C/typical hardware, a simple array is literally stored in memory "contiguously". An array only needs two pieces of information to define: what is the memory address of the first element, and how many elements are there. Thus, if you want the 6th element, the machine literally finds the first element, and then moves 6 memory "blocks" (C arrays can only have a single type, so each element needs the same amount of space to store) forwards to the 6th element, and then gives you back what it finds there. The simplest/fastest interpretations of a C array don't even care about how many elements there are, so if your array has 5 things, but you ask for the 6th thing, you just get back the data stored where the 6th thing should have been. Typically, this is garbage, but maybe it's super valuable (e.g. the first character of a stored password is put there..."overflow vulnerability"...to get the second character, ask for the 7th element, etc. etc. etc.) Okay, going off topic. Back to the point. Other data structures that can be built upon primitives are structs, enums, etc., but the array is important relevant to us because it's easy to think of an array as a linear algebra vector. C does not implement dot products natively, but one can easily write a function called "dot_product" that takes two lists of integers, or floats and returns a scalar integer or float. Some higher level languages (e.g. MATLAB) do exactly this, and save you the work of implementing all of linear algebra again. Matrices are more tricky: for usual non-parallelized hardware, they are still stored contiguously as an array of primitives, but some extra data might go along with this array to tell you how many elements you have to pass before you're onto the next row. Again, MATLAB just provides this sugar-coating. So where do GPUs come in? Well, think of a simple grayscale image on a monitor, and note that it is built up of little pixels: so, this image can be thought of as a matrix of integers that range from 0 to 255 inclusive (256 values total), where the (i, j) corresponds to how light/dark each (i, j) pixel is: if your monitor is a square that contains 1000 pixels by 1000 pixels, you can represent it by a 1000x1000 matrix. Using C-like languages on typical hardware, you are basically representing your 1000x1000 matrix as a 1,000,000 long array. Imagine you want to transform each pixel (independently of the others) by applying some function you have written --- in typical non-parallelized hardware, you would go through each of the million entries in that array, and apply your function, one after the other (serially). This is the sort of operation you typically have to do when you want to transform images on a screen. You can imagine that doing it serially would get more and more tiresome, as your images get more detailed, your monitors get higher resolution, etc., so purpose built hardware called GPUs were created, which can represent a matrix as a true machine type: a true "primitive", where memory is actually (i, j) addressable. You can pass matrices to the thing, and get matrices back. If you can do matrices, you can also obviously do vectors. Most importantly, you can give the GPU instructions that say "this functions should be applied to each pixel, and it doesn't care about other pixels when transforming one", and the GPU will apply that function in parallel. It will do a 1,000,000 calculations at once (assuming it can store a 1000x1000 matrix, otherwise it might have to get into some other abstractions, but now we are going off topic). Eventually people figured that any problem that can ultimately be thought of as "I have X data points, and I want to apply a function on each data point independent of the other" could be run efficiently on a GPU. Machine learning is just one application where you need to deal with lots of matrices. People have come up with languages that know about matrices as a machine type (early examples being graphics shader languages), and TensorFlow fits in somewhere in this space, with in-built sugar-coating for things the creators thought were "relevant". I have never used TensorFlow though, so someone else can probably give you more detail. -------------------------------------- Some things I have not mentioned: * vectorization: basically, how to do vector operations "smarter" on non-parallelized hardware, something that many languages (e.g. MATLAB, NumPy) and hardware now support -------------------------------------- Anyway: "transpiling" vector/matrix operations from MATLAB/NumPy to TensorFlow/OpenCL/Cuda is a breeze conceptually (but I bet it's kind of boring problem for advanced programmers). If a transpiler doesn't exist, it's probably because no one has put in the work to open source it. One example of a Python+NumPy to Cuda "transpiler" is Continuum's Numba: http://numba.pydata.org/numba-doc/0.38.0/cuda/index.html http://numba.pydata.org/numba-doc/0.38.0/cuda/index.html The devs there are also thinking about OpenCL, and some progress was made. This is the current state of that task: https://github.com/numba/numba/pull/582 https://github.com/numba/numba/pull/582 SPIR-V is basically a standardized "middle language" that "transpilers" can use. What you do is translate from language X -> SPIR-V -> OpenCL/Vulkan/whatever -> hardware drivers -> machine language And SPIR-V is work in progress: https://en.wikipedia.org/wiki/Standard_Portable_Intermediate_Representation https://en.wikipedia.org/wiki/Standard_Portable_Intermediate... There is also similar work for MATLAB, but its done by the owners: https://www.mathworks.com/matlabcentral/answers/25973-matlab-opencl https://www.mathworks.com/matlabcentral/answers/25973-matlab... Also, I have to echo other comments about how no one really cares about MATLAB that much in sci-comp, probably because it's proprietary and expensive. Heavy work is usually done in Fortran (legacy code), C/C++, or experimental languages like Julia, or languages that are more flexible/open-source like Python (NumPy/SciPy are just wrappers around lots of C/Fortran). People who use MATLAB tend to be those who were just "introduced to it" (e.g. through school) and have never hit any limitations that need them to switch to more flexible languages. Or they don't even realize that more flexible languages exist? It's a matter of comfort/goals (I don't advocate that everything should be done in C, and MATLAB is great for quickly doing many things, but many other things that have been done in MATLAB would have been less painful/faster if it used...say, Python+NumPy, because then you can use everything the Python ecosystem has to offer, and aren't limited to the MATLAB ecosystem).
dahart 8y ago
Maybe something like Jupyter or plain Python with GPU enabled numpy is what you’re looking for? Tensorflow is a library and not a language, it’s meant to be used from a host language that is Turing complete. It’s goal is to make construction of graphs of vector evaluations easier and more performant, but not really to provide a general purpose computation environment, it’s assumed you’re calling it from a general purpose computation environment. So, if you’re using Tensorflow, you normally have general purpose computing available to you, with the option to bake your vector tasks into graphs easily and/or speed them up. Jupyter is becoming a decent alternative to Matlab, and you have many options for running vector computations from python, with or without a GPU.
ianbertolacci 8y ago
> niches like AI or physics simulations or protein folding These are 99% of HPC workloads. Nothing `niche` about them.

harias 8y ago

CUDA libraries fit your description I guess.

fnbr 8y ago

At work, we have a bunch of vectorized computations that we run in Tensorflow, as it's a convenient way to get GPU-optimized code, so that is still an option (albeit an awkward one). You could also use something like CUDA, or OpenGL to do this; there are some Python libraries to do basic numerical work, such as PyCUDA or gnumpy.

geoalchimista 8y ago

If you are not confined by the MATLAB platform, then there are a few options. For example, Julia [1] is a general-purpose numerical computing language that has more or less similar syntax with MATLAB for vector/matrix computation, and there is a Julia package TensorFlow.jl [2] that allows you to call TensorFlow in Julia. There are also quite a few packages in development to adapt Julia to GPU-based computation. And, to be fair, the NumPy/SciPy stack of Python can also be seen as a general-purpose vector computing platform. My feeling is that MATLAB had its days in the 90s. It's just that the most cutting-edge technologies do not seem to be developed in MATLAB any more. [1] https://github.com/JuliaLang/julia https://github.com/JuliaLang/julia [2] https://github.com/malmaud/TensorFlow.jl https://github.com/malmaud/TensorFlow.jl

albertzeyer 8y ago

Why do you think TensorFlow is a subset of general purpose computing? What do you think what is missing? I think nothing is really missing, only that it's maybe more difficult to perform certain kind of tasks. But compared to Matlab/Octave, I don't really see much lacking (in the platform). I would even say the opposite, that the Matlab/Octave platform seems to me like a subset of what TensorFlow offers.

zackmorris 8y ago
I wasn't sure who was first, so see my reply to bmer here: https://news.ycombinator.com/item?id=17420821 https://news.ycombinator.com/item?id=17420821

joe_the_user 8y ago

I don't know about tensor flow in particular but are little-known methods of running "general purpose" parallel programs on GPUs. Specifically, H. Dietz' MOG, "Mimd on GPU". It's a shame the project hasn't gotten more attention imo. http://aggregate.org/MOG/ http://aggregate.org/MOG/ See: https://en.wikipedia.org/wiki/Flynn%27s_taxonomy https://en.wikipedia.org/wiki/Flynn%27s_taxonomy for explanations of terms.

zackmorris 8y ago
Sorry for my late reply, I just wanted to thank you because your comment is an exceptionally good example of what I was trying to get at with my longwinded explanations. Compiling MIMD to SIMD is the future of programming, although it seems that companies will try every other course of action before they realize that.

bmer 8y ago

What do you mean by "TensorFlow is a subset of general purpose computing, and thus will always be limited to niches"? It's not clear to me at all what one could mean by this. Doesn't TensorFlow have to use matrix math deep down (just like any other digital computing system)?

zackmorris 8y ago
I'm not sure if you or albertzeyer asked first, but what I meant by that is that MATLAB is similar to any other C-like language, except uses the vector as its primitive instead of something like integer or float. That's really all there is to it. Other than a few details about notation, every major concept of MATLAB stems from that and is easily understood and predictable. MATLAB (or non-proprietary analogs like GNU Octave) let you write C-like code and then its runtime deals with architecture and optimization details internally (so there are no limitations on vector size or number of samplers or anything like that). Whereas things like TensorFlow, CUDA, OpenCL, OpenGL etc seem to deal more with DSP processing of buffer(s). They all have their own abstractions and lingo which work extremely well for certain use cases, kind of like domain-specific languages (DSLs). The end result is that it's trivial (at least in theory) to go from a TensorFlow implementation to a MATLAB implementation. But it's very difficult to go the other direction. Another way to think of this is that any solution written in TensorFlow can be run by MATLAB, but the reverse is not necessarily true. Trying to run MATLAB code within TensorFlow might encounter hardware limitations or other restrictions that makes the code run thousands of times slower. Now I could be wrong about this - maybe they truly are equivalent. But until I'm able to transpile MATLAB code directly to TensorFlow or OpenCL or whatever and have it be performant, I'm going to continue working under this assumption.
- bmer 8y ago
  Yeah, I am pretty certain you're not right in this understanding. I am not the most knowledgeable, but basically all I know is scientific computing, so here are my two cents: Any C-like language does not use the "vector" as its "primitive" (not clear what you mean by primitive, and I am instead interpreting it as "machine type"), since integers, floats, characters, are the "primitives" (machine types), which are the things that the hardware itself "knows" how to store using a consistent system ultimately involving groups of bits. On top of these primitives, single-type arrays (e.g. basic C arrays) are built, for instance a string is an array of characters. Using C/typical hardware, a simple array is literally stored in memory "contiguously". An array only needs two pieces of information to define: what is the memory address of the first element, and how many elements are there. Thus, if you want the 6th element, the machine literally finds the first element, and then moves 6 memory "blocks" (C arrays can only have a single type, so each element needs the same amount of space to store) forwards to the 6th element, and then gives you back what it finds there. The simplest/fastest interpretations of a C array don't even care about how many elements there are, so if your array has 5 things, but you ask for the 6th thing, you just get back the data stored where the 6th thing should have been. Typically, this is garbage, but maybe it's super valuable (e.g. the first character of a stored password is put there..."overflow vulnerability"...to get the second character, ask for the 7th element, etc. etc. etc.) Okay, going off topic. Back to the point. Other data structures that can be built upon primitives are structs, enums, etc., but the array is important relevant to us because it's easy to think of an array as a linear algebra vector. C does not implement dot products natively, but one can easily write a function called "dot_product" that takes two lists of integers, or floats and returns a scalar integer or float. Some higher level languages (e.g. MATLAB) do exactly this, and save you the work of implementing all of linear algebra again. Matrices are more tricky: for usual non-parallelized hardware, they are still stored contiguously as an array of primitives, but some extra data might go along with this array to tell you how many elements you have to pass before you're onto the next row. Again, MATLAB just provides this sugar-coating. So where do GPUs come in? Well, think of a simple grayscale image on a monitor, and note that it is built up of little pixels: so, this image can be thought of as a matrix of integers that range from 0 to 255 inclusive (256 values total), where the (i, j) corresponds to how light/dark each (i, j) pixel is: if your monitor is a square that contains 1000 pixels by 1000 pixels, you can represent it by a 1000x1000 matrix. Using C-like languages on typical hardware, you are basically representing your 1000x1000 matrix as a 1,000,000 long array. Imagine you want to transform each pixel (independently of the others) by applying some function you have written --- in typical non-parallelized hardware, you would go through each of the million entries in that array, and apply your function, one after the other (serially). This is the sort of operation you typically have to do when you want to transform images on a screen. You can imagine that doing it serially would get more and more tiresome, as your images get more detailed, your monitors get higher resolution, etc., so purpose built hardware called GPUs were created, which can represent a matrix as a true machine type: a true "primitive", where memory is actually (i, j) addressable. You can pass matrices to the thing, and get matrices back. If you can do matrices, you can also obviously do vectors. Most importantly, you can give the GPU instructions that say "this functions should be applied to each pixel, and it doesn't care about other pixels when transforming one", and the GPU will apply that function in parallel. It will do a 1,000,000 calculations at once (assuming it can store a 1000x1000 matrix, otherwise it might have to get into some other abstractions, but now we are going off topic). Eventually people figured that any problem that can ultimately be thought of as "I have X data points, and I want to apply a function on each data point independent of the other" could be run efficiently on a GPU. Machine learning is just one application where you need to deal with lots of matrices. People have come up with languages that know about matrices as a machine type (early examples being graphics shader languages), and TensorFlow fits in somewhere in this space, with in-built sugar-coating for things the creators thought were "relevant". I have never used TensorFlow though, so someone else can probably give you more detail. -------------------------------------- Some things I have not mentioned: * vectorization: basically, how to do vector operations "smarter" on non-parallelized hardware, something that many languages (e.g. MATLAB, NumPy) and hardware now support -------------------------------------- Anyway: "transpiling" vector/matrix operations from MATLAB/NumPy to TensorFlow/OpenCL/Cuda is a breeze conceptually (but I bet it's kind of boring problem for advanced programmers). If a transpiler doesn't exist, it's probably because no one has put in the work to open source it. One example of a Python+NumPy to Cuda "transpiler" is Continuum's Numba: http://numba.pydata.org/numba-doc/0.38.0/cuda/index.html http://numba.pydata.org/numba-doc/0.38.0/cuda/index.html The devs there are also thinking about OpenCL, and some progress was made. This is the current state of that task: https://github.com/numba/numba/pull/582 https://github.com/numba/numba/pull/582 SPIR-V is basically a standardized "middle language" that "transpilers" can use. What you do is translate from language X -> SPIR-V -> OpenCL/Vulkan/whatever -> hardware drivers -> machine language And SPIR-V is work in progress: https://en.wikipedia.org/wiki/Standard_Portable_Intermediate_Representation https://en.wikipedia.org/wiki/Standard_Portable_Intermediate... There is also similar work for MATLAB, but its done by the owners: https://www.mathworks.com/matlabcentral/answers/25973-matlab-opencl https://www.mathworks.com/matlabcentral/answers/25973-matlab... Also, I have to echo other comments about how no one really cares about MATLAB that much in sci-comp, probably because it's proprietary and expensive. Heavy work is usually done in Fortran (legacy code), C/C++, or experimental languages like Julia, or languages that are more flexible/open-source like Python (NumPy/SciPy are just wrappers around lots of C/Fortran). People who use MATLAB tend to be those who were just "introduced to it" (e.g. through school) and have never hit any limitations that need them to switch to more flexible languages. Or they don't even realize that more flexible languages exist? It's a matter of comfort/goals (I don't advocate that everything should be done in C, and MATLAB is great for quickly doing many things, but many other things that have been done in MATLAB would have been less painful/faster if it used...say, Python+NumPy, because then you can use everything the Python ecosystem has to offer, and aren't limited to the MATLAB ecosystem).

dahart 8y ago

Maybe something like Jupyter or plain Python with GPU enabled numpy is what you’re looking for? Tensorflow is a library and not a language, it’s meant to be used from a host language that is Turing complete. It’s goal is to make construction of graphs of vector evaluations easier and more performant, but not really to provide a general purpose computation environment, it’s assumed you’re calling it from a general purpose computation environment. So, if you’re using Tensorflow, you normally have general purpose computing available to you, with the option to bake your vector tasks into graphs easily and/or speed them up. Jupyter is becoming a decent alternative to Matlab, and you have many options for running vector computations from python, with or without a GPU.

ianbertolacci 8y ago

> niches like AI or physics simulations or protein folding These are 99% of HPC workloads. Nothing `niche` about them.