3 ms·

The author seems to be confused and uninformed about how parallelism is implemented in ML libraries. Libraries don't mostly rely on spawning threads for paralle

The author seems to be confused and uninformed about how parallelism is implemented in ML libraries. Libraries don't mostly rely on spawning threads for parallel execution! Parallelization happens at the hardware level using SIMD by running the calculations for backprop on a GPU/TPU/DSP. Not by creating new threads on the CPU. Because of this, you get the same type of isolated execution you would have with a functional language. Which leads to the second point. Libraries like TF are already designed around a data flow execution model and provide functional APIs. It is nonsense to assert these libraries are missing something fundamental that Clojure and Haskell would magically fix.

K0SM0S 7y ago
This is beyond my level of understanding but I believe there are physical implementation paradigms in all languages that make them either capable or not of performing certain things. While you can always bend a turing-complete language to do the logic, it doesn't mean your abstractions carry all the way down to hardware — there is "overhead" or "loss of efficiency" if you will. You get the convenience, and perhaps organize the safety; but it only goes so far. For instance Go was designed based on SCP (notably), a theoretical language designed by Tony Hoare in 1978 which implements a workable and fairly efficient concurrent paradigm. At the compiler then runtime level (low-level things you'll never be able to 'change' with libraries), when the program is built then executed, the implementation is directly able to translate into the physical topology of multi-core machines (Go design began in the late 2000's, shortly after Intel released the Core Duo, first multi-core CPU). Now you can do parallel and/or concurrent in other languages, of course, but since they were not designed as such, it's a convoluted programming exercise to say the least. For proof, observe how little software is able to multi-thread efficiently today, how very little is built in a concurrent approach — it's just a mess in most languages, the added complexity not worth the cost. This is an example of how you can get the functionality of a fundamental feature but the cost, the implementation curve is so steep that it doesn't really fly — reality, money, skill pool, all these things. On topic, functional languages like Haskell do indeed "fix" or rather implement some features in more efficient ways than e.g. Python; it's always a trade off you know. No best tool, only tools best suited to a given case.
- codesushi42 7y ago
  The point was that the concurrency model of these ML libraries is not dependent on a CPU for backprop (ideally), so discussing language level concurrency as an advantage is irrelevant. The parallelism is not implemented at the language level, nor does it rely on language constructs for synchronizing shared memory. The parallelism happens at the hardware level (e.g. with SIMD). And in the model architecture (e.g. with convolutions). So the advantages you posited in your lengthy diatribe are meaningless, because you failed to take into account the problem domain, along with utterly failing to grasp what gets calculated during backprop. Plus, backprop doesn't get calculated in Python. The argument about functional vs imperative languages is old, tired and is not relevant to the subject. Please at least learn some fundamentals about NNs and ML frameworks before contributing a long, uninformed response.
  - K0SM0S 7y ago
    > the concurrency model of these ML libraries is not dependent on a CPU Gotcha, my bad. > Please at least learn some fundamentals about NNs and ML frameworks before contributing a long, uninformed response. My apologies. It's not like I enjoy wasting people's time, starting with mine. It was an honest mistake, point taken, and thanks for explaining. If you have a somewhat 'definitive' book or chapter/resource to share on "understanding concurrency"... please do so. I'm eager to learn, and I'd wager many reading such a thread.
    - codesushi42 7y ago
      See Udacity's course on parallel programming: https://www.youtube.com/watch?v=F620ommtjqk&list=PLGvfHSgImk4aweyWlhBXNF6XISY3um82_ https://www.youtube.com/watch?v=F620ommtjqk&list=PLGvfHSgImk... I don't think it touches upon ML, but it is relevant for understanding the difference between GPUs and CPUs. You can go through the intro to backprop lectures on deeplearning.ai for ML.
- joe_the_user 7y ago
  While you can always bend a turing-complete language to do the logic, it doesn't mean your abstractions carry all the way down to hardware — there is "overhead" or "loss of efficiency" if you will. You get the convenience, and perhaps organize the safety; but it only goes so far. Well, hmm, I think you're right about the "hardware bending" part but unfortunately you are completely wrong about "Turing complete" part (I'm not sure where that comes in, tb). Indeed, a big factor with all this is that a c/c++ vector extension is pretty good reflection of a GPUs operations (and I'd assume a TPU, it being similar). To wit. If you look at the assembly code of a GPU (Nvidia PTX), appears is reasonably similar to the machine code of CPUs but some extra sorts of primitives to allow communication between kernels. Otherwise, the code is "SIMD" (or STMD as Nvidia says); Most of a program is a sequence of imperative instructions that are different from CPU instruction in that they're acting on a huge "vector" of values. You really don't "threads" in the fashion that CPUs have - as the GP says. What you talking about is a system that's absolutely "Turing complete" (or as Turing complete as assembler or C on a CPU) but with extra facilities. The parallelism involved is absolutely different from the parallelism of current CPU systems. This is why just everything is translated to c++ which is then translated to PTX. All that said, people have indeed proposed ways to bend GPUs to make them look more like current (MIMD) systems - see: https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ etc. But if anything, the "bend the hardware argument" very much work against Haskell etc on the GPU because not all parallelism is concurrent system like.

K0SM0S 7y ago

This is beyond my level of understanding but I believe there are physical implementation paradigms in all languages that make them either capable or not of performing certain things. While you can always bend a turing-complete language to do the logic, it doesn't mean your abstractions carry all the way down to hardware — there is "overhead" or "loss of efficiency" if you will. You get the convenience, and perhaps organize the safety; but it only goes so far. For instance Go was designed based on SCP (notably), a theoretical language designed by Tony Hoare in 1978 which implements a workable and fairly efficient concurrent paradigm. At the compiler then runtime level (low-level things you'll never be able to 'change' with libraries), when the program is built then executed, the implementation is directly able to translate into the physical topology of multi-core machines (Go design began in the late 2000's, shortly after Intel released the Core Duo, first multi-core CPU). Now you can do parallel and/or concurrent in other languages, of course, but since they were not designed as such, it's a convoluted programming exercise to say the least. For proof, observe how little software is able to multi-thread efficiently today, how very little is built in a concurrent approach — it's just a mess in most languages, the added complexity not worth the cost. This is an example of how you can get the functionality of a fundamental feature but the cost, the implementation curve is so steep that it doesn't really fly — reality, money, skill pool, all these things. On topic, functional languages like Haskell do indeed "fix" or rather implement some features in more efficient ways than e.g. Python; it's always a trade off you know. No best tool, only tools best suited to a given case.

codesushi42 7y ago
The point was that the concurrency model of these ML libraries is not dependent on a CPU for backprop (ideally), so discussing language level concurrency as an advantage is irrelevant. The parallelism is not implemented at the language level, nor does it rely on language constructs for synchronizing shared memory. The parallelism happens at the hardware level (e.g. with SIMD). And in the model architecture (e.g. with convolutions). So the advantages you posited in your lengthy diatribe are meaningless, because you failed to take into account the problem domain, along with utterly failing to grasp what gets calculated during backprop. Plus, backprop doesn't get calculated in Python. The argument about functional vs imperative languages is old, tired and is not relevant to the subject. Please at least learn some fundamentals about NNs and ML frameworks before contributing a long, uninformed response.
- K0SM0S 7y ago
  > the concurrency model of these ML libraries is not dependent on a CPU Gotcha, my bad. > Please at least learn some fundamentals about NNs and ML frameworks before contributing a long, uninformed response. My apologies. It's not like I enjoy wasting people's time, starting with mine. It was an honest mistake, point taken, and thanks for explaining. If you have a somewhat 'definitive' book or chapter/resource to share on "understanding concurrency"... please do so. I'm eager to learn, and I'd wager many reading such a thread.
  - codesushi42 7y ago
    See Udacity's course on parallel programming: https://www.youtube.com/watch?v=F620ommtjqk&list=PLGvfHSgImk4aweyWlhBXNF6XISY3um82_ https://www.youtube.com/watch?v=F620ommtjqk&list=PLGvfHSgImk... I don't think it touches upon ML, but it is relevant for understanding the difference between GPUs and CPUs. You can go through the intro to backprop lectures on deeplearning.ai for ML.
joe_the_user 7y ago
While you can always bend a turing-complete language to do the logic, it doesn't mean your abstractions carry all the way down to hardware — there is "overhead" or "loss of efficiency" if you will. You get the convenience, and perhaps organize the safety; but it only goes so far. Well, hmm, I think you're right about the "hardware bending" part but unfortunately you are completely wrong about "Turing complete" part (I'm not sure where that comes in, tb). Indeed, a big factor with all this is that a c/c++ vector extension is pretty good reflection of a GPUs operations (and I'd assume a TPU, it being similar). To wit. If you look at the assembly code of a GPU (Nvidia PTX), appears is reasonably similar to the machine code of CPUs but some extra sorts of primitives to allow communication between kernels. Otherwise, the code is "SIMD" (or STMD as Nvidia says); Most of a program is a sequence of imperative instructions that are different from CPU instruction in that they're acting on a huge "vector" of values. You really don't "threads" in the fashion that CPUs have - as the GP says. What you talking about is a system that's absolutely "Turing complete" (or as Turing complete as assembler or C on a CPU) but with extra facilities. The parallelism involved is absolutely different from the parallelism of current CPU systems. This is why just everything is translated to c++ which is then translated to PTX. All that said, people have indeed proposed ways to bend GPUs to make them look more like current (MIMD) systems - see: https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ etc. But if anything, the "bend the hardware argument" very much work against Haskell etc on the GPU because not all parallelism is concurrent system like.