4 ms·

At the heart of the machine learning that I do with pytorch, most of the errors come from wrong sizes of matrix multiplications, or some matrix input that has a

by nafizh 8y ago

At the heart of the machine learning that I do with pytorch, most of the errors come from wrong sizes of matrix multiplications, or some matrix input that has a wrong size somewhere in the middle of the net. How much an extra layer of wrapper of a functional language help with that? Pytorch is not much helpful in terms of the error messages. You have to print out the matrix sizes generated from intermediate operations to find out what is really happening.

l-m-z 8y ago
(I'm obviously biased as being the author of ocaml-torch) Using functional programming is not of much help for size mismatches, the current bindings don't even use the type system to check the number of dimensions - although that should be reasonably easy to add. Maybe a better approach to help with this would be tensors with named dimensions http://nlp.seas.harvard.edu/NamedTensor http://nlp.seas.harvard.edu/NamedTensor It's possible that a strong type system would help here but I don't think there have been much attempts. However when using the Python api I often have errors because of: - Unused variables when refactoring my code, which are just me forgetting to use some parameters. - Comparing things of different types (and Python does not report any error and just return that they are different). - Making changes to some helper functions without adapting all the projects where I'm using them. Using a good Python linter probably helps with these, but that's a place where languages like ocaml naturally shine.
75dvtwin 8y ago
Octave checks for matrix mismatch, I believe. Not sure if it is integrated with pytorch.
jackpirate 8y ago
I've done quite a lot of work trying to solve this problem in Haskell, and I'm of the opinion that current type system technology is not strong enough for real world machine learning. At the most basic level, the sizes of tensors are often not known until runtime, so some sort of dependent typing is necessary. Idris is currently the most practical dependently typed language, and it's missing a number of features that would be needed for machine learning work. For example, it only supports 64bit floating point operations, whereas 32bit ops are standard and the industry is moving to 16 bit and fewer ops. There's ways in Haskell to get most of the benefits of dependent typing, but they're pretty ugly. My subhask library [1] I think did a reasonable job for vector/matrix math, but once you get into higher order tensors everything becomes either far too verbose or impossible to specify. For example, permuting the axes of a tensor takes roughly a full page of code, and it's not even properly type safe. At the bottom of the link [1], there's a list of type system features that I think would be necessary to add to Haskell before it has a chance of a good linear algebra system... but in all honestly, I'm not even convinced yet that a usable linear algebra system can even be implemented in a language with proper dependent types. [1] http://github.com/mikeizbicki/subhask http://github.com/mikeizbicki/subhask
jvican 8y ago
The Scala community has some clever techniques to ensure that those properties can be proven at compile-time, for example, https://github.com/eaplatanios/tensorflow_scala https://github.com/eaplatanios/tensorflow_scala for TensorFlow and https://github.com/ctongfei/nexus https://github.com/ctongfei/nexus

by nafizh 8y ago

l-m-z 8y ago

(I'm obviously biased as being the author of ocaml-torch) Using functional programming is not of much help for size mismatches, the current bindings don't even use the type system to check the number of dimensions - although that should be reasonably easy to add. Maybe a better approach to help with this would be tensors with named dimensions http://nlp.seas.harvard.edu/NamedTensor http://nlp.seas.harvard.edu/NamedTensor It's possible that a strong type system would help here but I don't think there have been much attempts. However when using the Python api I often have errors because of: - Unused variables when refactoring my code, which are just me forgetting to use some parameters. - Comparing things of different types (and Python does not report any error and just return that they are different). - Making changes to some helper functions without adapting all the projects where I'm using them. Using a good Python linter probably helps with these, but that's a place where languages like ocaml naturally shine.

75dvtwin 8y ago

Octave checks for matrix mismatch, I believe. Not sure if it is integrated with pytorch.

jackpirate 8y ago

I've done quite a lot of work trying to solve this problem in Haskell, and I'm of the opinion that current type system technology is not strong enough for real world machine learning. At the most basic level, the sizes of tensors are often not known until runtime, so some sort of dependent typing is necessary. Idris is currently the most practical dependently typed language, and it's missing a number of features that would be needed for machine learning work. For example, it only supports 64bit floating point operations, whereas 32bit ops are standard and the industry is moving to 16 bit and fewer ops. There's ways in Haskell to get most of the benefits of dependent typing, but they're pretty ugly. My subhask library [1] I think did a reasonable job for vector/matrix math, but once you get into higher order tensors everything becomes either far too verbose or impossible to specify. For example, permuting the axes of a tensor takes roughly a full page of code, and it's not even properly type safe. At the bottom of the link [1], there's a list of type system features that I think would be necessary to add to Haskell before it has a chance of a good linear algebra system... but in all honestly, I'm not even convinced yet that a usable linear algebra system can even be implemented in a language with proper dependent types. [1] http://github.com/mikeizbicki/subhask http://github.com/mikeizbicki/subhask

jvican 8y ago

The Scala community has some clever techniques to ensure that those properties can be proven at compile-time, for example, https://github.com/eaplatanios/tensorflow_scala https://github.com/eaplatanios/tensorflow_scala for TensorFlow and https://github.com/ctongfei/nexus https://github.com/ctongfei/nexus