6 ms·

This is great. Functional languages have such an elegant representation of so many mathematical concepts. It's a bit of a shame that they don't have more widesp

This is great. Functional languages have such an elegant representation of so many mathematical concepts. It's a bit of a shame that they don't have more widespread use in scientific computing.

mlthoughts2018 8y ago
I would also suggest looking into Keras and PyTorch too. I think they honestly achieve a greater degree of elegance and a greater degree of mapping the programming constructs into the mental model space of the domain expert, than any FP interface to neural nets that I’ve seen yet.
- phonebucket 8y ago
  I use PyTorch a lot; it's definitely my preferred framework at the moment. I just wish there was something as thoughtfully done and well-supported in a more functionally oriented language. Flux.jl on Julia is the frontrunner in this regard, IMO. The added benefit is that being written in Julia the whole way down makes it easy for practitioners to delve into the source code and extend it in a performant way without going into the C level nitty gritty.
  - dnautics 8y ago
    Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)
    - mlthoughts2018 8y ago
      This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integration hurdle to make it recognized by the backend, and then requires published research or some similar type of evidence that there are use cases which materially benefit from that new additional fixed data type, to get a PR approved to add it. The reason is that permitting arbitrary complexity growth in the form of “easy” custom data type support has two big downsides, (a) the mechanism that makes it easy had to consume maintenance and development resources even if it’s a very obscure form of customization, and (b) more importantly, it proliferates and worsens the already insane problems of being able to export / import models from one language/framework to another. It’s a case study of KISS and YAGNI: this is super premature abstraction especially if it’s for experiments. And the hurdle of making a branch and adding your new dtype in the backend is not (and should not be seen as) a significant engineering hurdle. Rather it’s a very good check on complexity growth.
      - dnautics 8y ago
        Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype). This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed. In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.
        mlthoughts2018 8y ago
        > “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?” This sounds like premature abstraction to me...
        phonebucket 8y ago
        Premature abstraction is a problem for engineering. But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
        dnautics 8y ago
        At some point one of those explorations becomes useful and it's no longer premature.
  - amelius 8y ago
    My biggest wish is that new research implementations in the field of ML come out in 1 framework, in 1 language. But for engineering purposes, it's nice that there is an ocaml framework now.
  - xvilka 8y ago
    Julia is very good, the only problem is the requirement for patched LLVM (patches they provide are not yet merged in the upstream), which can cause the conflicts with other frameworks if there is no separation.
    - dnautics 8y ago
      This is fairly easily dealt with using containers.
flavio81 8y ago
> It's a bit of a shame that they don't have more widespread use in scientific computing In truth, they have. Lisp was the first functional language (or the first language that allowed that paradigm), and has been used a lot in scientific computing, for example doing symbolic calculus and manipulation.
- nikofeyn 8y ago
  that doesn't matter when very few scientists have even heard of an ml (standard ml, ocaml, f#) or a lisp/scheme (common lisp, racket), much less have an inkling to use them. their use does of course exist but by no measure is it widespread.
  - ced 8y ago
    That's part of what makes Julia promising, it's a numerically-focused Lisp, without the parentheses.

This is great. Functional languages have such an elegant representation of so many mathematical concepts. It's a bit of a shame that they don't have more widespread use in scientific computing.

mlthoughts2018 8y ago

I would also suggest looking into Keras and PyTorch too. I think they honestly achieve a greater degree of elegance and a greater degree of mapping the programming constructs into the mental model space of the domain expert, than any FP interface to neural nets that I’ve seen yet.

phonebucket 8y ago
I use PyTorch a lot; it's definitely my preferred framework at the moment. I just wish there was something as thoughtfully done and well-supported in a more functionally oriented language. Flux.jl on Julia is the frontrunner in this regard, IMO. The added benefit is that being written in Julia the whole way down makes it easy for practitioners to delve into the source code and extend it in a performant way without going into the C level nitty gritty.
- dnautics 8y ago
  Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)
  - mlthoughts2018 8y ago
    This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integration hurdle to make it recognized by the backend, and then requires published research or some similar type of evidence that there are use cases which materially benefit from that new additional fixed data type, to get a PR approved to add it. The reason is that permitting arbitrary complexity growth in the form of “easy” custom data type support has two big downsides, (a) the mechanism that makes it easy had to consume maintenance and development resources even if it’s a very obscure form of customization, and (b) more importantly, it proliferates and worsens the already insane problems of being able to export / import models from one language/framework to another. It’s a case study of KISS and YAGNI: this is super premature abstraction especially if it’s for experiments. And the hurdle of making a branch and adding your new dtype in the backend is not (and should not be seen as) a significant engineering hurdle. Rather it’s a very good check on complexity growth.
    - dnautics 8y ago
      Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype). This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed. In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.
      - mlthoughts2018 8y ago
        > “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?” This sounds like premature abstraction to me...
        phonebucket 8y ago
        Premature abstraction is a problem for engineering. But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
        dnautics 8y ago
        At some point one of those explorations becomes useful and it's no longer premature.
- amelius 8y ago
  My biggest wish is that new research implementations in the field of ML come out in 1 framework, in 1 language. But for engineering purposes, it's nice that there is an ocaml framework now.
- xvilka 8y ago
  Julia is very good, the only problem is the requirement for patched LLVM (patches they provide are not yet merged in the upstream), which can cause the conflicts with other frameworks if there is no separation.
  - dnautics 8y ago
    This is fairly easily dealt with using containers.

flavio81 8y ago

> It's a bit of a shame that they don't have more widespread use in scientific computing In truth, they have. Lisp was the first functional language (or the first language that allowed that paradigm), and has been used a lot in scientific computing, for example doing symbolic calculus and manipulation.

nikofeyn 8y ago
that doesn't matter when very few scientists have even heard of an ml (standard ml, ocaml, f#) or a lisp/scheme (common lisp, racket), much less have an inkling to use them. their use does of course exist but by no measure is it widespread.
- ced 8y ago
  That's part of what makes Julia promising, it's a numerically-focused Lisp, without the parentheses.