6 ms·

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “

by mlthoughts2018 8y ago

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integration hurdle to make it recognized by the backend, and then requires published research or some similar type of evidence that there are use cases which materially benefit from that new additional fixed data type, to get a PR approved to add it. The reason is that permitting arbitrary complexity growth in the form of “easy” custom data type support has two big downsides, (a) the mechanism that makes it easy had to consume maintenance and development resources even if it’s a very obscure form of customization, and (b) more importantly, it proliferates and worsens the already insane problems of being able to export / import models from one language/framework to another. It’s a case study of KISS and YAGNI: this is super premature abstraction especially if it’s for experiments. And the hurdle of making a branch and adding your new dtype in the backend is not (and should not be seen as) a significant engineering hurdle. Rather it’s a very good check on complexity growth.

dnautics 8y ago
Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype). This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed. In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.
- mlthoughts2018 8y ago
  > “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?” This sounds like premature abstraction to me...
  - phonebucket 8y ago
    Premature abstraction is a problem for engineering. But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
    - dnautics 8y ago
      At some point one of those explorations becomes useful and it's no longer premature.
  - improbable22 8y ago
    There is work on rotationally invariant networks, e.g. for identifying galaxies, or cells under a microscope. For example: https://arxiv.org/abs/1612.04642 https://arxiv.org/abs/1612.04642 https://arxiv.org/abs/1805.12301 https://arxiv.org/abs/1805.12301 I haven't looked closely enough to be sure if they literally had complex activations, but this seems like an obvious use. Maybe they would have, if only tensorflow made it easy.
    - mlthoughts2018 8y ago
      Why would you ever want to represent activations directly as complex numbers in that case? I can’t think of any good reason, compared with putting them in rotation matrix form or some other equivalent form that actually maps to the domain modeling problem. Even when working in signal processing problems that require complex arithmetic, the underlying representations are just based on tuples of doubles and operator conventions, and you always need to map to real spaces (real part, imaginary part, angle, or magnitude) for any type of analytical representation that can be human readable. In all these cases, the idea that what we should optimize for is overhead-free easy expression of cutesy math domain verbiage is a bad idea. Writing libraries that expose an API that matches the user’s domain mental model is a great thing. But enforcing a particular abstraction and extensibility hierarchy so those things can be “autogenerated” just by parameterizing over a new type turns out to be actually much worse than just writing that type separately, with helper functions and converters, and customizing its API to be efficient from a domain mental model perspective. A better way, for example, might be to use mixin patterns or decorators and other metaprogramming, while writing a custom data type and its associated methods.
one-more-minute 8y ago
I can see where you're coming from in any other language. But in Julia it's important to realise that this "premature abstraction" isn't actually any extra work, it's just the default. If we write `f(x) = x+x` then `f` takes anything that can be added, which can be any custom number or matrix type, or really anything else. Adding type restrictions to make it work with only a limited set of types is completely doable, but actually more work than just leaving it generic. We didn't at any point decide "it's worth the extra effort/complexity to make Flux work with custom number types"; it's just inadvertently been that way from day one, and I didn't even know anyone one was making use of it until today.
- mlthoughts2018 8y ago
  This is the same for many languages that treat operators with type class patterns. It’s still usually bad design choice. I find a lot of language designers & programmers like it, but they are super disconnected from the realities. For an example consider breeze and spire in Scala. There’s so much effort to create these bloated numeric type hierarchies that abstract out things like monoids, rings, fields, iterability, sortability, etc. It’s not good. Just having really boring repetitive implementations for each distinct data structure would be better! No joke! Being able to write type generic functions over sortable matrix subclasses turns out to not be valuable unless you’re also writing a highly abstracted library, which is never, certainly not when you’re using it for experiments. Nobody needs to be able to make a DenseMatrix[Quaternion] and get it to automatically pick up implementations of fancy indexing. No. You can just write your own helper methods, and this is better, more convenient, applies less pressure for DenseMatrix to have some indecipherably complicated abstract implementation so it can be more free to just specialize on linear algebra functionality that works for DenseMatrix[Double] which is what is needed 99.999999999% of the time.
  - dnautics 8y ago
    You really should try Julia, before making claims about its complexity. The numeric type systems are simple, and designed for convenience, not to satisfy mathematical theory. In the case of FloatX, It's basically Any <: Number <: AbstractReal <: AbstractFloat <: FloatX For complex datatypes, like vectors, matrices, dicts, etc, you have templatable datatypes, but that is no more complex than C++, and actually far cleaner in implementation. For the most part, you do not NEED to make a Matrix{Quaternion}. And that's fine. However, if you do, the standard library will do the right thing, as if you had made a Matrix{Int32} or a Matrix{8BitGaloisField}. And if you choose to use Matrix{Float32}, the type system interacts with the compiler, and in the standard library it picks up the fortran BLAS library so you get faster-than-c performance. On the other hand, you might be deploying a really large matrix on a supercomputing cluster, and it might be useful to re-index the matrix as a datatype that fits in the L1 cache of your Knights Landing chips. In which case, you have the option of redeploying as an AbstractMatrix{Float64}, implementing index catching functions, and dropping it in to you code (probably about 100 lines of code total, if even) without having to rewrite every single matrix operation everywhere.
ChrisRackauckas 8y ago
I'm using the abstract numbers for encodings of uncertainty and probability distributions. This goes far beyond FloatX types and is really helping spawn a whole new area of research. The number type abstractions in Flux are really one of a kind. Kudos to the developers.
- mlthoughts2018 8y ago
  I work on large scale MCMC and causal inference problems. Would be very interested to know what this area of research is where you require abstract numeric types themselves to represent uncertainty, and why it would be different than other inference algorithms that handle uncertainty. I admit, at first blush I am extremely skeptical. It sounds like a silly sort of thing where instead of parameterizing over a numeric type, you might parameterize over a number-from-a-distribution type, and then try to make the type system represent how everything will flow in an MCMC setup, similar to tools like pymc, except where those are declarative and procedural, this would attempt to embed that into the type system. I can scarcely think of a worse way to represent manipulating uncertainty though. I hope I’m just reading your comment incorrectly.
  - ChrisRackauckas 8y ago
    It's for differential equations. Getting uncertainty without parameter sampling saves a lot of computational time and really opens up the problems that can be solved.
    - mlthoughts2018 8y ago
      But why does “getting uncertainty without parameter sampling” have anything to do with parametric numeric types? The latter is just a possible manner of implementation (that I’d argue is too cutesy), but there are many other ways to design a system like that, for example like fused types in Cython combined with numba class jitting. I still see no reason to believe that something that parametrizes differential equation functions or linalg functions over “uncertainty primitives” would be anything but a functional programming hot take on something that could be more straightforwardly done many other ways not relying on parametric abstraction.

dnautics 8y ago

Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype). This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed. In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.

mlthoughts2018 8y ago
> “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?” This sounds like premature abstraction to me...
- phonebucket 8y ago
  Premature abstraction is a problem for engineering. But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
  - dnautics 8y ago
    At some point one of those explorations becomes useful and it's no longer premature.
- improbable22 8y ago
  There is work on rotationally invariant networks, e.g. for identifying galaxies, or cells under a microscope. For example: https://arxiv.org/abs/1612.04642 https://arxiv.org/abs/1612.04642 https://arxiv.org/abs/1805.12301 https://arxiv.org/abs/1805.12301 I haven't looked closely enough to be sure if they literally had complex activations, but this seems like an obvious use. Maybe they would have, if only tensorflow made it easy.
  - mlthoughts2018 8y ago
    Why would you ever want to represent activations directly as complex numbers in that case? I can’t think of any good reason, compared with putting them in rotation matrix form or some other equivalent form that actually maps to the domain modeling problem. Even when working in signal processing problems that require complex arithmetic, the underlying representations are just based on tuples of doubles and operator conventions, and you always need to map to real spaces (real part, imaginary part, angle, or magnitude) for any type of analytical representation that can be human readable. In all these cases, the idea that what we should optimize for is overhead-free easy expression of cutesy math domain verbiage is a bad idea. Writing libraries that expose an API that matches the user’s domain mental model is a great thing. But enforcing a particular abstraction and extensibility hierarchy so those things can be “autogenerated” just by parameterizing over a new type turns out to be actually much worse than just writing that type separately, with helper functions and converters, and customizing its API to be efficient from a domain mental model perspective. A better way, for example, might be to use mixin patterns or decorators and other metaprogramming, while writing a custom data type and its associated methods.

one-more-minute 8y ago

I can see where you're coming from in any other language. But in Julia it's important to realise that this "premature abstraction" isn't actually any extra work, it's just the default. If we write `f(x) = x+x` then `f` takes anything that can be added, which can be any custom number or matrix type, or really anything else. Adding type restrictions to make it work with only a limited set of types is completely doable, but actually more work than just leaving it generic. We didn't at any point decide "it's worth the extra effort/complexity to make Flux work with custom number types"; it's just inadvertently been that way from day one, and I didn't even know anyone one was making use of it until today.

mlthoughts2018 8y ago
This is the same for many languages that treat operators with type class patterns. It’s still usually bad design choice. I find a lot of language designers & programmers like it, but they are super disconnected from the realities. For an example consider breeze and spire in Scala. There’s so much effort to create these bloated numeric type hierarchies that abstract out things like monoids, rings, fields, iterability, sortability, etc. It’s not good. Just having really boring repetitive implementations for each distinct data structure would be better! No joke! Being able to write type generic functions over sortable matrix subclasses turns out to not be valuable unless you’re also writing a highly abstracted library, which is never, certainly not when you’re using it for experiments. Nobody needs to be able to make a DenseMatrix[Quaternion] and get it to automatically pick up implementations of fancy indexing. No. You can just write your own helper methods, and this is better, more convenient, applies less pressure for DenseMatrix to have some indecipherably complicated abstract implementation so it can be more free to just specialize on linear algebra functionality that works for DenseMatrix[Double] which is what is needed 99.999999999% of the time.
- dnautics 8y ago
  You really should try Julia, before making claims about its complexity. The numeric type systems are simple, and designed for convenience, not to satisfy mathematical theory. In the case of FloatX, It's basically Any <: Number <: AbstractReal <: AbstractFloat <: FloatX For complex datatypes, like vectors, matrices, dicts, etc, you have templatable datatypes, but that is no more complex than C++, and actually far cleaner in implementation. For the most part, you do not NEED to make a Matrix{Quaternion}. And that's fine. However, if you do, the standard library will do the right thing, as if you had made a Matrix{Int32} or a Matrix{8BitGaloisField}. And if you choose to use Matrix{Float32}, the type system interacts with the compiler, and in the standard library it picks up the fortran BLAS library so you get faster-than-c performance. On the other hand, you might be deploying a really large matrix on a supercomputing cluster, and it might be useful to re-index the matrix as a datatype that fits in the L1 cache of your Knights Landing chips. In which case, you have the option of redeploying as an AbstractMatrix{Float64}, implementing index catching functions, and dropping it in to you code (probably about 100 lines of code total, if even) without having to rewrite every single matrix operation everywhere.

ChrisRackauckas 8y ago

I'm using the abstract numbers for encodings of uncertainty and probability distributions. This goes far beyond FloatX types and is really helping spawn a whole new area of research. The number type abstractions in Flux are really one of a kind. Kudos to the developers.

mlthoughts2018 8y ago
I work on large scale MCMC and causal inference problems. Would be very interested to know what this area of research is where you require abstract numeric types themselves to represent uncertainty, and why it would be different than other inference algorithms that handle uncertainty. I admit, at first blush I am extremely skeptical. It sounds like a silly sort of thing where instead of parameterizing over a numeric type, you might parameterize over a number-from-a-distribution type, and then try to make the type system represent how everything will flow in an MCMC setup, similar to tools like pymc, except where those are declarative and procedural, this would attempt to embed that into the type system. I can scarcely think of a worse way to represent manipulating uncertainty though. I hope I’m just reading your comment incorrectly.
- ChrisRackauckas 8y ago
  It's for differential equations. Getting uncertainty without parameter sampling saves a lot of computational time and really opens up the problems that can be solved.
  - mlthoughts2018 8y ago
    But why does “getting uncertainty without parameter sampling” have anything to do with parametric numeric types? The latter is just a possible manner of implementation (that I’d argue is too cutesy), but there are many other ways to design a system like that, for example like fused types in Cython combined with numba class jitting. I still see no reason to believe that something that parametrizes differential equation functions or linalg functions over “uncertainty primitives” would be anything but a functional programming hot take on something that could be more straightforwardly done many other ways not relying on parametric abstraction.