12 ms·

I am a big fan of much of the work going on with statistics in Julia. I'd like to point out another (related) thing though that is wrong with the state of stat

by gajomi 12y ago

I am a big fan of much of the work going on with statistics in Julia. I'd like to point out another (related) thing though that is wrong with the state of statistics in Julia, in my opinion. Actually my problem has less to do with statistics, per se, than the mathematical foundations of statistical calculations. There are, of course, many different approaches to statistical inference (the Bayesian vs. frequentist camps infamous among these), but the calculations all come down to reasoning about probabilities which is a well posed but not always easy task. The Julia developers, in their wisdom, have recognized this, and as such have put together Distributions.jl the purpose of which is to provide datatypes and utility functions over probability distributions (plus some vestigial methods about maximum likelihood inference, which thankfully stay out of the way). If you haven't seen it, check it out. It's got a nice design, I think. But there is presently a clear server limitation: the parameterized type hierarchy. The requirement is that every distribution have support over a set in which all members are either Univariate, Multivariate or Matrixvariate with elements in the fields being either Discrete or Continuous. This obviously misses the general picture of the kinds of sets from which one draws random variables, which plays into issues that the article mentions (if the probability distribution datatype can't model the data you have to do ad hoc things to account for it). Indeed a huge chunk of the issues currently open in the Distributions github page essential boil down to problems with representing the sets and/or spaces from which elements in the distribution are drawn: https://github.com/JuliaStats/Distributions.jl/issues/147 https://github.com/JuliaStats/Distributions.jl/issues/147 https://github.com/JuliaStats/Distributions.jl/issues/309 https://github.com/JuliaStats/Distributions.jl/issues/309 https://github.com/JuliaStats/Distributions.jl/issues/224 https://github.com/JuliaStats/Distributions.jl/issues/224 https://github.com/JuliaStats/Distributions.jl/issues/283 https://github.com/JuliaStats/Distributions.jl/issues/283 End users have a variety of well developed ideas in mind about the sets that their samples belong to and even the spaces from which they are drawn from which currently exceed the representational capacity of the existing types. In my opinion the way to fix this is to start a separate library that focuses on types and methods for representing and manipulating sets and spaces (i.e. topological information attached to sets). This could then be consumed by the Distributions people as well as others modeling things outside of probability.

johnmyleswhite 12y ago
I think it would be great to develop support for generic sets in a separate package that could eventually be merged into Distributions. I'm not sure that I think all of the issues you've linked to are related to that problem, though. For example, the issue about using Vector{Rational} as an input to the Categorical distribution type's constructor is a relatively minor issue since there are very few cases in which you'd want to evaluate the PDF of a Categorical variable using rational numbers rather than floating-point numbers. As such, it's not clear that you'd want to have a Categorical{Rational} type that's separate from the current Categorical type (which could be called the Categorical{Float64} type).
- gajomi 12y ago
  >I'm not sure that I think all of the issues you've linked to are related to that problem, though It's probably true that many of these issues could be resolved without the need for this generic sets fantasy of mine. The resolution that you propose for Categorical{Rational} I think makes the right choice, given the tradeoffs that one currently has to make in designing these things. The particular reason that I considered this issue to be related to the problem of the representation of sets is that if it is true that both the support of such a distribution and the values of the pdf were rationals one could support exact computation of moments and/or cumulants and/or coefficients of generating functions and the like. The "exactness" of these procedures becomes important if there is important information in the factorization of said moments/coefficients/whatever. In fact in these cases one actually might care to know not just that they are Rationals but Rationals with such and such restrictions, which is where more general type parameters would really start to shine. So perhaps someday somebody will want it...
  - johnmyleswhite 12y ago
    That's true if Rational means Rational{BigInt}, but the default in Julia is for Rational to mean Rational{Int}, which can potentially produce less reliable answers that using Float64.

by gajomi 12y ago

johnmyleswhite 12y ago

I think it would be great to develop support for generic sets in a separate package that could eventually be merged into Distributions. I'm not sure that I think all of the issues you've linked to are related to that problem, though. For example, the issue about using Vector{Rational} as an input to the Categorical distribution type's constructor is a relatively minor issue since there are very few cases in which you'd want to evaluate the PDF of a Categorical variable using rational numbers rather than floating-point numbers. As such, it's not clear that you'd want to have a Categorical{Rational} type that's separate from the current Categorical type (which could be called the Categorical{Float64} type).

gajomi 12y ago
>I'm not sure that I think all of the issues you've linked to are related to that problem, though It's probably true that many of these issues could be resolved without the need for this generic sets fantasy of mine. The resolution that you propose for Categorical{Rational} I think makes the right choice, given the tradeoffs that one currently has to make in designing these things. The particular reason that I considered this issue to be related to the problem of the representation of sets is that if it is true that both the support of such a distribution and the values of the pdf were rationals one could support exact computation of moments and/or cumulants and/or coefficients of generating functions and the like. The "exactness" of these procedures becomes important if there is important information in the factorization of said moments/coefficients/whatever. In fact in these cases one actually might care to know not just that they are Rationals but Rationals with such and such restrictions, which is where more general type parameters would really start to shine. So perhaps someday somebody will want it...
- johnmyleswhite 12y ago
  That's true if Rational means Rational{BigInt}, but the default in Julia is for Rational to mean Rational{Int}, which can potentially produce less reliable answers that using Float64.