6 ms·

Show HN: Golang statistics package with 100% code coverage

40 points by anonfunction 11y ago

anonfunction 11y ago
Hey Hackers, this is my first full-featured Golang package. I took a lot of care to ensure it was well tested and provided a lot of common functions that I felt were missing from the standard library. In addition it has a lot of other statistics functionality like regressions, outliers, correlation and more.
- Mithaldu 11y ago
  The coverage seems to be only line-based. I don't know if it's relevant for Go, but you might want to see how the coverage looks like for branch-based analysis (if Go has software for that). Otherwise, nice to see someone take quality serious. :)
  - anonfunction 11y ago
    The coverage is based on the built in go tooling, which supports line or function based analysis. In addition it can start a webserver where you can browse coverage by file with color output. # Generate coverage data go test -coverprofile=coverage.out # View coverage by function go tool cover -func="coverage.out" # View coverage line by line go tool cover -html="coverage.out"
- faitswulff 11y ago
  This seems like a really cool project to hack on. I noticed you have no issues on the github repo, but where do you want to go with it next? Or what would you like more eyes on?
  - howeman 11y ago
    I mentioned elsewhere in the thread, but if you're interested in help buliding statistical tools for go, Gonum is always looking for more hands. Look at what we have in github.com/gonum, and/or come introduce yourself at gonum-dev
  - anonfunction 11y ago
    Thanks for your interest! I think the biggest thing to do is finalize the public API and release v1.0.0. I started with only functions but have since added types, so I'd like to figure out if I should support both or stick to one or the other. I have a feeling I could use an Interface to make stats work for both []float64 and []int but I'm not sure about that. Another thing I'd like to do is add more test suites, I know they are out there for more mature statistics packages in other languages that could be ported to Go. Speaking of mature packages, I think the Gnu GSL Statistics[1] could be a good roadmap for future functionality and API design decisions. 1. http://www.gnu.org/software/gsl/manual/html_node/Statistics.html#Statistics http://www.gnu.org/software/gsl/manual/html_node/Statistics....
djs070 11y ago
Nicely done. Is returning error common in go (as opposed to raising)?
- anonfunction 11y ago
  Thanks! Go doesn't have exceptions and it is idiomatic[1] to return the error type. Dave Cheney has a really great blog post[2] which he describes why go does away with exceptions. The Golang wiki on GitHub has a great section[3] on errors as well. 1. https://gobyexample.com/errors https://gobyexample.com/errors 2. http://dave.cheney.net/2012/01/18/why-go-gets-exceptions-right http://dave.cheney.net/2012/01/18/why-go-gets-exceptions-rig... 3. https://github.com/golang/go/wiki/Errors https://github.com/golang/go/wiki/Errors
  - dragonwriter 11y ago
    > Go doesn't have exceptions Go has panics, which are pretty much the same thing as exceptions in other languages (the details of how they are used are slightly different from, say, Java/Python/Ruby exceptions, but much more than, say, Perl 6 exceptions.) Go has a strong stylistic recommendation not to expose panics to external consumers of code, and it is idiomatic to instead use error returns from publicly-exposed library code. But "Go doesn't have exceptions" is mostly misleading.
    - anonfunction 11y ago
      Rob Pike, one of the three creators of Golang, disagrees with you. In his blog post less is exponentially more[1] he shares a list of significant simplifications in Go over C and C++ which includes: > no exceptions 1. http://commandcenter.blogspot.pt/2012/06/less-is-exponentially-more.html http://commandcenter.blogspot.pt/2012/06/less-is-exponential...
      - dragonwriter 11y ago
        And yet "panics" exist, with very similar semantics to exceptions in C++/Java/Python/Ruby. Sure, there's differences between Go panics and any one of those implementations of exceptions (just as there are between them, or between any of them and Perl 6 exceptions.) But panics are clearly a variation on the exception theme, just as each of those forms of exceptions is. The strong idiom of not having panics exposed across public interfaces of libraries is a cultural change, which to the extent that it is followed does present a simplification compared to ecosystems where you have to deal with both returns and exceptions in library code. But panics very much are exceptions, and changing the name doesn't change that.
howeman 11y ago
Gonum has many of these capabilities, but with a slightly different mentality (gonum has weights and sticks to pure []float64 while this defines types and methods, i.e. Coordinate, Float64Data) gonum/floats for max, min, round, sum, etc. gonum/stat for mean, std, variance, etc. gonum/stat/sample for sampling without replacement
- howeman 11y ago
  If you're interested in developing more stats stuff, we'd love to have you as part of the effort. gonum/stat/dist in particular needs a lot more functionality.
- fogleman 11y ago
  Why are these all separate repos? And is there any centralized documentation? Can't find any.
  - howeman 11y ago
    They are separate repos because they have different mentalities. Stat is basic statistics for a (weighted) set of values. Floats is closer to dealing with properties of a vector (norm, distance, etc.). Sample has sophisticated sampling from distributions (metropolis hastings, etc.). gonum/stat/dist has sampling distributions themselves. Documentation can be found in godoc, i.e. godoc.org/github.com/gonum/stat, godoc.org/github.com/gonum/matrix/mat64, etc.
- anonfunction 11y ago
  Stats also works fine with pure []float64 data. The types allow for a very clean API for working with data and statistics. You can compare using functions[1] or methods[2] in /examples. I started with only []float64 but wanted to allow stuff like this: func getData() []float64 { // Do stuff to get the data and // store it in var data []float64 return data } var d Float64Data = getData() fmt.Println(d.Max()) 1. https://github.com/montanaflynn/stats/blob/master/examples/main.go https://github.com/montanaflynn/stats/blob/master/examples/m... 2. https://github.com/montanaflynn/stats/blob/master/examples/methods.go https://github.com/montanaflynn/stats/blob/master/examples/m...
fogleman 11y ago
Regarding your examples, shouldn't Round return 3.17 and shouldn't Median be 3.5?
- anonfunction 11y ago
  You are correct, it's been fixed with this commit[1]. My excuse is it was late when I added those output comments. 1. https://github.com/montanaflynn/stats/commit/a3efee7551d14a95342cfddd45ccee4ee745b9b8 https://github.com/montanaflynn/stats/commit/a3efee7551d14a9...
jaytaylor 11y ago
I've been using this package for awhile now and I can attest to the quality: It is great! It has powerful and easy to use statistics tools which IMHO should've been included in the standard library in the first place. Thank you anonfunction et. al. for your sharing the fruits of your labor with us! :D
- anonfunction 11y ago
  Thanks a lot, it means a bunch to me :D
jaytaylor 11y ago
I've been using this package for awhile now and I can attest to the quality: It is great! It has powerful and easy to use statistics tools which IMHO should've been included in the standard library in the first place. Thank you anonfunction for your sharing the fruits of your labor with us! :D

40 points by anonfunction 11y ago

Hey Hackers, this is my first full-featured Golang package. I took a lot of care to ensure it was well tested and provided a lot of common functions that I felt were missing from the standard library. In addition it has a lot of other statistics functionality like regressions, outliers, correlation and more.

Mithaldu 11y ago
The coverage seems to be only line-based. I don't know if it's relevant for Go, but you might want to see how the coverage looks like for branch-based analysis (if Go has software for that). Otherwise, nice to see someone take quality serious. :)
- anonfunction 11y ago
  The coverage is based on the built in go tooling, which supports line or function based analysis. In addition it can start a webserver where you can browse coverage by file with color output. # Generate coverage data go test -coverprofile=coverage.out # View coverage by function go tool cover -func="coverage.out" # View coverage line by line go tool cover -html="coverage.out"
faitswulff 11y ago
This seems like a really cool project to hack on. I noticed you have no issues on the github repo, but where do you want to go with it next? Or what would you like more eyes on?
- howeman 11y ago
  I mentioned elsewhere in the thread, but if you're interested in help buliding statistical tools for go, Gonum is always looking for more hands. Look at what we have in github.com/gonum, and/or come introduce yourself at gonum-dev
- anonfunction 11y ago
  Thanks for your interest! I think the biggest thing to do is finalize the public API and release v1.0.0. I started with only functions but have since added types, so I'd like to figure out if I should support both or stick to one or the other. I have a feeling I could use an Interface to make stats work for both []float64 and []int but I'm not sure about that. Another thing I'd like to do is add more test suites, I know they are out there for more mature statistics packages in other languages that could be ported to Go. Speaking of mature packages, I think the Gnu GSL Statistics[1] could be a good roadmap for future functionality and API design decisions. 1. http://www.gnu.org/software/gsl/manual/html_node/Statistics.html#Statistics http://www.gnu.org/software/gsl/manual/html_node/Statistics....

djs070 11y ago

Nicely done. Is returning error common in go (as opposed to raising)?

anonfunction 11y ago
Thanks! Go doesn't have exceptions and it is idiomatic[1] to return the error type. Dave Cheney has a really great blog post[2] which he describes why go does away with exceptions. The Golang wiki on GitHub has a great section[3] on errors as well. 1. https://gobyexample.com/errors https://gobyexample.com/errors 2. http://dave.cheney.net/2012/01/18/why-go-gets-exceptions-right http://dave.cheney.net/2012/01/18/why-go-gets-exceptions-rig... 3. https://github.com/golang/go/wiki/Errors https://github.com/golang/go/wiki/Errors
- dragonwriter 11y ago
  > Go doesn't have exceptions Go has panics, which are pretty much the same thing as exceptions in other languages (the details of how they are used are slightly different from, say, Java/Python/Ruby exceptions, but much more than, say, Perl 6 exceptions.) Go has a strong stylistic recommendation not to expose panics to external consumers of code, and it is idiomatic to instead use error returns from publicly-exposed library code. But "Go doesn't have exceptions" is mostly misleading.
  - anonfunction 11y ago
    Rob Pike, one of the three creators of Golang, disagrees with you. In his blog post less is exponentially more[1] he shares a list of significant simplifications in Go over C and C++ which includes: > no exceptions 1. http://commandcenter.blogspot.pt/2012/06/less-is-exponentially-more.html http://commandcenter.blogspot.pt/2012/06/less-is-exponential...
    - dragonwriter 11y ago
      And yet "panics" exist, with very similar semantics to exceptions in C++/Java/Python/Ruby. Sure, there's differences between Go panics and any one of those implementations of exceptions (just as there are between them, or between any of them and Perl 6 exceptions.) But panics are clearly a variation on the exception theme, just as each of those forms of exceptions is. The strong idiom of not having panics exposed across public interfaces of libraries is a cultural change, which to the extent that it is followed does present a simplification compared to ecosystems where you have to deal with both returns and exceptions in library code. But panics very much are exceptions, and changing the name doesn't change that.

howeman 11y ago

Gonum has many of these capabilities, but with a slightly different mentality (gonum has weights and sticks to pure []float64 while this defines types and methods, i.e. Coordinate, Float64Data) gonum/floats for max, min, round, sum, etc. gonum/stat for mean, std, variance, etc. gonum/stat/sample for sampling without replacement

howeman 11y ago
If you're interested in developing more stats stuff, we'd love to have you as part of the effort. gonum/stat/dist in particular needs a lot more functionality.
fogleman 11y ago
Why are these all separate repos? And is there any centralized documentation? Can't find any.
- howeman 11y ago
  They are separate repos because they have different mentalities. Stat is basic statistics for a (weighted) set of values. Floats is closer to dealing with properties of a vector (norm, distance, etc.). Sample has sophisticated sampling from distributions (metropolis hastings, etc.). gonum/stat/dist has sampling distributions themselves. Documentation can be found in godoc, i.e. godoc.org/github.com/gonum/stat, godoc.org/github.com/gonum/matrix/mat64, etc.
anonfunction 11y ago
Stats also works fine with pure []float64 data. The types allow for a very clean API for working with data and statistics. You can compare using functions[1] or methods[2] in /examples. I started with only []float64 but wanted to allow stuff like this: func getData() []float64 { // Do stuff to get the data and // store it in var data []float64 return data } var d Float64Data = getData() fmt.Println(d.Max()) 1. https://github.com/montanaflynn/stats/blob/master/examples/main.go https://github.com/montanaflynn/stats/blob/master/examples/m... 2. https://github.com/montanaflynn/stats/blob/master/examples/methods.go https://github.com/montanaflynn/stats/blob/master/examples/m...

fogleman 11y ago

Regarding your examples, shouldn't Round return 3.17 and shouldn't Median be 3.5?

anonfunction 11y ago
You are correct, it's been fixed with this commit[1]. My excuse is it was late when I added those output comments. 1. https://github.com/montanaflynn/stats/commit/a3efee7551d14a95342cfddd45ccee4ee745b9b8 https://github.com/montanaflynn/stats/commit/a3efee7551d14a9...

jaytaylor 11y ago

I've been using this package for awhile now and I can attest to the quality: It is great! It has powerful and easy to use statistics tools which IMHO should've been included in the standard library in the first place. Thank you anonfunction et. al. for your sharing the fruits of your labor with us! :D

anonfunction 11y ago
Thanks a lot, it means a bunch to me :D

jaytaylor 11y ago

I've been using this package for awhile now and I can attest to the quality: It is great! It has powerful and easy to use statistics tools which IMHO should've been included in the standard library in the first place. Thank you anonfunction for your sharing the fruits of your labor with us! :D