6 ms·

Does D have code for: plotting, optimization, probability distributions, machine learning, Fourier transformations, masked arrays, finanial calculations, struct

by RogerL 11y ago

Does D have code for: plotting, optimization, probability distributions, machine learning, Fourier transformations, masked arrays, finanial calculations, structured arrays (read a CSV from disk, get named columns based on the header), SVD, QR and Cholesky decomposition, eigens, least squares, Levenberg Marquardt, matrix inverse and pseudoinverses, integration, Runge Kutta, interpolation, bsplines, fft convolves, multidimensional images, KDTrees, symbolic equation solvers, merge/join of data sets, etc.? Because I use almost all of these every single day (I don't do multidimensional images or b-splines much at all). Are those all in standard libraries, fully documented, backed by 60 year old, fully debugge code (LAPACK, etc), that I reliably email to anyone across the world and they can immediately run and modify my code because it is such a standard? I honestly don't know, but I'm guessing not. I use Python/Numpy/Scipy/Pandas/Matplotlib because everyone else in the world knows and uses them; they are a standard. Yes, my np.mean() might be slower than your map(). I almost always don't care. That misses the forest for the trees. The article might be a good argument for why library writers might consider building out D's standard library to support numerical computation, I dunno. But no one is going to use D for serious number crunching without that infrastructure in place. People moved from Fortran and Matlab to Python not because it is fast, but for the environment. These language tricks are cute and all (I like D well enough, don't get me wrong), but it ain't why we are using Python. At this point, if I were to switch languages to something without a lot of adoption I'd lean towards Julia. It also have a modern language design, but it is written from the ground up for numerical computation. I can't think of any reason I'd ever reach for D.

vidarh 11y ago
> Does D have code for: [...] He covered this, specifically out that Numpy has an advantage in the number of libraries and other resources available for it. > because everyone else in the world knows and uses them; No, everyone working in some specialized niches does. And those niches don't even scratch the surface of the number of people who regularly do simple operations on multi-dimensional arrays but who don't need most of what you list. And sometimes people even in your niche that does depend on the other stuff run into things that are just way too slow, that they have to rewrite in a lower level language. There are plenty of people for whom this will be very useful without having all the support infrastructure around Numpy in place. > I can't think of any reason I'd ever reach for D. D is for people who wants a "C++ done right", where it is implicit that you want a C-like language, and share ideals of C++ such as to whatever extent possible not paying for what you don't use. Specifically in this case, you'd reach for D if you need multi-dimensional arrays and either prefer a C++-like language over Python or your Python code is too slow and you prefer to avoid a lower level language like C. That may not be for you, and that's fine. Just like Numpy isn't nearly universal for numerical computing. > Yes, my np.mean() might be slower than your map(). I almost always don't care. I find this an amusing objection... I did my MSc thesis on methods for reducing error rates in OCR, and included methods that used various nearest-neighbour variations and statistical methods on bitmap slices and similar. I did it mainly in Ruby (with some inline C where absolutely necessary for performance - if I remember correctly it was about a dozen lines of code in C) because I cared far more about readable code than fast code, and Python would in my eyes have been a double loss: not the speed of C, yet still ugly and annoying to work with in my opinion. Seeing the D code examples, I might very well have picked D for it today.
n00b101 11y ago
>Are those all in standard libraries, fully documented, backed by 60 year old, fully debugge code (LAPACK, etc) ... yes, my np.mean() might be slower than your map(). I almost always don't care. Hmm ... LAPACK itself is only 23 years old (1992). Its was implemented to be a faster replacement of the "standard" LINPACK library, by effectively exploiting the caches on modern processors. For most serious users of numerical computing, performance is extremely important. Also, I think you might be putting far too much faith in an arbitrary assortment of Python libraries. If you think that NumPy is just a thin wrapper on LAPACK, and you are using it in mission or life critical operations, then I think you have some code vetting to do. Even if some of your chosen libraries do use LAPACK, and even if there is some code in LAPACK that is 60 years old, that does not suddenly mean that all of your chosen Python libraries (and your code on top of them) is all correct. Most serious users of many of the routines that you mention (e.g. "machine learning" and "financial calculations") implement and test these routines themselves in order to ensure their correctness. In very serious model vetting operations, it would be customary to even invest in two different independent implementations of the same routine. > I reliably email to anyone across the world and they can immediately run and modify my code because it is such a standard? I don't think "reliable" and "emailing scripts" normally go together. It would be more customary to use a source code control system in a high reliability environment. Personally I have found reproducing Python installations to be very tricky and I would take extreme care in any critical operations that assume two different Python installations produce the exactly the same results (high reliability, like some of the 60+year old codes run by NASA, would even require bug for bug reproducibility). "Python/Numpy/Scipy/Pandas/Matplotlib" are not the last word on anything, theyt are not standardized as you imply, and they leave vast room for improvement in all dimensions (performance, correctness, reliability, productivity, etc).
dallbee 11y ago
A lot of that is up and coming. https://github.com/DlangScience https://github.com/DlangScience [x] Plotting: http://code.dlang.org/packages/plot2kill http://code.dlang.org/packages/plot2kill [+] Optimization: Nothing directly for the purpose, but the intermediate steps are done. And a little bit of work from http://code.dlang.org/packages/atmosphere http://code.dlang.org/packages/atmosphere [x] Probability Distributions: http://dlangscience.github.io/dstats/api/dstats/random.html http://dlangscience.github.io/dstats/api/dstats/random.html and http://dlang.org/phobos/std_mathspecial.html http://dlang.org/phobos/std_mathspecial.html [-] Machine Learning: Not that I'm aware of, but I vaguely recall some work being done on this [-] Financial calculations: I couldn't find anything, but I'd be surprised if this wasn't implemented already. [x] Masked Arrays: Accomplished via language features in combination with ndslice [x] Structured Arrays: Absolutely. Standard library support as well as a number of 3rd-party libs can do this. [-] SVD [?] QR: Afraid I'm not sure what you're referring to. [+] Cholesky Decomposition: Trivial to implement [x] Eigenvalues/Eigenvectors: Yes. SciD provides Eigens as well as two other third party packages. https://dlangscience.github.io/scid/api/scid/linalg/eigenvalues.html https://dlangscience.github.io/scid/api/scid/linalg/eigenval... [x] Least Squares (linear): Trivial to implement, but also implemented by a few of the 2-d & 3-d graphics libraries. [x] Least Squares (nonlinear): Nothing, but we both know that this is easily done by applying a function to your independent variable to linearize it in many cases. [-] Levenberg Marquardt [x] Matrix Inverse: Provided by SciD [?] pseudoinverses: I'm afraid my math background doesn't cover that, and I couldn't find anything with a similar naming in the package repository [x] Integration: SciD provides integration [-] Range Kutta [-] Interpolation: Sort of. A few of the graphics libraries have this, but it's asking a bit much to include a graphics library to do interpolation. [-] Bsplines [-] fft convoles [?] Multidimensional images [?] KDTrees: Probably in one of the 3d graphics libs [+] Symbolic equation solvers: Some of this was done for Tango back in the day, and it has been ported, but the project looks fairly dead. https://github.com/opticron/libdmathexpr/blob/master/source/libdmathexpr/mathexpr.d https://github.com/opticron/libdmathexpr/blob/master/source/... [x] Merge/join of data sets: Can be done efficiently and easily with core language features Additionally, anything written for C can be trivially wrapped and used in D. An example is http://scimath.com/ http://scimath.com/, which completes much of what you mentioned. Are they in standard libraries? SciD needs more time Fully Documented? D tends to encourage good API documentation, and the algorithms are largely ports of "69 year old, fully debugged code". I think you're point about "reliably email to anyone across the world" is an overstatement. Python is popular, not ubiquitous. D is obviously much less popular. Can anyone run the code? Yes. For them to modify it, they need to learn the language. You might not care about how long computations take, but I know back at my old university there were dozens of researchers complaining about the resources that had to be spent on supercomputer time, and the annoying amount of time that many calculations involving things like molecular configurations can take. Speeding this up saves money and makes the research less painful. "No one is ever going to use D for serious number crunching without the infrastructure in place" - Yeah, that's totally true. The infrastructure is a WIP. There's nothing wrong with that. Your post comes across to me as being rather cynical - but why not be supportive of the good work that's being done?
9il 11y ago
Yes, Julia is amazing! In the same time, if you want to write a package for Julia you _may_ need to use C/C++. D is going to have integration with Julia in 2016 ;) D already have good integration with Python. You may want to read this article http://d.readthedocs.org/en/latest/examples.html#plotting-with-matplotlib-python http://d.readthedocs.org/en/latest/examples.html#plotting-wi... (it may be a little bit outdated).
- tadlan 11y ago
  What would one need to use c or c++ to write a julia package? It's as fast as native code so no need to use multiple languages.
  - 9il 11y ago
    Julia is really fast in 95% cases, but 5% still "make the weather". The pairwise summation is an example. I will post benchmarks D vs Julia next week ;)
    - skariel1 11y ago
      I would say the biggest benefit of D is static typing. In Julia you can run a simulation and discover only after half an hour that you misspelled a function
    - tadlan 11y ago
      Make sure you use the devect macro or parallel accelerator if you're going to use vector iced code :)

by RogerL 11y ago

vidarh 11y ago

> Does D have code for: [...] He covered this, specifically out that Numpy has an advantage in the number of libraries and other resources available for it. > because everyone else in the world knows and uses them; No, everyone working in some specialized niches does. And those niches don't even scratch the surface of the number of people who regularly do simple operations on multi-dimensional arrays but who don't need most of what you list. And sometimes people even in your niche that does depend on the other stuff run into things that are just way too slow, that they have to rewrite in a lower level language. There are plenty of people for whom this will be very useful without having all the support infrastructure around Numpy in place. > I can't think of any reason I'd ever reach for D. D is for people who wants a "C++ done right", where it is implicit that you want a C-like language, and share ideals of C++ such as to whatever extent possible not paying for what you don't use. Specifically in this case, you'd reach for D if you need multi-dimensional arrays and either prefer a C++-like language over Python or your Python code is too slow and you prefer to avoid a lower level language like C. That may not be for you, and that's fine. Just like Numpy isn't nearly universal for numerical computing. > Yes, my np.mean() might be slower than your map(). I almost always don't care. I find this an amusing objection... I did my MSc thesis on methods for reducing error rates in OCR, and included methods that used various nearest-neighbour variations and statistical methods on bitmap slices and similar. I did it mainly in Ruby (with some inline C where absolutely necessary for performance - if I remember correctly it was about a dozen lines of code in C) because I cared far more about readable code than fast code, and Python would in my eyes have been a double loss: not the speed of C, yet still ugly and annoying to work with in my opinion. Seeing the D code examples, I might very well have picked D for it today.

n00b101 11y ago

>Are those all in standard libraries, fully documented, backed by 60 year old, fully debugge code (LAPACK, etc) ... yes, my np.mean() might be slower than your map(). I almost always don't care. Hmm ... LAPACK itself is only 23 years old (1992). Its was implemented to be a faster replacement of the "standard" LINPACK library, by effectively exploiting the caches on modern processors. For most serious users of numerical computing, performance is extremely important. Also, I think you might be putting far too much faith in an arbitrary assortment of Python libraries. If you think that NumPy is just a thin wrapper on LAPACK, and you are using it in mission or life critical operations, then I think you have some code vetting to do. Even if some of your chosen libraries do use LAPACK, and even if there is some code in LAPACK that is 60 years old, that does not suddenly mean that all of your chosen Python libraries (and your code on top of them) is all correct. Most serious users of many of the routines that you mention (e.g. "machine learning" and "financial calculations") implement and test these routines themselves in order to ensure their correctness. In very serious model vetting operations, it would be customary to even invest in two different independent implementations of the same routine. > I reliably email to anyone across the world and they can immediately run and modify my code because it is such a standard? I don't think "reliable" and "emailing scripts" normally go together. It would be more customary to use a source code control system in a high reliability environment. Personally I have found reproducing Python installations to be very tricky and I would take extreme care in any critical operations that assume two different Python installations produce the exactly the same results (high reliability, like some of the 60+year old codes run by NASA, would even require bug for bug reproducibility). "Python/Numpy/Scipy/Pandas/Matplotlib" are not the last word on anything, theyt are not standardized as you imply, and they leave vast room for improvement in all dimensions (performance, correctness, reliability, productivity, etc).

dallbee 11y ago

A lot of that is up and coming. https://github.com/DlangScience https://github.com/DlangScience [x] Plotting: http://code.dlang.org/packages/plot2kill http://code.dlang.org/packages/plot2kill [+] Optimization: Nothing directly for the purpose, but the intermediate steps are done. And a little bit of work from http://code.dlang.org/packages/atmosphere http://code.dlang.org/packages/atmosphere [x] Probability Distributions: http://dlangscience.github.io/dstats/api/dstats/random.html http://dlangscience.github.io/dstats/api/dstats/random.html and http://dlang.org/phobos/std_mathspecial.html http://dlang.org/phobos/std_mathspecial.html [-] Machine Learning: Not that I'm aware of, but I vaguely recall some work being done on this [-] Financial calculations: I couldn't find anything, but I'd be surprised if this wasn't implemented already. [x] Masked Arrays: Accomplished via language features in combination with ndslice [x] Structured Arrays: Absolutely. Standard library support as well as a number of 3rd-party libs can do this. [-] SVD [?] QR: Afraid I'm not sure what you're referring to. [+] Cholesky Decomposition: Trivial to implement [x] Eigenvalues/Eigenvectors: Yes. SciD provides Eigens as well as two other third party packages. https://dlangscience.github.io/scid/api/scid/linalg/eigenvalues.html https://dlangscience.github.io/scid/api/scid/linalg/eigenval... [x] Least Squares (linear): Trivial to implement, but also implemented by a few of the 2-d & 3-d graphics libraries. [x] Least Squares (nonlinear): Nothing, but we both know that this is easily done by applying a function to your independent variable to linearize it in many cases. [-] Levenberg Marquardt [x] Matrix Inverse: Provided by SciD [?] pseudoinverses: I'm afraid my math background doesn't cover that, and I couldn't find anything with a similar naming in the package repository [x] Integration: SciD provides integration [-] Range Kutta [-] Interpolation: Sort of. A few of the graphics libraries have this, but it's asking a bit much to include a graphics library to do interpolation. [-] Bsplines [-] fft convoles [?] Multidimensional images [?] KDTrees: Probably in one of the 3d graphics libs [+] Symbolic equation solvers: Some of this was done for Tango back in the day, and it has been ported, but the project looks fairly dead. https://github.com/opticron/libdmathexpr/blob/master/source/libdmathexpr/mathexpr.d https://github.com/opticron/libdmathexpr/blob/master/source/... [x] Merge/join of data sets: Can be done efficiently and easily with core language features Additionally, anything written for C can be trivially wrapped and used in D. An example is http://scimath.com/ http://scimath.com/, which completes much of what you mentioned. Are they in standard libraries? SciD needs more time Fully Documented? D tends to encourage good API documentation, and the algorithms are largely ports of "69 year old, fully debugged code". I think you're point about "reliably email to anyone across the world" is an overstatement. Python is popular, not ubiquitous. D is obviously much less popular. Can anyone run the code? Yes. For them to modify it, they need to learn the language. You might not care about how long computations take, but I know back at my old university there were dozens of researchers complaining about the resources that had to be spent on supercomputer time, and the annoying amount of time that many calculations involving things like molecular configurations can take. Speeding this up saves money and makes the research less painful. "No one is ever going to use D for serious number crunching without the infrastructure in place" - Yeah, that's totally true. The infrastructure is a WIP. There's nothing wrong with that. Your post comes across to me as being rather cynical - but why not be supportive of the good work that's being done?

9il 11y ago

Yes, Julia is amazing! In the same time, if you want to write a package for Julia you _may_ need to use C/C++. D is going to have integration with Julia in 2016 ;) D already have good integration with Python. You may want to read this article http://d.readthedocs.org/en/latest/examples.html#plotting-with-matplotlib-python http://d.readthedocs.org/en/latest/examples.html#plotting-wi... (it may be a little bit outdated).

tadlan 11y ago
What would one need to use c or c++ to write a julia package? It's as fast as native code so no need to use multiple languages.
- 9il 11y ago
  Julia is really fast in 95% cases, but 5% still "make the weather". The pairwise summation is an example. I will post benchmarks D vs Julia next week ;)
  - skariel1 11y ago
    I would say the biggest benefit of D is static typing. In Julia you can run a simulation and discover only after half an hour that you misspelled a function
  - tadlan 11y ago
    Make sure you use the devect macro or parallel accelerator if you're going to use vector iced code :)