4 ms·

I've been doing a whole lot of scientific Python lately, and every time I let my program run for like an hour only to crash on an array of the wrong dimensions,

I've been doing a whole lot of scientific Python lately, and every time I let my program run for like an hour only to crash on an array of the wrong dimensions, or pass in a scalar where an array is expected or vice versa, I swear and wish I was using a strongly typed language like Haskell. If only it had the same tools as SciPy I'd be all over doing my scientific work in Haskell. (Not to mention it would have faster run times.) (There are a few array libraries, but I'm pretty sure nothing as complete as NumPy+SciPy+matplotlib exists for Haskell..)

aaronblohowiak 15y ago
save more intermediate values to disk so resuming upon failure is easier.
- TheEzEzz 15y ago
  It's painful even when it only takes a single time step to reach the error (sometimes upwards of a minute or more). It's as if I've gone from an interpreted language with no compile time to a compiled language with horrendous compile times.
nightski 15y ago
I agree, and this is something I am investigating also. Data Parallel Haskell is an awesome base for something like this, but it is hard to tell if anyone is actually still working on it - and even harder to tell how one would contribute. Especially since there are so many spin off projects like Repa which have some of the functionality but just don't quite make the cut (lack primitives, hard to implement new parallel operations, etc...). The potential is there, it just needs to be exploited.
eeperson 15y ago
It's not Haskell but its closer than Python. Have you seen Scalala for Scala: https://github.com/scalala/Scalala https://github.com/scalala/Scalala
kisielk 15y ago
Sounds to me like you're doing it wrong. I write scientific Python code every single day for work and I (nor anyone else who works with me) never run in to those kinds of problems. Why are you running simulations that take an hour before you even know your code work? Where are your unit and integration tests? Every piece of our software stack has (or should have...) tests that verify all the bits fit together correctly in a matter of seconds or perhaps minutes tops. Strong typing would be nice to have sometimes, but in practice I think it limits the amount of ad-hoc data analysis and exploration you can do while trying to piece together a piece of software. I think the scientists in our company would murder the software engineering team if they had to deal with strong typing to get things done.
- gaius 15y ago
  The point is that a whole class of things that you would have to write unit tests for, you get "for free" with strong typing, and for a lot of the others, there's QuickCheck. Or as the old saying goes, why get a dog and bark yourself?
  - vidarh 15y ago
    Except you don't get them "for free". You get them in the form a ton of restrictions on you that you need to be mindful of when writing it in the first place. You just pay for it in different ways. The biggest difference is that in dynamic languages, you can sometimes get away with being lazy and not writing the tests to verify the things static languages enforce to varying degrees (whether or not that's a good thing is another issue).
    - SkyMarshal 15y ago
      Haskell frontloads type considerations and tests via a required, tightly-coupled type system. Dynamic languages backload them via optional testing. In either case you still have to think about those issues when correctness is a requirement. When it isn't (ad-hoc analysis, prototyping), dynamic languages are more pleasant to work with. When it is a requirement, Haskell is uniquely powerful. Problem is, many projects are subject to both requirements at the different times. Exploratory at the start, strict once the problem to be solved is identified. Know the tradeoffs and pick your poison.
cscheid 15y ago
I think strong type systems for arrays are very much an open problem (as soon as you stick integers in there you go into dependent types land), so for dimension matching you'd probably end up with runtime type errors even in Haskell.

aaronblohowiak 15y ago

save more intermediate values to disk so resuming upon failure is easier.

TheEzEzz 15y ago
It's painful even when it only takes a single time step to reach the error (sometimes upwards of a minute or more). It's as if I've gone from an interpreted language with no compile time to a compiled language with horrendous compile times.

nightski 15y ago

I agree, and this is something I am investigating also. Data Parallel Haskell is an awesome base for something like this, but it is hard to tell if anyone is actually still working on it - and even harder to tell how one would contribute. Especially since there are so many spin off projects like Repa which have some of the functionality but just don't quite make the cut (lack primitives, hard to implement new parallel operations, etc...). The potential is there, it just needs to be exploited.

eeperson 15y ago

It's not Haskell but its closer than Python. Have you seen Scalala for Scala: https://github.com/scalala/Scalala https://github.com/scalala/Scalala

kisielk 15y ago

Sounds to me like you're doing it wrong. I write scientific Python code every single day for work and I (nor anyone else who works with me) never run in to those kinds of problems. Why are you running simulations that take an hour before you even know your code work? Where are your unit and integration tests? Every piece of our software stack has (or should have...) tests that verify all the bits fit together correctly in a matter of seconds or perhaps minutes tops. Strong typing would be nice to have sometimes, but in practice I think it limits the amount of ad-hoc data analysis and exploration you can do while trying to piece together a piece of software. I think the scientists in our company would murder the software engineering team if they had to deal with strong typing to get things done.

gaius 15y ago
The point is that a whole class of things that you would have to write unit tests for, you get "for free" with strong typing, and for a lot of the others, there's QuickCheck. Or as the old saying goes, why get a dog and bark yourself?
- vidarh 15y ago
  Except you don't get them "for free". You get them in the form a ton of restrictions on you that you need to be mindful of when writing it in the first place. You just pay for it in different ways. The biggest difference is that in dynamic languages, you can sometimes get away with being lazy and not writing the tests to verify the things static languages enforce to varying degrees (whether or not that's a good thing is another issue).
  - SkyMarshal 15y ago
    Haskell frontloads type considerations and tests via a required, tightly-coupled type system. Dynamic languages backload them via optional testing. In either case you still have to think about those issues when correctness is a requirement. When it isn't (ad-hoc analysis, prototyping), dynamic languages are more pleasant to work with. When it is a requirement, Haskell is uniquely powerful. Problem is, many projects are subject to both requirements at the different times. Exploratory at the start, strict once the problem to be solved is identified. Know the tradeoffs and pick your poison.

cscheid 15y ago

I think strong type systems for arrays are very much an open problem (as soon as you stick integers in there you go into dependent types land), so for dimension matching you'd probably end up with runtime type errors even in Haskell.