4 ms·

Pandas is usable? I had no idea.. Pandas is really badly designed, in the same way that most Python libraries are: each function has so many parameters. And a

by smabie 7y ago

Pandas is usable? I had no idea.. Pandas is really badly designed, in the same way that most Python libraries are: each function has so many parameters. And a parameter can often be a bunch of different types. Pandas is useful, especially for time-series data, but no one particularly loves it. And, it’s embarrassingly slow. Maybe PandaPy is better, but I doubt it. When you start trying to use Python implemented functions (vs C ones) things are going to get bad no matter what you do. Speaking of which, I decided to port over a statistical model for betting from Python to Julia week ago. I’m not done yet, and this is my first major experience with Julia, but it’s been so much nicer than using Python. The performance can easily be 10x-50x faster without really doing any extra work. Also the language feels explicitly designed for scientific computing and really meshes well with the domain. Python the language never really was good for this, but the libraries were pretty compelling. Julia libraries have almost caught up (or in some domains, like linear algebra) have actually exceeded what’s available dor Python. Moreover, if you need to, PyCall is really easy to use. I’m going to go out on a limb and say that people shouldn’t be using Python for new scientific computing projects. Julia has arrived, and is better in everyway (I’m still unsure about the 1-based indexing, but I’m sure I’ll get over it. 0-based waa never that great in the first place).

beowulfey 7y ago
HN hasn’t really taken to Julia yet, but I am also a big fan. I’m not even a programmer really, just had some python experience, and I was able to hack together a program that extracted molecular contact data from protein structures and draw contact maps for them all. It runs in a few minutes for 10000 structures. I did zero optimization on it! The language is incredibly fast and I really think it’s easy to use.
- ChrisRackauckas 7y ago
  I think it's partly just the audience. HN seems to have a lot of professional programmers, which is similar to but not quite the same as computational scientists. That doesn't mean there's no overlap, but for example I haven't met a people (other than one Google person) at a recent physics-informed learning or scientific machine learning conference who is a HN reader, which really explained why none of the stories in those domains have been getting HN traction even though HN seems very ML/AI-positive. Julia is in this weird middle ground where its ML/AI stories seem to get a lot of HN traction as a programming languages topic given the HN audience, but in reality it has a lot more people doing things like research in numerical linear algebra, numerical differential equations, statistics, bioinformatics, etc. but fast methods for block-banded matrices won't hit science press.
playing_colours 7y ago
Ha, I see Julia Evangelism Task Force is grouping up in another thread! I am waiting for the arrival of Commander socialdemocrat soon :) I am a big fan of Julia as well, and it is quite amusing to see recent growing presence of enthusiasts roaming between topics.
unishark 7y ago
I also found the "speed of numpy" part of the title amusing. Reminded me of a quote that went something like "C has the low-level power of an assembly language combined with the high-level usability of an assembly language."
woah 7y ago
How’s the package management story on Julia ? Python package management is a fractal of badness
- smabie 7y ago
  I’m not sure I just do using Pkg; Pkd.add(“foo”) And it works. I’m sure there’s some dependency file for projects but I don’t know how it works. Thinking about build tools and package management makes me sad, so I try and avoid it as long as possible.
- ddragon 7y ago
  Usually pretty good. The package management is completely centered around Pkg.jl, which is integrated in the REPL and you can also import it to your program for more advanced scripting. If you don't create an environment, everything is added to the global user library, and if you do create it, it will automatically manage your projects dependency files, and each environment/package can have it's own independent versions of each library (so you don't really have dependency hell issues, but you might end up with more disk space due to multiple versions of the same library, although it will respect semver when keeping multiple libraries). Pkg.jl is based on git/github with a central registry that can I believe it's automatically updated with new packages using bots. The current version also has native supports for automatically deploying binaries and other stuff like datasets, that can be optionally loaded on demand. Most troubles I hear is with more strict enterprise firewall scenarios and perhaps Julia JIT making it compile the libraries every time (though that's not an issue with the package manager).
julosflb 7y ago
Me.I particularly love pandas. Sure its API is quite complex and I need to refer to the documentation more often than I would like. But the doc. is be pretty solid. And the set of features unmatched in the python ecosystem. Nothing can really beats pandas when you need to quickly do some medium sized (ie. in RAM) data exploration in python.

by smabie 7y ago

beowulfey 7y ago

HN hasn’t really taken to Julia yet, but I am also a big fan. I’m not even a programmer really, just had some python experience, and I was able to hack together a program that extracted molecular contact data from protein structures and draw contact maps for them all. It runs in a few minutes for 10000 structures. I did zero optimization on it! The language is incredibly fast and I really think it’s easy to use.

ChrisRackauckas 7y ago
I think it's partly just the audience. HN seems to have a lot of professional programmers, which is similar to but not quite the same as computational scientists. That doesn't mean there's no overlap, but for example I haven't met a people (other than one Google person) at a recent physics-informed learning or scientific machine learning conference who is a HN reader, which really explained why none of the stories in those domains have been getting HN traction even though HN seems very ML/AI-positive. Julia is in this weird middle ground where its ML/AI stories seem to get a lot of HN traction as a programming languages topic given the HN audience, but in reality it has a lot more people doing things like research in numerical linear algebra, numerical differential equations, statistics, bioinformatics, etc. but fast methods for block-banded matrices won't hit science press.

playing_colours 7y ago

Ha, I see Julia Evangelism Task Force is grouping up in another thread! I am waiting for the arrival of Commander socialdemocrat soon :) I am a big fan of Julia as well, and it is quite amusing to see recent growing presence of enthusiasts roaming between topics.

unishark 7y ago

I also found the "speed of numpy" part of the title amusing. Reminded me of a quote that went something like "C has the low-level power of an assembly language combined with the high-level usability of an assembly language."

woah 7y ago

How’s the package management story on Julia ? Python package management is a fractal of badness

smabie 7y ago
I’m not sure I just do using Pkg; Pkd.add(“foo”) And it works. I’m sure there’s some dependency file for projects but I don’t know how it works. Thinking about build tools and package management makes me sad, so I try and avoid it as long as possible.
ddragon 7y ago
Usually pretty good. The package management is completely centered around Pkg.jl, which is integrated in the REPL and you can also import it to your program for more advanced scripting. If you don't create an environment, everything is added to the global user library, and if you do create it, it will automatically manage your projects dependency files, and each environment/package can have it's own independent versions of each library (so you don't really have dependency hell issues, but you might end up with more disk space due to multiple versions of the same library, although it will respect semver when keeping multiple libraries). Pkg.jl is based on git/github with a central registry that can I believe it's automatically updated with new packages using bots. The current version also has native supports for automatically deploying binaries and other stuff like datasets, that can be optionally loaded on demand. Most troubles I hear is with more strict enterprise firewall scenarios and perhaps Julia JIT making it compile the libraries every time (though that's not an issue with the package manager).

julosflb 7y ago

Me.I particularly love pandas. Sure its API is quite complex and I need to refer to the documentation more often than I would like. But the doc. is be pretty solid. And the set of features unmatched in the python ecosystem. Nothing can really beats pandas when you need to quickly do some medium sized (ie. in RAM) data exploration in python.