4 ms·

> I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) Please do! I'm a PhD student in CS, and I d

• kaeluka 10y ago
  +1, I'm a PhD student in programming languages/PL design
• enriquto 10y ago
  You want to process (typically, with simple arithmetical operations) huge arrays of numbers. Imagine that you want to sum two arrays of 800 MB of floating point numbers each. This is one step inside the loop of your algorithm. You can do that natively in C, and the result is very fast. You can do that natively in Fortran. And in Matlab, etc. You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower.
  • kevin_thibedeau 10y ago
    > You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower. NumPy + Cython would beg to differ.
    • enriquto 10y ago
      Of course, but not natively. Try doing that in plain python using a list of 8000 lists of 8000 lists of 3 numbers (e.g. a mid-resolution microscopy image).
    • dagss 10y ago
      NumPy is about one order of magnitude slower in very many situations. Cython is in some ways a reimplementation of Fortran in places -- I mean, you can consider it a better alternative but have to use features that are not in Python. Personally I gave up writing numerical code in Cython and instead wrote it in Fortran and merely wrapped it with Cython... But yes, being 2x-10x slower is something one can live with for productivity, vs sometimes 1000x slower of Python.
• alcidesfonseca 10y ago
  I am a PhD in CS, specifically in Programming Languages and Parallel Programming and I believe I do get scientific computing. Most people doing it did not have a formal CS education. They are biology, physics, mathematics or chemistry majors that have had one or two courses on programming, from other scientific programmers. There are two main families, one that comes from the Fortran background, which still writes programs like they did in the 80s, with almost no new tooling. Programs are written for some time, and then they are scheduled for clusters that spend months calculating whatever it is. The other family of scientific programmers, which I believe is the majority, uses a tool like Matlab, or more recently R, to dynamically inspect and modify data (RStudio is a Matlab/Mathematica-like friendly environment for this task) and use libraries written by more proficient programmers to perform some kind of analysis (either machine learning, DNA segmentation, plotting or just basic statistics). Most of these programmers know 1 or 2 languages (maybe plus python and bash for basic scripting). They write programs that are relatively small and the chances of someone else using that code is low. Thus, the deadline pressure is high and code maintainability is not a priority. For a non-CS programmer, learning a new programming language is almost impossible, because they are used to that way of doing things, and those libraries. They take much more time to adjust to new languages because they do not see the language logically, like anyone who had a basic compiler course. Given this context, web apps, rest APIs and all the other trending tech in IT are not commonly used in scientific programming, because they typically do not need it (when they do, they learn it). Datasets are retrieved and stored in CSV and processed in one of those environments (or even in julia or python-pandas).
  • matt4077 10y ago
    You're painting an awfully dark picture of scientists' skills. Having been on both sides, I believe the deciding factor is simply the availability of libraries. If you're doing web development you have an insane amount of languages to chose from because after String, Array, and File are implemented, HTTP is next. Having done a bit of web development, I'd also say a typical project only uses a subset of libraries that is surprisingly small. Scientific computing is quite different: a paper in structural biology (my former stomping grounds) can easily require a few dozen algorithms that each once filled a 10-page paper. These could easily be packaged as libraries, but it's a niche so it rarely happens. Newer language quite often don't even have a robust numeric library. Leave the beaten tracks and your workload just increased by a magnitude. That's also why science, unlike "general purpose" programming, often uses a workflow that connects five or more languages or so: a java GUI, python for network/string/fileIO, maybe R for larger computations, all held together by a (typically too long) shell script. But these workflows are getting better. There's a build tool that formalizes the pipeline somewhat (I forgot the name) and APIs are surprisingly common. The reason why csv will never die is that the data fetched from APIs is usually more static than it is in a typical web app (-> local cache needed) and that scientists often work with data that just isn't a good fit for a database. Postgres just doesn't offer anything that enriches a 15MB gene sequence.