6 ms·

If you know actual scientists, this isn't counter intuitive at all. My partner is a scientist, so now I know tons of them, and I have done a bunch of Python cod

If you know actual scientists, this isn't counter intuitive at all. My partner is a scientist, so now I know tons of them, and I have done a bunch of Python coding and support for scientists, have been a Python programmer (as well as other languages) since 2005-ish. I saw this coming (as did many) 15 years ago. Most scientists, and their grad students, are trying to do a whole bunch of things in their research, and programming is just one of them. Field work, experiments, data wrangling, writing papers, defending papers, teaching, etc. And most of them do not have access to budgets for programmers or when they do, it's for a limited amount of time and work, meaning they need to be able to pick up and run with whatever the programmer did. So the fact that with Python they and their grad students (who might be there for only 2 years) can be working productively, and figure out what the hell the code did when they come back to it months later, is HUGE. As in, literally blows every other consideration to smithereens. This has meant that over the last 20 years the scientific libraries in Python got mature faster than in any other language, and this in turn has had a snowball effect. And when speed is necessary, C++ extensions can be written. But honestly, most of the time speed is not the main factor. The downside of Python in my experience is that junior teams can make heinous atrocities when a project gets really big (I have had to step in as CTO to one of those messes, so much as I love Python, I must admit this is true!) But the stuff the scientists are doing is very rarely that big. It's tools programming, scripting, making utilities, data analysis and so on. Readability counts. In some fields, it counts more than anything. I've worked in about 10 languages now over the last 20 years, and Python is still the easiest to read when you come back to some old code or have to pick up code for a small job, or hand it to a beginner to extend without having them create an unreadable mess. This is what scientists need to do all the time. Re other people's comments on Python packaging and setup being hard, well honestly I've had just as much pain with Ruby or Node. The shining exception there is R, which is giving Python a run for its money in many scientific areas. R Studio has the best "hit the ground running" experience out there and is really slick for data programming.

analog31 5y ago
My partner's partner is a scientist too. ;-) In addition to not having budgets for programmers, we also don't know how to manage them, for instance how to communicate our needs, decide if their implementation plans make sense, or gauge their progress. Nearly half a century after The Mythical Man Month, managing software development is still generally acknowledged to be an unsolved problem. The other two obstacles are that most programmers hate the scientific work environment, with its ever-changing requirements and frequent dead ends. And, the programmers who can work on math related stuff are in the highest demand.
- iainctduncan 5y ago
  Spot on with my experience! Much of our work was helping them manage the project and figure out how to work with us. And someone went on sabbatical, and then someone dropped their program, and someone else left for another school, and someone was stuck managing the program for a semester who had literally no time or experience doing that, etc. It's a Dynamic Environment. lol. There is no other language I have used that makes it as easy to read code from somebody else, especially where that contributor is likely to be a domain expert with very limited programming experience. It's not actually my favourite language anymore (hello Scheme!) but if you want me to do work in that environment, I'll reach for Python first.
- protomolecule 5y ago
  "the programmers who can work on math related stuff are in the highest demand." Could you suggest the best way for finding a remote job for such programmer?
  - iainctduncan 5y ago
    Network with scientists. Doing some small jobs or favours for scientists who will tell other scientists about you is the way to go. Universities are a good source of connections.
    - Icathian 5y ago
      If labs are really struggling to find math-literate programmers, I would imagine it's in part because the process for matching them with the work is so terrible. Generally speaking, skilled programmers do not want to (and certainly don't have to) shake hands and do favors to find work. I wonder if there's any concerted effort to fix that for academia, or if the "shortage" of math-literate programmers just isn't a problem worth fixing.
      - iainctduncan 5y ago
        That was kind of my point. What skilled programmers do to find work is one thing, what scientists, who just need some help for a short project, do to find people is another. Assuming you are a programmer who wants to do work for scientists for some reason, you need to go where they are - they won't find you in your regular tech recruiting circles, which tend to be all about full time jobs. I happen to like doing some work for scientists so that my career isn't entirely about making private equity companies richer, but I don't expect them to pay my enterprise rates or find me on Linked In.
zozbot234 5y ago
> Readability counts. In some fields, it counts more than anything. I've worked in about 10 languages now over the last 20 years, and Python is still the easiest to read when you come back to some old code or have to pick up code for a small job, or hand it to a beginner to extend without having them create an unreadable mess. This is what scientists need to do all the time. Meh. Python might be readable at the smallest scale, but then COBOL is even more readable. What matters is large-scale development, and your implied point that large Python projects turn into unstructured big-ball-of-mud monstrosities is well taken. A big ball of mud is not surveyable, or "readable". Which is where other modern languages (e.g. Julia in the scientific programming domain, heck even Go or Rust) will probably have an advantage.
- iainctduncan 5y ago
  think what you will, the scientists disagree. Which was the point. Not holding my breath to find many scientists matching my description who would rather learn Go or Rust...
  - shpongled 5y ago
    I'll be the counter example. PhD in life sciences, but I've also been programming since I was a teen. Rust is by far my most used language for both general fun projects and in my role as a programmer in the life-sciences. Python is OK for ad-hoc analyses, but I cannot stand to use dynamically typed languages for anything "real" given how much difficulty dynamic typing imposes on reading and understanding code.
    - iainctduncan 5y ago
      Sure, but by your description, you aren't really the people I'm describing. If you've been doing this since you were a teen, you're a "Real Programmer". My point was that people who have to do this as item 7 of 10 things in their job description are very much less likely to learn something like Rust than Python. That is undeniably a bigger lift to a non-programmer. Python's success in the sciences is in large part due to how good a fit as a language it is for part-time occasional programmers. I like all kinds of languages, but the only ones I would encourage my partner to bother with as tools for her science work would be R and Python.
ip26 5y ago
Not to mention, what they are working on is often very abstract compared to the math many programmers are used to doing. I write a lot of boolean; my scientist partner writes regressions, surface transformations, eigenmaps, linear algebra, and so on. Imagine being something other than a programmer by trade, and trying to apply linear algebra to your problem without good tools or libraries.
liamwestray 5y ago
Literally can't add anything to this. You nailed it. I suspect MicroPython is going to do the same thing to Arduino/C as Python just did here in Academia as well.
- rleigh 5y ago
  I'm afraid I don't agree. MicroPython is neat, but if there's one thing Python is not suited for, it's microcontrollers. Coupling one of the slowest scripting languages with low latency near-realtime requirements is not a recipe for success. It might be useful for teaching basic concepts, but it is not going to be useful for real applications. And Arduino already has the teaching of basic concepts nailed very effectively. I certainly think that MicroPython serves a niche, primarily very simple hobbyist/educational roles. However, I do not regard it as suitable for anything beyond this. It's the wrong tool for the job, and if you want a scripting language for low-latency low-overhead use, there are smaller and more efficient languages which fit better into an embedded role.
  - liamwestray 5y ago
    You’re an engineer. And engineering programs will continue to use engineering focused tools. The goals of engineering program projects are vastly different than other areas of academia. Developing the tool is the project for engineering. Bioscience / environmental science programs will find micropython good enough for their needs. The tool itself is just the means to the end if real science. Micropython let’s you deploy in lower power applications without having to learn much beyond what you already know from Jupyter notebooks. I really don’t know any PhD students or post-docs in microbiology/environmental sciences who have the time to learn embedded C or similar languages.
blunte 5y ago
> Python is still the easiest to read when you come back to some old code Lucky you. You must not have seen the "pythonic" monstrosities I've seen. Python has such a low barrier for entry that one can "get stuff done" with absolutely atrocious and often very overly complicated OOP-ish code. Ruby is not my favorite language, but I would bet real money that without dependence on libraries, nobody could show me Python code which I could not show more logical, consistent, and readable Ruby code which solves the same problem. I say Ruby because it's of the same "type" and follows similar methodologies. Python suffers from far too many years under the leadership of one odd person. It has a cult-like following, whereby anyone who disagrees is an outcast. Where else could you hear comments like, "why would you ever need a switch statement? if/if else works fine!" That's just the tip of the iceberg. Python is great for integration glue code, but only because of the libraries it has. But now it is becoming more Javascript like, and the dependencies are multiplying to the point where you're better off writing your own left-pad instead (or even re-evaluating your approach) instead of taking on new duct tape like django-database-view. Sometimes the bar needs to be high enough to force the juniors to actually learn something before they start building "MVP" startups. On the other hand, who cares if the MVP is a horror show as long as you get that IPO and take your f-u money and leave.
- iainctduncan 5y ago
  So my real job is technical due diligence on companies being purchased. I get the keys to the kingdom when we do a diligence and trust me, there are just as many people making unmaintainable monstrosities that get bogged down in tech debt in Ruby. Looking at this scenario is literally my job, and the company I work for does more of these than anyone in the world. Bad coders can make terrible stuff in any language, and with two as similar as Python and Ruby, the minor differences are a drop in the bucket in the grand scheme of things. Both Django-database code and RoR's Active Record have bogged down many a startup when they got big enough that DB size and query performance mattered. None of which, as I pointed out, is relevant to the vast majority of scientists writing code.
  - danielmarkbruce 5y ago
    In your experience, what are most buyers looking for when they get you to do technical DD? Is there a specific set of things they are worried about? Specifically looking to confirm, etc?

analog31 5y ago

My partner's partner is a scientist too. ;-) In addition to not having budgets for programmers, we also don't know how to manage them, for instance how to communicate our needs, decide if their implementation plans make sense, or gauge their progress. Nearly half a century after The Mythical Man Month, managing software development is still generally acknowledged to be an unsolved problem. The other two obstacles are that most programmers hate the scientific work environment, with its ever-changing requirements and frequent dead ends. And, the programmers who can work on math related stuff are in the highest demand.

iainctduncan 5y ago
Spot on with my experience! Much of our work was helping them manage the project and figure out how to work with us. And someone went on sabbatical, and then someone dropped their program, and someone else left for another school, and someone was stuck managing the program for a semester who had literally no time or experience doing that, etc. It's a Dynamic Environment. lol. There is no other language I have used that makes it as easy to read code from somebody else, especially where that contributor is likely to be a domain expert with very limited programming experience. It's not actually my favourite language anymore (hello Scheme!) but if you want me to do work in that environment, I'll reach for Python first.
protomolecule 5y ago
"the programmers who can work on math related stuff are in the highest demand." Could you suggest the best way for finding a remote job for such programmer?
- iainctduncan 5y ago
  Network with scientists. Doing some small jobs or favours for scientists who will tell other scientists about you is the way to go. Universities are a good source of connections.
  - Icathian 5y ago
    If labs are really struggling to find math-literate programmers, I would imagine it's in part because the process for matching them with the work is so terrible. Generally speaking, skilled programmers do not want to (and certainly don't have to) shake hands and do favors to find work. I wonder if there's any concerted effort to fix that for academia, or if the "shortage" of math-literate programmers just isn't a problem worth fixing.
    - iainctduncan 5y ago
      That was kind of my point. What skilled programmers do to find work is one thing, what scientists, who just need some help for a short project, do to find people is another. Assuming you are a programmer who wants to do work for scientists for some reason, you need to go where they are - they won't find you in your regular tech recruiting circles, which tend to be all about full time jobs. I happen to like doing some work for scientists so that my career isn't entirely about making private equity companies richer, but I don't expect them to pay my enterprise rates or find me on Linked In.

zozbot234 5y ago

> Readability counts. In some fields, it counts more than anything. I've worked in about 10 languages now over the last 20 years, and Python is still the easiest to read when you come back to some old code or have to pick up code for a small job, or hand it to a beginner to extend without having them create an unreadable mess. This is what scientists need to do all the time. Meh. Python might be readable at the smallest scale, but then COBOL is even more readable. What matters is large-scale development, and your implied point that large Python projects turn into unstructured big-ball-of-mud monstrosities is well taken. A big ball of mud is not surveyable, or "readable". Which is where other modern languages (e.g. Julia in the scientific programming domain, heck even Go or Rust) will probably have an advantage.

iainctduncan 5y ago
think what you will, the scientists disagree. Which was the point. Not holding my breath to find many scientists matching my description who would rather learn Go or Rust...
- shpongled 5y ago
  I'll be the counter example. PhD in life sciences, but I've also been programming since I was a teen. Rust is by far my most used language for both general fun projects and in my role as a programmer in the life-sciences. Python is OK for ad-hoc analyses, but I cannot stand to use dynamically typed languages for anything "real" given how much difficulty dynamic typing imposes on reading and understanding code.
  - iainctduncan 5y ago
    Sure, but by your description, you aren't really the people I'm describing. If you've been doing this since you were a teen, you're a "Real Programmer". My point was that people who have to do this as item 7 of 10 things in their job description are very much less likely to learn something like Rust than Python. That is undeniably a bigger lift to a non-programmer. Python's success in the sciences is in large part due to how good a fit as a language it is for part-time occasional programmers. I like all kinds of languages, but the only ones I would encourage my partner to bother with as tools for her science work would be R and Python.

ip26 5y ago

Not to mention, what they are working on is often very abstract compared to the math many programmers are used to doing. I write a lot of boolean; my scientist partner writes regressions, surface transformations, eigenmaps, linear algebra, and so on. Imagine being something other than a programmer by trade, and trying to apply linear algebra to your problem without good tools or libraries.

liamwestray 5y ago

Literally can't add anything to this. You nailed it. I suspect MicroPython is going to do the same thing to Arduino/C as Python just did here in Academia as well.

rleigh 5y ago
I'm afraid I don't agree. MicroPython is neat, but if there's one thing Python is not suited for, it's microcontrollers. Coupling one of the slowest scripting languages with low latency near-realtime requirements is not a recipe for success. It might be useful for teaching basic concepts, but it is not going to be useful for real applications. And Arduino already has the teaching of basic concepts nailed very effectively. I certainly think that MicroPython serves a niche, primarily very simple hobbyist/educational roles. However, I do not regard it as suitable for anything beyond this. It's the wrong tool for the job, and if you want a scripting language for low-latency low-overhead use, there are smaller and more efficient languages which fit better into an embedded role.
- liamwestray 5y ago
  You’re an engineer. And engineering programs will continue to use engineering focused tools. The goals of engineering program projects are vastly different than other areas of academia. Developing the tool is the project for engineering. Bioscience / environmental science programs will find micropython good enough for their needs. The tool itself is just the means to the end if real science. Micropython let’s you deploy in lower power applications without having to learn much beyond what you already know from Jupyter notebooks. I really don’t know any PhD students or post-docs in microbiology/environmental sciences who have the time to learn embedded C or similar languages.

blunte 5y ago

> Python is still the easiest to read when you come back to some old code Lucky you. You must not have seen the "pythonic" monstrosities I've seen. Python has such a low barrier for entry that one can "get stuff done" with absolutely atrocious and often very overly complicated OOP-ish code. Ruby is not my favorite language, but I would bet real money that without dependence on libraries, nobody could show me Python code which I could not show more logical, consistent, and readable Ruby code which solves the same problem. I say Ruby because it's of the same "type" and follows similar methodologies. Python suffers from far too many years under the leadership of one odd person. It has a cult-like following, whereby anyone who disagrees is an outcast. Where else could you hear comments like, "why would you ever need a switch statement? if/if else works fine!" That's just the tip of the iceberg. Python is great for integration glue code, but only because of the libraries it has. But now it is becoming more Javascript like, and the dependencies are multiplying to the point where you're better off writing your own left-pad instead (or even re-evaluating your approach) instead of taking on new duct tape like django-database-view. Sometimes the bar needs to be high enough to force the juniors to actually learn something before they start building "MVP" startups. On the other hand, who cares if the MVP is a horror show as long as you get that IPO and take your f-u money and leave.

iainctduncan 5y ago
So my real job is technical due diligence on companies being purchased. I get the keys to the kingdom when we do a diligence and trust me, there are just as many people making unmaintainable monstrosities that get bogged down in tech debt in Ruby. Looking at this scenario is literally my job, and the company I work for does more of these than anyone in the world. Bad coders can make terrible stuff in any language, and with two as similar as Python and Ruby, the minor differences are a drop in the bucket in the grand scheme of things. Both Django-database code and RoR's Active Record have bogged down many a startup when they got big enough that DB size and query performance mattered. None of which, as I pointed out, is relevant to the vast majority of scientists writing code.
- danielmarkbruce 5y ago
  In your experience, what are most buyers looking for when they get you to do technical DD? Is there a specific set of things they are worried about? Specifically looking to confirm, etc?