7 ms·

I was a Python dev for 14 years or so (now doing mostly personal projects in C/C++, Scheme, Clojure), and 10 years ago met my partner who is a biologist. I thin

I was a Python dev for 14 years or so (now doing mostly personal projects in C/C++, Scheme, Clojure), and 10 years ago met my partner who is a biologist. I think one thing programmers underestimate is how many scientists and academics are using Python. It's not just ML, it's pretty much every kind of scientist and tons of non-science academics too. The fact that it's a great "casual" or part-time language is huge for these people. They aren't full time (or even self-identifying) programmers, but their programs are non-trivial. The readability, portability, tooling, and massive package ecosystem is huge for them. In that world, being somewhat comfortable in R and Python is becoming table stakes for really large swathes of academia. Personally I believe this is exactly why Python is dominant in ML now. We are harvesting after 20 years of academics happily writing libraries and tools for academic work in Python. :-) So I have no idea how the article got their numbers, but I'm not at all surprised to see the sheer number of people programming in Python exploding.

wainstead 6y ago
As to how they get the numbers, this is TIOBE's methodology: [1] Since there are many questions about the way the TIOBE index is assembled, a special page is devoted to its definition. Basically the calculation comes down to counting hits for the search query +"<language> programming" In the next few sections it is explained what search engines qualify, what programming languages qualify and how the ratings are exactly calculated. Search Engines There are 25 search engines that are used to calculate the TIOBE index. The selected search engines are the 25 highest ranked websites of Alexa that meet the following conditions: The entry page of the site contains a search facility The result of querying the site contains an indication of the number of page hits The results should be available in HTML with clear tags Search engines in languages with special characters should be encoded properly The search engine should at least return 1 hit for 1 query The results of querying the site shouldn't contain too many outliers Porn sites are excluded [1] https://www.tiobe.com/tiobe-index/programming-languages-definition/ https://www.tiobe.com/tiobe-index/programming-languages-defi...
- iratewizard 6y ago
  > Chaturbate.com: PORN_SITE Gave me a laugh. Engineers would initially qualify a porn search engine to search for this sort of experiment
  - geewee 6y ago
    That'd certainly give "Python" a leg up as a search term.
  - EEMac 6y ago
    Well, yes. Why wouldn't you? They hire engineers, and they have programming projects. Meets all requirements.
    - iratewizard 6y ago
      I don't usually search pornhub for programming help
vslira 6y ago
From what I've seen, jupyter notebooks reach convoluted excel formulas-level of accessibility and efficacy for lay people. Most of us really underestimate what "non" programmers can achieve with if/else, loops, equality tests, a repl and autocompletion.
- qorrect 6y ago
  Yeah I'm guessing code quality is _tanking_ around the world. Lots more code , a lot less quality.
  - twox2 6y ago
    Not every programmer is a "software engineer" for many, code is a tool to get a job done, it's a means to an end, and that's wonderful.
  - Veen 6y ago
    Perhaps, but when I built the shed in my back yard I didn't enforce the same standards a construction engineer would for a skyscraper. I increased the quantity of "shitty construction" in the world, but it works great for my purposes.
  - oblio 6y ago
    And a lot more stuff being done. Cut it out with the elitism. It's like being an anaerobic bacteria complaining about the abundance of oxygen in during the Cambrian explosion. "Back in my day no stinking, puny, oxygen breathing things were around, only sturdy creatures such as myself" :-) Yeah, it was messy, but that's how we were able to have humans, hundreds of millions of years later.
  - mmcnl 6y ago
    What is "code quality" anyway?
vmchale 6y ago
> I think one thing programmers underestimate is how many scientists and academics are using Python. It's not just ML, it's pretty much every kind of scientist and tons of non-science academics too. The fact that it's a great "casual" or part-time language is huge for these people. They aren't full time (or even self-identifying) programmers, but their programs are non-trivial. The readability, portability, tooling, and massive package ecosystem is huge for them. I've found the reliability and tooling really unsatisfactory, at least compared to Haskell. The libraries are there, though, and familiarity is in the departments (psychology (maybe more R?), physics, &c.)
asdfman123 6y ago
Oh yeah. Lots of scientists just want to turn their text file of data points into a graph, and Python is great for that. Some people just stick to the basic stuff but there's plenty of other, more complicated programs that are developed by academics/students. Guess what they're going to use? The language they first used to make that pretty graph.
TrackerFF 6y ago
Ya, I think it's great that Python has become a sorts of universal / unified language these days. When I was a student, people used a bunch of different languages to write software, depending on what group they were in, or what the culture there was. You had Matlab, R, Octave, Java, C++, Lisp, etc. all depending on what type of scientist or engineers you worked with (Statisticians used R/SPSS/etc., Physicists and Electrical Engineers used Matlab or C++, Computer Scientists used Lisp etc., and the list goes on) At least today, most is written in Python - only the libraries are different. Also, sharing code is a breeze today. Back then you'd have to download or manually transfer the codebase from other users, check dependencies, install software, check licensing, and what not. Today you can fire up a Python notebook, and that's it - very nice for smaller stuff. Even larger applications is a easy as pie.
- Jugurtha 6y ago
  >Physicists and Electrical Engineers used Matlab or C++ Trained as an EE and this is true. We used MATLAB/Simulink for signal processing and control systems. Octave, too (because our sensor technology teacher was more comfortable with that). x86 and Microship Assembly, C, and Pascal. Used LTSpice and NI Multisim for circuit design and simulation. Cadsoft EAGLE for printed circuit board (PCB) work. Played a bit with LabVIEW. However, I started coding as a child with BASIC/x86 Assembly/C/Pascal/VB/Delphi. Doing mostly Python for the past several years. I suppose one could find reasons to "love" or "hate" anything, but I always found that nuance keeps sanity.
marmaduke 6y ago
Just want to bump this from another perspective: I’m an engineer in a big EU science infrastructure project and everything has a Python API. The main lab environment is JupyterLab. No one wonders whether what languages scripts or workflows will be written in. Julia is on the radar but even in Julians’ wildest dreams Python remains relevant through PyCall. Why rewrite it if you can reuse it? (Obv I refer to boring-to-write non-performance-sensitive code here)
analog31 6y ago
I'm one of those scientists, working in industry R&D. I use Python for everything from Jupyter notebooks to a roughly 8000 line program that runs a prototype of a product. Python pulled me away from Visual Basic several years ago, and then I have a longer history going back through Turbo Pascal and ultimately originating with primordial BASIC. On the hardware side, I use C within the Arduino ecosystem. I've got roughly a half dozen other scientists at my work site using Python at various levels. Confession: Code quality is a problem. I care a lot about code quality. My team recently brought on a couple of people with real programming background, and I've asked them to help me bring myself up to date on good techniques, and it has helped. I was lucky that some of my former training, e.g., in Pascal, was not just about how to program, but actually how to write good code (for its time). But here's my view. Don't just tell us that our code sucks. We already know it, but we're under the same pressures as anybody else, and programming is a force multiplier that we won't give up. We need instructions that we can understand at our level, that will help us do this stuff better, perhaps incrementally. Starting with a strict discipline is hard, but Python supports the ability to go back through working code and improve it.
- The_Amp_Walrus 6y ago
  I'm working with some researches on about ~40k lines of Python disease modelling code and these are the things that helped us becomes more productive: - tests (unit, smoke, acceptance) (eg. https://mattsegal.dev/alternate-test-styles.html https://mattsegal.dev/alternate-test-styles.html, https://understandlegacycode.com/approval-tests/ https://understandlegacycode.com/approval-tests/) - continuous integration (via GitHub actions eg: https://mattsegal.dev/pytest-on-github-actions.html https://mattsegal.dev/pytest-on-github-actions.html) - 10x performance improvements via cProfile (https://julien.danjou.info/guide-to-python-profiling-cprofile-concrete-case-carbonara/ https://julien.danjou.info/guide-to-python-profiling-cprofil...) - getting fast feedback on code changes by visualizing outputs with Streamlit (https://www.streamlit.io/ https://www.streamlit.io/) - input parameter validation (eg. https://pydantic-docs.helpmanual.io/ https://pydantic-docs.helpmanual.io/, https://mattsegal.dev/cerberus-config-validation.html https://mattsegal.dev/cerberus-config-validation.html) - total automation of data processing job (eg. https://www.prefect.io/ https://www.prefect.io/ - we don't use this but that's the kind of thing we have)

wainstead 6y ago

As to how they get the numbers, this is TIOBE's methodology: [1] Since there are many questions about the way the TIOBE index is assembled, a special page is devoted to its definition. Basically the calculation comes down to counting hits for the search query +"<language> programming" In the next few sections it is explained what search engines qualify, what programming languages qualify and how the ratings are exactly calculated. Search Engines There are 25 search engines that are used to calculate the TIOBE index. The selected search engines are the 25 highest ranked websites of Alexa that meet the following conditions: The entry page of the site contains a search facility The result of querying the site contains an indication of the number of page hits The results should be available in HTML with clear tags Search engines in languages with special characters should be encoded properly The search engine should at least return 1 hit for 1 query The results of querying the site shouldn't contain too many outliers Porn sites are excluded [1] https://www.tiobe.com/tiobe-index/programming-languages-definition/ https://www.tiobe.com/tiobe-index/programming-languages-defi...

iratewizard 6y ago
> Chaturbate.com: PORN_SITE Gave me a laugh. Engineers would initially qualify a porn search engine to search for this sort of experiment
- geewee 6y ago
  That'd certainly give "Python" a leg up as a search term.
- EEMac 6y ago
  Well, yes. Why wouldn't you? They hire engineers, and they have programming projects. Meets all requirements.
  - iratewizard 6y ago
    I don't usually search pornhub for programming help

vslira 6y ago

From what I've seen, jupyter notebooks reach convoluted excel formulas-level of accessibility and efficacy for lay people. Most of us really underestimate what "non" programmers can achieve with if/else, loops, equality tests, a repl and autocompletion.

qorrect 6y ago
Yeah I'm guessing code quality is _tanking_ around the world. Lots more code , a lot less quality.
- twox2 6y ago
  Not every programmer is a "software engineer" for many, code is a tool to get a job done, it's a means to an end, and that's wonderful.
- Veen 6y ago
  Perhaps, but when I built the shed in my back yard I didn't enforce the same standards a construction engineer would for a skyscraper. I increased the quantity of "shitty construction" in the world, but it works great for my purposes.
- oblio 6y ago
  And a lot more stuff being done. Cut it out with the elitism. It's like being an anaerobic bacteria complaining about the abundance of oxygen in during the Cambrian explosion. "Back in my day no stinking, puny, oxygen breathing things were around, only sturdy creatures such as myself" :-) Yeah, it was messy, but that's how we were able to have humans, hundreds of millions of years later.
- mmcnl 6y ago
  What is "code quality" anyway?

vmchale 6y ago

> I think one thing programmers underestimate is how many scientists and academics are using Python. It's not just ML, it's pretty much every kind of scientist and tons of non-science academics too. The fact that it's a great "casual" or part-time language is huge for these people. They aren't full time (or even self-identifying) programmers, but their programs are non-trivial. The readability, portability, tooling, and massive package ecosystem is huge for them. I've found the reliability and tooling really unsatisfactory, at least compared to Haskell. The libraries are there, though, and familiarity is in the departments (psychology (maybe more R?), physics, &c.)

asdfman123 6y ago

Oh yeah. Lots of scientists just want to turn their text file of data points into a graph, and Python is great for that. Some people just stick to the basic stuff but there's plenty of other, more complicated programs that are developed by academics/students. Guess what they're going to use? The language they first used to make that pretty graph.

TrackerFF 6y ago

Ya, I think it's great that Python has become a sorts of universal / unified language these days. When I was a student, people used a bunch of different languages to write software, depending on what group they were in, or what the culture there was. You had Matlab, R, Octave, Java, C++, Lisp, etc. all depending on what type of scientist or engineers you worked with (Statisticians used R/SPSS/etc., Physicists and Electrical Engineers used Matlab or C++, Computer Scientists used Lisp etc., and the list goes on) At least today, most is written in Python - only the libraries are different. Also, sharing code is a breeze today. Back then you'd have to download or manually transfer the codebase from other users, check dependencies, install software, check licensing, and what not. Today you can fire up a Python notebook, and that's it - very nice for smaller stuff. Even larger applications is a easy as pie.

Jugurtha 6y ago
>Physicists and Electrical Engineers used Matlab or C++ Trained as an EE and this is true. We used MATLAB/Simulink for signal processing and control systems. Octave, too (because our sensor technology teacher was more comfortable with that). x86 and Microship Assembly, C, and Pascal. Used LTSpice and NI Multisim for circuit design and simulation. Cadsoft EAGLE for printed circuit board (PCB) work. Played a bit with LabVIEW. However, I started coding as a child with BASIC/x86 Assembly/C/Pascal/VB/Delphi. Doing mostly Python for the past several years. I suppose one could find reasons to "love" or "hate" anything, but I always found that nuance keeps sanity.

marmaduke 6y ago

Just want to bump this from another perspective: I’m an engineer in a big EU science infrastructure project and everything has a Python API. The main lab environment is JupyterLab. No one wonders whether what languages scripts or workflows will be written in. Julia is on the radar but even in Julians’ wildest dreams Python remains relevant through PyCall. Why rewrite it if you can reuse it? (Obv I refer to boring-to-write non-performance-sensitive code here)

analog31 6y ago

I'm one of those scientists, working in industry R&D. I use Python for everything from Jupyter notebooks to a roughly 8000 line program that runs a prototype of a product. Python pulled me away from Visual Basic several years ago, and then I have a longer history going back through Turbo Pascal and ultimately originating with primordial BASIC. On the hardware side, I use C within the Arduino ecosystem. I've got roughly a half dozen other scientists at my work site using Python at various levels. Confession: Code quality is a problem. I care a lot about code quality. My team recently brought on a couple of people with real programming background, and I've asked them to help me bring myself up to date on good techniques, and it has helped. I was lucky that some of my former training, e.g., in Pascal, was not just about how to program, but actually how to write good code (for its time). But here's my view. Don't just tell us that our code sucks. We already know it, but we're under the same pressures as anybody else, and programming is a force multiplier that we won't give up. We need instructions that we can understand at our level, that will help us do this stuff better, perhaps incrementally. Starting with a strict discipline is hard, but Python supports the ability to go back through working code and improve it.

The_Amp_Walrus 6y ago
I'm working with some researches on about ~40k lines of Python disease modelling code and these are the things that helped us becomes more productive: - tests (unit, smoke, acceptance) (eg. https://mattsegal.dev/alternate-test-styles.html https://mattsegal.dev/alternate-test-styles.html, https://understandlegacycode.com/approval-tests/ https://understandlegacycode.com/approval-tests/) - continuous integration (via GitHub actions eg: https://mattsegal.dev/pytest-on-github-actions.html https://mattsegal.dev/pytest-on-github-actions.html) - 10x performance improvements via cProfile (https://julien.danjou.info/guide-to-python-profiling-cprofile-concrete-case-carbonara/ https://julien.danjou.info/guide-to-python-profiling-cprofil...) - getting fast feedback on code changes by visualizing outputs with Streamlit (https://www.streamlit.io/ https://www.streamlit.io/) - input parameter validation (eg. https://pydantic-docs.helpmanual.io/ https://pydantic-docs.helpmanual.io/, https://mattsegal.dev/cerberus-config-validation.html https://mattsegal.dev/cerberus-config-validation.html) - total automation of data processing job (eg. https://www.prefect.io/ https://www.prefect.io/ - we don't use this but that's the kind of thing we have)