4 ms·

Having worked in the field long enough, I always feel the need to add: A) If you're writing high-performance code DON'T you _will_ hit bottlenecks which are no

by rob_c 5y ago

Having worked in the field long enough, I always feel the need to add: A) If you're writing high-performance code DON'T you _will_ hit bottlenecks which are non-trivial to the language unless you're a guru. If you're a scientist you're going to have to work to get to this level. Apart2) Python is excellent for saying "perform analysis A with variables B,C,D,E,F on X,Y,Z 30 times using this random seed library". What Python is not excellent at is saying, "I have 20GB of raw doubles and I need to push these through non-euclidian algebra as fast as a modern CPU can cope". The biggest surprise is that modern Python really can do both, it just struggles a lot with the latter over the former. IPython and JuPyter notebooks as well are amazing for the former! B) For the love of whatever deity you worship. PLEASE DOCUMENT YOUR DEPENDENCIES CORRECTLY. This is non-trivial and doesn't mean "Works on CentOS6". I mean at least tested on Python x.y with depA version 1.2.3. Python isn't as bad a npm, but it's getting there... Bpart2) Unless you're starting out, _avoid_conda_. Installing/maintaining things through this is not nice. This is the sort of system that makes your local sysadmin groan when you come to him and say "it's easy because it has a UI". Behind the scenes it makes so many assumptions that are great for people starting out, but are often painfully wrong for people looking to run code at hyper-scale on clusters. I recommend investing _some_ time in learning how python packages work (not much!, packaging, is not coding). But as a result you will learn or employ a lot of skills which will help you with larger python codebases. There are large performant python packages/tools out-there and they all tend to leverage advanced features for good reasons. C) Never! re-invent the wheel unless you know you explicitly need to. And avoiding pulling in a dependency (unless the dependency is _huge_ is not always the correct answer). D) Ask an expert. It's fine to acknowledge you don't know how to use the tool that is modern python. It's a swiss-knife with a shotgun attached pointed at your face. When it goes wrong it will come back to bite, but when you get it correct it's so efficient you will marvel at how much you can do with just 5/6 lines of code. E) Have fun, just try things and see if they don't work or work really well, python is great for avoiding too much boilerplate for mid-scale projects :)

jonnycomputer 5y ago
>B) For the love of whatever deity you worship. PLEASE DOCUMENT YOUR DEPENDENCIES CORRECTLY. This is non-trivial and doesn't mean "Works on CentOS6". I mean at least tested on Python x.y with depA version 1.2.3. Python isn't as bad a npm, but it's getting there... That is why for R code, I really like the renv project. https://www.rstudio.com/blog/renv-project-environments-for-r/ https://www.rstudio.com/blog/renv-project-environments-for-r...
- rob_c 5y ago
  very similar to virtualenv by the sounds of it,, and doesn't really address things any more/less I would argue other than maybe snapshotting things which I'd argue isn't the same as understanding and testing. Although I like the separation between deployment and OS that both solutions provide.
  - jonnycomputer 5y ago
    Doesn't address documenting dependencies? This is scientific code we are talking about right? Perhaps we're talking past each other. This is what I mean. For application or package development, say in python, or even in R, then you will want to test for a wide range of dependencies. Then a snapshot such as provided by virtualenv or conda is inadequate. Better to have a requirements.txt file, or something equivalent (like a poetry toml file). But for a lot of scientific code, such as is published along side a paper, a snapshot is sufficient, and more than is usually provided. How many times have I downloaded a paper's R code only to find it won't run because of some dependency kludge that isn't explicitly stated? Too often.
O_H_E 5y ago
Insightful writeup! I am just not sure what did you mean to discourage in part A. > A) If you're writing high-performance code DON'T you _will_ hit bottlenecks which are non-trivial to the language unless you're a guru. DON'T do what? don't try to hyper-optimize your code? don't use all types of "fancy" constructs like how the post shows?
- rob_c 5y ago
  Don't write code designed to burn CPU with numerical calculations. There is hyper optimisation and then there is the right tool for the job. The speed loss from doing mathS in an interpreter is huge and frankly usually not worth paying. Python like this is great for low CPU hyper responsive code, or for increasing efficiency of existing constructs. As a rule of advice to anyone who asks me this, go away and read about the benefits of compiled over interpreted code. There's to much to conver in 30s and frankly if you're worrying about extreme performance you shouldn't be using python... That's not too say I don't think python is awesome, but, write a module in something else :p

by rob_c 5y ago

jonnycomputer 5y ago

>B) For the love of whatever deity you worship. PLEASE DOCUMENT YOUR DEPENDENCIES CORRECTLY. This is non-trivial and doesn't mean "Works on CentOS6". I mean at least tested on Python x.y with depA version 1.2.3. Python isn't as bad a npm, but it's getting there... That is why for R code, I really like the renv project. https://www.rstudio.com/blog/renv-project-environments-for-r/ https://www.rstudio.com/blog/renv-project-environments-for-r...

rob_c 5y ago
very similar to virtualenv by the sounds of it,, and doesn't really address things any more/less I would argue other than maybe snapshotting things which I'd argue isn't the same as understanding and testing. Although I like the separation between deployment and OS that both solutions provide.
- jonnycomputer 5y ago
  Doesn't address documenting dependencies? This is scientific code we are talking about right? Perhaps we're talking past each other. This is what I mean. For application or package development, say in python, or even in R, then you will want to test for a wide range of dependencies. Then a snapshot such as provided by virtualenv or conda is inadequate. Better to have a requirements.txt file, or something equivalent (like a poetry toml file). But for a lot of scientific code, such as is published along side a paper, a snapshot is sufficient, and more than is usually provided. How many times have I downloaded a paper's R code only to find it won't run because of some dependency kludge that isn't explicitly stated? Too often.

O_H_E 5y ago

Insightful writeup! I am just not sure what did you mean to discourage in part A. > A) If you're writing high-performance code DON'T you _will_ hit bottlenecks which are non-trivial to the language unless you're a guru. DON'T do what? don't try to hyper-optimize your code? don't use all types of "fancy" constructs like how the post shows?

rob_c 5y ago
Don't write code designed to burn CPU with numerical calculations. There is hyper optimisation and then there is the right tool for the job. The speed loss from doing mathS in an interpreter is huge and frankly usually not worth paying. Python like this is great for low CPU hyper responsive code, or for increasing efficiency of existing constructs. As a rule of advice to anyone who asks me this, go away and read about the benefits of compiled over interpreted code. There's to much to conver in 30s and frankly if you're worrying about extreme performance you shouldn't be using python... That's not too say I don't think python is awesome, but, write a module in something else :p