8 ms·

Three of us (including me), in an academic lab, are making the jump from matlab to python. We use py3 (of Anaconda, on Windows10). We like it, but we are const

by pen2l 9y ago

Three of us (including me), in an academic lab, are making the jump from matlab to python. We use py3 (of Anaconda, on Windows10). We like it, but we are constantly screaming and bitching about the complications of 2<->3 ("Ahhh X and Y module only works on py2!", "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!"). Package management ("Do I use conda? pip? easy_install? compile?") is annoying, I hope that something like hatch (https://github.com/ofek/hatch https://github.com/ofek/hatch) maybe takes over everything -- because I think the mess that is package management is THE biggest deterrent for newcomers. I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache. All of that said -- python is nice, it's amazing, it's the newer, funner Java. It is incredibly approachable, it can do pretty much EVERYTHING because of the expansive availability of libraries, it's finally a programming language that is great to get my 7 year old niece started on.

BeetleB 9y ago
>I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache. Meh. I use python on Windows all the time - it's not a problem. Mostly pip just works. Of course, I have over a decade of experience with Python - albeit only the last few years in Windows.
- princeb 9y ago
  the #1 issue i have with py on win is having to download a million different C++ compilers esp for packages not distributed on pip, and esp the more obscure ones with no support whatsoever (e.g. data processing or math packages that were put together by some chap who was working on it for a specific set of research papers and has since moved on). chris gohlke's website was a real lifesaver back then, although now pip seems much better. nowadays it can be a bit difficult replicating environments across different machines especially since conda is moving rapidly from python ver to ver while some obscure packages i rely on are still stuck on 3.4 or earlier... or if c++ 10.0 or 9.0 or 14.0 is not set up correctly. then it becomes an issue of crossing fingers and hoping "python setup.py install" doesn't fail.
- cfusting 9y ago
  What's the point of using Windows as a Dev or data science environment anyway? Everything is built and works well in Linux, scripting is powerful, it's free, etc.. seriously asking.
  - BeetleB 9y ago
    When you work for a company that gives you Windows PC's, and have a horrible Linux environment (through a network, not on your laptop). From a Python standpoint, there is no down side to developing in Windows. It's easier for me to install the Python libraries I need in Windows than to install them on Linux where I don't have root privileges.
erikb 9y ago
Actually I always complain about Python packaging when I'm only using Python. But the moment I start using it in other languages I dream about the comfort and professionalism of tools in Python. Of course in scientific Py you need compiled C attachments, which makes it more complex. But in normal Py usage you only need pip. Easy Install is only needed when you don't have pip installed. Pip is a drop in replacement with additional features. And at least from what I heard for everything with C compiled stuff inside ("binary packages") you use conda. So not that hard actually.
- mixmastamyk 9y ago
  Easy install hasn't been needed for years, consider it deprecated.
  - erikb 9y ago
    Well, as said before sometimes you need it because pip isn't installed by default. Just had such a situation yesterday at work. A guy from QA (not a dev) needs to setup an environment to install our tool for testing. He doesn't know python, so explaining to him what a virtualenv is would take way too much time. He probably also doesn't know how to use the OS's package manager to install pip. So, what to do? Give him the oneliner "$ sudo easy_install whatever". It works, he doesn't need to understand it, and actual system maintenance doesn't matter because it's just a VM that gets deleted after he's done anyways.
    - mixmastamyk 9y ago
      You don't. First, if your QA peeps can't figure out an OS package manager, they need training. It's also a one-liner. Without a package manager it is only a two-step process. 1) download get-pip.py from the source 2) run it with sudo
yladiz 9y ago
I think you're right, the biggest pain for a general developer for Python is determining which package manager to use. I'm thankfully insulated from it a bit due to using Docker for a lot of my development, but I hate the fact that Pip won't install to your home/project directory unless you coax it to (and sometimes still won't even if you give it all the proper environment variables), and I think virtual environments are not the solution (I shouldn't have to think about how to install a package or work on a project). Especially coming from Node-land, and before that Ruby-land, Python seemed like a wonderfully designed language but with terrible package managers. From what I remember Pip is going to change the default to install on the home directory sometime in the future, but it shouldn't have taken years of discussion to come to the conclusion that it's the right thing to do.
- thanksgiving 9y ago
  I like python3 -m venv myfolder; source myfolder/bin/activate; It is the easiest way to use python3 because even fedora (one of the more bleeding edge) likely won't finish migrating 2 to 3 in current year. Why do you not like venv? What could be done better?
  - yladiz 9y ago
    I don't really like virtualenv because you have to activate it to do anything and deactivate it if you want to go to another project or work on something unrelated. I'd much rather it work by the project directory so I don't have to worry about the activation part (this is why Node's package manager, for all its issues, is great: you install in a local folder unless you explicitly supply the -g flag). At least I should be able to install packages to my home directory.
    - thanksgiving 9y ago
      Ah yes. Local by default would be nicer.
      - yladiz 9y ago
cfusting 9y ago
Yeah use Ubuntu, your life will get way easier.
heydonovan 9y ago
Package management was so frustrating coming from Ruby to Python. There are so many programs, and version conflicts that I run into all the time. With gems and bundler, it was rare if things didn't work.
- cshenton 9y ago
  If you get into the habit of making a virtual environment and requirements.txt in every repo then it's smooth. Just activate the env when you're using the repo and use vanilla pip install to install dependencies. Then pip freeze them into the requirements file. It's a very similar workflow to a gemfile and bundler.
  - dasil003 9y ago
    I worked on python for a few months last year, and although I didn't find anything representing a canonical description of that workflow, I did eventually come to this conclusion. The only thing was that pip freeze > requirements.txt didn't really give the same power as I was used to with Gemfile.lock and the various permutations of bundle update. I forget the specifics, but I remember being unable to get the finer points of optimistic version locking to work in a way I found acceptable. That said, I have been doing ruby since before Bundler, and I really have to take my hat off to what Yehuda and company accomplished with Bundler. It was both a technical and open source community triumph to get Bundler done, stable and covering the breadth of use cases it applies to.
  - yes_or_gnome 9y ago
    You're going to have a hard time supporting multiple versions of Python that way. If I started a new package today I would target 3.4, 3.5, 3.6, 3.7-dev, and likely, 2.7. A lot of well maintained packages have different dependencies based on the exact version of Python. If you're crazy enough to support 2.6, then there will be a lot of additional packages in pip freeze. Ideally, packages would have just one set of dependencies and the packages would be version locked, but that's just not the case in the Python community.
    - 9y ago
joshuamorton 9y ago
I'm curious, what issues do you run into with python3 incompatibility? Most scientific stuff I've done works in python3. In fact, its now possible to run a tensorflow/opencv/scipy environment entirely from pip in a virtualenv, though it won't be the fastest. which is amazing. Doing that kind of thing in any other language would require docker. In general, if you're sticking to python3, the answer will always be "use pip", everything should work with pip, and everything should install as a binary without needing to build source. Edit: There's exactly one context where I use python2, and its for a robotics project where some transitive dependencies are python2 only.
- photojosh 9y ago
  Why won't it be the fastest? I didn't think there would be any difference between "native" in the system paths and in a virtualenv.
  - joshuamorton 9y ago
    So, for things like tensorflow, you have to build from source to enable certain optimizations. Wheel's aren't source builds essentially.
    - kranner 9y ago
      I think GP is talking about virtualenv, not pip. Using an optimized build in a virtualenv environment should not be any slower than using it outside the environment, right?
      - joshuamorton 9y ago
        Ah yes, that's true as far as I know.
chrisseaton 9y ago
> it's the newer ... Java Python (1991) is older than Java (1995).
- mark-r 9y ago
  But Java was an overnight success, while it took Python years to get traction. I was exposed to both around the same time, probably 1995, but I ignored Python until the second time I encountered it in 2003. It just didn't have the hype.
  - Twirrim 9y ago
    Java was by no means an overnight success. It was way too slow for several years until they really started to do anything more than run the bytecode in the JVM.
    - nyrikki 9y ago
      McNealy freely admitted in an interview that: "we were able to hook Java runtime onto the free, open source Netscape browser. And it was a collision of two things that accelerated and launched each other together. It was fascinatingly lucky timing for both companies." While it may have taken time to mature at that time it was an "overnight success". Sun was considering dropping it and considered it an expense before that.
      - nol13 9y ago
        Did Sun ever make any money on it?
    - goatlover 9y ago
      It was an overnight success in the hype department.
highd 9y ago
I highly recommend switching to the conda distributions and package ecosystem. A lot of those issues have gotten a lot better for me since I did that.
shepardrtc 9y ago
If you're using Windows, then WinPython (http://winpython.sourceforge.net/ http://winpython.sourceforge.net/) is an absolute necessity. I have two separate installations, one for 2.x and one for 3.x, and I have zero issues. Works amazingly well and comes with just about everything you need already installed and working. Can't say enough good things about it.
epage 9y ago
> "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!"). What kind of devices or libraries do you use from LabVIEW but there isn't Python support or examples? If its an NI product, some are starting to get Python support DAQmx https://github.com/ni/nidaqmx-python/blob/master/nidaqmx_examples/ai_voltage_sw_timed.py https://github.com/ni/nidaqmx-python/blob/master/nidaqmx_exa... XNET https://github.com/ni/nixnet-python/blob/master/nixnet_examples/can_signal_single_point_io.py https://github.com/ni/nixnet-python/blob/master/nixnet_examp... DMM https://github.com/ni/nimi-python/blob/master/src/nidmm/examples/nidmm_measurement.py https://github.com/ni/nimi-python/blob/master/src/nidmm/exam... FPGA Interface https://nifpga-python.readthedocs.io/en/latest/examples/basic_examples.html#opening-a-session https://nifpga-python.readthedocs.io/en/latest/examples/basi... EDIT: All with Python3 support.
- pen2l 9y ago
  Andor cameras: http://www.andor.com/scientific-software/software-development-kit http://www.andor.com/scientific-software/software-developmen... Labview, Matlab, lots of other frameworks, no python PI instruments: https://www.physikinstrumente.com/en/products/linear-stages-and-actuators/piezo-stages/p-736-pinano-z-microscope-scanner-for-well-plates-200405/ https://www.physikinstrumente.com/en/products/linear-stages-... Matlab, Labview, etc. yes. No python Thorlabs equipment: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=8184 https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=81... Same story, no python. In fact, it's always a _SURPRISE_ when I see Python being supported. I would say after having gone through about a hundred major lab devices, less than 2% I found to be supporting Python. Importantly, example code should be made available (as it generally is for matlab, and labview) because creating things from scratch in a move-fast environment is a little too much.

by pen2l 9y ago

BeetleB 9y ago

>I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache. Meh. I use python on Windows all the time - it's not a problem. Mostly pip just works. Of course, I have over a decade of experience with Python - albeit only the last few years in Windows.

princeb 9y ago
the #1 issue i have with py on win is having to download a million different C++ compilers esp for packages not distributed on pip, and esp the more obscure ones with no support whatsoever (e.g. data processing or math packages that were put together by some chap who was working on it for a specific set of research papers and has since moved on). chris gohlke's website was a real lifesaver back then, although now pip seems much better. nowadays it can be a bit difficult replicating environments across different machines especially since conda is moving rapidly from python ver to ver while some obscure packages i rely on are still stuck on 3.4 or earlier... or if c++ 10.0 or 9.0 or 14.0 is not set up correctly. then it becomes an issue of crossing fingers and hoping "python setup.py install" doesn't fail.
cfusting 9y ago
What's the point of using Windows as a Dev or data science environment anyway? Everything is built and works well in Linux, scripting is powerful, it's free, etc.. seriously asking.
- BeetleB 9y ago
  When you work for a company that gives you Windows PC's, and have a horrible Linux environment (through a network, not on your laptop). From a Python standpoint, there is no down side to developing in Windows. It's easier for me to install the Python libraries I need in Windows than to install them on Linux where I don't have root privileges.

erikb 9y ago

Actually I always complain about Python packaging when I'm only using Python. But the moment I start using it in other languages I dream about the comfort and professionalism of tools in Python. Of course in scientific Py you need compiled C attachments, which makes it more complex. But in normal Py usage you only need pip. Easy Install is only needed when you don't have pip installed. Pip is a drop in replacement with additional features. And at least from what I heard for everything with C compiled stuff inside ("binary packages") you use conda. So not that hard actually.

mixmastamyk 9y ago
Easy install hasn't been needed for years, consider it deprecated.
- erikb 9y ago
  Well, as said before sometimes you need it because pip isn't installed by default. Just had such a situation yesterday at work. A guy from QA (not a dev) needs to setup an environment to install our tool for testing. He doesn't know python, so explaining to him what a virtualenv is would take way too much time. He probably also doesn't know how to use the OS's package manager to install pip. So, what to do? Give him the oneliner "$ sudo easy_install whatever". It works, he doesn't need to understand it, and actual system maintenance doesn't matter because it's just a VM that gets deleted after he's done anyways.
  - mixmastamyk 9y ago
    You don't. First, if your QA peeps can't figure out an OS package manager, they need training. It's also a one-liner. Without a package manager it is only a two-step process. 1) download get-pip.py from the source 2) run it with sudo

yladiz 9y ago

I think you're right, the biggest pain for a general developer for Python is determining which package manager to use. I'm thankfully insulated from it a bit due to using Docker for a lot of my development, but I hate the fact that Pip won't install to your home/project directory unless you coax it to (and sometimes still won't even if you give it all the proper environment variables), and I think virtual environments are not the solution (I shouldn't have to think about how to install a package or work on a project). Especially coming from Node-land, and before that Ruby-land, Python seemed like a wonderfully designed language but with terrible package managers. From what I remember Pip is going to change the default to install on the home directory sometime in the future, but it shouldn't have taken years of discussion to come to the conclusion that it's the right thing to do.

thanksgiving 9y ago
I like python3 -m venv myfolder; source myfolder/bin/activate; It is the easiest way to use python3 because even fedora (one of the more bleeding edge) likely won't finish migrating 2 to 3 in current year. Why do you not like venv? What could be done better?
- yladiz 9y ago
  I don't really like virtualenv because you have to activate it to do anything and deactivate it if you want to go to another project or work on something unrelated. I'd much rather it work by the project directory so I don't have to worry about the activation part (this is why Node's package manager, for all its issues, is great: you install in a local folder unless you explicitly supply the -g flag). At least I should be able to install packages to my home directory.
  - thanksgiving 9y ago
    Ah yes. Local by default would be nicer.
    - yladiz 9y ago

cfusting 9y ago

Yeah use Ubuntu, your life will get way easier.

heydonovan 9y ago

Package management was so frustrating coming from Ruby to Python. There are so many programs, and version conflicts that I run into all the time. With gems and bundler, it was rare if things didn't work.

cshenton 9y ago
If you get into the habit of making a virtual environment and requirements.txt in every repo then it's smooth. Just activate the env when you're using the repo and use vanilla pip install to install dependencies. Then pip freeze them into the requirements file. It's a very similar workflow to a gemfile and bundler.
- dasil003 9y ago
  I worked on python for a few months last year, and although I didn't find anything representing a canonical description of that workflow, I did eventually come to this conclusion. The only thing was that pip freeze > requirements.txt didn't really give the same power as I was used to with Gemfile.lock and the various permutations of bundle update. I forget the specifics, but I remember being unable to get the finer points of optimistic version locking to work in a way I found acceptable. That said, I have been doing ruby since before Bundler, and I really have to take my hat off to what Yehuda and company accomplished with Bundler. It was both a technical and open source community triumph to get Bundler done, stable and covering the breadth of use cases it applies to.
- yes_or_gnome 9y ago
  You're going to have a hard time supporting multiple versions of Python that way. If I started a new package today I would target 3.4, 3.5, 3.6, 3.7-dev, and likely, 2.7. A lot of well maintained packages have different dependencies based on the exact version of Python. If you're crazy enough to support 2.6, then there will be a lot of additional packages in pip freeze. Ideally, packages would have just one set of dependencies and the packages would be version locked, but that's just not the case in the Python community.
  - 9y ago

joshuamorton 9y ago

I'm curious, what issues do you run into with python3 incompatibility? Most scientific stuff I've done works in python3. In fact, its now possible to run a tensorflow/opencv/scipy environment entirely from pip in a virtualenv, though it won't be the fastest. which is amazing. Doing that kind of thing in any other language would require docker. In general, if you're sticking to python3, the answer will always be "use pip", everything should work with pip, and everything should install as a binary without needing to build source. Edit: There's exactly one context where I use python2, and its for a robotics project where some transitive dependencies are python2 only.

photojosh 9y ago
Why won't it be the fastest? I didn't think there would be any difference between "native" in the system paths and in a virtualenv.
- joshuamorton 9y ago
  So, for things like tensorflow, you have to build from source to enable certain optimizations. Wheel's aren't source builds essentially.
  - kranner 9y ago
    I think GP is talking about virtualenv, not pip. Using an optimized build in a virtualenv environment should not be any slower than using it outside the environment, right?
    - joshuamorton 9y ago
      Ah yes, that's true as far as I know.

chrisseaton 9y ago

> it's the newer ... Java Python (1991) is older than Java (1995).

mark-r 9y ago
But Java was an overnight success, while it took Python years to get traction. I was exposed to both around the same time, probably 1995, but I ignored Python until the second time I encountered it in 2003. It just didn't have the hype.
- Twirrim 9y ago
  Java was by no means an overnight success. It was way too slow for several years until they really started to do anything more than run the bytecode in the JVM.
  - nyrikki 9y ago
    McNealy freely admitted in an interview that: "we were able to hook Java runtime onto the free, open source Netscape browser. And it was a collision of two things that accelerated and launched each other together. It was fascinatingly lucky timing for both companies." While it may have taken time to mature at that time it was an "overnight success". Sun was considering dropping it and considered it an expense before that.
    - nol13 9y ago
      Did Sun ever make any money on it?
  - goatlover 9y ago
    It was an overnight success in the hype department.

highd 9y ago

I highly recommend switching to the conda distributions and package ecosystem. A lot of those issues have gotten a lot better for me since I did that.

shepardrtc 9y ago

If you're using Windows, then WinPython (http://winpython.sourceforge.net/ http://winpython.sourceforge.net/) is an absolute necessity. I have two separate installations, one for 2.x and one for 3.x, and I have zero issues. Works amazingly well and comes with just about everything you need already installed and working. Can't say enough good things about it.

epage 9y ago

> "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!"). What kind of devices or libraries do you use from LabVIEW but there isn't Python support or examples? If its an NI product, some are starting to get Python support DAQmx https://github.com/ni/nidaqmx-python/blob/master/nidaqmx_examples/ai_voltage_sw_timed.py https://github.com/ni/nidaqmx-python/blob/master/nidaqmx_exa... XNET https://github.com/ni/nixnet-python/blob/master/nixnet_examples/can_signal_single_point_io.py https://github.com/ni/nixnet-python/blob/master/nixnet_examp... DMM https://github.com/ni/nimi-python/blob/master/src/nidmm/examples/nidmm_measurement.py https://github.com/ni/nimi-python/blob/master/src/nidmm/exam... FPGA Interface https://nifpga-python.readthedocs.io/en/latest/examples/basic_examples.html#opening-a-session https://nifpga-python.readthedocs.io/en/latest/examples/basi... EDIT: All with Python3 support.

pen2l 9y ago
Andor cameras: http://www.andor.com/scientific-software/software-development-kit http://www.andor.com/scientific-software/software-developmen... Labview, Matlab, lots of other frameworks, no python PI instruments: https://www.physikinstrumente.com/en/products/linear-stages-and-actuators/piezo-stages/p-736-pinano-z-microscope-scanner-for-well-plates-200405/ https://www.physikinstrumente.com/en/products/linear-stages-... Matlab, Labview, etc. yes. No python Thorlabs equipment: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=8184 https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=81... Same story, no python. In fact, it's always a _SURPRISE_ when I see Python being supported. I would say after having gone through about a hundred major lab devices, less than 2% I found to be supporting Python. Importantly, example code should be made available (as it generally is for matlab, and labview) because creating things from scratch in a move-fast environment is a little too much.