14 ms·
Migrating to Python 3 with pleasure
- coppolaemilio 9y agothe Path thing is incredible! I wasn't aware of it. I'll definitively be replacing my os.path.join calls for a more readable version of it.
- ioddly 9y agoWow, I switched to Python 3 really early on and never knew about pathlib.
- has2k1 9y agoI have been conditioned to write code that is 2/3 compatible, that even when I am writing specifically for the PY3 interpreter the code turns out to be a __future__ import away from being valid PY2. I did not think much of them at first, but very recently, f-strings have changed that. I think many people get imprinted with writing the PY3 code using only the features available at the time they switched over.
- orf 9y agoYou can use some backports of libraries though, surely? `pathlib` is in pip.
- has2k1 9y agoBack-ports are an option but they are an extra dependency. When you have not used a new feature, the cost of an extra dependency out weighs the unknown benefits that would be realised by using a back-port.
- sametmax 9y agoAnd the multiprocessing and threading pools. And collections.ChainMap. And f-strings. And yield from. And type hints. And statistics. And ipadress. And secrets. And matmul. And subprocess.run. Come on, Python 3 is packed with awesomeness !
- dozzie 9y agoEvery time somebody mentions how awsome f-strings are makes me laugh. It was around ten years ago when Python community was looking down at Perl and shell with their string interpolation, but now that Python got pretty much the same it's suddenly not considered a misfeature.
- has2k1 9y agoHave you considered that maybe the language had not evolved enough for them to be appealing. With b'' and u'' string syntax coming into the language, there is a realisation that string interpolation can be an opt-in feature among other reasons. What you have is decision making analogous to the function evaluate(string_interpolation, python_ecosystem) which is different from evaluate(string_interpolation)
- dozzie 9y ago> With b'' and u'' string syntax coming into the language, there is a realisation that string interpolation can be an opt-in feature among other reasons. This argument won't fly. Did you know that string interpolation in Perl and shell was always an opt-in feature? And despite that it was frowned upon by Python community.
- sametmax 9y agoWe were wrong. We aknowledged it and improved. It's a good thing.
- dagw 9y agomultiprocessing and threading pools While certainly awesome, these are in python 2.7 as well. Although I don't remember if they where in 2.7 first of back-ported to 2.7 from 3.
- sametmax 9y ago__Reliable__ pools are only in 3.
- quietbritishjim 9y agoThe read_bytes() and read_text() methods are particularly useful. They're new in Python 3.5.
- jstimpfle 9y ago> test_path = datasets_root / dataset / 'test' > Previously it was always tempting to use string concatenation (concise, but obviously bad), now with pathlib the code is safe, concise, and readable. This is the kind of feature that I'm wary to use even in scripts: questionable benefit, and probably too clever.
- metaobject 9y agoI don't have a machine available right now, but I wonder what happens if two adjacent path elements are integers? Does it perform division instead of path/string concatenation?
- lgas 9y agoTypeError: expected str, bytes or os.PathLike object, not int
- maxerickson 9y agoThe Path object won't construct a path from an integer. >>> from pathlib import Path >>> p=Path(1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.6/pathlib.py", line 979, in __new__ self = cls._from_parts(args, init=False) File "/usr/lib/python3.6/pathlib.py", line 654, in _from_parts drv, root, parts = self._parse_args(args) File "/usr/lib/python3.6/pathlib.py", line 638, in _parse_args a = os.fspath(a) TypeError: expected str, bytes or os.PathLike object, not int >>> So what happens if the paths are numbers? They are treated like any other characters.
- sametmax 9y agoPath(__file__).absolute().parent.glob('/*.py') is just wonderful.
- minimaxir 9y agoI'm not sure why this is targeted toward data scientists; these tips are useful for any Python user.
- tomlock 9y agoI think it's because the remaining python 2.7 users are mostly people using the packages that were unsupported in 3, which I think were mostly packages used in data science afaik
- crimsonalucard 9y agoIf it were targeted towards developers I would highly, highly emphasize type annotations with a type checker.
- iandanforth 9y agoIs the np.dot -> @ tip reliable? I thought these were fairly different in practice.
- deleted 9y ago[deleted]
- gh02t 9y agoWhat do you mean? `A.dot(B)` and `A @ B` are the same thing for NumPy arrays. You might be mixing it up with the weirdness of `array` versus `matrix`, but that's totally separate.
- iandanforth 9y agoI was thinking of this - https://stackoverflow.com/questions/34142485/difference-between-numpy-dot-and-python-3-5-matrix-multiplication https://stackoverflow.com/questions/34142485/difference-betw...
- pletnes 9y agoThe @ operator is the same as np.matmul which is different from np.dot for matrices of rank >=3.
- dagw 9y agoAnother difference is that B can be a scalar in the statement A.dot(B), but not in A @ B
- gh02t 9y agoAh you're right, I had that mixed up. matmul should generally be the desired behavior anyway I'd think, the previous behavior of dot was a bit weird IMO, especially the behavior with a scalar. It's not backwards compatible, but I think it'd be better to prefer @/matmul in the future anyway. I almost think it'd be nice to make matmul undefined for ranks higher than 2, since it's not really matrix multiplication and if you want to do that (or the previous behavior of dot) it can be achieved with einsum, with the advantage that you have to be a lot more explicit about what sort of tensor multiplication you want. That's probably a bit too purist though.
- breatheoften 9y agoDidn’t know about the enforce library (https://github.com/RussBaz/enforce/blob/master/README.md https://github.com/RussBaz/enforce/blob/master/README.md) — but have been wanting something like this. Thanks for the useful list!
- AlexCoventry 9y agoI've moved to python 3 over the past couple of months, after resisting for the better part of a decade. I like it. One surprising thing I learned from this document is that dicts now iterate in assignment order, not hash order. That's going to break some code for people.
- StavrosK 9y agoDicts never had an iteration order, though. If you relied on dicts iterating in order, you were kind of asking for it.
- jstimpfle 9y agoI believe newer Python versions have at least deterministic iteration order (correct me if I'm wrong), while some previous ones had a non-deterministic one (non-deterministic between mutiple invocations of the interpreter). But there is also OrderedDict which iterates in assignment order.
- StavrosK 9y agoNo, you're right. I believe dict orders were made deterministic (by insertion order) in 3.6. Before that, it was undefined.
- rtpg 9y agoCPython interates in this fashion since 3.6, but it is not part of the Python spec. Might be a bit contentious but it's not supposed to be relied upon
- thijsvandien 9y agoIt was recently decided that as of 3.7 it will be. (Source: https://mail.python.org/pipermail/python-dev/2017-December/151283.html https://mail.python.org/pipermail/python-dev/2017-December/1...)
- ggm 9y agoCame here to validate my own "I moved" experience: learned stuff I hadn't checked on. TL;DR its never too late to learn what you can do, once you can deprecate the past.
- brett40324 9y agoI realized, just today, that the secrets module is new to 3.6 after trying to pip install it. This being provided directly by the language is a game changer, IMHO.
- orf 9y agoIt's interesting for sure, and I'd like to see where it goes, but right now there isn't much to it: https://github.com/python/cpython/blob/3.6/Lib/secrets.py https://github.com/python/cpython/blob/3.6/Lib/secrets.py
- re 9y agoI've been toying around with Python 3 and using it for most of my personal/hack projects, but I somehow missed the unpacking improvements: https://www.python.org/dev/peps/pep-0448/ https://www.python.org/dev/peps/pep-0448/ In particular, being able to create an updated copy of a dict with a single expression is pretty cool: return {**old, 'foo': 'bar'} # Old way new = old.copy new['foo'] = ['bar'] return new
- miracle2k 9y agoMy mind is blown. Ever since JavaScript added this, I've been wanting it in Python... and somehow it was there all along. It works for lists too! [*a, *b, *c]
- agumonkey 9y agobear in mind that most things javascript got recently are old. literals, closure syntax, destruct, spread .. (lisp, scheme of course, but yeah python took it a while back too) this is all very very old but now ECMAxxxx is bringing it to the mainstream.
- orf 9y agoAnd the performance is pretty much the same, just a lot nicer syntax In [1]: x = {1:2, 3:4} In [2]: %timeit x[3] = 5 48.6 ns ± 1.18 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [3]: %timeit y = x.copy(); y[3] = 5 189 ns ± 3.23 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [4]: %timeit {**x, 3: 5} 182 ns ± 3.3 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) Edit: It also seems to be pretty constant time if you're just merging: In [16]: %timeit {**x, **y, **z} 180 ns ± 1.29 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [17]: %timeit {**x, **y, **z, 3: 5} 278 ns ± 18.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [19]: dis.dis(lambda: {**x, **y, **z, 3: 5}) 0 LOAD_GLOBAL 0 (x) 2 LOAD_GLOBAL 1 (y) 4 LOAD_GLOBAL 2 (z) 6 LOAD_CONST 1 (3) 8 LOAD_CONST 2 (5) 10 BUILD_MAP 1 12 BUILD_MAP_UNPACK 4 14 RETURN_VALUE In [20]: dis.dis(lambda: {**x, **y, **z}) 0 LOAD_GLOBAL 0 (x) 2 LOAD_GLOBAL 1 (y) 4 LOAD_GLOBAL 2 (z) 6 BUILD_MAP_UNPACK 3 8 RETURN_VALUE
- Noumenon72 9y agoIs there some kind of slash operator making this statement work? Is this a way to concatenate things? train_path = datasets_root / dataset / 'train'
- Cyphase 9y agopathlib[1] has been in the standard library since Python 3.4. It can do this.[2] [1] https://docs.python.org/3/library/pathlib.html https://docs.python.org/3/library/pathlib.html [2] https://docs.python.org/3/library/pathlib.html#basic-use https://docs.python.org/3/library/pathlib.html#basic-use
- necavi 9y agoPython allows the overriding of just about every operator. For Pathlib they overrode the division operator to instead perform path addition in a platform agnostic manner.
- glandium 9y ago> Python allows the overriding of just about every operator. Except the boolean operators (and, or, not). For instance __and__ overrides the binary and (&), not the boolean and.
- Cyphase 9y agoThat's True, but you _can_ override __bool__.[1] [1] https://docs.python.org/3/reference/datamodel.html#object.__bool__ https://docs.python.org/3/reference/datamodel.html#object.__...
- sdsad46464 9y agoType hinting isn't part of Python 3, it's part of Python 3.5+.
- ubernostrum 9y agoSyntactic support for annotating functions and exposing the annotations existed as of 3.0. The 'typing' module in the standard library was new as of 3.5. Syntactic support for annotating variables was new as of 3.6. Support for delaying resolution of annotations is new in 3.7 with a __future__ import. Originally the annotation feature was seen as a possible way to add type hints to Python, but other potential uses were envisioned and no immediate preference was given to types over other uses of annotations.
- viraptor 9y agoWith type comments you can use it in 2.7 as well: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code https://www.python.org/dev/peps/pep-0484/#suggested-syntax-f... Mypy supports it: http://mypy.readthedocs.io/en/latest/python2.html http://mypy.readthedocs.io/en/latest/python2.html
- makecheck 9y agoI’m a little surprised at this point that Apple still doesn’t include a default Python 3.x on macOS. It’s the single thing keeping me from moving (as there’s a big difference between “just run this” and “first download this, then run this”).
- rajacombinator 9y agoDownloading it is definitely not the biggest hurdle to moving to python 3.
- semi-extrinsic 9y agoI infer GP was talking about his users on OSX, who would have to download/install python3 to use the distributed software.
- jimwalsh 9y agoThere are arguments against doing anything against system python installs in the first place.
- meddlepal 9y agoI used PyInstaller for the first time this week to build a single binary executable for Linux. So far in limited use the result has been great. I think I'm gonna do OS X next.
- viraptor 9y agoMy theory is that this is not going to be a nice change when they drop 2.7. Maybe one version will be released with both installed by default, then they'll use only 3.7/3.8 for the next decade. MacOS doesn't seem to care a lot about backwards compatibility recently. This is of course pure speculation...
- makecheck 9y agoOther OSes just install a “python3” binary, I’d expect Apple to do the same.
- amyjess 9y agoI'd been a 2.7 holdout for ages, but when f-strings were greenlit for 3.6, I decided then and there that all my new personal projects would be written in 3. I'm glad I did. F-strings are wonderful, as is pathlib and the enhanced unpacking syntax. Since I started my current job, I've also been writing as many scripts as I can in Python 3 as well (and Docker has been a godsend for that because I can now deploy 3.6 scripts to the few servers we have that are still running RHEL 6).
- memco 9y agoCould you provide more info on your setup for this? I work on some EL 6 servers and would be interested in using this setup.
- Leynos 9y agoNot the OP, but replying to offer a suggestion in this respect. I'll often do something similar to this, where I have a CLI tool that I'm not ready to deploy server-wide yet, and has hefty dependencies. The pattern I use is to have a wrapper shell script that calls: docker run -it --rm --volume "$(pwd)/$1:/file_to_process:z" --user $(id -u) container-image /opt/command_to_run /fileToProcess This runs "/opt/command_to_run /fileToProcess" inside a container as the current uid, mounting the parameter to the shell script as "/file_to_process" inside the container. The :z mount parameter may or may not be needed depending upon whether you have SELinux enabled or not (by default, SELinux prevents countainers accessing any file on the host, and :z changes the SE context to allow access). I don't know if this is the case with EL6 tho. The -t parameter shouldn't be used if your script is running in a pipeline (it creates a pseudo-tty). So it may be worth having some kind of conditional to remove this. The wrapper I use also has a conditional to add the "$(pwd)" prefix to the call parameter only if the parameter is a relative path.
- aepiepaey 9y agoIf Python 3.4 is good enough, you can get it from the EPEL repos.
- amyjess 9y agoHonestly, I just wrote my Dockerfile using the example from the official Python container: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/ Then I push the image to my company's internal Docker registry, ssh into the server, and pull the image. (also, aside from using Python 3 on RHEL 6, it also means using Python 3.6 on RHEL 7 without having to install python36u) Edit: Dockerfile (script name redacted) FROM python:3 WORKDIR /usr/src/app COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD [ "python", "./redacted.py" ] Build and push commands (company and script names redacted): docker build --rm -t dreg.example.com/redactedproject/redacted . docker push dreg.example.com/redactedproject/redacted Pull and run on the server (same stuff redacted as above, plus I redacted the actual port number to be on the safe side): docker pull dreg.example.com/redactedproject/redacted docker run -p 1337:1337 -d --name redacted dreg.example.com/redactedproject/redacted And at some point, I'll make proper startup scripts for them. On RHEL 7 boxes, I've made systemd unit files. On RHEL 6... well, I suppose I'll be writing initscripts soon.
- jtchang 9y agoI had so many issues trying to install python3 in an existing server that I ended up having to go back. pip kept complaining and it was just really annoying. Then some libraries were not compatible and it felt like it wasn't worth it.
- setr 9y agoIsn't python 3 usually installed alongside python/pip 2.7 as python3/pip3, and everything kept seperately?
- gesman 9y ago<3 python 3 :)
- pletnes 9y agoSeveral posters indicate that they’ve stuck to python 2.7 even for small side projects until now. I cannot understand why? Python 3 seems to have been technically superior for a few years, and side projects must surely be good for learning something new?
- CJefferson 9y agoI distribute python programs. Macs only have Python 2 by default.
- wyclif 9y agoBut surely that's not stopping anyone. As someone above said, just use pyenv to run Python 2.7x and Python 3. It's not as if anyone has to settle for using only legacy Python.
- CJefferson 9y agoAt the moment my instructions are "grab this script and run 'python script.py'". That is ok for about everyone. I don't want to have to start teaching pyenv to every academic or student I want to send a script to.
- cup-of-tea 9y agoIt's so strange to me that data scientists would need to be convinced to move to Python 3. It's superior in every way to legacy Python. I can understand maintaing Python 3 compatbility for legacy systems if you don't want to have Python 3 as a dependancy, but data scientists will be writing mostly ad hoc code and using Jupyter notebooks. The people around me are not allowed to use Python 2, in fact they're generally required to use the latest version of Python 3. For anyone having trouble with maintaing multiple Python versions, I recommend Pyenv. You can install multiple local versions and switch between them. The selected version then uses the standard commands "python" and "pip", etc. which you can use to make your virtualenv from.
- pletnes 9y agoIf you’re doing data science, do consider the anaconda python distribution with conda environments instead. Intel MKL linked numpy gives a lot better performance, and conda can install binary dependencies for libs that are tricky to compile even on linux. Pip packages are just more work and less performance in my experience (~10x)
- newen 9y agoFor me, it's as simple as python is not a moving target. Whatever new changes there are in python 3 doesn't affect me since I won't be using any of the new features even if I were to use python 3.
- cup-of-tea 9y agoWhy on earth would you not use the new features?
- newen 9y agoTradition? ;) I only use python for deep learning, which is not really that intensive in terms of code. Also most of the code out there for deep learning works on both python 2 and 3. Also, your phrasing invites the question. Why on earth would I use a new feature in a turing complete language? Unless using the new feature results in tangible improvements in my code, I don't see a reason to use it.
- julox 9y agoalso you can use pythonconverter.com
- donarb 9y agoAnother feature is that the mock module is now part of the standard library, no need to install.
- eindiran 9y agoIf you would like to write Python 3 but need to maintain support for any particular version of Python 2, something that I can personally recommend is using the Coconut transpiler: http://coconut-lang.org/ http://coconut-lang.org/ The Coconut language is a superset of Python 3, so any valid Python 3 is valid Coconut. But the advantage of the transpiler extends beyond the language itself, in that it can target any version of Python from 2.6 onwards on the Python 2 branch and 3.2 onwards on the Python 3 branch. It has been really useful for me in that I want the advantages of type annotations and f-strings and other python 3.5+ features but I have to support running in an environment with only 2.6 installed. So when I target a 3.5+ version, all of those features are maintained, but when I target 2.6, the transpiler does all the work in converting to running 2.6 code for me.