23 ms·
Python Standard Library changes in recent years
- rolobio 4y agoDid not know about functools.singledispatchmethod, this could lead to some clean code in specific situations. I know I’ve missed overloading a few times!
- mumblemumble 4y agoI've been making extensive use of it lately; it's treating me quite well.
- tonyg 4y ago`singledispatchmethod` definitely looks useful. The name isn't great, though. It actually offers double dispatch, not single dispatch! `divider.divide(10, 2)` first dispatches on `divider`, then on `10`. As so often with Python, I'm left wondering why they didn't follow through and properly complete the design, in this case perhaps offering multiple dispatch (multimethods) instead of just double dispatch. Casting an eye over the functools code, it looks like they have all the necessary C3 infrastructure in there.
- formerly_proven 4y agoPython sort of has multimethods, but only for type-annotations, not for actually dispatching to code.
- isoprophlex 4y agoDefinitely useful. That code looks ugly as fuck though
- make3 4y agothis feels really unpythonic, would be very surprised if I saw this in code
- kzrdude 4y agoFor a property setter, you have to give the method the same name as the original property, otherwise it mysteriously breaks. Here for singledispatchmethod they give it the name _. Why no consistency? Such things seem ad-hoc in python. (This criticism comes from an avid Python user.)
- ssl232 4y agoI guess to indicate that the method the @mydispatch.register() decorator wraps will not be added to the class namespace; it will only be accessible via the function wrapped by @singledispatchmethod. As is typical in Python, an underscore indicates an anonymous/unused name.
- kzrdude 4y agoThis explanation doesn't seem to cover the case with @property def foo(self) and the following @foo.setter def foo(self, value) - I wonder why the name of the foo.setter method matters, it doesn't make sense to me.
- ris 4y agoThough I'm guessing it won't get as much use now pattern matching has arrived and is a more general way of achieving the same thing.
- gfaure 4y agoint.bit_count, but str.removesuffix -- consistency has never been a strength of the Python standard library.
- kzrdude 4y agosimilarly, I was just looking at functools.cached_property.. In the usual style of Python, it's a smart special case but not the same as the composition of @functools.cache and @property. Maybe it should have had a different name, for that reason, and the issue would be less prominent.
- ehsankia 4y agoWhat do you mean? That one's a verb and the other isn't? I think it's fine here. The latter returns a modified version of the string, whereas the former is more like a property of the int, like it's "length", although Python also doesn't have .length so I guess that's a bad example. Think of it like datetime.year vs datetime.replace
- rcfox 4y agoPython's style guide prefers words separated by underscores, but makes an exception to match the prevailing style of the module. `removesuffix` is consistent with the naming style of the rest of the `str` methods. `str` is such a fundamental class that it likely predates the current style guide. (All of the `int` named methods seem to be new to Python 3.)
- pansa2 4y ago`bit_count` is a terrible name for a function that returns the number of one-bits. Zero-bits are bits too!
- Waterluvian 4y agoI love the memoization/caching stuff. I know someone will blurt out “Haskell!” or whatnot, but I wish memoization was more first class in popular languages. It’s a great way to maintain performance without having to reason about the implementation details of your code. A decorator in the stdlib gets pretty close.
- cozzyd 4y agoNow I'm wondering, can this be done in c++ with variadic templates?
- dataflow 4y agoThe problem with memoization too easy is that you generally want a sane cache policy for the problem, and that's generally problem dependent. Otherwise you blow up program memory usage.
- aastronaut 4y agoI only need invalidation policies if i want to cache functions that aren't referentially transparent.
- dataflow 4y agoYou also need invalidation policies if the number of items you cache might end up too large in practice. That has nothing to do with referential transparency.
- pen2l 4y agoAre folks still excited about py? I see talk of rust and Julia and it all goes over my head, but when I watch a talk given by Raymond about basic data types and iterators (his creation), I get excited again (I’m glad to see he’s still contributing! Boy does that man have a talent for teaching) I think while the experienced here might scoff at it, it still remains one of the most accessible and -fun- languages. Also I just adore iterators and any language without it is a no-go.
- pid-1 4y agoI am. Python is a joy to work with. Moreover, new features are being delivered at incredible speed.
- throwaway0x7E6 4y ago
- pbourke 4y agoPython just gets out of your way like almost nothing else. I work in graph ML, and Python allows us to glue together runtimes and technologies with ease (eg PySpark).
- sph 4y agoNot that much to be honest. Python is the language that gave dynamic typing its bad name. When I code in Python I must have unit tests even for the simpler of scripts because it's so easy to get a NoneType where you don't want it to. I'm not sure dynamic typing is the fault by itself, I guess it's from a lack of compilation phase, as other languages such as Elixir seems to protect you from typing mistakes much more than Python does. My running theory is that dynamic typing is a sprectrum, and Python is on the side of "it's very easy to get it wrong" compared to other languages. I haven't used the recent versions with type hints, so it might've gotten better. Also, I'm not a fan of how it feels. It is afraid of functional constructs: come on, itertools.accumulate is a simple reduce summing the elements. Why is Python so against map, reduce and ergonomic lambdas? Like Go, it seems Python wants to be simple enough to be understood in large teams of varying developer skills, but unlike Go, Python is much harder to understand and debug in hairier codebases when you add metaclasses, weird backward compatible behaviours, monkey patching, typing woes and an often substandard stdlib.
- thrdbndndn 4y agoQuestion about this part: key = operator.itemgetter("name") people = sorted([p1, p2, p3], key=key) What's the advantage of using operator module instead of just key=lambda x: x['name'] ?
- kzrdude 4y agoIt's faster (better performance). It could also be preferred just as a coding style.
- kzrdude 4y agoThis makes me wonder, couldn't python have a peephole optimization that replaces certain easy labmdas like this with operator functions? Maybe it's too noticeable to work as a transparent optimization (function kind and name etc changes).
- winstonewert 4y agoSuch optimizations can't be transparent in python (at least not easily) because you could very well reassigned everything in `operator` to do whatever you want. PyPy can do it with its jit because it supports invalidating compiled code when such changes are made.
- eesmith 4y agoAssuming the implementation has access to the "real" operator.itemgetter, I can't think of a case where lambda x: x['name'] cannot be replaced by a operator.itemgetter("name") and get a different result. ... except for the stack trace: >>> f = lambda x: x["A"] >>> f({"A": 3}) 3 >>> f({"B": 3}) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <lambda> KeyError: 'A' >>> import operator >>> f = operator.itemgetter("A") >>> f({"A": 3}) 3 >>> f({"B": 3}) Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 'A' So if 'x' implemented its own getitem and it inspected the stack (eg, how Pandas uses the 'level' parameter in 'eval' and 'query' methods to get locals() and globals()), then this change would cause different results.
- est 4y agoI hope the new itertools.pairwise could be added to strings and lists with split() e.g. 'abcdef'.split(2) == ['ab', 'cd', 'ef'] # not the same as .split("2") [1, 2, 3, 4, 5, 6].split(3) == [[1,2,3], [4,5,6]] Or one can use the `grouper` from the recipe https://docs.python.org/3/library/itertools.html#itertools-recipes https://docs.python.org/3/library/itertools.html#itertools-r...
- robertlagrant 4y ago> 'abcdef'.split(2) == ['ab', 'cd', 'ef'] # not the same as .split("2") Pairwise would give you ['ab', 'bc', 'cd', 'de', 'ef']. It's not a split.
- srvmshr 4y agoSimilar: Ned Batchelder recently posted about major Python changes introduced in every point-release - https://nedbatchelder.com/text/which-py.html https://nedbatchelder.com/text/which-py.html
- ehsankia 4y agoI was really excited to see graphlib in stdlib (despite the very limited use so far), but wow is it me or that API is.... awful. I guess it's super focused on parallel task processing but that has to be one of the least Pythonic and user-friendly APIs I've seen. Create a class, then call .prepare(), then loop over .is_active(), what the hell were they thinking? I was hoping to have a nice alternative to networkx, which is big and slow, but this is honestly a mess. Also is ZoneInfo basically stdlib version of pytz? Why not just put it under datetime.timezone like utc?
- petters 4y agoI guess they wanted to raise an exception if the graph has cycles, while at the same time still be able to use the object partially. Then the constructor can not raise. Also, you don't have to call is_active(), the object itself is convertible to bool. Not sure if that is an improvement. graphlib is definitely underwhelming at the moment. A standard Python vocabulary for graphs would be nice.
- lifthrasiir 4y ago> Also is ZoneInfo basically stdlib version of pytz? Why not just put it under datetime.timezone like utc? A good thing about Python is that pretty much every decision and rationale is documented by PEP, and this is no exception: https://peps.python.org/pep-0615/#using-datetime-zoneinfo-instead-of-zoneinfo https://peps.python.org/pep-0615/#using-datetime-zoneinfo-in... In short the heavy implementation of zoneinfo disfavors a nested module, since it is not a norm in Python that importing A doesn't automatically make A.B available as well.
- gorgoiler 4y agoI’ve only read the top level part of the PEP, but it sounds like they’d rather keep the Python source code clean than reuse `datetime.timezone`. They optimised for, what, 20 people instead of optimising for thousands. I am not impressed at all. “Optimising” here is quite a euphemism too. The omission of timezones from the standard library, alongside allowing timezone naive datetimes, has been a major implementation flaw for a long time.
- assbuttbuttass 4y agoI'm disappointed base32hex isn't called duotrigesimal...
- adontz 4y agoOh, come in, who did the graphlib? It's not cycles, it's strongly connected components.
- eterevsky 4y agoI think Python needs much fewer packages in its standard library: - array: everyone uses numpy for the same purposes - bisect: very niche feature, not a good fit for standard library - glob: should probably be part of os - graphlib: why is it part of the standard library? even if you need to work with graphs, chances are you have different requirements or data format - shlex: again, not really necessary in standard library (unless it's used in shutil, I don't know) - statistics: should be handled by scipy And that's just the modules mentioned in this article. There are tons of other modules that are outdated, unused, don't fit into standard library: getopt, curses, urllib (everyone uses requests), xml*, html, tkinter. What Python should be doing is deprecating those modules and/or moving them outside of the standard library. Not adding more of them.
- shiredude95 4y agobinary search is standard imo.
- macleginn 4y agoI agree, Python's stdlib doesn't have a lot of standard algorithms and data structures; bisect is a very welcome exception.
- wenc 4y agoglob is part of pathlib in Python 3. Pathlib is great.
- divbzero 4y agoWhat is the use case for int.bit_count [1] which returns the number of ones in the binary representation of the integer? [1]: https://docs.python.org/3/library/stdtypes.html#int.bit_count https://docs.python.org/3/library/stdtypes.html#int.bit_coun...
- sio8ohPi 4y agohttps://en.wikipedia.org/wiki/Hamming_weight https://en.wikipedia.org/wiki/Hamming_weight I guess it's included as an easy way to access the fast hardware supported implementation, on architectures where it is available.
- saagarjha 4y agoSadly it does not call into the hardware implementation: https://github.com/python/cpython/blob/f62ad4f2c4214fdc05cc45c27a5c068553c7942c/Objects/longobject.c#L5711 https://github.com/python/cpython/blob/f62ad4f2c4214fdc05cc4...
- orlp 4y agoIt does. Note that it calls popcount_digit on each digit of the integer (a 'digit' here is 32 bits), which calls _Py_popcount32, which does map to a hardware instruction if possible: https://github.com/python/cpython/blob/f62ad4f2c4214fdc05cc45c27a5c068553c7942c/Include/internal/pycore_bitutils.h#L95 https://github.com/python/cpython/blob/f62ad4f2c4214fdc05cc4...
- saagarjha 4y agoAh, you are correct. Although I still take issue with "fast" in the comment I was replying to ;)
- newbye3 4y agoIf you are using integers to represent subsets then int.bit_count is the number of elements of the set. There is a machine instruction to count the number of ones in the binary representation, maybe popcnt is the name, I am not sure. Edited, more info: AMD's Barcelona architecture introduced the advanced bit manipulation (ABM) ISA introducing the POPCNT instruction as part of the SSE4a extensions in 2007. Intel Core processors introduced a POPCNT instruction with the SSE4.2 instruction set extension, first available in a Nehalem-based Core i7 processor, released in November 2008. https://www.rosettacode.org/wiki/Population_count https://www.rosettacode.org/wiki/Population_count
- ssl232 4y agoTIL about dict reverse mapping in 3.10. This will be rather useful: people = { "Diane": 70, "Bob": 78, "Emma": 84 } keys = people.keys() # dict_keys(['Diane', 'Bob', 'Emma']) keys.mapping["Bob"] # 78 EDIT: I actually misunderstood this; see comments. The keys.mapping field does not point to the original dictionary. I guess this is actually just about consistency with the way other mappings work? Not so useful (to me) after all.
- matjet 4y agoI do not see the use for this, over the standard access methods. people = { "Diane": 70, "Bob": 78, "Emma": 84 } people.get("Bob") # 78 The common usecase is: for key in people.keys(): people.get(key) vs keys = people.keys() # dict_keys(['Diane', 'Bob', 'Emma']) for key in keys: keys.mapping[key] What is the advantage?
- ssl232 4y agoIt gives you access to the original dictionary from a view. I've not played with it yet but I guess that means you can modify it. EDIT: apparently not: >>> keys.mapping mappingproxy({'Diane': 70, 'Bob': 78, 'Emma': 84}) In this case I actually don't know what the use case is :shrug:
- mjw1007 4y agoThe rationale was here. https://github.com/python/cpython/issues/85067#issuecomment-1093874030 https://github.com/python/cpython/issues/85067#issuecomment-... It seems a bit vague: « Exposing this attribute would help with introspection, making it possible to write efficient functions that operate on dict views. »
- aidos 4y agoI don’t quite get the usecase for it. You’ve got the original dictionary that gives you exactly the same thing. There’s obviously something I’m not thinking of here but I’m not sure why I would want to trade the dictionary for a set of keys, that’s also the dictionary.
- deleted 4y ago[deleted]
- Kalanos 4y agothank you for this practical, hype-free guide!!! couldn't agree more with your sentiment about yawnsync.io. i wonder how the speed/dimensionality of array compares to numpy
- tdstein 4y agoGot ‘em!!! (0.2).as_integer_ratio() # (3602879701896397, 18014398509481984) # oopsie