6 ms·
I find defaultdict, OrderedDict, namedtuple among other data structures/classes in the collections module to be incredibly useful. Another module that's packag
by judicious 2y ago
I find defaultdict, OrderedDict, namedtuple among other data structures/classes in the collections module to be incredibly useful.
Another module that's packaged with the stdlib that's immensely useful is itertools. I especially find takewhile, cycle, and chain to be incredibly useful building blocks for list-related functions. I highly recommend a quick read.
EDIT: functools is also great! Fantastic module for higher-order functions on callable objects.
https://docs.python.org/3/library/itertools.html https://docs.python.org/3/library/itertools.html
- padthai 2y agoWhy do you use OrderedDict for now that regular dicts are ordered by default?
- judicious 2y agoI work with different versions of Python3 (and 2 unfortunately) and some code is still in 3.6, hence I used OrderedDicts.
- mixmastamyk 2y ago3.6 was the first with the new ordered by default dicts, even though wasn't specc'd until 3.7.
- Izkata 2y agoIt worked as an accidental implementation detail in CPython from some other optimization, but it wasn't intentional at the time. Because it wasn't intentional and wasn't part of the spec, that code could be incompatible with other interpreters like pypy or jython.
- ericvsmith 2y agoSee my comment and the linked email at https://github.com/ericvsmith/dataclasses?tab=readme-ov-file#compatibility https://github.com/ericvsmith/dataclasses?tab=readme-ov-file... for dataclasses and 3.6. I think it's still true.
- raymondh 2y agoThe reason Guido didn't want 3.6 to guarantee dict ordering was to protect 3.5 projects from mysteriously failing when using code that implicitly relied on 3.6 behaviors (for example, cutting and pasting a snippet from StackOverflow). He thought that one cycle of "no ordering assumptions" would give a smoother transition. All 3.6 implementations would have dict ordering, but it was safer to not have people rely on it right away.
- masklinn 2y agopypy implemented naturally ordered dict before cpython did. jython never released a P3 version so is irrelevant, ironpython has yet to progress beyond 3.4 so is also irrelevant.
- sgarland 2y agoAs someone who just had to backport a fairly large script to support 3.6, I found myself surprised at how much had changed. Dataclasses? Nope. `__future__.annotations`? Nope. `namedtuple.defaults`? Nope. It's also been frustrating with the lack of tooling support. I mean, I get it – it's hideously EOL'd – but I can't use Poetry, uv, pytest... at least it still has type hints.
- neves 2y agoNot even VSCode extension works anymore
- heavyset_go 2y agoOrderedDicts have some convenience methods and features that ordinary dicts don't have.
- rbanffy 2y agoAlso, dicts can become unordered at any time in the future. Right now the OrderedDict implementation is a thin layer over dict, but there are no guarantees it’ll always be that.
- wodenokoto 2y agoThey can, but ordered dict can also become unordered in the future, should the steering committee decide. But seriously: It’s no longer an implementation detail that dictionaries are ordered in Python. It’s a specification of how Python works.
- rbanffy 2y agoI missed that in the 3.7 release notes.
- judicious 2y agoThere in lies another reason why OrderedDicts are still useful even in 3.12
- rbanffy 2y agoNot really. It was pointed out that since 3.7 the order preserving behaviour is part of the spec for dicts.
- judicious 2y agoI guess for most purposes, OrderedDicts are then obsolete, but I believe there are some extra convenience methods that they have, but I've only really needed to preserve order. Makes you think what other parts of Python have become obsolete.
- d0mine 2y agoIt may be more explicit: OrderedDict has move_to_end() which may be useful e.g., for implementing lru_cache-like functionality (like deque.rotate but with arbitrary keys).
- masklinn 2y agoOTOH that’s a lot less useful now that functools.lru_cache exists: it’s more specialised so it’s lighter, more efficient, and thread-safe. So unless you have extended flexibility requirements around your LRU, OD loses a lot there. And if you’re using a FIFO cache, threading a regular dict through a separate fifo (whether linked list or deque) is more efficient in my experience of implementing both S3 and Sieve.
- Flimm 2y agoTwo dictionaries with equal keys and values are considered equal in Python, even if the order of the entries differ. By contrast, two OrderedDict objects are only equal if their respective entries are equal and if their order does not differ.
- BerislavLopac 2y agoChainMap might be the most underrated bit in the standard library.
- stevesimmons 2y agoFor anyone wanting some more explanation, ChainMap can be used to build nested namespaces from a series of dicts without having to explicitly merge the names in each level. Updates to the whole ChainMap go into the top-level dict. The docs are here [0]. Some simple motivating applications: - Look up names in Python locals before globals before built-in functions: `pylookup = ChainMap(locals(), globals(), vars(builtins))` - Get config variables from various sources in priority order: `var_map = ChainMap(command_line_args, os.environ, defaults)` - Simulate layered filesystems - etc [0] https://docs.python.org/3/library/collections.html#collections.ChainMap https://docs.python.org/3/library/collections.html#collectio...
- BerislavLopac 2y agoI found it perfect for structured logging, where you might want to modify some details of the logged structures (e.g. a password) without changing the underlying data.
- mont_tag 2y agoMy faves are the lru_cache, namedtuples, deques, chainmap, and all of the itertools.
- sevensor 2y agoI mostly migrated to frozen dataclasses from namedtuples when dataclasses became available. I’m curious about your preference for the namedtuple. Is it the lighter weight, the strong immutability, the easy destructing? Or is it that most tuples might as well be namedtuples? Those are the advantages I can think of anyway :)
- sgarland 2y agoThe main thing I find myself using them for is `_make()`. From the canonical [0] example: import sqlite3 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade') conn = sqlite3.connect('/companydata') cursor = conn.cursor() cursor.execute('SELECT name, age, title, department, paygrade FROM employees') for emp in map(EmployeeRecord._make, cursor.fetchall()): print(emp.name, emp.title) You could of course accomplish the same with a dictionary comprehension, but I find this to be less noisy. Also, they have `_asdict()` should you want to have the contents as a dict. [0]: https://docs.python.org/3/library/collections.html#collections.namedtuple https://docs.python.org/3/library/collections.html#collectio...
- gcr 2y agoHoly shit that’s really clever. Didn’t know about _make, thank you!
- sgarland 2y agoI didn’t either until I read docs tbf. It’s just kind of thrown in as an afterthought for the section, too.
- judicious 2y agoDictionary comprehensions can be very elegant. List and dictionary comprehensions are very powerful and expressive abstractions. In fact, while not good practice you can pretty much write all Python code inside comprehensions including stuff regarding mutation. This is valid(as in it will run, but highly unidiomatic) code: quicksort = lambda arr: [pivot:=arr[0], left:= [x for x in arr[1:] if x < pivot], right := [x for x in arr[1:] if x >= pivot], quicksort(left) + [pivot] + quicksort(right)][-1] if len(arr) > 1 else arr print(quicksort([1, 33, -4, -2, 110, 5, 88]))
- tpoacher 2y agoalso more_itertools ! even less known than itertools, but equally useful.
- matsemann 2y agoI just wish python had some better ergonomics/syntactic sugar working with itertools and friends. Grouping and mapping and filtering and stuff quickly become so unwieldy without proper lambdas etc, especially as the typing is quite bad so after a few steps you're not sure what you even have. Just as recent as today I went to Kotlin to process something semicomplex even though we're a python shop, just because I wanted to bash my head in after a few attempts in python. A DS could probably solve it minutes with pandas or something, but again stringly typed and lots of guesswork. (It was actually a friendly algorithmic competition at work, I won, and even found a bug in the organizer's code that went undetected exactly because of this)
- judicious 2y agoI find converting things from map objects or filter objects back to lists to be a bit clunky. Not to mention chaining operations makes it even more clunky. Some syntatic sugar would go a long way.
- mturmon 2y agoI use defaultdict a lot - for accumulators when you're not sure about what is coming. Here's a simplified example: # a[star_name][instrument] = set of (seed, planet index) of visited planets a = defaultdict(lambda: defaultdict(set)) for row in rows: a[row.star][row.inst].add((row.seed, row.planet)) This is a dict-of-dict-of-set that is accumulating from a stream of rows, and I don't know what stars and instruments will be present. Another related tool is Counter (https://docs.python.org/3/library/collections.html#collections.Counter https://docs.python.org/3/library/collections.html#collectio...)
- daniel_grady 2y agoAlthough it's not part of the standard library, toolz is wonderful for rounding out these modules. https://toolz.readthedocs.io/en/latest/ https://toolz.readthedocs.io/en/latest/