10 ms·
Performance Analysis of Python's Dict() and {}
- pphysch 3y agoThe first "benchmark" shows dict() being twice as fast, but the rest of the article concludes "The {} is always faster than dict."
- michaelhoffman 3y agoI wonder if they swapped the benchmark results somehow. Because on my computer, I get roughly the same times but with {} twice as fast as dict().
- nathiss 3y agoYes, you're absolutely right. I apologize for this copy-paste mistake. It's fixed now.
- Epa095 3y agoDamb am I happy you commented this! I saw the same, checked it twice, and a final time before posting. But then the results were suddenly as expected (dict() being slower). At least I am not going mad, he must have changed it just now.
- bigbillheck 3y agoThe difference is 20ns, and if that's enough for you to care about you've got much bigger worries (like "using python").
- IshKebab 3y agoI agree. If you're benchmarking this then you're using the wrong language. But I hadn't considered the fact that `dict` can be overridden. That was interesting.
- frakt0x90 3y agoI guess I see this post as more about profiling, discovering unintuitive differences, and exploring python internals than offering 20ns optimizations. I personally appreciated it.
- sjwhevvvvvsj 3y ago(+20ns x # calls made) can be significant if you’re dealing with billions of executions though. If nothing else a search/replace for “dict()” to “{}” could yield an aggregate performance bump with basically zero refactoring cost. Also re Python in general: love it or hate it, it “won” as the language of ML. If you want to use ML libraries you’ll be in python.
- ayhanfuat 3y agoIf you are building billions of dicts, the bottleneck will not be how you initialize them as empty dicts for sure.
- sjwhevvvvvsj 3y agoDistributed systems can get very big.
- arsome 3y agoYeah I would be much more concerned about things which cause exponential time increases, like developers who brute force search lists instead of using dictionaries than how they declare their dictionaries unless I was at the point of profiling and the dict() constructor was somehow near the top of the list.
- queuebert 3y agoI have a book on the shelf called "High Performance Python". Seriously, though, if every Python program in the world was 0.1% faster, that would probably save a lot of energy.
- viraptor 3y agoYou have to compare that to the amount of energy spent while making it 0.1% faster. On average, I'm not sure we'd be saving any energy. The one bit that would get any remotely relevant gain would be offset by lots of tools where that improvement happens once a week. How many minutes / how much energy would it take you to setup a project, run a profiler, do the fix, run tests/CI, commit, release, etc. -vs- how much energy would that 0.1% change save over years?
- nomel 3y agoBring globals into local scope, never use '.' in tight loops, avoid function calls, etc [1], sure make ugly python code. But, wow, it really can make a difference. [1] https://wiki.python.org/moin/PythonSpeed/PerformanceTips https://wiki.python.org/moin/PythonSpeed/PerformanceTips
- physPop 3y agoI'm not sure how many of those are valid any more.
- ezo 3y agoI’m wonder, who’s thinking that dict() is more readable???
- shoo 3y ago`{x, y}` is the set containing the two elements `x` and `y`, `{x}` is the set containing only the element `x`, so "surely `{}` is the empty set" !
- pdonis 3y agoIf Python had had built-in set notation from the start, {} might indeed have been a notation for an empty set instead of an empty dict. However, Python didn't even have sets at all until version 2.3, and they were in a stdlib module instead of being a built-in type until version 2.6. By that time dict notation was well entrenched.
- colpabar 3y agoYou could argue that since `{}` is also technically an empty set, `dict()` is more explicit.
- masklinn 3y agoExcept `{}` is never an empty set.
- colpabar 3y ago> x = set() > x set() Well now I know! However I still think the shared usage of curly braces for dicts and sets could be somewhat confusing. > x = {1, 2, 3} > x {1, 2, 3} (But maybe no one should listen to me because I don't know how to do code blocks on HN.)
- tczMUFlmoNk 3y agoIndent by four spaces. :-)
- trostaft 3y agoUnless I've misunderstood the comparison, the lists example does not appear to be comparing apples to apples. Reproducing here: $ python -m timeit "list((1, 2, 'a'))" 5000000 loops, best of 5: 53 nsec per loop $ python -m timeit "[(1, 2, 'a')]" 10000000 loops, best of 5: 30.4 nsec per loop The first command produces a list of three elements, whereas the second produces a list of one element (being the tuple).
- masklinn 3y agoYou are correct. To µbench this, you'd want to use `-s` to set up the base object which both snippets then convert. > python -m timeit -s "t = (1, 2, 'a')" "list(t)" 10000000 loops, best of 5: 39.8 nsec per loop > python -m timeit -s "t = (1, 2, 'a')" "[*t]" 10000000 loops, best of 5: 25.1 nsec per loop > python -m timeit -s "[1, 2, 'a']" 50000000 loops, best of 5: 4.28 nsec per loop (this is 3.11.2 on an M1 Pro)
- charlieyu1 3y agoThe third method should be clearly the fastest one, the other two we are building a tuple then convert to a list
- masklinn 3y ago`-s` stands for "setup", the building of the tuple is only done once, and it is not part of the benchmark. All three versions only bench the construction of the list, two of them from a tuple, while the third is a list literal. However if you plug it into `dis` you'll see that it compiles to loading a const tuple and creating a list from that >>> dis.dis("[1, 2, 'a']") 0 0 RESUME 0 1 2 BUILD_LIST 0 4 LOAD_CONST 0 ((1, 2, 'a')) 6 LIST_EXTEND 1 8 RETURN_VALUE >>> dis.dis("[*t]") 0 0 RESUME 0 1 2 BUILD_LIST 0 4 LOAD_NAME 0 (t) 6 LIST_EXTEND 1 8 RETURN_VALUE
- 3y ago
- t8sr 3y agoWhile it points at interesting questions about Python's internals, I hope people writing Python realize that optimizing it is pointless, except for cases where you change the complexity class of an algorithm. The performance of pure python code is orders of magnitude worse than non-interpreted languages, there's no point trying to shave off 0.5% off a 5000% difference.
- pyuser583 3y agoIt’s also pointless because Python internals have no guarantees. Changing version or even platform can change the absolute efficiency.
- H8crilA 3y ago(I generally agree that everyone should almost always choose the more readable version) In this case the speed difference will likely always be there, since `dict()` can be overriden (monkey-patched) - hence the interpreter needs to resolve it at runtime.
- pyuser583 3y agoIsn’t some logic necessary to determine if you’re dealing with a dictionary or set? Or dictionary/set comprehension? You could probably override it using Python’s extension system. Forgive me, I know Python very well, but not C/C++. I’ve learned to make no assumptions.
- H8crilA 3y agoCheck out the Python bytecode in the article, and remember that bytecode is only generated once from the text of the program (but executed many times). "{}" is translated into a simple "make a dict" instruction.
- Spivak 3y agoThis logic makes no sense, you're saying that trying to boost performance by small increments 0-10% isn't worthwhile on a Civic because it'll never be a Bugatti? That's the least helpful advice when you have a real-life application written in Python and want to get some easy wins on your tight loops. Also Python internals do have some guarantees but in this case it's a semantic guarantee. Because builtins can be shadowed you'll always have to pay the performance cost of looking them up which isn't true for {}.
- sweezyjeezy 3y agoI don't think this will refute the article, but I would have found it more convincing if the benchmark had included a single setitem assignment as well, to be sure that the difference wasn't python doing a lazy dict-or-set type assignment on {}
- extasia 3y agoI've also had this thought, but found that inspecting the type shows its by default a dictionary, and that it only is interpreted as a set if you treat it as such (eg add comma-seperated elements when instantiating) assert type({}) == type(dict())) assert type({1,2,3} == type(set())
- ayhanfuat 3y agoFrom Alex Martelli , a prominent Python expert [1]: > I'm one of those who prefers words to punctuation -- it's one of the reasons I've picked Python over Perl, for example. "Life is better without braces" (an old Python motto which went on a T-shirt with a cartoon of a smiling teenager;-), after all (originally intended to refer to braces vs indentation for grouping, of course, but, hey, braces are braces!-). > "Paying" some nanoseconds (for the purpose of using a clear, readable short word instead of braces, brackets and whatnots) is generally affordable (it's mostly the cost of lookups into the built-ins' namespace, a price you pay every time you use a built-in type or function, and you can mildly optimize it back by hoisting some lookups out of loops). > So, I'm generally the one who likes to write dict() for {}, list(L) in lieu of L[:] as well as list() for [], tuple() for (), and so on -- just a general style preference for pronounceable code. When I work on an existing codebase that uses a different style, or when my teammates in a new project have strong preferences the other way, I can accept that, of course (not without attempting a little evangelizing in the case of the teammates, though;-). [1] https://stackoverflow.com/a/2745292/2285236 https://stackoverflow.com/a/2745292/2285236
- mgraczyk 3y agoI find {} much more readable then dict() because the former is more common. On the other hand list(L) more readable than L[:] because the fact that slicing copies is more obscure, and these are not equivalent when L is not a list. My take would be to ignore performance and generally do what is more commonly done.
- nephanth 3y agoMy problem with {} is that the syntax for dicts {x1:y1, x2:y2 ...} and sets {x1, x2 ...} Are very close, and {} is slightly ambiguous. I've run into bugs because i wrote {} for the empty set. Since then i write dict() and set()
- xarope 3y ago+1. I've run into this too, and ended up using set() to clarify.
- mgaunard 3y agodoes your colleague also uses list instead of []? If you're in Python, you should embrace the syntax.
- justinl33 3y agoBest way to optimize a python dict is to switch languages and use a dict there
- paulddraper 3y agoYour perf will get killed by overhead.
- __loam 3y agoPython is filled with these kinds of traps where there's more than one way to do the same thing but one way is faster for no reason. It's such a bad language.
- encomiast 3y agoCompared to what?
- __loam 3y agoGo for example has fewer ways to write things. With Python, you have to guess what the idiomatic pattern is and it's often not obvious and specific to python. Go went as far removing the ternary operator to cut down on this kind of thing. Writing things in the idiomatic way is more obvious because there's usually only one way to do a given thing.
- Sohcahtoa82 3y ago> With Python, you have to guess what the idiomatic pattern is and it's often not obvious and specific to python. This is true of every language. You just don't know Python. You probably decided a long time ago that you hate it because of X reason (usually whitespace-as-syntax, as that's probably the most controversial Python design choice), and so refuse to learn anything more and decide you just hate everything. > Go went as far removing the ternary operator to cut down on this kind of thing. It surprises me when people get confused by the ternary operator. I never thought it was hard. I wish Python had it. Yes, I know, Python has "a = b if c else d", but that's just weird as it flips the order of expressions around to something that doesn't make much sense.
- __loam 3y agoI've been using python for 11 years bud. I know all the idioms. When I say it's a bad language, it's coming from years of experience using it in production environments. But yeah the problem is I never learned a language I've probably written more code in than any other language. The ternary is just one example. It's not confusing, it's just indicative of the philosophy of python's design, and it's hardly the worst example. E: it's actually been 11 years. I'm getting old
- wyldfire 3y agoA previous analysis of the same question (2.7, 2012): https://doughellmann.com/posts/the-performance-impact-of-using-dict-instead-of-in-cpython-2-7-2/ https://doughellmann.com/posts/the-performance-impact-of-usi...
- Cannabat 3y agoIf you want to implement the suggested changes across your codebase, you can use the `flake8` plugin `flake8-comprehensions`, or `ruff`'s `C4` ruleset, which is an implementation of `flake8-comprehensions`.