4 ms·
While everyone is on the subject, I thought I'd mention some weird behavior I saw today. I just switched my program from using a defaultdict to a regular dict.
by mrfusion 13y ago
While everyone is on the subject, I thought I'd mention some weird behavior I saw today.
I just switched my program from using a defaultdict to a regular dict.
I.e., from defaultdict(lambda:'NA') to regular dict using get(val, 'NA') for access.
And I got a something like a 100x speedup. It runs in two minutes instead of two hours. I had no idea a defaultdict would be so much slower.
Unless there's something funny going on in my program and it's unusual behavior.
- zo1 13y agoThe docs for the defaultdict are here: https://docs.python.org/2/library/collections.html#collections.defaultdict https://docs.python.org/2/library/collections.html#collectio... And they pretty much say that there is only one method that's been overridden from the normal dict class. And all it does is it executes the anonymous method (lamda) you supplied instead of returning the default value you pass in. Also, this depends on your usage, and what you have in the lambda expression. I'm curious about your code, care to paste/link some of it for us?
- cefstat 13y agoI would guess that the code tries to get the value for keys that do not exist in the defaultdict. Well, if the key does not exist then it is being added with the default value 'NA'. So in this way you can end up with a huge dictionary where most of the values are just 'NA'.
- mrfusion 13y agoOh, wait, so defaultdicts actually insert missed keys?? I thought they simply gave you a default value instead of a keyerror. If you're right, then that would certainly explain it.
- bryanh 13y agoYep, take a look! >>> import collections >>> dd = collections.defaultdict(int) >>> print dd['doesnt exist'] 0 >>> print dd defaultdict(<type 'int'>, {'doesnt exist': 0})
- kansface 13y agoUse .get() instead of [] to avoid setting the default value.
- ForHackernews 13y agoThat's what they said they're doing.
- zo1 13y agoCorrect. The behavior you are looking for is already in the standard dict class. You just pass in an extra parameter to the get method like so: dict.get("key", "my_default_value_if key_is_not_inserted_already")
- Someone 13y agoThey have to, because they cannot, in general, decide whether the lambda you gave it is idempotent _and_ returns an immutable value. For example, if you provide a function that returns an empty list, and do: l = foo['bar'] l.append('baz') print len(foo['bar']) That should print 2 (apologies if this isn't correct Python) If one had a defaultdict that took a default value in a language with enough reflection, you might be able to deduce that the lambda always returns a simple value such as 3.1415927, and not store copies in the dictionary.
- maxerickson 13y agoWhat you have there will print 1.
- deleted 13y ago[deleted]
- deleted 13y ago[deleted]
- deleted 13y ago[deleted]
- cynwoody 13y agoIf you are predominately adding entries to a dictionary, the standard dict will be faster. Where defaultdict shines is the case where you hit an entry a bunch of times, but you'd like to automate initializing an entry the first time it is hit. The most common example is probably the frequency distribution, where the frequency needs to start out zero and be incremented each time a key is seen. Using int or lambda:0 as defaultdict's argument works fine for the simple case. If you want to tabulate more than just the count, e.g. a count and one or more sums, you can pass a constructor that produces an object with an update method, to which you pass whatever your key represents. Simple microbenchmark: http://pastebin.com/vkacByzp http://pastebin.com/vkacByzp It appears that defaultdict saves time as well as code for the normal use case.