6 ms·

Widely useful Python utilities by Peter Norvig

95 points by madars 17y ago

th0ma5 17y ago
These are pretty good, some common sense or helper like things that I have implemented here or there at one time or another. Granted, some default programming styles may alleviate the need for some of these things, or ultimately you may want something different in your specific application. Good to see!
arithmetic 17y ago
These are useful! It always helps to take the monotony out of writing code (especially with methods that check boundary conditions and overflows) and spend more time on the actual design.
madars 17y ago
Newer version of these can be found on AIMA-Python site, which is an excellent read alone: http://code.google.com/p/aima-python/source/browse/trunk/utils.py http://code.google.com/p/aima-python/source/browse/trunk/uti...
amackera 17y ago
On a semi-related note, the code for AIMA taught me a lot about Python, just from reading through it at trying to understand. http://code.google.com/p/aima-python/source/browse/#svn/trunk http://code.google.com/p/aima-python/source/browse/#svn/trun...
yummyfajitas 17y ago
A useful utility I've rarely seen included in such a list is inverse_image(collection, function) which computes the inverse image of the function restricted to the collection. A simple example: students_by_grade = inverse_image(students, lambda s: s.grade) Then students_by_gender["A"] contains a list of all the A students (and so on).
- jongraehl 17y ago
  Agreed - sometimes known as "group by".
masklinn 17y ago
Note that more of them have been merged to recent Pythons than just the top: the `dict` constructor can now take kwargs (and a first dict argument, which means you can create an extended copy of dict d with `dict(d, foo='bar', baz='buzz')`), there's a `collections.defaultdict` in 2.5, `namedtuple` in 2.6 is pretty similar to his Struct (though not exactly, `namedtuple` is a type builder, Norvig's `struct` is basically a dict with args instead of items). `every` and `some` can be emulated by composing `map` (or a listcomp) with the builtin `any` and `all` (which, for some reason, don't take an optional predicate)
- smanek 17y ago
  `every` and `some` can be emulated by composing `map` (or a listcomp) with the builtin `any` and `all` (which, for some reason, don't take an optional predicate) No they can't - this version wouldn't short circuit evaluation, which is completely the wrong behavior. Map would have to apply the predicate to every element of the list before any/all kicked in. As a trivial example, consider I had a million item list and want to find out if every element is prime. If 8 is the first element, 'every' would return false after looking at the first item - while your composition would apply the prime predicate to all million elements (which would be ridiculously slow ...) before even starting to apply 'all' (which would short circuit, after seeing the first false).
  - knome 17y ago
    If you have enough elements to desire short circuit evaluation you can, as `masklinn` mentioned, use a list comprehension. Specifically a generator list comprehension. http://www.python.org/dev/peps/pep-0289/ http://www.python.org/dev/peps/pep-0289/ big_list = make_big_list() if any( not prime( a ) for a in big_list ) : do_thing() When you surround the list comprehension syntax with parenthesis instead of square brackets, either as a lone argument to a function or independently, you create a generator that will iterate once over the given data. This allows for functions accepting sequences to short circuit without completing the function mapping.
  - masklinn 17y ago
    > No they can't - this version wouldn't short circuit evaluation, which is completely the wrong behavior. Map would have to apply the predicate to every element of the list before any/all kicked in. Just use the lazy map itertools.imap (or Python 3, where `map` is lazy) or replace listcomps by gencomps, any & all take any iterable if you're worried about that. Stop creating problems where there is none.
spolsky 17y ago
somebody want to explain this? infinity = 1.0e400
- nearestneighbor 17y ago
  the largest representable float in Python is around 10^300 something or so. Anything bigger is considered infinite. Peter Norvig wanted a name for the float bit pattern that represents infinity.
  - mattj 17y ago
    float("inf")?
    - beza1e1 17y ago
      >>> 1.0e400 == float("inf") True I don't know, if there is a difference. Maybe the interpreter can evaluate 1.0e400 a little faster?
peripatetic 17y ago
http://urchin.earth.li/~twic/twic.py http://urchin.earth.li/~twic/twic.py excellent. and great comment: """This is Tom's personal Python subroutine library. You are not expected to understand this. Warning: this file contains HIGH MAGIC and DEEP VOODOO, as well as quite a bit of QUESTIONABLE CODE and very few COMMENTS. Use with caution. ha!
- diN0bot 17y ago
  so....what's excellent about it.... or do you mean you find it useful to use? i'm personally very cautious about introducing "magic" and "voodoo" into my repositories. i many times have to tweak libraries (add features, fix bugs), so it's worth them making sense without me having to refactor them first (i did this with qunit, tho...it's hard to generalize, eh?). more importantly, it's important to me that new developers ramp up fast. there's something about collaboration that many lone programmers don't grok. similarly, the code must make sense to me next month. extensibility and maintainability are almost inversely related to voodoo and magic. good abstraction, handling complexity through elegance, now these are worth having. it can be quite hard separating good programming from bad. it is even harder when the work itself is complex in nature, although that is when good programming really shines.

95 points by madars 17y ago

These are pretty good, some common sense or helper like things that I have implemented here or there at one time or another. Granted, some default programming styles may alleviate the need for some of these things, or ultimately you may want something different in your specific application. Good to see!

arithmetic 17y ago

These are useful! It always helps to take the monotony out of writing code (especially with methods that check boundary conditions and overflows) and spend more time on the actual design.

madars 17y ago

Newer version of these can be found on AIMA-Python site, which is an excellent read alone: http://code.google.com/p/aima-python/source/browse/trunk/utils.py http://code.google.com/p/aima-python/source/browse/trunk/uti...

amackera 17y ago

On a semi-related note, the code for AIMA taught me a lot about Python, just from reading through it at trying to understand. http://code.google.com/p/aima-python/source/browse/#svn/trunk http://code.google.com/p/aima-python/source/browse/#svn/trun...

yummyfajitas 17y ago

A useful utility I've rarely seen included in such a list is inverse_image(collection, function) which computes the inverse image of the function restricted to the collection. A simple example: students_by_grade = inverse_image(students, lambda s: s.grade) Then students_by_gender["A"] contains a list of all the A students (and so on).

jongraehl 17y ago
Agreed - sometimes known as "group by".

masklinn 17y ago

Note that more of them have been merged to recent Pythons than just the top: the `dict` constructor can now take kwargs (and a first dict argument, which means you can create an extended copy of dict d with `dict(d, foo='bar', baz='buzz')`), there's a `collections.defaultdict` in 2.5, `namedtuple` in 2.6 is pretty similar to his Struct (though not exactly, `namedtuple` is a type builder, Norvig's `struct` is basically a dict with args instead of items). `every` and `some` can be emulated by composing `map` (or a listcomp) with the builtin `any` and `all` (which, for some reason, don't take an optional predicate)

smanek 17y ago
`every` and `some` can be emulated by composing `map` (or a listcomp) with the builtin `any` and `all` (which, for some reason, don't take an optional predicate) No they can't - this version wouldn't short circuit evaluation, which is completely the wrong behavior. Map would have to apply the predicate to every element of the list before any/all kicked in. As a trivial example, consider I had a million item list and want to find out if every element is prime. If 8 is the first element, 'every' would return false after looking at the first item - while your composition would apply the prime predicate to all million elements (which would be ridiculously slow ...) before even starting to apply 'all' (which would short circuit, after seeing the first false).
- knome 17y ago
  If you have enough elements to desire short circuit evaluation you can, as `masklinn` mentioned, use a list comprehension. Specifically a generator list comprehension. http://www.python.org/dev/peps/pep-0289/ http://www.python.org/dev/peps/pep-0289/ big_list = make_big_list() if any( not prime( a ) for a in big_list ) : do_thing() When you surround the list comprehension syntax with parenthesis instead of square brackets, either as a lone argument to a function or independently, you create a generator that will iterate once over the given data. This allows for functions accepting sequences to short circuit without completing the function mapping.
- masklinn 17y ago
  > No they can't - this version wouldn't short circuit evaluation, which is completely the wrong behavior. Map would have to apply the predicate to every element of the list before any/all kicked in. Just use the lazy map itertools.imap (or Python 3, where `map` is lazy) or replace listcomps by gencomps, any & all take any iterable if you're worried about that. Stop creating problems where there is none.

spolsky 17y ago

somebody want to explain this? infinity = 1.0e400

nearestneighbor 17y ago
the largest representable float in Python is around 10^300 something or so. Anything bigger is considered infinite. Peter Norvig wanted a name for the float bit pattern that represents infinity.
- mattj 17y ago
  float("inf")?
  - beza1e1 17y ago
    >>> 1.0e400 == float("inf") True I don't know, if there is a difference. Maybe the interpreter can evaluate 1.0e400 a little faster?

peripatetic 17y ago

http://urchin.earth.li/~twic/twic.py http://urchin.earth.li/~twic/twic.py excellent. and great comment: """This is Tom's personal Python subroutine library. You are not expected to understand this. Warning: this file contains HIGH MAGIC and DEEP VOODOO, as well as quite a bit of QUESTIONABLE CODE and very few COMMENTS. Use with caution. ha!

diN0bot 17y ago
so....what's excellent about it.... or do you mean you find it useful to use? i'm personally very cautious about introducing "magic" and "voodoo" into my repositories. i many times have to tweak libraries (add features, fix bugs), so it's worth them making sense without me having to refactor them first (i did this with qunit, tho...it's hard to generalize, eh?). more importantly, it's important to me that new developers ramp up fast. there's something about collaboration that many lone programmers don't grok. similarly, the code must make sense to me next month. extensibility and maintainability are almost inversely related to voodoo and magic. good abstraction, handling complexity through elegance, now these are worth having. it can be quite hard separating good programming from bad. it is even harder when the work itself is complex in nature, although that is when good programming really shines.