8 ms·

Boltons: over 100 Python utilities

282 points by mhashemi 12y ago

drawkbox 12y ago
Very nice collection. I like the cacheutils LRI and LRU setups [1]. Interested in the StringBuffer [2]. The Table object looks useful [3]. Overall some great additions in utils that you can tell were added through usage and shipping needs especially in strutils, listutils, formatutils etc. Lots of common ones in our own python or other platform libs. In any review there is always a few WTFs like this one [4]. I am surprised you didn't get arrested during that development [1] https://github.com/mahmoud/boltons/blob/master/boltons/cacheutils.py#L5 https://github.com/mahmoud/boltons/blob/master/boltons/cache... [2] https://github.com/mahmoud/boltons/blob/master/boltons/strutils.py#L282 https://github.com/mahmoud/boltons/blob/master/boltons/strut... [3] https://github.com/mahmoud/boltons/blob/master/boltons/tableutils.py#L180 https://github.com/mahmoud/boltons/blob/master/boltons/table... [4] https://github.com/mahmoud/boltons/blob/master/boltons/easterutils.py https://github.com/mahmoud/boltons/blob/master/boltons/easte...
- mhashemi 12y ago
  Hahaha, so glad someone spotted that. :)
  - RyanMcGreal 12y ago
    What to do about boltons - and you will get boltons.
hamiltonkibbe 12y ago
I'd been wondering how to do this... https://github.com/mahmoud/boltons/blob/master/boltons/easterutils.py https://github.com/mahmoud/boltons/blob/master/boltons/easte...
nuclear_eclipse 12y ago
Really like the concept, and would love to use this in some of my projects, but no Python 3 support.... :'(
- mhashemi 12y ago
  Python 3 support is pretty trivial for all the ones that come to mind. I'll look into it tonight.
- mhashemi 12y ago
  Python 3 support has been added as of Boltons 0.6.
wrsh07 12y ago
Where are the tests?
- mhashemi 12y ago
  They're around. Doctests cover most of the library, many modules have main()s that take care of some more complex unit tests, then the applications that use the utilities cover a lot of regression cases. Tests are kept as close to the code as possible to keep each module independent and self-explanatory, as explained here: https://boltons.readthedocs.org/en/latest/architecture.html https://boltons.readthedocs.org/en/latest/architecture.html
geoelectric 12y ago
Very nice. These solve a number of small but real problems and it looks like a great utility package! I think the docs of iterutils.split() vs. iterutils.split_iter() might be a little off. They both claim to return lists but split_iter() says if you want lists, see split(). Without having installed the package yet, my guess is split_iter() returns an iterator. Either way, the docs are contradictory. https://boltons.readthedocs.org/en/latest/iterutils.html#iteration https://boltons.readthedocs.org/en/latest/iterutils.html#ite... Thanks for this!
- detaro 12y ago
  From my understanding the docs are correct: > split_iter() yields lists of non-separator values. -> returns a generator of lists vs for split() > Returns a list of lists. Note that the examples in the docs always wrap split_iter() in list(), hiding the generator step from the visible output.
- mhashemi 12y ago
  One returns a list of lists, the other an iterator of lists. split_iter() docs say, "split_iter() yields lists of non-separator values," whereas split() docs say, "Returns a list of lists." Still, I'll look at clarifying it. ;)
  - geoelectric 12y ago
    You're absolutely correct. I read "yields" colloquially, not in terms of "yield." Written as designed. :)
pyre 12y ago
This doesn't seem portable: https://github.com/mahmoud/boltons/blob/master/boltons/tests/table_html_app.py#L8 https://github.com/mahmoud/boltons/blob/master/boltons/tests...
- TazeTSchnitzel 12y ago
  That probably doesn't even work on other people's computers.
- mhashemi 12y ago
  Hahaha, true! Get good look, that'll be gone tonight.
- has2k1 12y ago
  Some irony there at clastic [1] "A functional Python web framework that streamlines explicit development practices while eliminating global state." Meanwhile somewhere else, it comes in as an overbearing global solutions provider. Hope that does not spell the end for the table_html utility. 1. https://github.com/mahmoud/clastic https://github.com/mahmoud/clastic
- sciurus 12y ago
  A more permanent link: https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf3d30f20e1fd48209520e/boltons/tests/table_html_app.py#L8 https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf... See https://help.github.com/articles/getting-permanent-links-to-files/ https://help.github.com/articles/getting-permanent-links-to-...
  - pyre 12y ago
    I know about getting permalinks to Github. Also: * You can shorten the commit id: https://github.com/mahmoud/boltons/blob/e5b7582/boltons/tests/table_html_app.py#L8 https://github.com/mahmoud/boltons/blob/e5b7582/boltons/test... * You can highlight ranges: https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf3d30f20e1fd48209520e/boltons/tests/table_html_app.py#L8-L14 https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf... * You can ignore whitespace in diffs: https://github.com/blog/967-github-secrets https://github.com/blog/967-github-secrets
eiopa 12y ago
These are great! You know, this makes me thing about how there are so many things that I wish Python would come with: requests, lxml, gevents and a good amount of stuff from the package. I wish there was Python 2.8.
- XorNot 12y ago
  I wish the regular threading interface was as nice as gevents. Gevents is stunningly simple to use for a lot of simple things where I find trying to use the threading library (I.e. when you want to deal with paramiko) is much harder.
ekimekim 12y ago
So glad someone did something like this and made it presentable enough for widespread usage. In contrast, my own "utils" libraries[1] are often too specific, or not well documented. However, there's already some overlap (eg. I had my own version of funutils.mro_items()), and later today I'll take a look through and see what of mine might be a candidate for inclusion. [1] https://github.com/ekimekim/pylibs/tree/master/libs https://github.com/ekimekim/pylibs/tree/master/libs
- mhashemi 12y ago
  Please do! Quality is not uniform throughout the library, and while mro_items covered my use cases, I fully anticipate others might have more comprehensive approaches.
cool-RR 12y ago
Quite similar to my project, Python Toolbox: https://github.com/cool-RR/python_toolbox https://github.com/cool-RR/python_toolbox It supports Python 3.
- akx 12y ago
  4,800 commits without commit messages? Whoa.
  - cool-RR 12y ago
    Yeah, I get that a lot. I don't believe in commit messages. I know that 99% of developers are disturbed by this, but I think it's the best way to work so I'm willing to deal with people not liking it.
    - michaelmior 12y ago
      Why do you think it's best to give no indication of what was changed by a particular commit? While I feel strongly against this, I'm really asking out of genuine curiousity, not to bash whatever seems to be working for you :)
      - cool-RR 12y ago
        I could write an entire article about this (possibly someday I will), but to explain it briefly: I feel it hurts my creative process to have to take the time to write these commit messages at every commit. It's making me not want to commit. I treat a commit as an extended save operation, so unconsciously avoiding saving is definitely not something you want to do. I know most people would object to my use of a commit as an extended save, but I stand by that opinion as well, especially in these days of deploy-by-commit systems like Heroku. Sometimes I make commits that are smaller than a commit message would be so having to write a message will be a big waste. Also, commit messages are a form of documentation; but they are one of the least useful ones. Much less useful in my opinion than docstrings, topical guides, references and blog posts. So I think that spending time on them is a much lower priority than spending time on the other kinds of documentation.
justanother 12y ago
Is that their real name? Are they related to that singer guy?
lqdc13 12y ago
I like the spirit, however a lot of these can be done with a simple one liner... Example for chunked iteration: ``` def chunk_iter(some_seq, n): return (seq[pos:pos + n] for pos in range(0, len(seq), n)) ``` example for multi-dict: (unless I'm missing something) ``` from collections import defaultdict d = defaultdict(list) d['a'].append('b') d['a'].append('c') dict(d) {'a': ['b', 'c']} ```
- has2k1 12y ago
  While your version are usable. They differ in the requirements fulfilled. e.g. chunk_iter - iterables not just sequences multi-dict - usability Plus nice error messages. The goal is to be standard library quality.
  - mhashemi 12y ago
    That's right! Boltons are meant to replace those one liners, unless you really really think downstream customers are going to be impressed with code golf birdies. ;)
- father_of_two 12y ago
  Here's my own version of an iterating chunker (itertools needed imports omitted). def chunks_of(it, n): return takewhile(len, imap(list, imap(islice, repeat(it), repeat(n)))) But this library functions adds error checking and are more expressive (I wrote the above and always have to look at it lenghty to understand it). I've not paid much attention yet to the license scheme, but if it's friendly to be used on commercial projects, I might take the plunge.
  - mhashemi 12y ago
    BSD, go nuts!
thinxer 12y ago
Python really need something like Go's official libraries. I mean those with a prefix "golang.org/x/". Those libraries update more often than stdlibs but still provide a concrete common ground for developers. It is like "evergreen" stdlibs.
- davidism 12y ago
  This is being talked about now with [PEP 411 - Provisional packages in the Python standard library][1]. [1]: https://www.python.org/dev/peps/pep-0411/ https://www.python.org/dev/peps/pep-0411/
HZet0r 12y ago
There is a LRU cache in the standard library starting in 3.2: https://docs.python.org/3/library/functools.html#functools.lru_cache https://docs.python.org/3/library/functools.html#functools.l... Doesn't have quite the same options, but arguably nicer to use because it's a decorator.
- mhashemi 12y ago
  boltons has an all-purpose cache decorator, which gives you an arrangement with a lot more in terms of flexibility and transparency: https://boltons.readthedocs.org/en/latest/cacheutils.html#automatic-function-caching https://boltons.readthedocs.org/en/latest/cacheutils.html#au...
lifeisstillgood 12y ago
So glad to see this - there are things here I thought I was weird for having to make and want myself - liked chunked iterations. I assumed there as something wrong with me ... Glad to see it's not just me Nice one
asah 12y ago
wow, this is awesome. boltons.tableutils has a lot of TODOs - if you like the concept, the littletable package [1] takes the concept of a python query language pretty far. I put it in production and extended it even further [2]. We'd both welcome help / pull requests. [1] https://www.google.com/search?q=python+littletable https://www.google.com/search?q=python+littletable [2] https://github.com/asah/littletable https://github.com/asah/littletable
_0ffh 12y ago
"The :class:`set` type brings brings the practical expressiveness of set theory to Python. It has a very rich API overall, but lacks a couple of fundamental features. For one, sets are not ordered. On top of this, sets are not indexable" I suppose that may be because sets are not supposed to be ordered. Nor indexable.
- coldtea 12y ago
  In which context? I guess not in math: http://mathworld.wolfram.com/TotallyOrderedSet.html http://mathworld.wolfram.com/TotallyOrderedSet.html
  - _0ffh 12y ago
    You guessed wrong. From your own link: >>A total order (or "totally ordered set," or "linearly ordered set") is a set plus a relation on the set<< Please note that "set" (unqualified) means the general concept of a set. You can extend that general concept, but then what you get is a new thing (in this case a "totally ordered set").
    - coldtea 12y ago
      And who said that our use in Python doesn't need this extended concept?
      - _0ffh 12y ago
        Did anybody say it doesn't? I am sure I didn't. But claiming that Python's implementation of sets "lacks fundamental features" is unfair, wrong, and making the world a dumber place. Some might want to live in that place. I don't.
        coldtea 12y ago
        It seems that your confusion stems from the premise that because this Python collection is called a "set" it should have the absolute minimum functionality of the mathematical notion of a set. Python collections are data structures (containers etc). The standard library is not a symbolic math library, and they are not supposed to model math entities perfectly (e.g. they can me mutable), they are used for their utility, which might or might not need them to be ordered.

282 points by mhashemi 12y ago

Very nice collection. I like the cacheutils LRI and LRU setups [1]. Interested in the StringBuffer [2]. The Table object looks useful [3]. Overall some great additions in utils that you can tell were added through usage and shipping needs especially in strutils, listutils, formatutils etc. Lots of common ones in our own python or other platform libs. In any review there is always a few WTFs like this one [4]. I am surprised you didn't get arrested during that development [1] https://github.com/mahmoud/boltons/blob/master/boltons/cacheutils.py#L5 https://github.com/mahmoud/boltons/blob/master/boltons/cache... [2] https://github.com/mahmoud/boltons/blob/master/boltons/strutils.py#L282 https://github.com/mahmoud/boltons/blob/master/boltons/strut... [3] https://github.com/mahmoud/boltons/blob/master/boltons/tableutils.py#L180 https://github.com/mahmoud/boltons/blob/master/boltons/table... [4] https://github.com/mahmoud/boltons/blob/master/boltons/easterutils.py https://github.com/mahmoud/boltons/blob/master/boltons/easte...

mhashemi 12y ago
Hahaha, so glad someone spotted that. :)
- RyanMcGreal 12y ago
  What to do about boltons - and you will get boltons.

hamiltonkibbe 12y ago

I'd been wondering how to do this... https://github.com/mahmoud/boltons/blob/master/boltons/easterutils.py https://github.com/mahmoud/boltons/blob/master/boltons/easte...

nuclear_eclipse 12y ago

Really like the concept, and would love to use this in some of my projects, but no Python 3 support.... :'(

mhashemi 12y ago
Python 3 support is pretty trivial for all the ones that come to mind. I'll look into it tonight.
mhashemi 12y ago
Python 3 support has been added as of Boltons 0.6.

wrsh07 12y ago

Where are the tests?

mhashemi 12y ago
They're around. Doctests cover most of the library, many modules have main()s that take care of some more complex unit tests, then the applications that use the utilities cover a lot of regression cases. Tests are kept as close to the code as possible to keep each module independent and self-explanatory, as explained here: https://boltons.readthedocs.org/en/latest/architecture.html https://boltons.readthedocs.org/en/latest/architecture.html

geoelectric 12y ago

Very nice. These solve a number of small but real problems and it looks like a great utility package! I think the docs of iterutils.split() vs. iterutils.split_iter() might be a little off. They both claim to return lists but split_iter() says if you want lists, see split(). Without having installed the package yet, my guess is split_iter() returns an iterator. Either way, the docs are contradictory. https://boltons.readthedocs.org/en/latest/iterutils.html#iteration https://boltons.readthedocs.org/en/latest/iterutils.html#ite... Thanks for this!

detaro 12y ago
From my understanding the docs are correct: > split_iter() yields lists of non-separator values. -> returns a generator of lists vs for split() > Returns a list of lists. Note that the examples in the docs always wrap split_iter() in list(), hiding the generator step from the visible output.
mhashemi 12y ago
One returns a list of lists, the other an iterator of lists. split_iter() docs say, "split_iter() yields lists of non-separator values," whereas split() docs say, "Returns a list of lists." Still, I'll look at clarifying it. ;)
- geoelectric 12y ago
  You're absolutely correct. I read "yields" colloquially, not in terms of "yield." Written as designed. :)

pyre 12y ago

This doesn't seem portable: https://github.com/mahmoud/boltons/blob/master/boltons/tests/table_html_app.py#L8 https://github.com/mahmoud/boltons/blob/master/boltons/tests...

TazeTSchnitzel 12y ago
That probably doesn't even work on other people's computers.
mhashemi 12y ago
Hahaha, true! Get good look, that'll be gone tonight.
has2k1 12y ago
Some irony there at clastic [1] "A functional Python web framework that streamlines explicit development practices while eliminating global state." Meanwhile somewhere else, it comes in as an overbearing global solutions provider. Hope that does not spell the end for the table_html utility. 1. https://github.com/mahmoud/clastic https://github.com/mahmoud/clastic
sciurus 12y ago
A more permanent link: https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf3d30f20e1fd48209520e/boltons/tests/table_html_app.py#L8 https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf... See https://help.github.com/articles/getting-permanent-links-to-files/ https://help.github.com/articles/getting-permanent-links-to-...
- pyre 12y ago
  I know about getting permalinks to Github. Also: * You can shorten the commit id: https://github.com/mahmoud/boltons/blob/e5b7582/boltons/tests/table_html_app.py#L8 https://github.com/mahmoud/boltons/blob/e5b7582/boltons/test... * You can highlight ranges: https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf3d30f20e1fd48209520e/boltons/tests/table_html_app.py#L8-L14 https://github.com/mahmoud/boltons/blob/e5b75821781b37ee8ccf... * You can ignore whitespace in diffs: https://github.com/blog/967-github-secrets https://github.com/blog/967-github-secrets

eiopa 12y ago

These are great! You know, this makes me thing about how there are so many things that I wish Python would come with: requests, lxml, gevents and a good amount of stuff from the package. I wish there was Python 2.8.

XorNot 12y ago
I wish the regular threading interface was as nice as gevents. Gevents is stunningly simple to use for a lot of simple things where I find trying to use the threading library (I.e. when you want to deal with paramiko) is much harder.

ekimekim 12y ago

So glad someone did something like this and made it presentable enough for widespread usage. In contrast, my own "utils" libraries[1] are often too specific, or not well documented. However, there's already some overlap (eg. I had my own version of funutils.mro_items()), and later today I'll take a look through and see what of mine might be a candidate for inclusion. [1] https://github.com/ekimekim/pylibs/tree/master/libs https://github.com/ekimekim/pylibs/tree/master/libs

mhashemi 12y ago
Please do! Quality is not uniform throughout the library, and while mro_items covered my use cases, I fully anticipate others might have more comprehensive approaches.

cool-RR 12y ago

Quite similar to my project, Python Toolbox: https://github.com/cool-RR/python_toolbox https://github.com/cool-RR/python_toolbox It supports Python 3.

akx 12y ago
4,800 commits without commit messages? Whoa.
- cool-RR 12y ago
  Yeah, I get that a lot. I don't believe in commit messages. I know that 99% of developers are disturbed by this, but I think it's the best way to work so I'm willing to deal with people not liking it.
  - michaelmior 12y ago
    Why do you think it's best to give no indication of what was changed by a particular commit? While I feel strongly against this, I'm really asking out of genuine curiousity, not to bash whatever seems to be working for you :)
    - cool-RR 12y ago
      I could write an entire article about this (possibly someday I will), but to explain it briefly: I feel it hurts my creative process to have to take the time to write these commit messages at every commit. It's making me not want to commit. I treat a commit as an extended save operation, so unconsciously avoiding saving is definitely not something you want to do. I know most people would object to my use of a commit as an extended save, but I stand by that opinion as well, especially in these days of deploy-by-commit systems like Heroku. Sometimes I make commits that are smaller than a commit message would be so having to write a message will be a big waste. Also, commit messages are a form of documentation; but they are one of the least useful ones. Much less useful in my opinion than docstrings, topical guides, references and blog posts. So I think that spending time on them is a much lower priority than spending time on the other kinds of documentation.

justanother 12y ago

Is that their real name? Are they related to that singer guy?

lqdc13 12y ago

I like the spirit, however a lot of these can be done with a simple one liner... Example for chunked iteration: ``` def chunk_iter(some_seq, n): return (seq[pos:pos + n] for pos in range(0, len(seq), n)) ``` example for multi-dict: (unless I'm missing something) ``` from collections import defaultdict d = defaultdict(list) d['a'].append('b') d['a'].append('c') dict(d) {'a': ['b', 'c']} ```

has2k1 12y ago
While your version are usable. They differ in the requirements fulfilled. e.g. chunk_iter - iterables not just sequences multi-dict - usability Plus nice error messages. The goal is to be standard library quality.
- mhashemi 12y ago
  That's right! Boltons are meant to replace those one liners, unless you really really think downstream customers are going to be impressed with code golf birdies. ;)
father_of_two 12y ago
Here's my own version of an iterating chunker (itertools needed imports omitted). def chunks_of(it, n): return takewhile(len, imap(list, imap(islice, repeat(it), repeat(n)))) But this library functions adds error checking and are more expressive (I wrote the above and always have to look at it lenghty to understand it). I've not paid much attention yet to the license scheme, but if it's friendly to be used on commercial projects, I might take the plunge.
- mhashemi 12y ago
  BSD, go nuts!

thinxer 12y ago

Python really need something like Go's official libraries. I mean those with a prefix "golang.org/x/". Those libraries update more often than stdlibs but still provide a concrete common ground for developers. It is like "evergreen" stdlibs.

davidism 12y ago
This is being talked about now with [PEP 411 - Provisional packages in the Python standard library][1]. [1]: https://www.python.org/dev/peps/pep-0411/ https://www.python.org/dev/peps/pep-0411/

HZet0r 12y ago

There is a LRU cache in the standard library starting in 3.2: https://docs.python.org/3/library/functools.html#functools.lru_cache https://docs.python.org/3/library/functools.html#functools.l... Doesn't have quite the same options, but arguably nicer to use because it's a decorator.

mhashemi 12y ago
boltons has an all-purpose cache decorator, which gives you an arrangement with a lot more in terms of flexibility and transparency: https://boltons.readthedocs.org/en/latest/cacheutils.html#automatic-function-caching https://boltons.readthedocs.org/en/latest/cacheutils.html#au...

lifeisstillgood 12y ago

So glad to see this - there are things here I thought I was weird for having to make and want myself - liked chunked iterations. I assumed there as something wrong with me ... Glad to see it's not just me Nice one

asah 12y ago

wow, this is awesome. boltons.tableutils has a lot of TODOs - if you like the concept, the littletable package [1] takes the concept of a python query language pretty far. I put it in production and extended it even further [2]. We'd both welcome help / pull requests. [1] https://www.google.com/search?q=python+littletable https://www.google.com/search?q=python+littletable [2] https://github.com/asah/littletable https://github.com/asah/littletable

_0ffh 12y ago

"The :class:`set` type brings brings the practical expressiveness of set theory to Python. It has a very rich API overall, but lacks a couple of fundamental features. For one, sets are not ordered. On top of this, sets are not indexable" I suppose that may be because sets are not supposed to be ordered. Nor indexable.

coldtea 12y ago
In which context? I guess not in math: http://mathworld.wolfram.com/TotallyOrderedSet.html http://mathworld.wolfram.com/TotallyOrderedSet.html
- _0ffh 12y ago
  You guessed wrong. From your own link: >>A total order (or "totally ordered set," or "linearly ordered set") is a set plus a relation on the set<< Please note that "set" (unqualified) means the general concept of a set. You can extend that general concept, but then what you get is a new thing (in this case a "totally ordered set").
  - coldtea 12y ago
    And who said that our use in Python doesn't need this extended concept?
    - _0ffh 12y ago
      Did anybody say it doesn't? I am sure I didn't. But claiming that Python's implementation of sets "lacks fundamental features" is unfair, wrong, and making the world a dumber place. Some might want to live in that place. I don't.
      - coldtea 12y ago
        It seems that your confusion stems from the premise that because this Python collection is called a "set" it should have the absolute minimum functionality of the mathematical notion of a set. Python collections are data structures (containers etc). The standard library is not a symbolic math library, and they are not supposed to model math entities perfectly (e.g. they can me mutable), they are used for their utility, which might or might not need them to be ordered.