14 ms·

Ask HN: Good Python codebases to read?

278 points by nekopa 11y ago

Hi all, I am currently going through Learn Python the Hard Way, and am on the review section where I have to read through source code. Can anyone recommend some good open source Python software I can look at? I am specifically looking to see ones that employ idiomatic Python, and maybe see how they approach testing (I am not completely new to programming, just rusty after being out of the field for a long time) Extra bonus points if the software is something you use regularly. Cheers!

nyddle 11y ago
Flask - https://github.com/mitsuhiko/flask https://github.com/mitsuhiko/flask. It's small, awesome and digestible.
- nekopa 11y ago
  Thanks! I am also looking into using Flask soon too. I dabbled in django for a while, but too much magic was happening and I am trying to learn.
  - collyw 11y ago
    Django is quite transparent when you learn enough of it. But there is a lot to learn.
    - nekopa 11y ago
      I was talking about just using django to make a web app. It is awesome in that regard. But if I'm making a web app just for learning about the issues regarding making web apps, django is not suitable because that's it's job :) But I am hoping to move into django eventually, but first I want tknow what is going on behind the curtain.
      - gknoy 11y ago
        I agree, Django is likely not the best if you're just starting learning, and writing a webapp from scratch. Working in an existing codebase, with other devs of whom you can ask questions, is a luxury that I was very thankful for. I use Django at work, and it was my first time with Python. The beauty of Django is, IT'S ALL SOURCE. When you run it locally, you have TONS of learning options: - Get an IDE, and read the sources if you don't understand how something works. "Find-Definition" all the way down. (I heart PyCharm.) - If something's broken, you can edit it! - You can put in `import ipdb; ipdb.set_trace()` calls anywhere and get a debugging prompt! (If you don't have IPython, you can use `pdb` instead of `ipdb`... shudder.) Being able to print What Actually Gets Passed Around can occasionally be very helpful. - You can put in debugging messages at multiple points in the flow! - make a `pygrep` alias to answer "where is ..." questions: # ignore south migrations, and external libraries. # You can alter this if you care about reading libs' codebase alias pygrep='grep -rin --include=*.py --exclude=*.pyc --exclude-dir=lib --exclude-dir=migrations'
- resc1440 11y ago
  Also MarkupSafe - https://github.com/mitsuhiko/markupsafe https://github.com/mitsuhiko/markupsafe It's a little bit meta, but I had lots of "wow" moments. Also, it's a nice example of using C to speed up certain operations.
  - nekopa 11y ago
    Thanks for this recommendation. I don't know if I will check it out right now, but it is on my list now because I am hoping to integrate C in Python when needs be. But I don't want to jump the gun :)
    - njbooher 11y ago
      For that you might want to look at Cython.
- liviu- 11y ago
  Armin Ronacher often writes pretty and compact Python — https://github.com/mitsuhiko https://github.com/mitsuhiko
- Kortaggio 11y ago
  Flask was the framework that finally made web frameworks "click" for me - the biggest advantage of learning about Flask is you can build up your knowledge of web services piece by piece, allowing you to take time and fully understand each component before moving on to the next. That way you're not trying to comprehend all the "magic" that's going on behind the scenes all at once.
clinth 11y ago
Requests - https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests. How to make a usable api. The decisions that went into each method call were fantastic. Great test coverage as well. I use package in most python development.
- nekopa 11y ago
  Thanks, I've been hearing a lot about this package, I think it's one I will check out. Cheers!
  - bbrazil 11y ago
    This presentation by the author explains the thinking behind it: https://speakerdeck.com/kennethreitz/python-for-humans https://speakerdeck.com/kennethreitz/python-for-humans
- Myrmornis 11y ago
  requests is very useful, but it always gets mentioned as a good python codebase and I'm not sure I agree. One example: The first thing many users will do is requests.get? Which tells them that it takes some kwargs, but doesn't tell them what those kwargs are. It's easy, especially for a newcomer, to read "optional arguments that `request` takes" and fail to understand that they should look up the docs on (not-really-encouraged-as-part-of-public-API) function `request`. That's pretty bad; those kwargs are important! (The reason is because requests.get is implemented as a call to request(method, ..., kwargs) but the user doesn't care what the implementation-level reason is.) Beyond that I did look into the codebase once to investigate a possible bug and there were a few python style things I wanted to fix, but I don't remember them so this comment probably sounds kind of annoying (it would annoy me if I were reading it not writing it...). It didn't strike me a really clean codebase. But yes the library is very useful and I'm sure it's a pretty decent python codebase. >>> requests.get? Signature: requests.get(url, params=None, **kwargs) Docstring: Sends a GET request. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param \*\*kwargs: Optional arguments that ``request`` takes. Here are all the kwargs the user probably wanted to know but failed to find out and was forced to either browse online docs or the source code: def request(method, url, **kwargs): """Constructs and sends a :class:`Request <Request>`. :param method: method for the new :class:`Request` object. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`. :param json: (optional) json data to send in the body of the :class:`Request`. :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload. :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. :param timeout: (optional) How long to wait for the server to send data before giving up, as a float, or a (`connect timeout, read timeout <user/advanced.html#timeouts>`_) tuple. :type timeout: float or tuple :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed. :type allow_redirects: bool :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided. :param stream: (optional) if ``False``, the response content will be immediately downloaded. :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair. :return: :class:`Response <Response>` object :rtype: requests.Response
SFjulie1 11y ago
Really the hard way? https://hg.python.org/cpython/file/3.5/Lib/collections/__init__.py https://hg.python.org/cpython/file/3.5/Lib/collections/__ini... Knowing specialized data structure in a language is always important and this is well coded. Well see the 2 lines I pointed in one of the standard library in python and you will understand that even in good language there will always be dust under the carpet at one point. https://hg.python.org/cpython/file/5c3812412b6f/Lib/email/_header_value_parser.py#l1329 https://hg.python.org/cpython/file/5c3812412b6f/Lib/email/_h...
- nekopa 11y ago
  Thanks for the recommendations, I think they may be a bit above me at the moment (I don't have enough knowledge of Python to be able to identify dust under the carpet :) But you did teach me one thing, I didn't know that the """ could be used for commenting. I thought it was only for multi-line print statements. These are exactly the type of things I hope to learn while code-reading. Cheers!
  - thatmalewoman 11y ago
    Even better: If you put the string right after your function/class definition, python automatically assigns it to the `__doc__` attribute of the object. This is how the `help` function works.
    - nekopa 11y ago
      Thanks! As ironic(?) as this sounds, where is this doc feature documented? (Part of my learning is that I am trying hard to learn how to navigate the various Python documentation. For example, I had a hell of a time trying to find a list of format strings, as different sources refer to it as different things)
      - denzil 11y ago
        PEP-257 is where you can find documentation on them. https://www.python.org/dev/peps/pep-0257/ https://www.python.org/dev/peps/pep-0257/
veddox 11y ago
1. The Python standard library (if you're on Linux, /usr/lib/python2.x or 3.x, depending on your version). 2. The Bazaar VCS is written entirely in Python, is very well documented and has a large test section. (www.launchpad.net/bzr)
- nekopa 11y ago
  I have been debating whether or not to read the standard library, pro - it's the standard library, if I learn it I may save myself innumerable hours from reinventing wheels, con - it may not be the best example of code (thinking about the 400 line function from vim to see if there is input from the keyboard. Justified in context, but would be a horrible way to learn some code) Bazaar I may look into, as I know it's a very capable piece of software, and now I know it's well documented with tests makes it very relevant to my interests :) Edit: I'm on windows, but I'm using vagrant with Ubuntu 14lts image for my development work. Specifically the data science vagrant box...
  - dochtman 11y ago
    I have some experience with the Mercurial code base, which I thought was pretty well engineered. It's not PEP 8, though, so that makes it slightly idiosyncratic. Might make an interesting comparison with Bazaar, though!
    - nekopa 11y ago
      That is actually a very cool idea, taking 2 codebases that work on the same domain and seeing the trade offs that each made. Thank you! (And I had no idea that either bazaar or mercurial were written in Python, because them being 'serious' software I automatically assumed they were written in C. Color me stupid)
    - veddox 11y ago
      I haven't used Mercurial yet, let alone glance at the code base, but it might be worth a look ;-) But honestly, the Bazaar code base is great. Great documentation at every level, and, as far as I can judge, some pretty good code too.
anon3_ 11y ago
SQLAlchemy - https://github.com/zzzeek/sqlalchemy https://github.com/zzzeek/sqlalchemy - http://www.aosabook.org/en/sqlalchemy.html http://www.aosabook.org/en/sqlalchemy.html - http://docs.sqlalchemy.org/en/rel_1_0/ http://docs.sqlalchemy.org/en/rel_1_0/
- nekopa 11y ago
  This one could be very interesting as I have decades of experience with DBs and SQL, so I was always wondering about ORMs, and sqlalchemy seems to be best in field at the moment. Thanks for the recommendation. Cheers!
  - nine_k 11y ago
    Large pieces of SQLAlchemy are about metaprogramming and some syntactic magic required for an easy-to-use ORM. It's mostly nice on the inside (to my cursory glance), but neither simple nor easy for an inexperienced person. Still it's a good example how to make many relatively complex parts easily composable and efficient.
- rch 11y ago
  You might actually find Dogpile (also by Mike Bayer) more approachable and useful in this context. Even the separation of the project into logically distinct packages is instructive. - https://bitbucket.org/zzzeek/dogpile.core https://bitbucket.org/zzzeek/dogpile.core - https://bitbucket.org/zzzeek/dogpile.cache https://bitbucket.org/zzzeek/dogpile.cache
ericjang 11y ago
NetworkX - https://networkx.github.io/ https://networkx.github.io/ Good example of object-oriented programming patterns (mixins) in Python and module organization.
- nekopa 11y ago
  Thanks! Not sure if I am ready to take on OOP and mixins in Python yet, but I do like the idea of module organization so it's definitely on my list now. Cheers!
danwakefield 11y ago
Openstack does a large amount of testing for their code[1] but they is a huge amount of it. Barbican[2] is one of the newer less crufty components. [1]: https://github.com/openstack/openstack https://github.com/openstack/openstack [2]: https://github.com/openstack/barbican https://github.com/openstack/barbican
- nekopa 11y ago
  Cool, thanks, didn't realize openstack was Python. It may be too much for me to take on at my level, but it is definitely on my 'to look at' list. Cheers!
- agentultra 11y ago
  Openstack is so large it also has a plethora of examples of bad code. Especially in vendor-contributed drivers and such. I wouldn't pay attention to the code so much as the incredible infrastructure and process that manages it and improves it over time. Seriously, I've seen "sychronization threads" that use no synchronization primitives that have a single bare exception handler trap the critical section. The handler just restarts the computation... leading to fun times. Cinder also has an interesting problem in the explosion of ABC "mixins" used for constructing a backend driver. It went from 5 in the previous release to something like 25 presently. A patch to fix it: https://review.openstack.org/#/c/201812/ https://review.openstack.org/#/c/201812/ But even the process isn't perfect. I've seen patches that change a couple of log lines get dog-piled with -1 nit-picks about inconsequential wording and take months to get through. Openstack is an interesting beast but it's not a good example if you're just learning, IMO.
  - danwakefield 11y ago
    I happen to agree, Ive been working on it for the past year and its sort of killed my drive to work with something 'big' again. That being said Barbican is quite good, which I think is due to its proof of concept being written in Go[1] before being ported. If you want to learn decent testing Openstack is a good example, the code as a whole not so much. This also only applies if you want to use unittest or their custom test module, testtools. py.test is a much nicer way to do testing IMO [1]: https://www.youtube.com/watch?v=245rSZBdm9s https://www.youtube.com/watch?v=245rSZBdm9s
mattwritescode 11y ago
The django project is a good example of a large opensource project which has aged well. http://github.com/django/django http://github.com/django/django
- nekopa 11y ago
  I know it is a good project, but would you recommend it for someone to read through the codebase? (Honestly, looking at the repo, I don't even know where to start if I wanted to do a read through. Has anyone created a map? :)
  - iorlas 11y ago
    No, definitely no. Django aged well in terms of being alive, usable and comfortable to use in most usecases. But the codebase... Let me explain. Currently it works as "pay-to-get-a-feature" by crowdfounding. It not a bad thing, it gives good features we all need, but it is a bit sad fact when you look at Rails. So, it doesn't mean Django developers doesn't care about code, they care. But codebase is so big and complicated(because some code needs to be refactored)... it is common to see that some bug cannot be simply fixed by two lines, because some core member insists "this all should be reworked".
  - hluska 11y ago
    Have you considered building something with Django and figuring out the magic as you go along??
  - subleq 11y ago
    I would start with django.core.handlers.base.BaseHandler (https://github.com/django/django/blob/master/django/core/handlers/base.py#L106 https://github.com/django/django/blob/master/django/core/han...), which is Django's Front Controller.
- rglullis 11y ago
  I don't know what would be of my life right now without Django and I can't praise it enough... but "good example of readable python-code", it is not.
aaronjgreenberg 11y ago
Jumping on the Kenneth Reitz train, you might check out The Hitchhiker's Guide to Python: http://docs.python-guide.org/en/latest/ http://docs.python-guide.org/en/latest/ He recommends the following Python projects for reading: * Howdoi (https://github.com/gleitz/howdoi https://github.com/gleitz/howdoi) * Flask (https://github.com/mitsuhiko/flask https://github.com/mitsuhiko/flask) * Werkzeug (https://github.com/mitsuhiko/werkzeug https://github.com/mitsuhiko/werkzeug) * Requests (https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests) * Tablib (https://github.com/kennethreitz/tablib https://github.com/kennethreitz/tablib) Hope that helps---good luck!
- nekopa 11y ago
  Holy crap! I don't know if this helps or hinders! ;) Thank you for this resource, much appreciated!
- pdeuchler 11y ago
  +1 to anything written by Kenneth Reitz, he's also a fantastic OSS maintainer and extremely welcoming of newbies and their PR's if you want to put some of your learning into practice
  - knite 11y ago
    Not the most responsive maintainer - still waiting on him to tag a 1.0.0 release for autoenv - https://github.com/kennethreitz/autoenv/issues/82 https://github.com/kennethreitz/autoenv/issues/82
- tallerholler 11y ago
  I would add the Django project to that list as it's a very large, mature, and successful open source python project - https://github.com/django/django https://github.com/django/django
  - rectangletangle 11y ago
    Django's source is very high quality. Though due to the large scope of the project, there are necessarily many layers of indirection, which may be a bit daunting for someone who is just starting out. However reading the less abstract parts may help. For instance, the paginator is pretty self contained. https://github.com/django/django/blob/master/django/core/paginator.py https://github.com/django/django/blob/master/django/core/pag...
feathj 11y ago
Check out boto. It's Amazon's official library for interacting with AWS. It is written and tested well. I use it every day. https://github.com/boto/boto https://github.com/boto/boto
- dynamicdispatch 11y ago
  I've had to read the source of boto since there were some boto exceptions I was seeing in stack traces in the fabric deploy process - and ugh, I found the code to be not very intuitive and the documentation poor. Anyone else run into the same issues with boto?
  - clebio 11y ago
    I've found the documentation (not the codebase itself) to be very hit-or-miss. Some components are very well documented, some not at all. Presumably due to maturity of different parts of the stack and the Boto library, but still frustrating from an end-user perspective.
- wldcordeiro 11y ago
  Boto3 might also be worth a read. https://github.com/boto/boto3 https://github.com/boto/boto3
  - ak217 11y ago
    Boto3 was made using a much more principled design approach, while Boto grew organically and frankly got a bit out of control. I love boto3, it makes AWS a joy to use.
NumberCruncher 11y ago
I´m just a user and not a contributor (I don´t know the source code), but the project is following good webdev techniques * https://github.com/web2py/web2py/ https://github.com/web2py/web2py/ * https://github.com/web2py/pydal https://github.com/web2py/pydal
misiti3780 11y ago
Django, Tornado, Sentry, Newsblur
rwar 11y ago
youtube-dl: https://github.com/rg3/youtube-dl https://github.com/rg3/youtube-dl
edoceo 11y ago
Gentoo Portage package manager. Its a big project, lots of moving parts, actively developed. Really helped me with learning Python
jonjacky 11y ago
Peter Norvig's examples. They are quite short and include much explanation in addition to code. They also include tests and benchmarking code. http://norvig.com/lispy.html http://norvig.com/lispy.html http://norvig.com/lispy2.html http://norvig.com/lispy2.html (Lisp interpreter) http://www.norvig.com/spell-correct.html http://www.norvig.com/spell-correct.html (Spelling corrector) http://norvig.com/sudoku.html http://norvig.com/sudoku.html (Sudoku solver) Also his online course Design of Computer programs includes many short, well-explained Python examples: https://www.udacity.com/wiki/cs212 https://www.udacity.com/wiki/cs212
- aaronchall 11y ago
  His coding style is not common to most writers of Python. He's using overly terse, poorly descriptive variable names, not using multiline strings for docstrings, not indenting where he should, and one-lining if/elif statements and function definitions. This style does not contribute to readability. This is not how I would want someone just learning Python to learn it.
  - gknoy 11y ago
    While that's true, he also tends to write fairly idiomatic Python -- using base data types a lot, list comprehensions, etc, and his comments and overall insight make them Very worthwhile to read -- especially once you're familiar enough reading it to recognize what he's doing. The spellchecker still blows my mind, and I don't yet understand the Sudoku puzzle. We can always take the code and re-format / re-name variables to aid our understanding.
  - merlincorey 11y ago
    Maybe it's because I am also something of a lisper (like Norvig), but I don't see anything wrong with inline ifs (after all, that is how if works in lisp, with returns for a conditional, like the ternary operator) or lambda functions. In fact, I find that improves readability dramatically because it more declaratively says what you are trying to accomplish in many cases. For example: absolute_path = lambda path: path if path.startswith('/') else '/' + path To me this is perfectly clearly a simple function whose only purpose is to prepend forward slashes to unix-style paths. Is the following really so much more readable? def absolute_path(path): if not path.startswith('/'): path = '/' + path return path To my eyes and mind, the second example is not any more readable at the expense of several lines of code.
jscottmiller 11y ago
Bottle: https://github.com/bottlepy/bottle https://github.com/bottlepy/bottle It's a nice, small, fast web framework. Great for building APIs. Also, it's one ~3k loc, readable file.[1] [1] https://github.com/bottlepy/bottle/blob/master/bottle.py https://github.com/bottlepy/bottle/blob/master/bottle.py
mapleoin 11y ago
Also, how about examples of good web applications built on python with available source code? Rather than seeing the code of great libraries, I sometimes want to see how people use them in the real world.
tzury 11y ago
1 - (web, network, file system, and more). tornado - tornadoweb.org github.com/facebook/tornado 2 - scapy The entire product runs on the python CLI secdev.org/scapy
shoyer 11y ago
I recommend PyToolz, "set of utility functions for iterators, functions, and dictionaries": https://github.com/pytoolz/toolz https://github.com/pytoolz/toolz The functions in PyToolz are short, well tested and idiomatic Python (thought the functional programming paradigm they support is not quite so idiomatic). I recommend starting with the excellent documentation: http://toolz.readthedocs.org/en/latest/ http://toolz.readthedocs.org/en/latest/ In particular, the API docs have links to the source code for each function: http://toolz.readthedocs.org/en/latest/api.html http://toolz.readthedocs.org/en/latest/api.html
giancarlostoro 11y ago
I see nobody has recommended CherryPy: http://www.cherrypy.org/ http://www.cherrypy.org/ It is a minimal web framework like Sinatra or Flask. The beautiful thing about CherryPy is you write code for it the same way you would write general Python code. I enjoy using it for small projects from time to time. edit: Bickbucket Repository: https://bitbucket.org/cherrypy/cherrypy/overview https://bitbucket.org/cherrypy/cherrypy/overview
mahouse 11y ago
For the web developers out there, what do you think of reddit? Any honest commentary on it? https://github.com/reddit/reddit https://github.com/reddit/reddit
spang 11y ago
The Nylas Sync Engine is a large Python codebase with a test suite: https://github.com/nylas/sync-engine https://github.com/nylas/sync-engine Lots of examples of SQLAlchemy, Flask, gevent, and pytest in action to build a REST API and sync platform for email/calendar/contacts data!
- jlees 11y ago
  Putting on my "Python teacher" hat for a moment (I also work at Nylas), our codebase is interesting for learning purposes because it's a fairly homogenous, 'live' product managed by a commercial engineering team. Definitely on the complex side of things, though -- let me know if you want any help figuring it out (we also have a community Slack at http://slack-invite.nylas.com/ http://slack-invite.nylas.com/). On the testing side, I was chatting to a fellow Pyladies attendee about how we do test fixture setup and teardown (especially around databases) -- you might find that interesting to look at too.
- aaronchall 11y ago
  I clicked through on this and browsed a few directories, none of which seemed likely. I do not see any `.py` files aside from an empty `__init__.py` and the `setup.py`. Are you sure a beginner just learning Python should see this?
  - spang 11y ago
    The main codebase is in `inbox/`, with launcher scripts and tools in `bin/`. (Might not be totally obvious, because the package namespace is called `inbox/` for legacy reasons.) If a beginner wants to see real production code, rather than toy examples, I think it's inevitable that there will be some points of confusion. Part of the learning process is diving in and exploring and being okay with not totally understanding everything that's going on. :)
jordigh 11y ago
Mercurial. By design, Mercurial has almost no dependencies, so it's very self-contained. I find this makes it a particularly easy codebase to get into. If you're interested, I would love to walk you (or anyone else!) trough it.
d0m 11y ago
The pep8 standard is also an easy read with so many useful explanations: https://www.python.org/dev/peps/pep-0008/ https://www.python.org/dev/peps/pep-0008/
- justizin 11y ago
  Don't let it turn you into a fucking pedant, though.
  - compostor42 11y ago
    When it comes to maintaining standards in a code base, one needs to be a pedant.
    - Thrymr 11y ago
      The mentality of someone who thinks that an 89-character line that ends with " # noqa" is better than the 81-character line without that ending because now it passes flake8 is one I'll never understand.
deepaksurti 11y ago
NLTK: https://github.com/nltk/nltk https://github.com/nltk/nltk with the documentation at http://www.nltk.org http://www.nltk.org. I found the code easy to follow through. I referred to it when adding tests for tokenizers in a common lisp NLP application: https://github.com/vseloved/cl-nlp/ https://github.com/vseloved/cl-nlp/.
- abecode 11y ago
  agreed: I also learned a lot from nltk too, esp. freqdist/probdist as well as the classifiers in nltk and sklearn
- rspeer 11y ago
  Agreed. NLTK's value is not that it should be your go-to source for NLP algorithms -- when there's a specific task you need to accomplish, there'll be a specific solution by now that works better than NLTK. NLTK's value is that it shows you how to write NLP algorithms, and gives you an understandable starting point when you need to do something that nobody has implemented yet.
notatoad 11y ago
I learned a lot about python by reading through the tornado codebase. it's pretty easy to read, well broken up into functions, and not too big.
- jehiah 11y ago
  ditto. https://github.com/tornadoweb/tornado/tree/master/tornado https://github.com/tornadoweb/tornado/tree/master/tornado
svieira 11y ago
Several good ones have already been suggested, but here's a few more: - https://github.com/mahmoud/boltons https://github.com/mahmoud/boltons : utility functions, but well documented - https://github.com/KeepSafe/aiohttp https://github.com/KeepSafe/aiohttp : a Python 3 async HTTP server - https://github.com/telefonicaid/di-py https://github.com/telefonicaid/di-py : a dependency injection framework - https://github.com/moggers87/salmon https://github.com/moggers87/salmon : a fork of Lamson (which was written by Zed) Python's internals are pretty darn open, so here's a few suggestions that push the boundaries of meta programming in Python - they're not the idiomatic code you're looking for right now, but later, when you know the best practices and you're wondering what is possible they'll be good to look at: - https://github.com/Suor/whatever https://github.com/Suor/whatever : Scala's magic `_` for Python - https://github.com/ryanhiebert/typeset https://github.com/ryanhiebert/typeset : Types as sets for Python - https://github.com/AndreaCensi/contracts https://github.com/AndreaCensi/contracts : Gradually typed Python (akin to MyPy) - http://mypy-lang.org http://mypy-lang.org : Gradually typed Python - the future (at least right now)
aaronchall 11y ago
Here's the link to the Pandas DataFrame source: https://github.com/pydata/pandas/blob/master/pandas/core/frame.py https://github.com/pydata/pandas/blob/master/pandas/core/fra... We spent a month of Sundays going through this in the NYC Python office hours. You learn a lot about this object by reading the source, and the WTF per minute rate is fairly low. The style is also fairly non-controversial.
travisfischer 11y ago
A large Python project that I haven't seen mentioned by others but that I find to be particularly well written and designed is the Pyramid web framework. * https://github.com/Pylons/pyramid/ https://github.com/Pylons/pyramid/
- rachbelaid 11y ago
  I agree with you. Pyramid is one of the most well designed codebase that I know.
thruflo 11y ago
> Those who don't study Zope are condemned to reinvent it http://dirtsimple.org/2007/01/where-zope-leads-python-follows.html http://dirtsimple.org/2007/01/where-zope-leads-python-follow... https://github.com/zopefoundation?tab=repositories https://github.com/zopefoundation?tab=repositories
rjusher 11y ago
I would also add *Twisted (https://github.com/twisted/twisted https://github.com/twisted/twisted) For async python.
- rspeer 11y ago
  I disagree. Twisted is a sprawling codebase, it started as a game library that turned into an async library along the way, you need to read books to get the full documentation, and some of it doesn't even have docstrings or comments.
  - rjusher 11y ago
    You are right it is not the best example of an open source project, and for what you say it neither is a good python project example. But is there any other place you can get the hold of working async with python, it may be hard, but you would learn a lot. But maybe there are other async python projects that I don't know of. If you know of any please post them, I would also like to learn more about the subject.
mkolodny 11y ago
Guido van Rossum, the creator of Python, co-wrote a web crawler in under 500 lines: https://github.com/aosabook/500lines/tree/master/crawler https://github.com/aosabook/500lines/tree/master/crawler It's especially interesting because it takes advantage of new Python features like asyncio.
cessor 11y ago
I would recommend reading about the Tornado WebServer. It features some nice stuff such as coroutines, async stuff. https://github.com/tornadoweb/tornado/ https://github.com/tornadoweb/tornado/
gcb0 11y ago
good examples with a (hopefully multiplatform) GUI?
patrickk 11y ago
youtube-dl: https://github.com/rg3/youtube-dl https://github.com/rg3/youtube-dl I fell in love with this project after discovering I don't need ad-choked, dodgy sites to download Youtube videos/mp3s. It also acts as a catch-all downloader for a huge amount of other video hosting sites, despite the name. If you want to learn how to scrape different videos from many platforms, look at this: https://github.com/rg3/youtube-dl/tree/master/youtube_dl/extractor https://github.com/rg3/youtube-dl/tree/master/youtube_dl/ext...
rasbt 11y ago
I wholeheartedly recommend [scikit-learn](https://github.com/scikit-learn/scikit-learn https://github.com/scikit-learn/scikit-learn) - the best organized and cleanest code I've seen so far. It is really organized and well thought-through.
rocho 11y ago
I think Radon (https://github.com/rubik/radon/ https://github.com/rubik/radon/) is a good example of clean code. It's quite a small project, but it also gets right all the setup chore (like setup.py, testing, coverage, travis integration, etc.) which a beginner may find interesting. Also, I second all the suggestions about Kenneth Reitz code, it's wonderful!

aaronchall 11y ago
The former is absolutely identical (semantically) to the latter, with two exceptions: 1) the former function does not know its own name and 2) the latter function can (and should) be documented with a docstring. I find the latter eminently more readable, and I work daily in a code base under development by 3000 Python developers for over 5 years. Considering that the creator of the Python language considered getting rid of lambdas because they are essentially limited functions and thus violate Python's "one obvious way to do it" philosophy, I'd rather those learning Python to be shown the latter, rather than the former. From a development and version control perspective, as soon as the lambda function requires more than a simple expression, i.e. a compound statement (https://docs.python.org/2/reference/compound_stmts.html https://docs.python.org/2/reference/compound_stmts.html), you have to trash the whole line, instead of adding perhaps a single extra line of content.
- merlincorey 11y ago
  > The former is absolutely identical to the latter Oh, really? Let's compare: >>> absolute_0 = lambda path: path if path.startswith('/') else '/' + path >>> def absolute_1(path): ... '''Return the absolute unix path from a given path name''' ... if not path.startswith('/'): ... path = '/' + path ... return path >>> import dis >>> dis.dis(absolute_0) 2 0 LOAD_FAST 0 (path) 3 LOAD_ATTR 0 (startswith) 6 LOAD_CONST 1 ('/') 9 CALL_FUNCTION 1 12 POP_JUMP_IF_FALSE 19 15 LOAD_FAST 0 (path) 18 RETURN_VALUE >> 19 LOAD_CONST 1 ('/') 22 LOAD_FAST 0 (path) 25 BINARY_ADD 26 RETURN_VALUE >>> dis.dis(absolute_1) 4 0 LOAD_FAST 0 (path) 3 LOAD_ATTR 0 (startswith) 6 LOAD_CONST 1 ('/') 9 CALL_FUNCTION 1 12 POP_JUMP_IF_TRUE 28 5 15 LOAD_CONST 1 ('/') 18 LOAD_FAST 0 (path) 21 BINARY_ADD 22 STORE_FAST 0 (path) 25 JUMP_FORWARD 0 (to 28) 6 >> 28 LOAD_FAST 0 (path) 31 RETURN_VALUE These functions are actually not identical in their computation - only their result. > 1) the former function does not know its own name If you think that is really important (hint: it's not [from a lisper perspective, anyway]), Python thankfully allows you to do this: >>> absolute_0.__name__ '<lambda>' >>> absolute_0.__name__ = '<lambda "absolute_0">' >>> absolute_0.__name__ '<lambda "absolute_0">' > 2) the latter function can be documented with a docstring >>> absolute_0.__doc__ >>> absolute_0.__doc__ = 'Return the absolute unix path from a given path name' >>> help(absolute_0) Help on function <lambda "absolute_0">: <lambda "absolute_0">(path) Return the absolute unix path from a given path name Of course the only reason you can't put docstrings on a lambda function in python is because the forced indentation of code and implicit return with no indented block available is what Guido went with for Lambda. > Considering that the creator of the Python language considered getting rid of lambdas Guido is not a proponent of functional programming in general and claims that map, reduce, and filter are so much harder to understand than list comprehensions (which implement some common map, reduce, and filter, operations with special optimized syntax) that he tried to get them removed from the language too. Thankfully for us users of the language, this view did not win through and we can still use map, reduce, and filter in python, if we choose.
  - aaronchall 11y ago
    Spare us the half-baked hackery. List comprehensions and generator expressions have replaced all need for map, filter, and lambdas, and are far more readable. For someone new to Python, halfway through LPTHW, they don't need those things. Hey, maybe you can help me decipher this, I've always wondered exactly what's going on here: https://docs.python.org/2/faq/programming.html#is-it-possible-to-write-obfuscated-one-liners-in-python https://docs.python.org/2/faq/programming.html#is-it-possibl... # Mandelbrot set print (lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y, Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM, Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro, i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y >=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr( 64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy ))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24) # \___ ___/ \___ ___/ | | |__ lines on screen # V V | |______ columns on screen # | | |__________ maximum of "iterations" # | |_________________ range on y axis # |____________________________ range on x axis
    - merlincorey 11y ago
      > Spare us the half-baked hackery. Ad hominem? I guess I win. > List comprehensions and generator expressions have replaced all need for map, filter, and lambdas Please explain how list comprehensions and generators have replaced the need for lambdas. >>> sorted(((x, -(x**2)) for x in xrange(10) if 0 == x % 2), key=lambda item: item[1]) [(8, -64), (6, -36), (4, -16), (2, -4), (0, 0)] Your "distaste" of functional programming constructs is right up there with Guido's.
      - aaronchall 11y ago
        Nice lambda. You've defended yourself admirably. Did you know there's an `operator.itemgetter` function that does that? So that's: >>> sorted(((x, -(x**2)) for x in xrange(10) if 0 == x % 2), key=operator.itemgetter(1)) [(8, -64), (6, -36), (4, -16), (2, -4), (0, 0)] Do note that good code and code golf are two different things! :D
srean 11y ago
To me the first is far more quicker to read and understand, and I am no lisper. There is something immediately gratifying about the first that I find missing in the more laborious ponderous prose of the latter.
Xephyrous 11y ago
I'd write that function like this: def absolute_path(path): if not path.startswith('/'): return '/' + path return path This way you don't have to keep track of mutating the path variable. I do find that more readable - it's clearer that there are two paths through the code. That said, this example is trivial enough that I'd probably do it with an inline if/else statement, although still probably not a lambda: def absolute_path(path): return path if path.startswith('/') else '/' + path This way it's more obvious at a glance that you're defining a function. It's easier to follow someone else's code if they generally adhere to standards, and Python is a very convention-oriented language.

srean 11y ago

Well, between a run of the mill programmer who happens to indent where he/she should vs Norvig, I will likely choose the latter If I could. Reminds me of that blog thread where this design pattern guru hemmed and hawed from his high horse over several posts about how to write constrained based solvers and still did not get to a piece of code that actually solved the problem, whereas Norvig just posted a simple solution. A few non-idiomatic indents here and there (although his style has never been a problem for me) are nothing really.

vram22 11y ago
>Well, between a run of the mill programmer who happens to indent where he/she should vs Norvig, I will likely choose the latter If I could. Right. >Reminds me of that blog thread where this design pattern guru hemmed and hawed from his high horse I read a similar story a while ago. Two programmers are given the task of writing a program for some non-trivial problem. One of them, Hoity Toity Harry, tries to apply many of the latest and greatest algorithms, techniques, paradigms, etc., to impress people (of course). The other, Down To Earth Dan, just strives for a good implementation, with reasonably good algorithms, etc. After a while, Dan finishes his program and does a test run. It works well enough for the task. Meanwhile, Harry is not even near to finishing his code, due to struggling with complexities of the techniques he has tried to use. The boss comes in, sees the results, and congratulates Down To Earth Dan. Hoity Toity Harry, of course, has to protest, trying to put down Down To Earth Dan's implementation, saying that it uses simple algorithms, etc., while his own code uses sophisticated, state of the art techniques. Dan replies: "Yes, I could also have used those things. But my program runs, and yours doesn't." Okay, I changed the programmers' names for fun and effect, but I really did read the story, in some good (and pragmatic) software book, a while ago.
- abecedarius 11y ago
  Speaking of which, I once saw a user review of Norvig's book PAIP complaining about the unsophisticated code -- no monads, etc. (I forget what other patterns the reviewer wanted to see.)
  - vram22 11y ago
    Ha ha, good one. I wonder if the reviewer knew who Norvig is.

allendoerfer 11y ago

>> Spare us the half-baked hackery. >Ad hominem? I guess I win. That was not ad hominem, because what you wrote is indeed half-baked hackery. Nobody said it cannot be done the way you did it. You were just pointed at the shortcomings of your approach and that the Python community generally prefers stupidly simple, easy to understand solutions. Using magic attributes to argue against it just makes it worse - remember this is a thread about idiomatic Python code bases.

merlincorey 11y ago
Now you're moving goalposts... > Nobody said it cannot be done the way you did it. You were just pointed at the shortcomings of your approach I "addressed" the shortcomings of "my approach" by showing that Python (the language, not the community) allows you to access and manipulate the data you claimed was important and missing. I don't believe it is necessary for most simple functions to know what their name is. I do believe the demonstrated code is self documenting enough to not require a documentation string. You made those "requirements". I never claimed that every function must be a lambda - you seem to be implying I am, so I am explicitly stating that I do not. Here's a place where you really do need a named function (due to deficiencies in Python's lambda implementation): >>> def named_lambda(procedure, name, documentation=''): ... procedure.__name__ = '<lambda {}>'.format(name) ... procedure.__doc__ = documentation ... return procedure ... >>> absolute_path = named_lambda(lambda path: path if path.startswith('/') else '/' + path, 'absolute_path', 'Return the absolute unix path from a given path name') >>> absolute_path.__name__ '<lambda absolute_path>' >>> absolute_path.__doc__ 'Return the absolute unix path from a given path name' Of course, that's completely silly... since the point of a lambda function generally is that the function is generally small enough and short-lived enough that it does not need a name or documentation. > the Python community generally prefers stupidly simple, easy to understand solutions. Which, despite your protests, includes using lambdas! > Using magic attributes to argue against it just makes it worse - remember this is a thread about idiomatic Python code bases. How else does a function "know its own name" unless it uses the "magic" attribute "__name__"? Oh, you prefer "func_name"? That's cute: >>> named_lambda.func_name 'named_lambda' >>> named_lambda.__name__ 'named_lambda' >>> named_lambda.__name__ = 'lol' >>> named_lambda.func_name 'lol' >>> named_lambda.func_name = 'named_lambda' >>> named_lambda.func_name 'named_lambda' So in this comment I am replying to, it is a bad thing that I made use of "__name__", but in the comment THAT was replying to, it was a bad thing that I did NOT use "__name__" or its linked "func_name". That's how you move goal posts!