7 ms·

Ask HN: High quality Python scripts or small libraries to learn from

• jihadjihad 2y ago
  For scripts, I've learned a couple tricks from OpenAI's Cookbook examples. This one came in handy not too long ago: https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py https://github.com/openai/openai-cookbook/blob/main/examples...
  • hopfenspergerj 2y ago
    This is an example of bad code.
    • parpfish 2y ago
      [flagged]
    • cinntaile 2y ago
      Maybe you can expand upon that. Now we have no way of knowing why you think it's bad code.
      • isoprophlex 2y ago
        those nested if - for - for - if loops are horrendously difficult to understand. take the fn starting at line 387. they comment why they do certain imports, but this function is comparatively underdocumented. it's not easy to wrap my head around the control flow. some bits are nested about 6 levels too deep for comfort, there are too many positions from which it can return or raise, and the function is about 3x too long really difficult to grok what is happening here.
      • radus 2y ago
        Quick critique: module contains functions with many parameters, many branches, deep nesting, and multiple return points.
    • mixmastamyk 2y ago
      It’s not horrible, but I found a few odd things, like f-strings w/o params, long cli options with underscores, non-pythonic if == 0, etc.
      • mixmastamyk 2y ago
        Also the main god function is incredibly long and nested, as others mentioned. Nested --> long lines --> black making a mess.
    • deleted 2y ago
      [deleted]
• mind-blight 2y ago
  The Django code base is excellent. I learned a ton early on by reading through it
  • 9dev 2y ago
    As a derivation of this, in general I advise reading framework source code. Not because you should write code like that, but to learn what the language can do. Framework source code has often been refined by several people over a longer period of time, honed to avoid rough edges. I think you can learn a lot about designing an API, writing good abstractions, and encapsulating complexity.
    • aynyc 2y ago
      I would caution this. Not because it's a bad idea, but because I've experienced juniors reading framework often ended up creating over-engineered codes that lean heavily into meta style programming. Their code become overly-complex due to massive amount of abstractions they put in. I like the old saying, "cook the recipe exactly as it is three times before you can add your own spin to it".
• SushiHippie 2y ago
  I think I mention this all the time when this comes up, but I learned the most 'best practices' through using ruff. https://docs.astral.sh/ruff/ https://docs.astral.sh/ruff/ I just installed and enabled all the rules by setting select = [ "ALL" ] And then looked at the 'errors' it showed me for my existing code base and then excluded some rules which don't interest me. I also have it set up in my IDE, so it'll show me the linting errors while coding. In a larger code base there will definitely be many 'errors', but using it cleaned up my code really good and it stopped me from some footguns.
  • sevensor 2y ago
    This is good advice. I learned Python long before ruff came on the scene, but I did the same with Pylint. I don't adhere rigidly to its recommendations any more, but I learned a lot about the language from trying. I fact, I think some of its recommendations are downright wrong, and what I learned was that I made my code harder to maintain by following them.
    • peteradio 2y ago
      I'm curious what recommendations you remember disagreeing with.
      • IshKebab 2y ago
        I can't say I've seen any but some of the code style ones are very prescriptive (function longer than N lines, short variable names etc.). Single letter variable names are totally fine in some cases. And while very long functions may be bad, it's pretty annoying when you're adding one line to a function for the linter to say "nope. have you considered dropping everything and refactoring this?" You can easily turn them off though. I can't remember any code based ones that are really wrong. Maybe some are prone to false positives, e.g. warning about a default `= []` argument. But you can waive them individually.
        • sevensor 2y ago
• rmorey 2y ago
  Everything @simonw has worked on, honestly: https://github.com/simonw https://github.com/simonw
• thelastbender12 2y ago
  Simon Willison's github would be a great place to get started imo - https://github.com/simonw/datasette https://github.com/simonw/datasette https://github.com/simonw/sqlite-utils https://github.com/simonw/sqlite-utils So, his code might not be a good place to find best patterns (for ex, I don't think they are fully typed), but his repos are very pragmatic, and his development process is super insightful (well documented PRs for personal repos!). Best part, he blogs about every non-trivial update, so you get all the context!
  • hiAndrewQuinn 2y ago
    simonw might be one of the best and most down to earth Pythonistas of our time. He was one of the co-creators of Django, and that was almost 2 decades ago by this point - getting better all the whole. I second this recommendation heartily.
• uneekname 2y ago
  I can't think of any (small) libraries I could recommend to learn best practices, but what does come to mind is click [0], the CLI library for Python. Their documentation is pretty great, there are tons of short example scripts to be found online, and in my experience making little apps with click can be a nice way to learn different python features like args/kwargs, decorators, string manipulation, etc. I agree with others that code formatters like black or ruff might be helpful to you. The literature surrounding them, such as PEPs concerning code formatting, often include examples you may find useful. [0] https://click.palletsprojects.com/en/8.1.x/ https://click.palletsprojects.com/en/8.1.x/
  • thenipper 2y ago
    I was just going to suggest this. Click is a great code base to learn form.
  • mixmastamyk 2y ago
    Beware, the pallets people are decorator supremacists. Everything is a decorator even when it arguably shouldn't be. DDD --> decorator driven development. It's a nice technique, too a point. Only exaggerating a bit. ;-)
• 0xbadcafebee 2y ago
  Mostly you should start by reading what's shipped with core. After that, look at larger projects with lots of committers, as they often end up getting "polished" down to something that's sort of generically useful without being too quirky or too simple, and the architecture tends to be more functional. Avoid anything developed/maintained by one corporation. In general, their organizational hierarchy leads to bad patterns and resists useful changes that don't conform to the goals and patterns of the business or engineering leads. Grassroots OSS projects aren't always better, but they're less likely to have a monoculture and perverse incentives.
  • martinky24 2y ago
    Python is an example of a language where, in general, the standard library probably isn't the best thing to read if you want to learn to write downstream Python applications/libraries. It can be terse, and not follow "modern" best practices in places (it was written at a time with different "best practices", but the code works so no need to change it). There are some exceptions... I'd say the `statistics` module is one [1], the `collections` module might be another [2]. But in general, it's probably not the best place to start. If you stumble upon the `multiprocessing` library source code as inspiration for "good Python code"... you're going to be in for a bad time and your future collaborators will not be happy. [1]: https://github.com/python/cpython/blob/3.12/Lib/statistics.py https://github.com/python/cpython/blob/3.12/Lib/statistics.p... [2]: https://github.com/python/cpython/blob/3.12/Lib/collections/__init__.py https://github.com/python/cpython/blob/3.12/Lib/collections/...
• begueradj 2y ago
  Take a look at PY4WEB: https://github.com/web2py/py4web https://github.com/web2py/py4web It is an improvement for the the tiny but efficient web2py web framework.
• pixelmonkey 2y ago
  I had 2 suggestions (plus a blog post) in my style guide here: https://github.com/amontalenti/elements-of-python-style#some-inspiration https://github.com/amontalenti/elements-of-python-style#some... The style guide itself, published a few years back, also has some suggestions with small code snippets.
• in9 2y ago
  if you are into ML libraries, take a look at fklearn, a scikitlearn-like lib, but written in a functional. Fun read to compare both side by side.
• zamubafoo 2y ago
  I think I've learned more reading bad code bases than reading good code bases. The entire point is not to just mindlessly consume a code base, but instead form an idea of how to approach the problem and then see if your hypothesis is correct. Then comparing your approach to the actual approach. This can show you things that you might've missed taking into account. For example, gallery-dl's incidental complexity all lies in centralizing persistent state, logging, and IO through the CLI. It doesn't have sufficient abstraction to allow it to be rewired to different UIs without relying on internal APIs that have no guarantee that won't change. Meanwhile a similar application in yt-dlp has that abstraction and works better, but has similar complexity in the configuration side of things.
  • sevensor 2y ago
    It's a pain, but you can definitely learn a lot from fixing a bad codebase. For that, I recommend trying to write type annotations and get the whole thing to type-check. I've found that bad codebases end up having very complex type annotations because their authors actually contradict themselves. One of my personal favorites in Python is mixing strings and UUIDs as dictionary keys. This positively guarantees a fun afternoon. Edit: speling
    • agumonkey 2y ago
      Another learning point is being sensitive to your psychology / mental energy. I can start with high quality well named, well abstracted code.. but after two weeks I find myself writing shitty code.. and having a hard time realising I should stop, take a pause, take a step back instead of piling on.
      • eternityforest 2y ago
        Pre-commit hooks help me immensely with this. At least there's some limits on crappiness.
    • mixmastamyk 2y ago
      I'd start with pyflakes and ruff check/format as step 0. Much easier to get started, and will have fixed a lot of stuff quickly. Next, add types.
• kunley 2y ago
  SQLAlchemy sources. Not exactly small, but great.
• mixmastamyk 2y ago
  Pyupgrade is a good tool that focuses on upgrading to newer idioms. Which is more important for learning than simple pep8 type stuff, which is useful but has its limits. On that subject, Raymond Hettinger has a great talk called “Beyond Pep8” that talks about how to de-java your codebase among other things. Also reading the book Fluent Python now and it is so far excellent.
  • claytonjy 2y ago
    And ruff has a bunch of the pyupgrade rules included, which has made it easy for me to start catching things like using List instead of list in py3.10+.
• shivekkhurana 2y ago
  At my university, I followed David Beazly's talks and tutorials. Just seeing him work and present improved my style and approach manifolds: https://www.dabeaz.com/tutorials.html https://www.dabeaz.com/tutorials.html
  • gabrielsroka 2y ago
    Beazley's videos on YouTube including his channel are great https://youtube.com/@dabeazllc https://youtube.com/@dabeazllc I also like Raymond Hettinger.
• koutetsu 2y ago
  If you're looking for some best practices related but limited to machine learning application code, you could have a look at Beyond Jupyter (https://github.com/aai-institute/beyond-jupyter https://github.com/aai-institute/beyond-jupyter) Here's an excerpt from the readme: "Beyond Jupyter is a collection of self-study materials on software design, with a specific focus on machine learning applications, which demonstrates how sound software design can accelerate both development and experimentation."
• achanda358 2y ago
  Peter Norvig's work is great to learn from https://github.com/norvig/pytudes https://github.com/norvig/pytudes
• nurettin 2y ago
  Any codebase that uses and respects type annotations is probably a good place to start. So grep for "from typing".
• byyoung3 2y ago
  just try to build stuff and then u will see over time what works and what causes issues. theres no shortcuts other than using chatgpt
• perfmode 2y ago
  Peter Norvig’s python scripts are quite beautiful.
• encoderer 2y ago
  You do what everybody does: Grind this out painstakingly and then copy and paste that as boilerplate for the rest of your natural life.
• michaeljx 2y ago
  The python requests library
  • oznt 2y ago
    Requests is really not small and it's full of backwards compatibility code. Would not recommend it.
• rsyring 2y ago
  I'd suggest Flask or some of the smaller projects in the Pallets ecosystem: Flask, in particular, has a very small number of open issues (2) for a project that is pretty popular. Its also maintained by a competent team and has a lot of project best practices. https://github.com/pallets/flask https://github.com/pallets/flask https://github.com/pallets https://github.com/pallets https://pypistats.org/packages/flask https://pypistats.org/packages/flask https://pypistats.org/packages/django https://pypistats.org/packages/django (for comparison)
• llandy3d 2y ago
  I don't know if I can consider my code "Great" but I dedicated way too many months on a prometheus library where I focused on quality since I did it for me. It's relatively small and I think the main take away would be the use of Protocols for the pluggable backend system. I hope you get something out of it :) https://github.com/Llandy3d/pytheus https://github.com/Llandy3d/pytheus
• srcreigh 2y ago
  what’s your experience level? what kind of python code do you work with (web dev? Data analysis? Algorithms?)
• rookie101 2y ago
  I've recently looked at tasktiger https://github.com/closeio/tasktiger https://github.com/closeio/tasktiger. It's a simple queue system that helped me understand how workers and schedulers work.
• vram22 2y ago
  The 500 Lines or Less section from https://aosabook.org/en/ https://aosabook.org/en/ The Architecture of Open Source Applications
• nmaleki 2y ago
  Check out https://github.com/recursion-computing/starcel-panda3D https://github.com/recursion-computing/starcel-panda3D for bleeding edge, but not refactored to be pythonic, Python OS development.
• vismit2000 2y ago
  Karpathy also writes beautiful python code and his GPT lecture series contains some of the finest python code to learn from. https://github.com/karpathy/nn-zero-to-hero https://github.com/karpathy/nn-zero-to-hero
• oznt 2y ago
  You should definitely read bottle.py, while full of hack to support python2 it still a very good code base to learn about many python features. Another one is stencil template engine.