6 ms·

Python Patterns (2013)

165 points by dedalus 11y ago

snuxoll 11y ago
Can I just state how awful this two-column layout is to read?
- BinaryIdiot 11y ago
  I'm glad I'm not the only one. Decided to skim it instead of reading it all the way through because it was annoying.
- Fuzzwah 11y ago
  I found the "toggle columns" option (top right, under the search box) and it made me happy.
  - dr_zoidberg 11y ago
    While it did away with the horrible 2-column layout, it looks like the article was writen with those columns in mind, so it uses short paragraphs that look good on 2-column, but feel a bit weird on a wall of text. My (barely existant) OCD is going haywire over this article as I read it...
    - zodiakzz 11y ago
      OT: Anyone know why all browser vendors implemented this CSS columns aberration so hurridly..? I have never seen a good use for them, so painful to read. It's always some designer trying to show off their "epic" CSS skillz.
      - anewhnaccount 11y ago
        Seems like it might be useful for a print style sheet. Haven't seen it used to good effect on a screen though.
- afarrell 11y ago
  It goes away if you decrease the width of the viewport
- praneshp 11y ago
  Also, the picture with the directory tree got cropped out for me, and I couldn't figure out how to scroll it horizontally. Good thing the columns can be toggled.
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
- agumonkey 11y ago
  Way better than the single column one on my 1024x768 lcd.
mixmastamyk 11y ago
Bottle, peewee, gevent workers, eh? Interesting choices. SimpleTemplate looks interesting, I hadn't seen it before. Would be nice to hear some discussion there, instead of the column layout.
- coleifer 11y ago
  I'm the author of Peewee and I've been running my blog with Flask, Peewee, gevent and sqlite. It's a very low-impact combination, lightweight and fun to use, although I'm a bit biased.
  - raymondh 11y ago
    Thank you for your work on Peewee. The code is nicely written and is easily read start to finish.
    - benhoyt 11y ago
      If Raymond Hettinger says that, it's probably true. :-) I agree -- I've been using peewee for a side project, and it's great. I also love the fact that the source is one file, as that makes it so easy to find stuff in and navigate. I've been working offline, so didn't have access to the docs, and just referred to the source to figure out how to do things.
    - coleifer 11y ago
      It's my pleasure! And thank you for the wonderful libraries, talks, and who could forget those snippets.
    - JustSomeNobody 11y ago
      Ok, now I'm going to go study Peewee's code...
      - coleifer 11y ago
        Ackkkk I'm actually rewriting it currently to try and ameliorate the cruft that's crept in since the 2.0 rewrite. I want to say "But wait! If you read the code in a couple weeks it'll be so much better!"
bendmorris 11y ago
Interesting as it's quite different from how I generally structure my own projects. A couple questions: 1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that) 2. You've chosen to separate your code into MVC-inspired directories instead of by function. Flask/Django have ways (apps, blueprints) of making application subcomponents modular. It seems like a single controllers directory and a single templates directory, etc. could get cluttered pretty quickly. What will you do as projects grow?
- zphds 11y ago
  Not the OP, but w.r.t. [1] vendored dependencies are especially helpful if you happen to deploy the software on premise where you might not have full system access and in some cases, no internet or also strict policies on what can be installed.
  - snuxoll 11y ago
    You can vendor your dependencies inside a proper OS package as well. Write an rpmspec that creates a virtualenv, runs your setup.py with the virtualenv python interpreter and boom - your dependencies are bundled without needing to go through acrobatics with manually vendoring them. If your application isn't able to be installed at the system level I would really recommend this for handling deployment, regardless of whether you have full access or not - deploying tarballs with configuration management tools like puppet are hackish at best, RPM's/DEB's really are the best way to roll your software out. EDIT: As an alternative to virtualenv which is actually a PITA since --relocatable is always broken, buildout works great.
    - techdragon 11y ago
      I can't honestly imagine any circumstances when this would be better than using tools that can manage my Python packages as part of deploying or just using solid Python environment management practices. Why would I want to add the encumbrance of needing to work with system package toolchains to create packages when I want to avoid platform specific mechanisms to minimise my exposure to unexpected issues and retain visibility into failure causes since system package tools can be extremely opaque as to the cause of install failure causes. For example I use PyEnv instead of virtalenv/venv because PyEnv is written in bash and has a much better level of isolation than virtalenv or venv. It's simple bash scripting and the only system dependencies it has are based on features you choose to use. If you want to build Python from sources you'll need compilers and Libs, etc, but other than that sort of thing, it's zero dependencies. Edit: PyEnv also has the ability for me to compress an entire Python environment and reuse it somewhere else provided I'm using the same OS and system libraries, so I can pre-build compilation steps and cut down on system package dependencies in production environments.
hcarvalhoalves 11y ago
Project scaffolding like this feels very 2005.
- deleted 11y ago
  [deleted]
Terr_ 11y ago
> Well, because JSON is about the only format I can read on all the languages I use without ambiguity or extra dependencies Alas, there is a downside: The JSON spec does not allow comments, which are often important for configuration files. (Though given how much that simplifies parsing, it may have been the right decision.) So you either omit comemnts, or end up trying to shove placeholder key-values into the closest place you can to whatever you want to comment about.
- creshal 11y ago
  For configuration files I just put in comments anyway and strip them before feeding the file to a JSON parser. (Like here: https://github.com/creshal/yspave/blob/master/yspave/pave.py#L33 https://github.com/creshal/yspave/blob/master/yspave/pave.py... ) Hackish, but beats having to bother with YAML parsers and their regular RCE vulnerabilities.
sanpan 11y ago
here's a link to the utils mentioned in the article: https://github.com/rcarmo/python-utils https://github.com/rcarmo/python-utils
rcarmo 11y ago
Hi, OP here. Brief clarification: some of those patterns were a consequence of us having to run Python 2.5 and 2.6 on older boxes - it's a survival technique, if you will - and having to vendor some dependencies to make it easier to deploy stuff under some other interesting constraints the article won't get into. On the whole, these patterns make it easier for other people to not break your code when they want to add, say, another set of REST endpoints (which was my main occupation at the time). You might also want to read my short piece on SSE: http://taoofmac.com/space/blog/2014/11/16/1940 http://taoofmac.com/space/blog/2014/11/16/1940 And yeah, I've been meaning to change the blog layout. Hardly any time for it these days, honestly.
- emehrkay 11y ago
  Adding overflow: auto to your code blocks (.syntax) would allow us to see the whole snippet. I love decorators too, glad I'm not the only crazy person :)
  - rcarmo 11y ago
    Riiiight. Let me see if I can toss that in without breaking something else in the CSS.
- euphemize 11y ago
  > My ground rule is that anything I easy_install will go into env, and env will quite often be (re)built on a separate machine that matches my target environment. So I treat it as a disposable folder, and it’s never committed to any repos. Was wondering, do you pip freeze > requirements.txt? I don't see the file in there, but I assume that's what you mean since you mention that none of the env folders make it into version control. On a semi-related note, I used to have tons of projects like this a year ago, and having moved most of our python code to AWS lambda, our repos have shrank to maybe 25% of their original size. Granted it's not fabulous if you need to run a JS app on top of your Bottle framework, but we used to have a lot of small services here and there, the provisioning and deployment was so much work.
andy_ppp 11y ago
I have to say the using Pika and RabbitMQ directly has been a lot easier than trying to use Celery. Just my experience.
- neuroid 11y ago
  Just out of interest, what were the main challenges you faced when using Celery? Celery is a task queue. Pika could definitely be used to build a task queue (similarly to how Celery is built on top of Kombu), but I can't think of many good reasons why that would pay off. Unless your use case was not a task queue...
  - andy_ppp 11y ago
    Flask application context problems, hiding exceptions within tasks, Json Serialisation sometimes does weird things and hard to configure (the default, pickle, seems to be warned against as a security vulnerability), weirdness with Gevent/Socket.io flask stuff, difficulty (impossibility??) to emit socket events from within a worker, finally the canvas stuff seems great on the surface but I found it mind blowingly difficult to debug weirdness with it. My Pika/RabbitMQ stuff was much easier to build, unit test and debug (i.e. it actually works). There is every chance the I am just a bit retarded when it comes to using celery, but the fact I've built the Pika version maybe these were just real issues? Dunno, it might all work flawlessly for you in which case it's a great tool.

165 points by dedalus 11y ago

snuxoll 11y ago

Can I just state how awful this two-column layout is to read?

BinaryIdiot 11y ago
I'm glad I'm not the only one. Decided to skim it instead of reading it all the way through because it was annoying.
Fuzzwah 11y ago
I found the "toggle columns" option (top right, under the search box) and it made me happy.
- dr_zoidberg 11y ago
  While it did away with the horrible 2-column layout, it looks like the article was writen with those columns in mind, so it uses short paragraphs that look good on 2-column, but feel a bit weird on a wall of text. My (barely existant) OCD is going haywire over this article as I read it...
  - zodiakzz 11y ago
    OT: Anyone know why all browser vendors implemented this CSS columns aberration so hurridly..? I have never seen a good use for them, so painful to read. It's always some designer trying to show off their "epic" CSS skillz.
    - anewhnaccount 11y ago
      Seems like it might be useful for a print style sheet. Haven't seen it used to good effect on a screen though.
afarrell 11y ago
It goes away if you decrease the width of the viewport
praneshp 11y ago
Also, the picture with the directory tree got cropped out for me, and I couldn't figure out how to scroll it horizontally. Good thing the columns can be toggled.
deleted 11y ago
[deleted]
deleted 11y ago
[deleted]
agumonkey 11y ago
Way better than the single column one on my 1024x768 lcd.

mixmastamyk 11y ago

Bottle, peewee, gevent workers, eh? Interesting choices. SimpleTemplate looks interesting, I hadn't seen it before. Would be nice to hear some discussion there, instead of the column layout.

coleifer 11y ago
I'm the author of Peewee and I've been running my blog with Flask, Peewee, gevent and sqlite. It's a very low-impact combination, lightweight and fun to use, although I'm a bit biased.
- raymondh 11y ago
  Thank you for your work on Peewee. The code is nicely written and is easily read start to finish.
  - benhoyt 11y ago
    If Raymond Hettinger says that, it's probably true. :-) I agree -- I've been using peewee for a side project, and it's great. I also love the fact that the source is one file, as that makes it so easy to find stuff in and navigate. I've been working offline, so didn't have access to the docs, and just referred to the source to figure out how to do things.
  - coleifer 11y ago
    It's my pleasure! And thank you for the wonderful libraries, talks, and who could forget those snippets.
  - JustSomeNobody 11y ago
    Ok, now I'm going to go study Peewee's code...
    - coleifer 11y ago
      Ackkkk I'm actually rewriting it currently to try and ameliorate the cruft that's crept in since the 2.0 rewrite. I want to say "But wait! If you read the code in a couple weeks it'll be so much better!"

bendmorris 11y ago

Interesting as it's quite different from how I generally structure my own projects. A couple questions: 1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that) 2. You've chosen to separate your code into MVC-inspired directories instead of by function. Flask/Django have ways (apps, blueprints) of making application subcomponents modular. It seems like a single controllers directory and a single templates directory, etc. could get cluttered pretty quickly. What will you do as projects grow?

zphds 11y ago
Not the OP, but w.r.t. [1] vendored dependencies are especially helpful if you happen to deploy the software on premise where you might not have full system access and in some cases, no internet or also strict policies on what can be installed.
- snuxoll 11y ago
  You can vendor your dependencies inside a proper OS package as well. Write an rpmspec that creates a virtualenv, runs your setup.py with the virtualenv python interpreter and boom - your dependencies are bundled without needing to go through acrobatics with manually vendoring them. If your application isn't able to be installed at the system level I would really recommend this for handling deployment, regardless of whether you have full access or not - deploying tarballs with configuration management tools like puppet are hackish at best, RPM's/DEB's really are the best way to roll your software out. EDIT: As an alternative to virtualenv which is actually a PITA since --relocatable is always broken, buildout works great.
  - techdragon 11y ago
    I can't honestly imagine any circumstances when this would be better than using tools that can manage my Python packages as part of deploying or just using solid Python environment management practices. Why would I want to add the encumbrance of needing to work with system package toolchains to create packages when I want to avoid platform specific mechanisms to minimise my exposure to unexpected issues and retain visibility into failure causes since system package tools can be extremely opaque as to the cause of install failure causes. For example I use PyEnv instead of virtalenv/venv because PyEnv is written in bash and has a much better level of isolation than virtalenv or venv. It's simple bash scripting and the only system dependencies it has are based on features you choose to use. If you want to build Python from sources you'll need compilers and Libs, etc, but other than that sort of thing, it's zero dependencies. Edit: PyEnv also has the ability for me to compress an entire Python environment and reuse it somewhere else provided I'm using the same OS and system libraries, so I can pre-build compilation steps and cut down on system package dependencies in production environments.

hcarvalhoalves 11y ago

Project scaffolding like this feels very 2005.

deleted 11y ago
[deleted]

Terr_ 11y ago

> Well, because JSON is about the only format I can read on all the languages I use without ambiguity or extra dependencies Alas, there is a downside: The JSON spec does not allow comments, which are often important for configuration files. (Though given how much that simplifies parsing, it may have been the right decision.) So you either omit comemnts, or end up trying to shove placeholder key-values into the closest place you can to whatever you want to comment about.

creshal 11y ago
For configuration files I just put in comments anyway and strip them before feeding the file to a JSON parser. (Like here: https://github.com/creshal/yspave/blob/master/yspave/pave.py#L33 https://github.com/creshal/yspave/blob/master/yspave/pave.py... ) Hackish, but beats having to bother with YAML parsers and their regular RCE vulnerabilities.

sanpan 11y ago

here's a link to the utils mentioned in the article: https://github.com/rcarmo/python-utils https://github.com/rcarmo/python-utils

rcarmo 11y ago

Hi, OP here. Brief clarification: some of those patterns were a consequence of us having to run Python 2.5 and 2.6 on older boxes - it's a survival technique, if you will - and having to vendor some dependencies to make it easier to deploy stuff under some other interesting constraints the article won't get into. On the whole, these patterns make it easier for other people to not break your code when they want to add, say, another set of REST endpoints (which was my main occupation at the time). You might also want to read my short piece on SSE: http://taoofmac.com/space/blog/2014/11/16/1940 http://taoofmac.com/space/blog/2014/11/16/1940 And yeah, I've been meaning to change the blog layout. Hardly any time for it these days, honestly.

emehrkay 11y ago
Adding overflow: auto to your code blocks (.syntax) would allow us to see the whole snippet. I love decorators too, glad I'm not the only crazy person :)
- rcarmo 11y ago
  Riiiight. Let me see if I can toss that in without breaking something else in the CSS.
euphemize 11y ago
> My ground rule is that anything I easy_install will go into env, and env will quite often be (re)built on a separate machine that matches my target environment. So I treat it as a disposable folder, and it’s never committed to any repos. Was wondering, do you pip freeze > requirements.txt? I don't see the file in there, but I assume that's what you mean since you mention that none of the env folders make it into version control. On a semi-related note, I used to have tons of projects like this a year ago, and having moved most of our python code to AWS lambda, our repos have shrank to maybe 25% of their original size. Granted it's not fabulous if you need to run a JS app on top of your Bottle framework, but we used to have a lot of small services here and there, the provisioning and deployment was so much work.

andy_ppp 11y ago

I have to say the using Pika and RabbitMQ directly has been a lot easier than trying to use Celery. Just my experience.

neuroid 11y ago
Just out of interest, what were the main challenges you faced when using Celery? Celery is a task queue. Pika could definitely be used to build a task queue (similarly to how Celery is built on top of Kombu), but I can't think of many good reasons why that would pay off. Unless your use case was not a task queue...
- andy_ppp 11y ago
  Flask application context problems, hiding exceptions within tasks, Json Serialisation sometimes does weird things and hard to configure (the default, pickle, seems to be warned against as a security vulnerability), weirdness with Gevent/Socket.io flask stuff, difficulty (impossibility??) to emit socket events from within a worker, finally the canvas stuff seems great on the surface but I found it mind blowingly difficult to debug weirdness with it. My Pika/RabbitMQ stuff was much easier to build, unit test and debug (i.e. it actually works). There is every chance the I am just a bit retarded when it comes to using celery, but the fact I've built the Pika version maybe these were just real issues? Dunno, it might all work flawlessly for you in which case it's a great tool.