3 ms·

Ask HN: Python framework advice

10 points by sounddust 16y ago

This has been asked on HN before, but it was a couple of years ago and the development of Python frameworks is moving rapidly. I'm looking for advice on which Python-based frameworks we should use on a new project. This project will have a high level of traffic (~50 million page views/day), and will be ported from another platform from scratch, but we will need to use an existing PostgreSQL database (therefore it might be preferable not to use ORMs). We'll also need memcached support, as well as templating. We want something lightweight, flexible, that doesn't try to do too much for us or at least makes it easy to replace functionality (for example, we have our own session management and rate-limiting systems that we want to preserve). Also important is an active community, good documentation, and proof that it can scale (that is, it's already being used by large sites). I've looked at many frameworks, but none I've found so far seems to meet all these requirements. (web.py seems to be the most appealing to me, but doesn't seem to have any large clients; Django seems to be the most developed, but I'm concerned about the "doesn't try to do too much for you" requirement I've listed above). Any advice would be appreciated!

zeemonkee 16y ago
You'll probably get a lot of replies from people saying "just use Django", but that's just because it's what they are using. First of, going by your requirements, I'd pick SQLAlchemy for handling your database. Although SQLAlchemy includes an ORM, it's not just an ORM. It has an excellent data mapper layer you can use with legacy databases, without having to drop down to raw SQL: http://www.sqlalchemy.org/docs/sqlexpression.html http://www.sqlalchemy.org/docs/sqlexpression.html If you go with SQLAlchemy the next step is the choice of framework. There are a few around, but I can only provide advice on 3: Django, Pylons and Werkzeug. Django + SQLAlchemy is frankly not worth the bother. The real benefit to Django is the apps that go with it - for example authentication, the admin site, 3rd party apps such as Pinax and so on. These are great, but if you are building on top of a legacy DB then you won't be able to make much benefit of them, and unfortunately the Django ORM is tightly coupled with these apps. Django without these apps is not IMHO that compelling, as other components - such as the template engine - are done better elsewhere. Pylons is the next in popularity - the biggest site I know of running on Pylons is Reddit. It has more flexibility in choice of components, but lacks Django's app ecosystem and documentation is sorely lacking compared to Django. That said, you can use Mako or Jinja2, both excellent template engines, and you can integrate SQLAlchemy very easily. This lack of good, up to date docs is a pain point, as is the auto-generated modules - there is a lot of generated boilerplate code and it takes a while to learn where everything goes (routing, middleware, etc). Finally Werkzeug - not really a full framework as such, more a toolkit for building WSGI applications. This is my personal favourite, as it's well documented and designed, and lets you structure your application the way you want to. The tutorial explains how to integrate Jinja2 and SQLAlchemy. That said, you will spend some time laying out a Werkzeug application before you get going, unlike Django and Pylons, and I wouldn't recommend it to Python newbies. There is a new microframework - Flask (http://flask.pocoo.org http://flask.pocoo.org) built on top of Werkzeug which makes things much easier, but it's a very new development and is probably not production-ready quite yet.
- alnayyir 16y ago
  >It has an excellent data mapper layer you can use with legacy databases, without having to drop down to raw SQL: http://docs.djangoproject.com/en/dev/ref/django-admin/#inspectdb http://docs.djangoproject.com/en/dev/ref/django-admin/#inspe... >These are great, but if you are building on top of a legacy DB then you won't be able to make much benefit of them, and unfortunately the Django ORM is tightly coupled with these apps. InspectDB > migration > populate new models My current job is converting old systems and old databases for use with Django. It's pretty easy to convert old schemas and migrate data in Django. >Django without these apps is not IMHO that compelling, as other components - such as the template engine - are done better elsewhere. Nobody said you have to give up those benefits. Please don't give advice on decisions when you haven't actually dealt with what's under discussion. It doesn't sound like you've ever actually migrated anything to Django. I'd say pylons or Django, fiddle with both, see which one catches your fancy better. Don't take the advice of others outright. The more time I've spent with Django, the more my complaints have faded away. The more I learn about it, the more tricks and tools I find to deal with anything I encounter. There's really no point in dismissing it just because you don't understand inspectdb and migrations. Edit: For the actual migration process, if you're dealing with a rather large table (millions of rows) you might want to drop down to SQL for convenience/speed, but that's still relatively optional. Most of our migrations are in python + django ORM, not in SQL. (All but 2, out of ~15)
  - zeemonkee 16y ago
    I'm well aware of inspectdb and migrations. The OP however stated that he didn't want to use an ORM. And what if you need to support composite primary keys ? The ticket on that is 5 years old with no fix in sight: http://code.djangoproject.com/ticket/373 http://code.djangoproject.com/ticket/373 Frankly, the more I've used Django the more I've run up against it's limits. One of these limits is that the apps such as auth and admin are tightly coupled to the ORM and template engine. I've given my experiences with different frameworks, the pluses and minuses of each. It's up to the original poster to decide naturally. However, yet again any criticism of the One Holy Web Framework brings ad hominem arguments and angry rebuttals.
stse 16y ago
I'm also in the hunt for a python framework. I've had a look at several, most have already been mentioned here, like Flask and web.py. Personally I'm probably going to go with custom framework built on Werkzeug or modifying Flask. I think Tornado Web (http://www.tornadoweb.org/ http://www.tornadoweb.org/) might be an interesting option for you. Should be highly scalable and is being used by FriendFeed. There were some things that I didn't like about it, but I've seemed to have forgotten what exactly.
khill 16y ago
Pylons would be my tool of choice for this type of project. I prefer both SQLAlchmeny and Mako to the Django equivalents anyway but, personal preference aside, the fact that you have an existing data model and don't want something that "tries to do too much" make Django less desirable for this project.
scorpioxy 16y ago
Well, I like Django too but you pretty much removed it from the equation when you said you want a framework that doesn't do too much for you. That pretty much means you want a minimal framework. So perhaps you'd want to go with web.py which if i understand correctly is what reddit uses. Or maybe you'd just want to go with something like Cherrypy and build things on top of that.
- zeemonkee 16y ago
  Reddit uses Pylons, not web.py (it was originally written in Lisp, then moved to web.py, then Pylons).
  - scorpioxy 16y ago
    Ah ok. I missed that last part then. But I'd still recommend web.py or something similar because it seems to me that he doesn't really need a framework, he wants to put one together from parts he prefers. Doing that with something like Django is a bit painful.
    - zeemonkee 16y ago
      It would be good to know how experienced he is with Python. I recommended Werkzeug elsewhere, which would fit that requirement quite well, but I wouldn't necessarily recommend it to someone quite new to Python.
towndrunk 16y ago
I have recently started looking into Bottle for a personal project. So far I really like it. http://bottle.paws.de/ http://bottle.paws.de/
davisp 16y ago
WebOb, Routes, SQLAlchemy, Mako. My favorite framework is no framework. The bottom line is that its a sliding scale. Django will give you a large amount of functionality for free but requires that you do things the Django way. As you move along the scale towards projects like Werkzeug you'll inevitably lose some of the free stuff but you'll be able to make a lot more choices on how things work. You'll want to check Ian Bicking's blog for some good reading on the no-framework approach. [1] is a great place to start. [1] http://blog.ianbicking.org/2010/03/12/a-webob-app-example/ http://blog.ianbicking.org/2010/03/12/a-webob-app-example/
sounddust 16y ago
Quick note: The "50 million page views/day" is a typo. I meant 50 million per month!

10 points by sounddust 16y ago

zeemonkee 16y ago

You'll probably get a lot of replies from people saying "just use Django", but that's just because it's what they are using. First of, going by your requirements, I'd pick SQLAlchemy for handling your database. Although SQLAlchemy includes an ORM, it's not just an ORM. It has an excellent data mapper layer you can use with legacy databases, without having to drop down to raw SQL: http://www.sqlalchemy.org/docs/sqlexpression.html http://www.sqlalchemy.org/docs/sqlexpression.html If you go with SQLAlchemy the next step is the choice of framework. There are a few around, but I can only provide advice on 3: Django, Pylons and Werkzeug. Django + SQLAlchemy is frankly not worth the bother. The real benefit to Django is the apps that go with it - for example authentication, the admin site, 3rd party apps such as Pinax and so on. These are great, but if you are building on top of a legacy DB then you won't be able to make much benefit of them, and unfortunately the Django ORM is tightly coupled with these apps. Django without these apps is not IMHO that compelling, as other components - such as the template engine - are done better elsewhere. Pylons is the next in popularity - the biggest site I know of running on Pylons is Reddit. It has more flexibility in choice of components, but lacks Django's app ecosystem and documentation is sorely lacking compared to Django. That said, you can use Mako or Jinja2, both excellent template engines, and you can integrate SQLAlchemy very easily. This lack of good, up to date docs is a pain point, as is the auto-generated modules - there is a lot of generated boilerplate code and it takes a while to learn where everything goes (routing, middleware, etc). Finally Werkzeug - not really a full framework as such, more a toolkit for building WSGI applications. This is my personal favourite, as it's well documented and designed, and lets you structure your application the way you want to. The tutorial explains how to integrate Jinja2 and SQLAlchemy. That said, you will spend some time laying out a Werkzeug application before you get going, unlike Django and Pylons, and I wouldn't recommend it to Python newbies. There is a new microframework - Flask (http://flask.pocoo.org http://flask.pocoo.org) built on top of Werkzeug which makes things much easier, but it's a very new development and is probably not production-ready quite yet.

alnayyir 16y ago
>It has an excellent data mapper layer you can use with legacy databases, without having to drop down to raw SQL: http://docs.djangoproject.com/en/dev/ref/django-admin/#inspectdb http://docs.djangoproject.com/en/dev/ref/django-admin/#inspe... >These are great, but if you are building on top of a legacy DB then you won't be able to make much benefit of them, and unfortunately the Django ORM is tightly coupled with these apps. InspectDB > migration > populate new models My current job is converting old systems and old databases for use with Django. It's pretty easy to convert old schemas and migrate data in Django. >Django without these apps is not IMHO that compelling, as other components - such as the template engine - are done better elsewhere. Nobody said you have to give up those benefits. Please don't give advice on decisions when you haven't actually dealt with what's under discussion. It doesn't sound like you've ever actually migrated anything to Django. I'd say pylons or Django, fiddle with both, see which one catches your fancy better. Don't take the advice of others outright. The more time I've spent with Django, the more my complaints have faded away. The more I learn about it, the more tricks and tools I find to deal with anything I encounter. There's really no point in dismissing it just because you don't understand inspectdb and migrations. Edit: For the actual migration process, if you're dealing with a rather large table (millions of rows) you might want to drop down to SQL for convenience/speed, but that's still relatively optional. Most of our migrations are in python + django ORM, not in SQL. (All but 2, out of ~15)
- zeemonkee 16y ago
  I'm well aware of inspectdb and migrations. The OP however stated that he didn't want to use an ORM. And what if you need to support composite primary keys ? The ticket on that is 5 years old with no fix in sight: http://code.djangoproject.com/ticket/373 http://code.djangoproject.com/ticket/373 Frankly, the more I've used Django the more I've run up against it's limits. One of these limits is that the apps such as auth and admin are tightly coupled to the ORM and template engine. I've given my experiences with different frameworks, the pluses and minuses of each. It's up to the original poster to decide naturally. However, yet again any criticism of the One Holy Web Framework brings ad hominem arguments and angry rebuttals.

stse 16y ago

I'm also in the hunt for a python framework. I've had a look at several, most have already been mentioned here, like Flask and web.py. Personally I'm probably going to go with custom framework built on Werkzeug or modifying Flask. I think Tornado Web (http://www.tornadoweb.org/ http://www.tornadoweb.org/) might be an interesting option for you. Should be highly scalable and is being used by FriendFeed. There were some things that I didn't like about it, but I've seemed to have forgotten what exactly.

khill 16y ago

Pylons would be my tool of choice for this type of project. I prefer both SQLAlchmeny and Mako to the Django equivalents anyway but, personal preference aside, the fact that you have an existing data model and don't want something that "tries to do too much" make Django less desirable for this project.

scorpioxy 16y ago

Well, I like Django too but you pretty much removed it from the equation when you said you want a framework that doesn't do too much for you. That pretty much means you want a minimal framework. So perhaps you'd want to go with web.py which if i understand correctly is what reddit uses. Or maybe you'd just want to go with something like Cherrypy and build things on top of that.

zeemonkee 16y ago
Reddit uses Pylons, not web.py (it was originally written in Lisp, then moved to web.py, then Pylons).
- scorpioxy 16y ago
  Ah ok. I missed that last part then. But I'd still recommend web.py or something similar because it seems to me that he doesn't really need a framework, he wants to put one together from parts he prefers. Doing that with something like Django is a bit painful.
  - zeemonkee 16y ago
    It would be good to know how experienced he is with Python. I recommended Werkzeug elsewhere, which would fit that requirement quite well, but I wouldn't necessarily recommend it to someone quite new to Python.

towndrunk 16y ago

I have recently started looking into Bottle for a personal project. So far I really like it. http://bottle.paws.de/ http://bottle.paws.de/

davisp 16y ago

WebOb, Routes, SQLAlchemy, Mako. My favorite framework is no framework. The bottom line is that its a sliding scale. Django will give you a large amount of functionality for free but requires that you do things the Django way. As you move along the scale towards projects like Werkzeug you'll inevitably lose some of the free stuff but you'll be able to make a lot more choices on how things work. You'll want to check Ian Bicking's blog for some good reading on the no-framework approach. [1] is a great place to start. [1] http://blog.ianbicking.org/2010/03/12/a-webob-app-example/ http://blog.ianbicking.org/2010/03/12/a-webob-app-example/

sounddust 16y ago

Quick note: The "50 million page views/day" is a typo. I meant 50 million per month!