3 ms·

Hire YC: Need high-traffic Django help ASAP

6 points by mattdennewitz 18y ago

hey, we just re-launched out site w/ Django, and are cant handle the peak traffic load. we have apache in the back (mod_python), nginx in the front, memcached. the database doesn't seem to be the bottleneck. restarting apache yields fast response times up front, but gradually drifts to about 6-12 seconds per request. we can't handle getting crushed with unacceptable load times much more without serious consequences, so if anyone is available to help, email me at `mattdennewitz at gmail` and lets talk.

erlanger 18y ago
mod_wsgi. If nginx is serving as a proxy or a file server too? If the former, I'd go with HAProxy. If the latter, Lighttpd works very well with Python and FastCGI.
- mattdennewitz 18y ago
  just switched to modwsgi response time is up to 30 sec with 1 thread and 50 processes. nginx is up front as a proxy and a file server.
  - erlanger 18y ago
    Have you benchmarked the same app running on Lighty and FCGI? You shouldn't have to change much, if anything.
    - mattdennewitz 18y ago
      no. is that a common solution for hi-traffic service? what would lighty/fcgi replace?
      - a_toker 18y ago
        Lighttpd would serve both static files and the application. Reddit uses web.py with Lighty+FCGI, and I see no reason that you couldn't run your Django app on it. I use Apache + mod_wsgi for the Apache modules, but I'm taking a good look at lighttpd+FCGI to get more out of my server for my Django and Trac installations.
suhail 18y ago
buy a server load balance with nginx and throw a duplicate copy of everything on that and use the same db since it's not your bottleneck. Start looking for code you can optimize such as bad loops.
chiffonade 18y ago
... why can't you just add more application servers?
maney 18y ago
The first thing you want to do is measure the problem in enough detail to identify the cause. I can recommend Rob's Debug Toolbar (though with reservations about the possible overhead, though I'd have been happy to chance that when I needed to instrument an underperforming site): http://github.com/robhudson/django-debug-toolbar/ http://github.com/robhudson/django-debug-toolbar/ though his tree doesn't have the injectable measurement points I've added to the timer panel yet. I hesitate to suggest this for a production setting only because it's not had much third party exposure (or if it has no one's let me know), but "it works for me": http://github.com/mmaney/django-debug-toolbar/ http://github.com/mmaney/django-debug-toolbar/ You might want to avoid the most recent commit (so use da1429...) - Rob had asked about adding support for the cache API's "add" method, and I threw that together based on the spec; that one part is completely untested other than that it doesn't cause a compilation error in my test setting.

6 points by mattdennewitz 18y ago

erlanger 18y ago

mod_wsgi. If nginx is serving as a proxy or a file server too? If the former, I'd go with HAProxy. If the latter, Lighttpd works very well with Python and FastCGI.

mattdennewitz 18y ago
just switched to modwsgi response time is up to 30 sec with 1 thread and 50 processes. nginx is up front as a proxy and a file server.
- erlanger 18y ago
  Have you benchmarked the same app running on Lighty and FCGI? You shouldn't have to change much, if anything.
  - mattdennewitz 18y ago
    no. is that a common solution for hi-traffic service? what would lighty/fcgi replace?
    - a_toker 18y ago
      Lighttpd would serve both static files and the application. Reddit uses web.py with Lighty+FCGI, and I see no reason that you couldn't run your Django app on it. I use Apache + mod_wsgi for the Apache modules, but I'm taking a good look at lighttpd+FCGI to get more out of my server for my Django and Trac installations.

suhail 18y ago

buy a server load balance with nginx and throw a duplicate copy of everything on that and use the same db since it's not your bottleneck. Start looking for code you can optimize such as bad loops.

chiffonade 18y ago

... why can't you just add more application servers?

maney 18y ago

The first thing you want to do is measure the problem in enough detail to identify the cause. I can recommend Rob's Debug Toolbar (though with reservations about the possible overhead, though I'd have been happy to chance that when I needed to instrument an underperforming site): http://github.com/robhudson/django-debug-toolbar/ http://github.com/robhudson/django-debug-toolbar/ though his tree doesn't have the injectable measurement points I've added to the timer panel yet. I hesitate to suggest this for a production setting only because it's not had much third party exposure (or if it has no one's let me know), but "it works for me": http://github.com/mmaney/django-debug-toolbar/ http://github.com/mmaney/django-debug-toolbar/ You might want to avoid the most recent commit (so use da1429...) - Rob had asked about adding support for the cache API's "add" method, and I threw that together based on the spec; that one part is completely untested other than that it doesn't cause a compilation error in my test setting.