4 ms·

The last time I looked under the hood, I observed this behavior. 1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so

The last time I looked under the hood, I observed this behavior. 1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way. 2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. 3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been checking every 5 seconds to make sure apache is ok, spots this, and restarts apache. The site shows a fail-whale image for 5-10 seconds as SupervisorD does its thing. Then apache comes back online and we repeat the cycle. 4. About every 3 days, the semaphores pile up, and SupervisorD is unable to restart apache. Apache goes into FATAL mode and I have to run a funky command line command a friend showed me to clear out the semaphores so Apache can be restarted and taken out of its tailspin.

spooneybarger 16y ago
Is there a reason you are using apache in a threaded mode rather than pre-fork? Basically... is threaded a requirement or a choice? If a choice, have you tried running in pre-fork and if yes, do you get the same issues?
- brandnewlow 16y ago
  I did not set up our hosting environment, a developer friend of mine did, so I can't answer that first question. I don't know if Drupal requires threaded or not or how to try running it in pre-fork mode.
  - spooneybarger 16y ago
    I would assume that it isnt a requirement but rather an attempt to allow apache to handle more w/ less memory usage. Apache is a memory pig especially when using mod_* modules and running with in pre-fork. Can you describe as best as you know, your hosting environment?
  - brandnewlow 16y ago
    My hosting environment: The site is hosted at Slicehost, on two slices. Webserver: 4GB slice. DB server: 2GB slice. We do about 3-5k uniques/day. I've tried scaling the DB server up to 4GB to see if that solved the problem, but we continued crashing every hour. Part of the problem though with my own efforts is that I don't have much experience with sysadmin stuff to be able to say for sure that any of my own observations and attempts to solve were done properly. For instance, I scaled up the slice, but maybe I didn't restart Apache in the right way afterward to know if that made a difference or not.
    - spooneybarger 16y ago
      hn thread isnt the best way to try and ask all these questions. i sent you an email at the contact address listed in your profile.
nl 16y ago
2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. To me this indicates some pages loads are blocking. I've seen this before where page loads triggered blocking network calls (web services) and the web service client code wasn't configured to use timeouts. I'd take a look for that. Another thing you can do is to treat the symptoms. Setup a cron job that runs every X minutes and kills the Y oldest threads. From memory there is a config setting in Apache that specifies the number of times a thread should be reused, so try tweaking that, too. (Edit: if this is the case, then switching to Ngnix won't solve it. You really need to understand why that threadcount is rising)
chrisbolt 16y ago
2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. Threads or processes? If using the worker mpm, switch to prefork. If problems persist, strace a frozen process to see what it's doing. strace is an incredibly useful diagnostic tool. You could even set up a script to parse the server-status page, find out which processes are hung based on the SS column (seconds since beginning of most recent request) and kill that PID... but that's not pretty.

spooneybarger 16y ago

Is there a reason you are using apache in a threaded mode rather than pre-fork? Basically... is threaded a requirement or a choice? If a choice, have you tried running in pre-fork and if yes, do you get the same issues?

brandnewlow 16y ago
I did not set up our hosting environment, a developer friend of mine did, so I can't answer that first question. I don't know if Drupal requires threaded or not or how to try running it in pre-fork mode.
- spooneybarger 16y ago
  I would assume that it isnt a requirement but rather an attempt to allow apache to handle more w/ less memory usage. Apache is a memory pig especially when using mod_* modules and running with in pre-fork. Can you describe as best as you know, your hosting environment?
- brandnewlow 16y ago
  My hosting environment: The site is hosted at Slicehost, on two slices. Webserver: 4GB slice. DB server: 2GB slice. We do about 3-5k uniques/day. I've tried scaling the DB server up to 4GB to see if that solved the problem, but we continued crashing every hour. Part of the problem though with my own efforts is that I don't have much experience with sysadmin stuff to be able to say for sure that any of my own observations and attempts to solve were done properly. For instance, I scaled up the slice, but maybe I didn't restart Apache in the right way afterward to know if that made a difference or not.
  - spooneybarger 16y ago
    hn thread isnt the best way to try and ask all these questions. i sent you an email at the contact address listed in your profile.

nl 16y ago

2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. To me this indicates some pages loads are blocking. I've seen this before where page loads triggered blocking network calls (web services) and the web service client code wasn't configured to use timeouts. I'd take a look for that. Another thing you can do is to treat the symptoms. Setup a cron job that runs every X minutes and kills the Y oldest threads. From memory there is a config setting in Apache that specifies the number of times a thread should be reused, so try tweaking that, too. (Edit: if this is the case, then switching to Ngnix won't solve it. You really need to understand why that threadcount is rising)

chrisbolt 16y ago

2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. Threads or processes? If using the worker mpm, switch to prefork. If problems persist, strace a frozen process to see what it's doing. strace is an incredibly useful diagnostic tool. You could even set up a script to parse the server-status page, find out which processes are hung based on the SS column (seconds since beginning of most recent request) and kill that PID... but that's not pretty.