5 ms·

Sounds like a dogpile ;-)

kelp 7y ago
I worked at Go2Net when they acquired Dogpile. I was directly involved in migrating it’s systems to our data centers after the acquisition closed. I don’t really remember the tech stack, but it was something like 10-15 web servers, probably pentium II or III. One detail I do remember is the original setup had an Apache log rotation script that would try several times to restart Apache and if it didn’t succeed, it would reboot the server. This ran nightly... Not sure what bug or issue they were working around. But I thought it was quite an aggressive solution!
- jaeh 7y ago
  "let's just leave this ugly hack here for tonight and get some logs about the restarts, then delete it tomorrow..." "oh, we forgot the print. lets retry tonight." "darn, did not happen. lets leave this part in our infra for now until we trigger the edgecase again." 5 weeks (or days...) later: "damn, why did we do that again? better not touch it" i love it when i find myself cargo culting code like that and finally have the time (and courage) to remove it :)
- toast0 7y ago
  Apache still doesn't know how to reopen log files when they're rotated :(
  - pushpop 7y ago
    That’s news to me. I’ve been managing Apache hosts for well over a decade and never had an issue. Used apache 1.3, 2.2 and 2.4. What’s your log rotation code look like?
    - toast0 7y ago
      You have to do a graceful restart after rotation, according to the docs, which then leads to scripts that try a couple times and then reboot. (It's been a while, but I did have an install where graceful restart would work once, but consistently fail on the second rotation). https://httpd.apache.org/docs/2.4/logs.html#rotation https://httpd.apache.org/docs/2.4/logs.html#rotation
      - pushpop 7y ago
        If you don’t change the inode then no restart should be required. There’s definitely a few tools out there that solve this problem without needing to restart httpd.
        toast0 7y ago
        How do you rotate without changing the inode?
        pushpop 7y ago
        Sorry, my previous comment was really unhelpful. What I meant was you should be using pipes and apaches own log rotate code. Read the “Logging Using Pipes” section at the very end. https://www.digitalocean.com/community/tutorials/how-to-configure-logging-and-log-rotation-in-apache-on-an-ubuntu-vps https://www.digitalocean.com/community/tutorials/how-to-conf... (Excuse the DO link, it was just the first result in Duck Duck Go. There’s nothing specific about DO or Ubuntu in this approach) I’ve used this method on large clusters of very heavily utilised web servers and it works great. No restarts required.

kelp 7y ago

I worked at Go2Net when they acquired Dogpile. I was directly involved in migrating it’s systems to our data centers after the acquisition closed. I don’t really remember the tech stack, but it was something like 10-15 web servers, probably pentium II or III. One detail I do remember is the original setup had an Apache log rotation script that would try several times to restart Apache and if it didn’t succeed, it would reboot the server. This ran nightly... Not sure what bug or issue they were working around. But I thought it was quite an aggressive solution!

jaeh 7y ago
"let's just leave this ugly hack here for tonight and get some logs about the restarts, then delete it tomorrow..." "oh, we forgot the print. lets retry tonight." "darn, did not happen. lets leave this part in our infra for now until we trigger the edgecase again." 5 weeks (or days...) later: "damn, why did we do that again? better not touch it" i love it when i find myself cargo culting code like that and finally have the time (and courage) to remove it :)
toast0 7y ago
Apache still doesn't know how to reopen log files when they're rotated :(
- pushpop 7y ago
  That’s news to me. I’ve been managing Apache hosts for well over a decade and never had an issue. Used apache 1.3, 2.2 and 2.4. What’s your log rotation code look like?
  - toast0 7y ago
    You have to do a graceful restart after rotation, according to the docs, which then leads to scripts that try a couple times and then reboot. (It's been a while, but I did have an install where graceful restart would work once, but consistently fail on the second rotation). https://httpd.apache.org/docs/2.4/logs.html#rotation https://httpd.apache.org/docs/2.4/logs.html#rotation
    - pushpop 7y ago
      If you don’t change the inode then no restart should be required. There’s definitely a few tools out there that solve this problem without needing to restart httpd.
      - toast0 7y ago
        How do you rotate without changing the inode?
        pushpop 7y ago
        Sorry, my previous comment was really unhelpful. What I meant was you should be using pipes and apaches own log rotate code. Read the “Logging Using Pipes” section at the very end. https://www.digitalocean.com/community/tutorials/how-to-configure-logging-and-log-rotation-in-apache-on-an-ubuntu-vps https://www.digitalocean.com/community/tutorials/how-to-conf... (Excuse the DO link, it was just the first result in Duck Duck Go. There’s nothing specific about DO or Ubuntu in this approach) I’ve used this method on large clusters of very heavily utilised web servers and it works great. No restarts required.