3 ms·

Reminds me of this story I heard about Yahoo. Their ads server had a memory leak and it would OOM after something like 10000 requests. Their solution: restart

Reminds me of this story I heard about Yahoo. Their ads server had a memory leak and it would OOM after something like 10000 requests. Their solution: restart the server after 8000 requests. This worked for a year or two. And then it started OOM-ing after 8000 requests. Next solution: restart the server after 6000 requests.

xandrius 2y ago
If that gives you extra time to move the problem to the future, I'd say that's a win :D
otabdeveloper4 2y ago
> Their solution: restart the server after 8000 requests. 8000 requests is something like 500 milliseconds for the average ad server. You need exceptionally fast restarts for that to work.
- wging 2y ago
  Perhaps now, but the story is about Yahoo, which means it could be from the early 2000s or late 90s. Traffic volumes were probably lower, computers were definitely slower, internet advertising was not as big as it is now, etc.
- toast0 2y ago
  This is in the context of (y)Apache. You set MaxRequestsPerChild, so when the request limit is hit, the child is killed, a new one started, and requests can be served by other children until the replacement is ready. In a pure config, idle children block on accept, so the kernel does all the coordination work required to get requests to children, the child exits after hitting the cap, and the parent catches the exit and spawns another. As long as all the children don't hit their cap at once, there's no gap in service. If they do all hit the cap at once, it's still not so bad. I don't know about ads, but on Yahoo Travel, I had no qualms about solving memory leaks that took months to show up with MaxRequestsPerChild 100000. I gave valgrind or something a whirl, but didn't find it right away and was tired of getting paged once a quarter for it, so... I did do some scaleout work for Shopping once, and found theirs set at 3. I don't remember where I left it, but much higher. Nobody knew why it was 3, so I took the chance that it would become aparrent if I set it to 100, and nothing obvious was wrong and the servers were much happier. fork() is fast, but it's not fast enough to do it every 3 requests.

xandrius 2y ago

If that gives you extra time to move the problem to the future, I'd say that's a win :D

otabdeveloper4 2y ago

> Their solution: restart the server after 8000 requests. 8000 requests is something like 500 milliseconds for the average ad server. You need exceptionally fast restarts for that to work.

wging 2y ago
Perhaps now, but the story is about Yahoo, which means it could be from the early 2000s or late 90s. Traffic volumes were probably lower, computers were definitely slower, internet advertising was not as big as it is now, etc.
toast0 2y ago
This is in the context of (y)Apache. You set MaxRequestsPerChild, so when the request limit is hit, the child is killed, a new one started, and requests can be served by other children until the replacement is ready. In a pure config, idle children block on accept, so the kernel does all the coordination work required to get requests to children, the child exits after hitting the cap, and the parent catches the exit and spawns another. As long as all the children don't hit their cap at once, there's no gap in service. If they do all hit the cap at once, it's still not so bad. I don't know about ads, but on Yahoo Travel, I had no qualms about solving memory leaks that took months to show up with MaxRequestsPerChild 100000. I gave valgrind or something a whirl, but didn't find it right away and was tired of getting paged once a quarter for it, so... I did do some scaleout work for Shopping once, and found theirs set at 3. I don't remember where I left it, but much higher. Nobody knew why it was 3, so I took the chance that it would become aparrent if I set it to 100, and nothing obvious was wrong and the servers were much happier. fork() is fast, but it's not fast enough to do it every 3 requests.