3 ms·
I once did a crawler that searched the content of a couple of large websites for phone numbers. I did this daemon in Perl, using fork() to search multiple webs
by bad_user 17y ago
I once did a crawler that searched the content of a couple of large websites for phone numbers.
I did this daemon in Perl, using fork() to search multiple websites in parallel. When a new search was initiated, the daemon broke the search in multiple packages (of 5000 items to be searched). After initiating 5000 searches (keeping the maximum number of active processes at 30), I did a wait for all the children to finish, so that I can mark the package as "done".
The trouble with fork is that opened sockets are sometimes not forked very well. And I forked my database connection (a DBD::Pg). Forking a DBD::Pg connection went well in my tests, but once in a few thousand forked processes, a process would stall. Sometimes it recovered after a few hours, sometimes it didn't.
I tried setting an ALARM that would make the process auto-kill itself after some time. Didn't work. The final workaround was to monitor the child processes from the parent, and kill -9 the ones that hanged. But the whole processing became too slow.
Finally I gave up on the idea of packages and on marking the search as "done". So I just marked the search as "done" after the last child was forked.
It was a difficult bug because the client had no idea it was a bug, and waited patiently for the results (sometimes as much as 2 days). It was also difficult because at first I had no idea why my processes hanged.