3 ms·
look through Indeed's pages of job results and click on all of the job links, but only in the center of the page where all of the jobs are posted (not on the ed
by logn 12y ago
look through Indeed's pages of job results and click on all of the job links, but only in the center of the page where all of the jobs are posted (not on the edges).
I wrote a toolkit to help solve this problem [1]. An issue with taking the approach of hard-coding the result pattern to scrape is that it can break when the page changes. E.g., the author's code has:
page_obj.find(id = 'resultsCol')
If Indeed ever changes that ID, the program won't work. In that respect, it's better to dynamically figure out where the results are.
And as far as cleaning up unicode and HTML entities, I like the "he" project [2]. Within text fields these HTML parsing libraries don't do a very good job so unfortunately this extra parsing is necessary. And sometimes properly stripping all duplicate whitespace involves getting very familiar with control characters (and corrupted control characters from bad encoding) as well as left-behind html tags/entities [3].
1. https://github.com/MachinePublishers/ScreenSlicer https://github.com/MachinePublishers/ScreenSlicer
2. https://github.com/mathiasbynens/he https://github.com/mathiasbynens/he
3. https://github.com/MachinePublishers/ScreenSlicer/blob/master/common/src/com/screenslicer/common/CommonUtil.java#L791 https://github.com/MachinePublishers/ScreenSlicer/blob/maste...
- zo1 12y agoCould you please elaborate on what you mean by "dynamically figure out where the results are"? Or how to go about doing it? #Edit. I see your first link sorta answers that. And correct me if I'm wrong, but when I went there, it seemed that the library caters more towards automatic searching + paging, rather than extracting results?
- logn 12y agoIt handles extraction too, trying to find where the results are and then extracting individually the title/summary/url/date. To elaborate on the general approach I used, it was to take each node in the web page and get stats about all of them (e.g., position on page, amount of freetext, etc) and plug those stats into a neural net. I worked on a different project some years ago that took the approach of looking for repeating tag patterns in the page, focusing especially on structural tags (as opposed to ones that are purely for formatting). Another possible approach might be to just plug the whole result page into something such as Boilerpipe (https://code.google.com/p/boilerpipe/ https://code.google.com/p/boilerpipe/) and look at the set of urls in the text block it identifies.