4 ms·

I wrote my own job search engine after getting fed up of the existing crap. It pulls in rss feeds and mailing lists and even does a bit of crawling via HN and R

by jamii 16y ago

I wrote my own job search engine after getting fed up of the existing crap. It pulls in rss feeds and mailing lists and even does a bit of crawling via HN and Reddit to find jobs. I used a trainable classifier to filter out most of the dross. The main problem with job sites is matching jobs to applicants. As far as I know noone has gone beyond simple keyword searching. If you had an actual userbase you could make use of collaborative filtering techniques (like Amazons recommendations) to provide a tailored service to each user. CVs, personal blogs and github accounts give you enough information to seed each users recommendations even before they start rating jobs.

lachyg 16y ago
This is what I'm trying to work on. How did you train your search engine? What sort of collaborative filtering techniques? How would that work? I'm intrigued! Would love to chat further via email.
- jamii 16y ago
  > How did you train your search engine? I used some pretty simple techniques: one bayesian filter to filter jobs from other posts in mailing lists etc, one bayesian filter to score jobs based on keywords. Both filters were trained by feedback from the console ui. The main problem is excluding site-specific keywords that distort the scoring (eg if a site with mostly crappy jobs includes its own name in the listing then even the good jobs will score low by association). A lot of job sites have manky markup so I also had a different scraping script for each site to extract text. All in all its only a couple of hours work. I've been thinking recently about extending it and adding a simple web ui, since finding freelance work is pretty time consuming. > What sort of collaborative filtering techniques? I didn't have any specific in mind but there are plenty of good machine learning books that cover different tecniques. If you don't already have a background in maths then 'Programming Collective Intelligence' is a good book to start with. 'The Elements of Statistical Learning' goes into a lot more detail but requires some basic maths. > Would love to chat further via email. Email is in my profile.
jlovelass 16y ago
I agree that most of the existing job search engines are crap. Keyword searches aren't effective and everyone involed gets bombarded with irrelevant responses. Check out www.wyckwyre.com we created this because we were fed uo just as you are.

by jamii 16y ago

lachyg 16y ago

This is what I'm trying to work on. How did you train your search engine? What sort of collaborative filtering techniques? How would that work? I'm intrigued! Would love to chat further via email.

jamii 16y ago
> How did you train your search engine? I used some pretty simple techniques: one bayesian filter to filter jobs from other posts in mailing lists etc, one bayesian filter to score jobs based on keywords. Both filters were trained by feedback from the console ui. The main problem is excluding site-specific keywords that distort the scoring (eg if a site with mostly crappy jobs includes its own name in the listing then even the good jobs will score low by association). A lot of job sites have manky markup so I also had a different scraping script for each site to extract text. All in all its only a couple of hours work. I've been thinking recently about extending it and adding a simple web ui, since finding freelance work is pretty time consuming. > What sort of collaborative filtering techniques? I didn't have any specific in mind but there are plenty of good machine learning books that cover different tecniques. If you don't already have a background in maths then 'Programming Collective Intelligence' is a good book to start with. 'The Elements of Statistical Learning' goes into a lot more detail but requires some basic maths. > Would love to chat further via email. Email is in my profile.

jlovelass 16y ago

I agree that most of the existing job search engines are crap. Keyword searches aren't effective and everyone involed gets bombarded with irrelevant responses. Check out www.wyckwyre.com we created this because we were fed uo just as you are.