5 ms·

Show HN: JobLens – AI-powered job search for 'Who Is Hiring'

108 points by hubraumhugo 3y ago

There are existing HN job aggregators, but I thought we could take it a step further. Inspired by an insightful comment on a previous thread (https://news.ycombinator.com/item?id=36163021 https://news.ycombinator.com/item?id=36163021), I built a tool that aggregates job postings and intelligently categorizes them based on user-specific preferences: * Country and remote work preferences * Employer type (e.g., startup, corporation, government) * Industry * Technologies used * Role type (developer, architect, product owner, etc.) * Salary range (where available) One of the superpowers of LLMs is reformatting information from any format X to any other format Y. We leverage this to map all the unstructured job postings into the same unified structure. The new GPT functions feature and the extended context windows are really helpful for this. Instead of having to build a custom NER pipeline, it works very well with GPT out-of-the box. One challenge is keeping the filters consistent and merging of duplicates. Embeddings help with that. What's next: * Integrate additional sources. We can generate web scrapers and data processing steps on the fly that extract and transform data into the same structure. * Add location distance filters. * Expand beyond jobs to monitor personalized data like events or real estate. Imagine using AI to rate local events from multiple sources based on your preferences, considering factors like your interests and distance from home. * Smaller improvements based on your feedback :)

rahimnathwani 3y ago
This is cool. I'm curious about this part: We leverage this to map all the unstructured job postings into the same unified structure. The new GPT functions feature and the extended context windows are really helpful for this. Does this mean that, rather than using the gpt-3.5-turbo chat API, and using a system prompt that describes the output format you want (JSON?), you instead describe your output format in the same way you would if you were going to call a function?
- hubraumhugo 3y ago
  Yes, you can essentially use GPT functions to reliably output JSON based on a predefined schema. See https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...
boxcarr 3y ago
JobLens is cool! This past month we had the same idea -- it's been a fun project: https://hnjobs.u-turn.dev https://hnjobs.u-turn.dev ChatGPT does an incredible job parsing, but then lots of effort goes into normalizing and deduping each field. Long story short, your results look quite good to me!
- hubraumhugo 3y ago
  Your project looks quite impressive as well, especially the extracted URLs to apply and the candidate profiles, didn't get that far yet :) Automating tedious work like data extraction and transformation is a great use case for LLMs.
  - boxcarr 3y ago
    Not sure if you found this as well, but gpt-3.5-turbo-016 does a poor job following instructions other than parsing. So, to work around this, we prompt gpt-3.5-turbo with the rules we want applied say an extracted field and then go back to gpt-3.5-turbo-016 to parse with chatgpt functions. Bottom line, every single post requires approximately 10 different prompts to refine the extraction.
    - hubraumhugo 3y ago
      yeah I had the same experience, divide and conquer works better than trying to do everything with a single prompt. Are you interested in comparing notes? Feel free to ping me via the email in my profile :)
SCUSKU 3y ago
Wow this is amazing. This is more or less what I was trying to build https://hnresumetojobs.com https://hnresumetojobs.com to look like, I am reconsidering whether or not to continue now. I tried using the OpenAI functions API to do structured text extraction, but found that it would hallunicate a lot of things. Do you guys have that problem? How did you guys go about solving that?
- OccamsMirror 3y ago
  Ask ChatGPT to improve your prompt to stop it hallucinating. I’ve had a lot of wins doing that.
- SomewhatLikely 3y ago
  Having the 'unchecked' visual actually have a checkmark threw me off. I was thinking, oh goodness, I have to deselect all of these.
timhigins 3y ago
Site is down for me, maybe the HN hug of death? Anyone else seeing this?
baxtr 3y ago
Nice! Is there something similar for freelancing gigs?
version_five 3y ago
Hi, just a thought, I backed out as soon as I saw I had to pick from a giant list of industries. Most people don't have rigid criteria like that, and if the do, a conventional search is easy. Conversational AI has a potential to more naturally arrive at interesting jobs, either through asking for a freeform description of what you want, or maybe a back and forth discussion.
- great_psy 3y ago
  I got turned off by that as well, but I submitted the form anyway, and it loads correctly. I guess no fill in means *
- hubraumhugo 3y ago
  You're right, I'll improve this.
digitalsanctum 3y ago
I was going to build something similar for funsies but now I don't have to :) I'm curious the page mentions not being able to scrape sites that require a login but then I saw LinkedIn as an example. Will LinkedIn eventually be supported but require a login or something?
Laaas 3y ago
All I want is a list of jobs that involve functional programming.
mmaia 3y ago
Looks like it's getting some fields wrong. For examples, "Global" location is matching other text not related to the job location. Still, a good example of OpenAI's functions. I was using direct prompt for YAML parsing before and migrated to functions for my job board https://aijobnetwork.com/ https://aijobnetwork.com/ I'm also using OpenAI's embeddings for recommendations: https://medium.com/@mauricio/how-to-add-recommendations-to-a-rails-app-with-pgvector-and-openai-881d87915fb2 https://medium.com/@mauricio/how-to-add-recommendations-to-a...
dsgnr 3y ago
Need a category for design. Right now all design jobs are scattered around, mostly in product management. Also as others have mentioned, having to make all these decisions before searching is hard to get through without abandoning.

108 points by hubraumhugo 3y ago

rahimnathwani 3y ago

This is cool. I'm curious about this part: We leverage this to map all the unstructured job postings into the same unified structure. The new GPT functions feature and the extended context windows are really helpful for this. Does this mean that, rather than using the gpt-3.5-turbo chat API, and using a system prompt that describes the output format you want (JSON?), you instead describe your output format in the same way you would if you were going to call a function?

hubraumhugo 3y ago
Yes, you can essentially use GPT functions to reliably output JSON based on a predefined schema. See https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...

boxcarr 3y ago

JobLens is cool! This past month we had the same idea -- it's been a fun project: https://hnjobs.u-turn.dev https://hnjobs.u-turn.dev ChatGPT does an incredible job parsing, but then lots of effort goes into normalizing and deduping each field. Long story short, your results look quite good to me!

hubraumhugo 3y ago
Your project looks quite impressive as well, especially the extracted URLs to apply and the candidate profiles, didn't get that far yet :) Automating tedious work like data extraction and transformation is a great use case for LLMs.
- boxcarr 3y ago
  Not sure if you found this as well, but gpt-3.5-turbo-016 does a poor job following instructions other than parsing. So, to work around this, we prompt gpt-3.5-turbo with the rules we want applied say an extracted field and then go back to gpt-3.5-turbo-016 to parse with chatgpt functions. Bottom line, every single post requires approximately 10 different prompts to refine the extraction.
  - hubraumhugo 3y ago
    yeah I had the same experience, divide and conquer works better than trying to do everything with a single prompt. Are you interested in comparing notes? Feel free to ping me via the email in my profile :)

SCUSKU 3y ago

Wow this is amazing. This is more or less what I was trying to build https://hnresumetojobs.com https://hnresumetojobs.com to look like, I am reconsidering whether or not to continue now. I tried using the OpenAI functions API to do structured text extraction, but found that it would hallunicate a lot of things. Do you guys have that problem? How did you guys go about solving that?

OccamsMirror 3y ago
Ask ChatGPT to improve your prompt to stop it hallucinating. I’ve had a lot of wins doing that.
SomewhatLikely 3y ago
Having the 'unchecked' visual actually have a checkmark threw me off. I was thinking, oh goodness, I have to deselect all of these.

timhigins 3y ago

Site is down for me, maybe the HN hug of death? Anyone else seeing this?

baxtr 3y ago

Nice! Is there something similar for freelancing gigs?

version_five 3y ago

Hi, just a thought, I backed out as soon as I saw I had to pick from a giant list of industries. Most people don't have rigid criteria like that, and if the do, a conventional search is easy. Conversational AI has a potential to more naturally arrive at interesting jobs, either through asking for a freeform description of what you want, or maybe a back and forth discussion.

great_psy 3y ago
I got turned off by that as well, but I submitted the form anyway, and it loads correctly. I guess no fill in means *
hubraumhugo 3y ago
You're right, I'll improve this.

digitalsanctum 3y ago

I was going to build something similar for funsies but now I don't have to :) I'm curious the page mentions not being able to scrape sites that require a login but then I saw LinkedIn as an example. Will LinkedIn eventually be supported but require a login or something?

Laaas 3y ago

All I want is a list of jobs that involve functional programming.

mmaia 3y ago

Looks like it's getting some fields wrong. For examples, "Global" location is matching other text not related to the job location. Still, a good example of OpenAI's functions. I was using direct prompt for YAML parsing before and migrated to functions for my job board https://aijobnetwork.com/ https://aijobnetwork.com/ I'm also using OpenAI's embeddings for recommendations: https://medium.com/@mauricio/how-to-add-recommendations-to-a-rails-app-with-pgvector-and-openai-881d87915fb2 https://medium.com/@mauricio/how-to-add-recommendations-to-a...

dsgnr 3y ago

Need a category for design. Right now all design jobs are scattered around, mostly in product management. Also as others have mentioned, having to make all these decisions before searching is hard to get through without abandoning.