5 ms·

https://www.unlistedjobs.com/ https://www.unlistedjobs.com/ Scraper of job listings directly from company websites. I found my last day job by using a scraper

by Jabbs 2y ago

https://www.unlistedjobs.com/ https://www.unlistedjobs.com/ Scraper of job listings directly from company websites. I found my last day job by using a scraper that visits company websites in search of job listings. Now I've turned it into an app for others to use and access jobs that are posted on company websites (rather than paid employer ads on Indeed or wherever). This gives the job searcher an advantage to find jobs not listed on job search sites and show the company you have taken time/interest to visit their site.

dchuk 2y ago
I’ve had a similar idea over the years. You should consider exploring whether competitive companies could be customers. As a competitor, getting alerts about roles another company is hiring for can be very interesting. Combine it with trends of postings over time…
- Jabbs 2y ago
  Oh that's a really interesting idea. Yea I dislike the idea of charging the job seeker but have not found a good way to monetize companies (not that they even know about me anyway)
stuckkeys 2y ago
What are you using to for the scraping? Playwright…selenium? I wanted to do something as a hobby but my IP kept getting reported lol. Also when you say companies…where are you getting the information from? Data brokers? Anyway, it is an interesting topic to me.
- Jabbs 2y ago
  Selenium, although I'm using a wrapper library that uses it. I only query each company every few days or so which probably helps to not get banned IP-wise but also rotate them. But many of the company job links are through external sources too (lever, greenhouse, etc.) which don't seem to mind The company data was gathered online for a long time until I found https://www.thecompaniesapi.com/ https://www.thecompaniesapi.com/ (which now is the source for much of that data)
  - stuckkeys 2y ago
    I tried to use my own desktop machine to process some of these tasks. I can see my fans go jet mode when the scraping was being done lol. Do you have discord or any way to connect? Would love to chat around this topic. Feel free to drop any social media handles. I’ll ping you.
    - Jabbs 2y ago
      Ohhh yea I run into this memory issue very quickly when scraping (especially if you have a large URL dataset then it will inevitably find a website with a giant bit of markup). So I have to set timeouts and blacklist timely requests but also completely reset the (headless) browser on 2-3 requests (which is overkill but I am restricted on memory for those workers). Feel free to drop me an email sometime (should be on my HN profile)
fusslo 2y ago
can I make feature requests? I would love a map of job postings to see where it might make sense to move to in the future. If there's 10 jobs within 50 miles... that might be a good place to buy a house. Additionally, if I filter by 'north america' I still get jobs from canada and india because they're remote only. I would LOVE to be able to filter out those positions. Also I would love to be able to AND 'remote' and 'north america'. I would like to work remotely, but only for US companies love your site <3
- Jabbs 2y ago
  Thank you :) I appreciate the request + feedback. I have a story in the backlog to add location-specific links to the landing page but I really like your idea of having a map (heatmap or something) to show densities of jobs. So the inclusive vs exclusive filtering is something that I struggle to perfect here. I'm tempted to throw both in the UI (since its ready to go on the backend) but its hard to explain to users. One thing you can do that is not so obvious is add a tag for "Canada" but click on the tag again which will put a line through Canada and exclude that location from your filter (still need to have helpers to show users how to do that). The 'remote' tag is probably the toughest one to parse of a job listing because it might appear anywhere within the text, so there is some inaccuracies for sure but its improving I hope! Ah I could probably add filters for company locations specifically too (so you can filter US companies), that's an interesting use case too. Thanks for the compliment too, it has been really fun to build
registeredcorn 2y ago
Here's a bit of feedback: * Job listings for "Quality Assurance" and "QA" are split into different listings in Job Search. * I really like the green highlight for Salary range! Personally, I would sort by jobs that list salary first, then by location (or relevance, or whatever). * The filter was a little confusing to use. I see you talked about it with other users in here. It needs some love, but it's getting there. :) * If you are going to target job searchers, it would be very helpful too see metrics based on the results. Here's a few examples I came up with Example 1: I select Help Desk -> Chicago I see a short-term graph showing whether demand has gone: up, down, or stayed the same - included is a red/green/yellow arrow giving me an idea at a glance. This helps me understand how many Help Desk postings are in Chicago Example 2: I select Cybersecurity -> I also select Information Security -> NYC I see a short-term graph showing demand for Cyber vs IS in NYC. This helps me understand if which job has more postings in NYC. Example 3: I select Python Developer -> Boston & Dallas. I see a medium-term graph showing demand for each location for Python Developer. This helps me decide whether demand is more consistent in Boston or Dallas. Example 4: I select Asia & Canada -> Advertising (Under Industry) I see a long-term graph showing the overall trend for that industry in each of those countries. This helps me track whether jobs are being outsourced, what I should expect in the coming years, and/or which country is the most competitive in that industry. Hope that helps! Good luck. :)
- Jabbs 2y ago
  This is great feedback, I'm very appreciative! Yea there is a much better version of the search bar soon-to-deploy (which accounts for aliases like QA -> Quality Assurance) and it will match by word rather than the entire phrase (currently "software engineer" will not query "software test engineer"). Appreciate the callout here You can find a toggle switch for "has salary" under the "other" filters which will show only those w salary, but good call perhaps that should be part of a sort feature (beyond just date) The filters do need more love for sure. I like your examples for various metrics displayed in the UI. I did think it would be cool to have a Github-like array of squares that represent units of time with colors that show how it has been changing over time, would have to figure out how heavy of calculations those would be in real-time but I really like your idea here. Or a line chart might be better. Many thanks for all the input!
windowshopping 2y ago
How did you design and build your homepage? I find that building the landing page and making it look like a professional, beautiful design is one of my biggest hurdles. I'm an experienced web developer but without a design to work off of - and especially accounting for mobile and dynamic sizing - I really struggle with this part of the work so I'm wondering what other people's workflows are for it.
- kmoser 2y ago
  Not OP but I have two solutions for this: 1) find an existing site and mimic their design, or 2) hire a designer
  - windowshopping 2y ago
    Yes, I know, that's very easy to say, but in my experience: 1.) Imitating a good design can take a lot of developer hours. Many landing pages have fancy css effects and support many screen sizes and it can take quite some time to build. 2.) Finding a good designer is not easy. I have tried hiring on most freelancer sites with very poor results. The applicant pool is typically of low quality. So I am wondering if others have found efficient/effective ways of going about #1 or #2, either by using certain tools or templates or by having a more clever hiring method.
    - kmoser 2y ago
      Certainly creating a robust design that supports multiple screen sizes is no trivial matter. There are tools to make it easier but as with anything else, no single solution fits all developers and all projects. You say you're an experienced web developer and yet you find yourself deficient in this particular skill, so I'm not sure what to think. Maybe your strength lies more in back end programming? In any case, I suggest looking into frameworks such as Bootstrap or Tailwind. Of course, there are also high-level solutions that don't require as much coding, e.g. WordPress, Wix, Squarespace, etc. WordPress plugins can make it very easy to apply fancy CSS effects without writing any CSS. As for finding freelancers, I agree that's also a difficult task. I don't have any magic bullet there, other than to prioritize people with good, timely communication skills, and to avoid people on the low end of the price range since they will usually be less experienced.
suchintan 2y ago
This is really cool. Do you have any interest in helping people auto apply to them? We can help you set it up with a really simple API call via Skyvern (https://github.com/Skyvern-AI/skyvern https://github.com/Skyvern-AI/skyvern)
- Jabbs 2y ago
  Thank you, appreciate this. I'm not certain about expanding this in that direction but that is certainly an interesting thought.
  - suchintan 2y ago
    Yep. Makes a lot of sense. Your side project is super noble -- thank you for that
NullPrefix 2y ago
I've tried searching for C++ but the search bar straight up refuses it
- Jabbs 2y ago
  The search bar is for job title only at the moment. Under the "Tech" filter you can find "C++" and add it as a filter
  - NullPrefix 2y ago
    thanks
zoke 2y ago
I see C++ and C#. Why no C?
- Jabbs 2y ago
  Yea that one is still a WIP (along with "R", "Rust", maybe a few more) because parsing out some of these tags is a little more difficult than others.
lippihom 2y ago
Cool site - very clean lightweight interface which is great. Have a few friends that are looking that would for sure have checked it out had there been a free trial.
- Jabbs 2y ago
  Thank you. Currently working on promo code functionality to give away to all the postings in Ask HN: Who wants to be hired? (December 2024) https://news.ycombinator.com/item?id=42297422 https://news.ycombinator.com/item?id=42297422
mfiro 2y ago
Great project! What steps did you take to address the legal implications of scraping from different websites?
- Jabbs 2y ago
  The data is destroyed and no content from the web pages are reused or repurposed (each listing is merely a link + various tags that are created/associated upon viewing. My understanding is that public websites scraping is legal but repurposing their content might not be
tpdly 2y ago
Great site. Small feedback: There's a category 'Closure'-- I'm not sure if that's something I don't know about, but it definitely isn't for jobs using 'Clojure'.
- Jabbs 2y ago
  Oh wow that’s not good, thank you so much for pointing that out!

by Jabbs 2y ago

dchuk 2y ago

I’ve had a similar idea over the years. You should consider exploring whether competitive companies could be customers. As a competitor, getting alerts about roles another company is hiring for can be very interesting. Combine it with trends of postings over time…

Jabbs 2y ago
Oh that's a really interesting idea. Yea I dislike the idea of charging the job seeker but have not found a good way to monetize companies (not that they even know about me anyway)

stuckkeys 2y ago

What are you using to for the scraping? Playwright…selenium? I wanted to do something as a hobby but my IP kept getting reported lol. Also when you say companies…where are you getting the information from? Data brokers? Anyway, it is an interesting topic to me.

Jabbs 2y ago
Selenium, although I'm using a wrapper library that uses it. I only query each company every few days or so which probably helps to not get banned IP-wise but also rotate them. But many of the company job links are through external sources too (lever, greenhouse, etc.) which don't seem to mind The company data was gathered online for a long time until I found https://www.thecompaniesapi.com/ https://www.thecompaniesapi.com/ (which now is the source for much of that data)
- stuckkeys 2y ago
  I tried to use my own desktop machine to process some of these tasks. I can see my fans go jet mode when the scraping was being done lol. Do you have discord or any way to connect? Would love to chat around this topic. Feel free to drop any social media handles. I’ll ping you.
  - Jabbs 2y ago
    Ohhh yea I run into this memory issue very quickly when scraping (especially if you have a large URL dataset then it will inevitably find a website with a giant bit of markup). So I have to set timeouts and blacklist timely requests but also completely reset the (headless) browser on 2-3 requests (which is overkill but I am restricted on memory for those workers). Feel free to drop me an email sometime (should be on my HN profile)

fusslo 2y ago

can I make feature requests? I would love a map of job postings to see where it might make sense to move to in the future. If there's 10 jobs within 50 miles... that might be a good place to buy a house. Additionally, if I filter by 'north america' I still get jobs from canada and india because they're remote only. I would LOVE to be able to filter out those positions. Also I would love to be able to AND 'remote' and 'north america'. I would like to work remotely, but only for US companies love your site <3

Jabbs 2y ago
Thank you :) I appreciate the request + feedback. I have a story in the backlog to add location-specific links to the landing page but I really like your idea of having a map (heatmap or something) to show densities of jobs. So the inclusive vs exclusive filtering is something that I struggle to perfect here. I'm tempted to throw both in the UI (since its ready to go on the backend) but its hard to explain to users. One thing you can do that is not so obvious is add a tag for "Canada" but click on the tag again which will put a line through Canada and exclude that location from your filter (still need to have helpers to show users how to do that). The 'remote' tag is probably the toughest one to parse of a job listing because it might appear anywhere within the text, so there is some inaccuracies for sure but its improving I hope! Ah I could probably add filters for company locations specifically too (so you can filter US companies), that's an interesting use case too. Thanks for the compliment too, it has been really fun to build

registeredcorn 2y ago

Here's a bit of feedback: * Job listings for "Quality Assurance" and "QA" are split into different listings in Job Search. * I really like the green highlight for Salary range! Personally, I would sort by jobs that list salary first, then by location (or relevance, or whatever). * The filter was a little confusing to use. I see you talked about it with other users in here. It needs some love, but it's getting there. :) * If you are going to target job searchers, it would be very helpful too see metrics based on the results. Here's a few examples I came up with Example 1: I select Help Desk -> Chicago I see a short-term graph showing whether demand has gone: up, down, or stayed the same - included is a red/green/yellow arrow giving me an idea at a glance. This helps me understand how many Help Desk postings are in Chicago Example 2: I select Cybersecurity -> I also select Information Security -> NYC I see a short-term graph showing demand for Cyber vs IS in NYC. This helps me understand if which job has more postings in NYC. Example 3: I select Python Developer -> Boston & Dallas. I see a medium-term graph showing demand for each location for Python Developer. This helps me decide whether demand is more consistent in Boston or Dallas. Example 4: I select Asia & Canada -> Advertising (Under Industry) I see a long-term graph showing the overall trend for that industry in each of those countries. This helps me track whether jobs are being outsourced, what I should expect in the coming years, and/or which country is the most competitive in that industry. Hope that helps! Good luck. :)

Jabbs 2y ago
This is great feedback, I'm very appreciative! Yea there is a much better version of the search bar soon-to-deploy (which accounts for aliases like QA -> Quality Assurance) and it will match by word rather than the entire phrase (currently "software engineer" will not query "software test engineer"). Appreciate the callout here You can find a toggle switch for "has salary" under the "other" filters which will show only those w salary, but good call perhaps that should be part of a sort feature (beyond just date) The filters do need more love for sure. I like your examples for various metrics displayed in the UI. I did think it would be cool to have a Github-like array of squares that represent units of time with colors that show how it has been changing over time, would have to figure out how heavy of calculations those would be in real-time but I really like your idea here. Or a line chart might be better. Many thanks for all the input!

windowshopping 2y ago

How did you design and build your homepage? I find that building the landing page and making it look like a professional, beautiful design is one of my biggest hurdles. I'm an experienced web developer but without a design to work off of - and especially accounting for mobile and dynamic sizing - I really struggle with this part of the work so I'm wondering what other people's workflows are for it.

kmoser 2y ago
Not OP but I have two solutions for this: 1) find an existing site and mimic their design, or 2) hire a designer
- windowshopping 2y ago
  Yes, I know, that's very easy to say, but in my experience: 1.) Imitating a good design can take a lot of developer hours. Many landing pages have fancy css effects and support many screen sizes and it can take quite some time to build. 2.) Finding a good designer is not easy. I have tried hiring on most freelancer sites with very poor results. The applicant pool is typically of low quality. So I am wondering if others have found efficient/effective ways of going about #1 or #2, either by using certain tools or templates or by having a more clever hiring method.
  - kmoser 2y ago
    Certainly creating a robust design that supports multiple screen sizes is no trivial matter. There are tools to make it easier but as with anything else, no single solution fits all developers and all projects. You say you're an experienced web developer and yet you find yourself deficient in this particular skill, so I'm not sure what to think. Maybe your strength lies more in back end programming? In any case, I suggest looking into frameworks such as Bootstrap or Tailwind. Of course, there are also high-level solutions that don't require as much coding, e.g. WordPress, Wix, Squarespace, etc. WordPress plugins can make it very easy to apply fancy CSS effects without writing any CSS. As for finding freelancers, I agree that's also a difficult task. I don't have any magic bullet there, other than to prioritize people with good, timely communication skills, and to avoid people on the low end of the price range since they will usually be less experienced.

suchintan 2y ago

This is really cool. Do you have any interest in helping people auto apply to them? We can help you set it up with a really simple API call via Skyvern (https://github.com/Skyvern-AI/skyvern https://github.com/Skyvern-AI/skyvern)

Jabbs 2y ago
Thank you, appreciate this. I'm not certain about expanding this in that direction but that is certainly an interesting thought.
- suchintan 2y ago
  Yep. Makes a lot of sense. Your side project is super noble -- thank you for that

NullPrefix 2y ago

I've tried searching for C++ but the search bar straight up refuses it

Jabbs 2y ago
The search bar is for job title only at the moment. Under the "Tech" filter you can find "C++" and add it as a filter
- NullPrefix 2y ago
  thanks

zoke 2y ago

I see C++ and C#. Why no C?

Jabbs 2y ago
Yea that one is still a WIP (along with "R", "Rust", maybe a few more) because parsing out some of these tags is a little more difficult than others.

lippihom 2y ago

Cool site - very clean lightweight interface which is great. Have a few friends that are looking that would for sure have checked it out had there been a free trial.

Jabbs 2y ago
Thank you. Currently working on promo code functionality to give away to all the postings in Ask HN: Who wants to be hired? (December 2024) https://news.ycombinator.com/item?id=42297422 https://news.ycombinator.com/item?id=42297422

mfiro 2y ago

Great project! What steps did you take to address the legal implications of scraping from different websites?

Jabbs 2y ago
The data is destroyed and no content from the web pages are reused or repurposed (each listing is merely a link + various tags that are created/associated upon viewing. My understanding is that public websites scraping is legal but repurposing their content might not be

tpdly 2y ago

Great site. Small feedback: There's a category 'Closure'-- I'm not sure if that's something I don't know about, but it definitely isn't for jobs using 'Clojure'.

Jabbs 2y ago
Oh wow that’s not good, thank you so much for pointing that out!