7 ms·

Show HN: HackerNews-new-jobs – insights into fresh and recurring job ads

128 points by nemanja_codes 2y ago

The website is meant for people that look for jobs on HackerNews "Who's Hiring" threads and want to focus more on fresh ads and companies, or to quickly look up ad history of any company. Github repository: https://github.com/nemanjam/hn-new-jobs https://github.com/nemanjam/hn-new-jobs Demo website: https://hackernews-new-jobs.arm1.nemanjamitic.com https://hackernews-new-jobs.arm1.nemanjamitic.com I used Algolia API as a data source, along with scheduled task that parses new threads few times at the beginning of each month. The extracted data is then stored in SQLite database for fast querying, and the results are cached with Keyv for faster page responses. I will see in the future how much traffic the website receives and if this stack is performant enough. For the website I used Next.js app with default ShadcnUI components and charts. I just wanted a quick functional prototype to test how much public interest is there for an app with functionality like this. If you are interested in more implementation details you can find them in the Readme file on Github. The project is free and open source. Feel free to use, self-host, fork and modify, and contribute. I would love to hear your impressions and suggestions and look forward to discussing features and technical details.

OccamsMirror 2y ago
Location / remote / hybrid would be interesting to visualize.
- nemanja_codes 2y ago
  True, I thought about it, technologies too, and other filters. Currently website has clear focus new/repeated job ads. I will see how much public interest is there for the website and if it is worth adding more features.
helloleo2024 2y ago
[dead]
xnx 2y ago
I don't know who's to thank for it, but someone set up a constantly updated ClickHouse database of all data from the Hacker News API at https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a... It gets you 80% of the way there on any HN data project.
- deleted 2y ago
  [deleted]
- nemanja_codes 2y ago
  This very is useful, thank you. Algolia API is also pretty generous, it made my work a lot easier. It returns even 1k comments in a single response. I started with low level Axios parser but I soon discovered that HackerNews webserver has very strict rate limiting. Good to know that there are already available data sources so you can focus on querying logic and data presentation.
- adhamsalama 2y ago
  This is very useful, thank you!
jcuenod 2y ago
This looks lovely! Good work. On my wishlist are some fuzzier categories: 1. Tech trends (rust, docker, postgres...) 2. Role trends ("ml engineer", "full-stack developer" ...)
- nemanja_codes 2y ago
  Thank you, I highly appreciate it. As is website has clear and strong focus, it's "fresh" vs "repeated" job ads. Naturally, you would always prefer to give a shot with company that just posted a fresh ad, then to apply for a role that is repeated in 10 months of previous 12 but company still hasn't found a candidate that satisfies criteria. For the additional filters, by technologies and role types it would be of great help if I could find some high level indexing and fuzzy search tools/libraries. I would probably need to migrate from SQLite to Postgres and when I am already there probably use ORM too. Certainly, I would need to do serious research, if there is enough public interest in the current website and I manage to find some contributors with data science and information retrieval experience maybe we can add many interesting filters like tech, roles, location, visas, remote, etc.
shermantanktop 2y ago
Me: hmm what’s the big dip in the middle? Spouse: uh, COVID? Me: ohhhh
- nemanja_codes 2y ago
  "HN Who's hiring" thread is just a random sample that pretty accurately describes trends in the entire industry.
andersa 2y ago
This is a rather depressing graph, what happened in 2022-2024?
- nemanja_codes 2y ago
  Collapse, unfortunately.
- robbie-c 2y ago
  end of ZIRP
  - game_the0ry 2y ago
    This and over-hiring.
- myth_drannon 2y ago
  And to put a more depressing angle to the graph, HN and Who's hiring were not that popular pre-2017, and now 2024 numbers are lower than the pre-2017 data
  - tredre3 2y ago
    > HN [...] were not that popular pre-2017 Is that true? What happened to change that in 2017? I've been around longer than that and I admit I haven't seen that much of a difference in engagement around here. But over such a long period it could easily have gone unnoticed by me! The only thing I've noticed is the increase in activity during the recent Reddit shenanigans (which resulted in a drop of quality of the conversation on HN, but it seems back to normal now).
- lubujackson 2y ago
  The tax change that causes companies to have to weirdly treat developer salaries as some sort of asset such that they can only write off 20% of it per year. Outcome of this is killer for startups and causing a huge issue everywhere. Bottom line: If a company makes $1MM in revenue and pays $1MM in salary, they owe taxes on $800k profit. Yes, this is actually the law now.
h1fra 2y ago
Nice to see the trend confirming my suspicion, the market has been complicated for the last year and it's not really improving (whether you are hiring or looking for a job)
- BillFranklin 2y ago
  I found the same - market is actually getting worse (if HN is representative of the market). This month there were more job seekers on HN than jobs for the first time since 2014. https://bilbof.com/posts/tech-hiring-is-bad-right-now https://bilbof.com/posts/tech-hiring-is-bad-right-now
  - daemonologist 2y ago
    This site I think counts only top-level comments, and the streams have been crossed since ~April 2023: https://www.hnhiringtrends.com/ https://www.hnhiringtrends.com/ . The overall trend is the same though - the market seems pretty historically bad. There's also Indeed postings data, which unfortunately only goes back to 2020 but is similarly bleak: https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE
welder 2y ago
This is awesome! But it's missing at least one company from 2024-08. Maybe a parsing error? Or does it exclude hiring freelance/contract posts?
- nemanja_codes 2y ago
  Thank you very much, I am glad you like it. I don't understand what is missing in 2024-08? You can link month by slug, I see nothing unusual here: https://hackernews-new-jobs.arm1.nemanjamitic.com/2024-08 https://hackernews-new-jobs.arm1.nemanjamitic.com/2024-08 But as a side note, yes, this is not meant as an exact analytic tool, rather just a best effort website that gives some interesting insights. To clarify, only the "Who's hiring" thread is parsed, you can see it clearly in this constants file: https://github.com/nemanjam/hn-new-jobs/blob/main/constants/algolia.ts https://github.com/nemanjam/hn-new-jobs/blob/main/constants/... Also, in there you can see how simple the parsing regex is, it just looks for "|" separator in the comment title. Another thing I noticed, some companies used different letter casing for their name in some comments, and the company name is part of the primary key, so same company is perceived as different, I should probably handle this better. https://github.com/nemanjam/hn-new-jobs/blob/main/modules/database/schema.ts https://github.com/nemanjam/hn-new-jobs/blob/main/modules/da... For example you can search for "ConsenSys" on the Search page: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?company=consens https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
  - welder 2y ago
    I didn't realize freelancer jobs go to a different HN post `Ask HN: Freelancer? Seeking freelancer?` Makes sense now, because the job I noticed missing was a freelance post.
    - nemanja_codes 2y ago
      You can try searching directly on Algolia, my website uses it as a data source: https://hn.algolia.com/ https://hn.algolia.com/ https://hn.algolia.com/api https://hn.algolia.com/api If it's there my website has it also. But Algolia does pretty good job parsing HackerNews, I am pretty confident 99.9% of comments are included.
mdaniel 2y ago
It would appear you have a non-trivial "company" parse problem, based on eyeballing the "First Time Companies" section. I do appreciate the problem of trying to do semantic analysis on HN comment text, but several of them follow the "X@Y" nomenclature and a few others are "X-Y" so it doesn't seem to be insurmountable to try and fix
- nemanja_codes 2y ago
  This is just a first iteration of a quick prototype to gauge public interest in the idea. It just looks for the "|" separator in the comment title, which is something that majority of posters enforce. https://github.com/nemanjam/hn-new-jobs/blob/main/constants/algolia.ts https://github.com/nemanjam/hn-new-jobs/blob/main/constants/... If people show constant interest I can iterate it further, enhance it, ad features, etc.
  - mdaniel 2y ago
    Well, if that's your attitude then I wish you all the luck with your project I was also going to say $(git commit -a) is evil based on <https://github.com/nemanjam/hn-new-jobs/blob/main/data/database/hn-new-jobs-database-dev.sqlite3 https://github.com/nemanjam/hn-new-jobs/blob/main/data/datab...> but it seems that you just want an always changing binary blob to make your git repo grow without bound :-( https://github.com/nemanjam/hn-new-jobs/blob/main/.gitignore#L46 https://github.com/nemanjam/hn-new-jobs/blob/main/.gitignore...
thih9 2y ago
Possible bug report, I noticed a company that posts regularly but I didn’t see it listed: https://news.ycombinator.com/threads?id=spark_CM https://news.ycombinator.com/threads?id=spark_CM
- nemanja_codes 2y ago
  Here it is: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?company=chartmogul https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
  - thih9 2y ago
    Something still looks off, the earliest ad is from 2024-07, but the company posted more recent ads too. I guess this is because they changed their ad’s headline from: ChartMogul | Remote (EU) | Full-time to: ChartMogul (https://chartmogul.com )| Remote | Full-time Other companies are similarly affected, e.g. Medusa: - listed: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?company=Medusa https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com... - earliest ad listed: https://news.ycombinator.com/item?id=34222858 https://news.ycombinator.com/item?id=34222858 - more recent ad on hn: https://news.ycombinator.com/item?id=42315828 https://news.ycombinator.com/item?id=42315828 Perhaps it’s the “)|” bit causing problems with some regexps.
    - nemanja_codes 2y ago
      Yes, there is a lot of room to improve parsing.
ryandrake 2y ago
This is great! I had long ago started a similar project to parse HN job ads month after month looking for the ones that just stay the same each month, which might suggest the company is probably "fishing" instead of "hiring". But it never really got off the ground. This platform seems more than capable of doing that kind of analysis and more. Great work.
- nemanja_codes 2y ago
  Thank you very much, very much appreciated. I was curious to see how much people would be interested in such website.
ofcrpls 2y ago
Nice work for Central Defender /s.
- nemanja_codes 2y ago
  Thank you very much. He is Chelsea legend that messes up my SEO.
Ancalagon 2y ago
I can maybe make a PR later - but would love to see the companies sorted alphabetically so I could just glance for specific ones!
- deleted 2y ago
  [deleted]
BerislavLopac 2y ago
This is nice, but I find this one more detailed: https://hnhiring.com/trends https://hnhiring.com/trends
EricaOrnelas92 2y ago
[dead]
vfulco2 2y ago
Really terrific contribution.
- nemanja_codes 2y ago
  Thank you, much appreciated.
artemkornilov 2y ago
[dead]
Jackosas 2y ago
[flagged]
rco8786 2y ago
Really shows the hiring slowdown over the last 2 years.

128 points by nemanja_codes 2y ago

OccamsMirror 2y ago

Location / remote / hybrid would be interesting to visualize.

nemanja_codes 2y ago
True, I thought about it, technologies too, and other filters. Currently website has clear focus new/repeated job ads. I will see how much public interest is there for the website and if it is worth adding more features.

helloleo2024 2y ago

[dead]

xnx 2y ago

I don't know who's to thank for it, but someone set up a constantly updated ClickHouse database of all data from the Hacker News API at https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a... It gets you 80% of the way there on any HN data project.

deleted 2y ago
[deleted]
nemanja_codes 2y ago
This very is useful, thank you. Algolia API is also pretty generous, it made my work a lot easier. It returns even 1k comments in a single response. I started with low level Axios parser but I soon discovered that HackerNews webserver has very strict rate limiting. Good to know that there are already available data sources so you can focus on querying logic and data presentation.
adhamsalama 2y ago
This is very useful, thank you!

jcuenod 2y ago

This looks lovely! Good work. On my wishlist are some fuzzier categories: 1. Tech trends (rust, docker, postgres...) 2. Role trends ("ml engineer", "full-stack developer" ...)

nemanja_codes 2y ago
Thank you, I highly appreciate it. As is website has clear and strong focus, it's "fresh" vs "repeated" job ads. Naturally, you would always prefer to give a shot with company that just posted a fresh ad, then to apply for a role that is repeated in 10 months of previous 12 but company still hasn't found a candidate that satisfies criteria. For the additional filters, by technologies and role types it would be of great help if I could find some high level indexing and fuzzy search tools/libraries. I would probably need to migrate from SQLite to Postgres and when I am already there probably use ORM too. Certainly, I would need to do serious research, if there is enough public interest in the current website and I manage to find some contributors with data science and information retrieval experience maybe we can add many interesting filters like tech, roles, location, visas, remote, etc.

shermantanktop 2y ago

Me: hmm what’s the big dip in the middle? Spouse: uh, COVID? Me: ohhhh

nemanja_codes 2y ago
"HN Who's hiring" thread is just a random sample that pretty accurately describes trends in the entire industry.

andersa 2y ago

This is a rather depressing graph, what happened in 2022-2024?

nemanja_codes 2y ago
Collapse, unfortunately.
robbie-c 2y ago
end of ZIRP
- game_the0ry 2y ago
  This and over-hiring.
myth_drannon 2y ago
And to put a more depressing angle to the graph, HN and Who's hiring were not that popular pre-2017, and now 2024 numbers are lower than the pre-2017 data
- tredre3 2y ago
  > HN [...] were not that popular pre-2017 Is that true? What happened to change that in 2017? I've been around longer than that and I admit I haven't seen that much of a difference in engagement around here. But over such a long period it could easily have gone unnoticed by me! The only thing I've noticed is the increase in activity during the recent Reddit shenanigans (which resulted in a drop of quality of the conversation on HN, but it seems back to normal now).
lubujackson 2y ago
The tax change that causes companies to have to weirdly treat developer salaries as some sort of asset such that they can only write off 20% of it per year. Outcome of this is killer for startups and causing a huge issue everywhere. Bottom line: If a company makes $1MM in revenue and pays $1MM in salary, they owe taxes on $800k profit. Yes, this is actually the law now.

h1fra 2y ago

Nice to see the trend confirming my suspicion, the market has been complicated for the last year and it's not really improving (whether you are hiring or looking for a job)

BillFranklin 2y ago
I found the same - market is actually getting worse (if HN is representative of the market). This month there were more job seekers on HN than jobs for the first time since 2014. https://bilbof.com/posts/tech-hiring-is-bad-right-now https://bilbof.com/posts/tech-hiring-is-bad-right-now
- daemonologist 2y ago
  This site I think counts only top-level comments, and the streams have been crossed since ~April 2023: https://www.hnhiringtrends.com/ https://www.hnhiringtrends.com/ . The overall trend is the same though - the market seems pretty historically bad. There's also Indeed postings data, which unfortunately only goes back to 2020 but is similarly bleak: https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE

welder 2y ago

This is awesome! But it's missing at least one company from 2024-08. Maybe a parsing error? Or does it exclude hiring freelance/contract posts?

nemanja_codes 2y ago
Thank you very much, I am glad you like it. I don't understand what is missing in 2024-08? You can link month by slug, I see nothing unusual here: https://hackernews-new-jobs.arm1.nemanjamitic.com/2024-08 https://hackernews-new-jobs.arm1.nemanjamitic.com/2024-08 But as a side note, yes, this is not meant as an exact analytic tool, rather just a best effort website that gives some interesting insights. To clarify, only the "Who's hiring" thread is parsed, you can see it clearly in this constants file: https://github.com/nemanjam/hn-new-jobs/blob/main/constants/algolia.ts https://github.com/nemanjam/hn-new-jobs/blob/main/constants/... Also, in there you can see how simple the parsing regex is, it just looks for "|" separator in the comment title. Another thing I noticed, some companies used different letter casing for their name in some comments, and the company name is part of the primary key, so same company is perceived as different, I should probably handle this better. https://github.com/nemanjam/hn-new-jobs/blob/main/modules/database/schema.ts https://github.com/nemanjam/hn-new-jobs/blob/main/modules/da... For example you can search for "ConsenSys" on the Search page: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?company=consens https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
- welder 2y ago
  I didn't realize freelancer jobs go to a different HN post `Ask HN: Freelancer? Seeking freelancer?` Makes sense now, because the job I noticed missing was a freelance post.
  - nemanja_codes 2y ago
    You can try searching directly on Algolia, my website uses it as a data source: https://hn.algolia.com/ https://hn.algolia.com/ https://hn.algolia.com/api https://hn.algolia.com/api If it's there my website has it also. But Algolia does pretty good job parsing HackerNews, I am pretty confident 99.9% of comments are included.

mdaniel 2y ago

It would appear you have a non-trivial "company" parse problem, based on eyeballing the "First Time Companies" section. I do appreciate the problem of trying to do semantic analysis on HN comment text, but several of them follow the "X@Y" nomenclature and a few others are "X-Y" so it doesn't seem to be insurmountable to try and fix

nemanja_codes 2y ago
This is just a first iteration of a quick prototype to gauge public interest in the idea. It just looks for the "|" separator in the comment title, which is something that majority of posters enforce. https://github.com/nemanjam/hn-new-jobs/blob/main/constants/algolia.ts https://github.com/nemanjam/hn-new-jobs/blob/main/constants/... If people show constant interest I can iterate it further, enhance it, ad features, etc.
- mdaniel 2y ago
  Well, if that's your attitude then I wish you all the luck with your project I was also going to say $(git commit -a) is evil based on <https://github.com/nemanjam/hn-new-jobs/blob/main/data/database/hn-new-jobs-database-dev.sqlite3 https://github.com/nemanjam/hn-new-jobs/blob/main/data/datab...> but it seems that you just want an always changing binary blob to make your git repo grow without bound :-( https://github.com/nemanjam/hn-new-jobs/blob/main/.gitignore#L46 https://github.com/nemanjam/hn-new-jobs/blob/main/.gitignore...

thih9 2y ago

Possible bug report, I noticed a company that posts regularly but I didn’t see it listed: https://news.ycombinator.com/threads?id=spark_CM https://news.ycombinator.com/threads?id=spark_CM

nemanja_codes 2y ago
Here it is: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?company=chartmogul https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
- thih9 2y ago
  Something still looks off, the earliest ad is from 2024-07, but the company posted more recent ads too. I guess this is because they changed their ad’s headline from: ChartMogul | Remote (EU) | Full-time to: ChartMogul (https://chartmogul.com )| Remote | Full-time Other companies are similarly affected, e.g. Medusa: - listed: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?company=Medusa https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com... - earliest ad listed: https://news.ycombinator.com/item?id=34222858 https://news.ycombinator.com/item?id=34222858 - more recent ad on hn: https://news.ycombinator.com/item?id=42315828 https://news.ycombinator.com/item?id=42315828 Perhaps it’s the “)|” bit causing problems with some regexps.
  - nemanja_codes 2y ago
    Yes, there is a lot of room to improve parsing.

ryandrake 2y ago

This is great! I had long ago started a similar project to parse HN job ads month after month looking for the ones that just stay the same each month, which might suggest the company is probably "fishing" instead of "hiring". But it never really got off the ground. This platform seems more than capable of doing that kind of analysis and more. Great work.

nemanja_codes 2y ago
Thank you very much, very much appreciated. I was curious to see how much people would be interested in such website.

ofcrpls 2y ago

Nice work for Central Defender /s.

nemanja_codes 2y ago
Thank you very much. He is Chelsea legend that messes up my SEO.

Ancalagon 2y ago

I can maybe make a PR later - but would love to see the companies sorted alphabetically so I could just glance for specific ones!

deleted 2y ago
[deleted]

BerislavLopac 2y ago

This is nice, but I find this one more detailed: https://hnhiring.com/trends https://hnhiring.com/trends

EricaOrnelas92 2y ago

[dead]

vfulco2 2y ago

Really terrific contribution.

nemanja_codes 2y ago
Thank you, much appreciated.

artemkornilov 2y ago

[dead]

Jackosas 2y ago

[flagged]

rco8786 2y ago

Really shows the hiring slowdown over the last 2 years.