7 ms·

Show HN: Tech jobs on the command line

80 points by nico 3y ago

Hi HN! I'm Nico and lately I've been struggling looking for jobs. I've gotten specially frustrated with having to spend hours reading through listings just to find good matches. So I built something to scratch my own itch: Command Jobs, a terminal-based job finder and application tracker designed specifically for me, and maybe also software engineers The app scrapes web listings, saves them, processes them with GPT[1], then based on your resume and job preferences, it gives you the best matches This is my first open source project, and also my first public project using LLMs I'm really excited to share this with the HN community and would love to hear your feedback, suggestions, and any features you'd like to see in the future. Please check it out and let me know what you think! https://github.com/nicobrenner/commandjobs https://github.com/nicobrenner/commandjobs I'm currently working on adding more sources for job listings, periodic scraping via cronjobs, and alerts for new matches Looking forward to your questions, feedback and requests. Thank you [1]: Bring Your Own (OpenAPI) Key

grahamg 3y ago
[dead]
nico 3y ago
Here’s a brief explainer video https://www.tiktok.com/t/ZTLRFQs2j/ https://www.tiktok.com/t/ZTLRFQs2j/
avantdang 3y ago
[flagged]
- deleted 3y ago
  [deleted]
  - avantdang 3y ago
    [flagged]
  - deleted 3y ago
    [deleted]
bagels 3y ago
Most of the jobs on Who's Hiring threads don't include any compensation, so I don't see how searching that source would be useful for many folks.
- nico 3y ago
  That’s true, and it would be a great feature to have. The scraper could “double click” on the listings and try to find salary ranges from other websites where the listing might be posted, like the careers page of the company I’m going one step at a time, the first tedious task I wanted to automate was searching and second-pass filtering Most job listing sites have tools to filter by many different factors, including role, location, remote, salary range and seniority level, but usually the filters are either too strict or too permissive and I still end up having to read through tens or hundreds of listings to select just a few Command Jobs gives me better matches faster by using GPT to match my resume and job preferences with the listings
- brailsafe 3y ago
  Seems like you're just critiquing the Who's Hiring thread, not this app. At this point though, I'd wager that a huge amount, if not a majority of people looking, aren't thinking about compensation first, and have accepted the reality that if they can get something that aligns even remotely with what they're interested in or are skilled at, they'd be lucky. Salaries are not going to be what they used to be. Not defending the exclusion of salary, but I think it would be pretty naive to dismiss the fact that a profound amount of people have been forced into unemployment and can negotiate if it gets there.
  - bagels 3y ago
    Yes, it's mostly a critique of those threads as a source. There's no forum to discuss them though. Not here, and the threads discourage feedback about the format. I think folks know there's downward pressure, but I'd rather not anyone have to waste time engaging with employers offering minimum wage for software engineers.
    - 3y ago
nico 3y ago
One of the most challenging aspects is having a proper interface for prompt management. Right now there are 5 different elements that are combined together to form the prompt in a specific format. The prompt elements are stored in the .env file, are are read from there when running the app Would love to get some tips or ideas on how to allow the user to manage the prompt, to make changes and test it (without braking it), from the command line
- john-radio 3y ago
  Well, you could throw in something like this - https://github.com/nicobrenner/commandjobs/compare/master...BenQuigley:commandjobs:master https://github.com/nicobrenner/commandjobs/compare/master...... You can also go further by writing code that prompts the user for these values, one by one, and stores them in a config file. But for a CLI application, you might reasonably wonder if the juice is worth the squeeze there. CLI users tend to be comfortable editing a config file to hold a secret. Edit: or if some of those values are actually optional, then you would maybe exclude them from this check.
  - nico 3y ago
    Thank you so much Excellent contribution, definitely need a ton of validations! Also pending is getting some tests in there, especially to test consistent LLM/GPT-dependent behavior Totally right about the difference in expectations from a cli user pov. I just want to make sure this tool is helpful to more people than just myself
toomuchtodo 3y ago
Are you looking for contributors?
- nico 3y ago
  For sure, would love any contributions. Feel free to create a ticket, or a PR on Github Let me know if you have any issues getting it up and running
thomasfromcdnjs 3y ago
Hey Nico, this looks awesome. I built out pretty much the exact same thing for JSON Resume (far uglier) but uses vector embeddings to rank relevant jobs -> https://registry.jsonresume.org/thomasdavis/jobs https://registry.jsonresume.org/thomasdavis/jobs My prompt when running it through GPT-3.5 sucks, doesn't always get location and salary. GPT-4 is way better but $$$$. I've seen about 3-4 projects doing this with HN WhoIsHiring projects now, I wonder if we should all just share an API (of GPT parsed HN job posts) and pool some $$$. (In my version, I also created a JSON Schema that represents a Job Description) scripts: https://github.com/jsonresume/jsonresume.org/tree/master/apps/registry/scripts/jobs https://github.com/jsonresume/jsonresume.org/tree/master/app... api code: https://github.com/jsonresume/jsonresume.org/blob/master/apps/registry/pages/api/jobs.js https://github.com/jsonresume/jsonresume.org/blob/master/app...
- nico 3y ago
  Amazing, thank you! Very cool Did you do any sort of benchmark between embeddings and direct inference? How did you tune the distance for relevance, how did you decide on chunking and how many results would you include? I want to try filtering using the distance of embeddings RAG style, but I wonder about how that changes the results The current method the app uses feels pretty good, it does get you the best matches, and sometimes it also includes a few “extra” suggestions that typical filters would have left out, but that end up being potentially interesting matches
  - thomasfromcdnjs 3y ago
    I'm not exactly sure what you mean by direct inference? Do you mean that you pass the resume and job posting to the initial GPT prompt? and rely on the returned "fit_for_resume" property? (I'm curious if you tried using OpenAI function outputs too) If so I didn't try that. And thinking about it, I like your style. (There's no reason why you couldn't do both anyway) Another question; Did you try asking for a "fit_rating"? Here is the prompt/call I used for GPT'ing the HN posts -> https://github.com/jsonresume/jsonresume.org/blob/master/apps/registry/scripts/jobs/gpted.js https://github.com/jsonresume/jsonresume.org/blob/master/app... As for the distance of the vector similarity search. I am just using Supabase's pg_vector plugin support, with 3704 dimensions using the default search (cosine).
ab_testing 3y ago
Excellent project. Is there a way to scrape and pull jobs from workday portals. I see lots of companies using workday and usually it is a pain to go through their job postings?
- nico 3y ago
  Thank you! It’s been fun building it and I’m looking forward to keep improving it I definitely want to add scrapers for more sources Haven't tried workday, is it popular for tech jobs (ie. with people that use the command line)? Do you have a sample url with listings you could share?
  - ydant 3y ago
    Here's some with tech jobs listed: https://workday.wd5.myworkdayjobs.com/Workday https://workday.wd5.myworkdayjobs.com/Workday https://shipt.wd1.myworkdayjobs.com/en-US/Shipt_External https://shipt.wd1.myworkdayjobs.com/en-US/Shipt_External It doesn't seem like there's a global listing across myworkdayjobs.com sites - each company using it has its own sub-domain. Which sort of makes sense - Workday is an internal HR/employee portal tool licensed to companies.
    - nico 3y ago
      Excellent, thank you so much for the links, it's super interesting It should be pretty straightforward to create a scraper for one of those portals If you feel like taking a crack at it, checkout the code of the Ask HN who's hiring scraper here: https://github.com/nicobrenner/commandjobs/blob/47b5c89402a37ea2929a6481301434a224f86b10/src/hn_scraping.py https://github.com/nicobrenner/commandjobs/blob/47b5c89402a3... I added a comment under the Add Sources issue, to support Workday's career portals: https://github.com/nicobrenner/commandjobs/issues/23#issuecomment-1995738457 https://github.com/nicobrenner/commandjobs/issues/23#issueco...
3abiton 3y ago
One of the best application of LLM!
RyanShook 3y ago
This is awesome. I've been looking for a way to scrape Who's Hiring for my site https://200kjobs.com https://200kjobs.com, thanks for sharing.
- nico 3y ago
  Thank you for posting your site! Will definitely explore it. Great and simple concept Might try scraping it too ;) Btw, random observation that surprised me when scraping Ask HN, most jobs (100+) are posted the first day of the month, but there are about 15-25 additional jobs posted everyday. This means it’s worth it as a user to get those extra daily listings, but it’s a pain to search through them
  - RyanShook 3y ago
    I came across this site which scrapes and shares job postings. The challenge is finding salary ranges: https://hnhiring.com/ https://hnhiring.com/ Feel free to scrape 200kjobs: https://200kjobs.com/feed https://200kjobs.com/feed
    - nico 3y ago
      Thank you! Will definitely go through the feed and try to add it to Command Jobs :) As an aggregator it’s true that finding salary ranges is quite the challenge As an applicant, I’ve settled for just only applying to the positions that have a range If Im really curious about a company without ranges in their listings, GPT/Google sometimes give decent estimates But also, after seeing 100+ listings, I’ve noticed that for the positions I’m looking for, there’s a pretty standard range, which doesn’t vary too much, unless it’s a FAANG or super hot company In any case, using Command Jobs I only get 5-20 job matches, which are also pretty easy to check manually, and it’s not too annoying, given that usually the matches are very interesting and also it’s not 100s of jobs to r
- kpandit 3y ago
  Your site looks really nice and addresses a real problem. Are you targeting US only remote roles? If not then including any domicile or timezone restrictions in job snippets may be helpful. Also thanks for providing the feed.
cheesehead 3y ago
Nice work! I did something similar with a personal project a few months ago using an open source llm. Also, not sure if you know, there is an api you can use. https://github.com/HackerNews/API https://github.com/HackerNews/API
- nico 3y ago
  Awesome, did not know about the HN api, nice! What open source LLM did you use? Anything that would run on a CPU of a 5 year old MacBook Air (which I’m using as my home server) Do you have the code for your project? Or maybe some features/ideas that you could share? Thank you
  - cheesehead 3y ago
    I'm using https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and currently mistral 7b (on a m1 macbook pro). I'm sure with some prompt examples you can get pretty good results on a smaller model. At the moment I don't have it open sourced due to it being part of a larger project that I'm working on that contains tailwindui licensed components. A cool feature that I'm working on is creating a firefox plugin so you can save/index job postings from other sites and extract out meta information via an LLM. Very similar to this chrome plugin. https://chromewebstore.google.com/detail/huntr-job-search-tracker/mihdfbecejheednfigjpdacgeilhlmnf https://chromewebstore.google.com/detail/huntr-job-search-tr...
    - nico 3y ago
      Love the extension idea, definitely useful for tracking opportunities Thank you for the perspective on running local LLMs Do you think that saas or local LLMs will be the most popular in the next 5 years?
      - cheesehead 3y ago
        Hoping it's local :)
kpandit 3y ago
Congrats Nico. Came here to tell you about the HN API (https://github.com/HackerNews/API https://github.com/HackerNews/API) but I see cheesehead beat me to it. Using the API it is much easier to pick up the latest thread each month and setup a cron job.
- nico 3y ago
  That’s an excellent tip, thank you very much. It will definitely come in handy soon when implementing the cron :) Are you using the API on some project?
  - kpandit 3y ago
    Yes, I am building an application tracking system(ATS) that also includes a job board. Among other sources, I used HN to populate the job board. My project is not open source, but I am happy to share the HN scraping code with you if you like. My email is in my profile.
hn2424 3y ago
[dead]
azophy_2 3y ago
would be interesting to turn this into something like `npx commandjobs` or `ssh commandjobs.com` so you wouldn't even need to install new software
- nico 3y ago
  Love the ideas, especially the ssh one, it's something that I had been fantasizing about My first year of college, the school had a Telnet-based application to request classes, it would be amazing to build something like that Any pointers on how to build the npx installer?
- cookiemonsieur 3y ago
  I've always wondered how to build SSH based applications. If anyone has any pointers I'd be extremely grateful.
  - nico 3y ago
    My uninformed understanding, is that you allow people to login into an actual user account within the system, but then you replace their shell (usually you change something like /bin/sh to /usr/bin/ssh_application in the entry for that account within the /etc/passwd file), so when someone logs into that account, instead of running an "open" shell, they get the application instead, and when the application exits, the session ends So essentially you should be able to create an ssh application with just a shell script
- thomasfromcdnjs 3y ago
  There is something similar https://github.com/alexpovel/ancv/ https://github.com/alexpovel/ancv/
  - nico 3y ago
    This looks really cool, thank you for sharing!
mfiro 3y ago
Very nice project. Thanks for sharing it here. I saw you used beautifulsoup to scrape, but HN has a public API which works great and it doesn't need any token or auth. I recently wrote a minimal wrapper for that: https://github.com/mfiro/hnconnector https://github.com/mfiro/hnconnector
- nico 3y ago
  Really appreciate it Super cool wrapper, love the methods that make it super easy to get the content What would be the best way to for example, get the latest Ask HN: who's hiring page and all it's top level comments? Thank you
  - mfiro 3y ago
    Here's a quick rundown, you can pull a list of up to 200 "Ask HN" stories using this endpoint: ask_stories = client.get_askstories() then you have the ids and you can sort them reversed to catch the latest stories. client.get_item(<id>) will give you the story's details, in which you'll find item['kids'], which are the IDs for the top-level comments. You can then loop through these comment IDs to get more info on each comment. Another option is to use Algolia's API for HN. But that's not yet supported in my wrapper.
hughdbrown 3y ago
I have not read all the code closely but it looks like you are scraping the HTML directly. True? It would be easier to use the JSON feed for a couple of reasons: - the data is segregated into types of things that you otherwise have to back out from the HTML/CSS - you don't need logic for parsing multiple pages Have a look at code I wrote on this: https://github.com/hughdbrown/who-is-hiring https://github.com/hughdbrown/who-is-hiring The code to get a month of Who Is Hiring posts just hits `https://hn.algolia.com/api/v1/items/{article_id} https://hn.algolia.com/api/v1/items/{article_id}` with a GET request for a particular article_id. Finding the monthly article_ids is the only manual maintenance. The code is split into multiple commandline tools that extract JSON from HN, preprocess the JSON, and filter on what I was interested in (rust job postings). You should be able to replace downstream parts like preprocessing and filtering. See if any of this is helpful.
- nico 3y ago
  Cool, thank you Yes, scraping the html directly was relatively easy Will try Algolia and the api when doing a cron job for it

80 points by nico 3y ago

grahamg 3y ago

[dead]

nico 3y ago

Here’s a brief explainer video https://www.tiktok.com/t/ZTLRFQs2j/ https://www.tiktok.com/t/ZTLRFQs2j/

avantdang 3y ago

[flagged]

deleted 3y ago
[deleted]
- avantdang 3y ago
  [flagged]
- deleted 3y ago
  [deleted]

bagels 3y ago

Most of the jobs on Who's Hiring threads don't include any compensation, so I don't see how searching that source would be useful for many folks.

nico 3y ago
That’s true, and it would be a great feature to have. The scraper could “double click” on the listings and try to find salary ranges from other websites where the listing might be posted, like the careers page of the company I’m going one step at a time, the first tedious task I wanted to automate was searching and second-pass filtering Most job listing sites have tools to filter by many different factors, including role, location, remote, salary range and seniority level, but usually the filters are either too strict or too permissive and I still end up having to read through tens or hundreds of listings to select just a few Command Jobs gives me better matches faster by using GPT to match my resume and job preferences with the listings
brailsafe 3y ago
Seems like you're just critiquing the Who's Hiring thread, not this app. At this point though, I'd wager that a huge amount, if not a majority of people looking, aren't thinking about compensation first, and have accepted the reality that if they can get something that aligns even remotely with what they're interested in or are skilled at, they'd be lucky. Salaries are not going to be what they used to be. Not defending the exclusion of salary, but I think it would be pretty naive to dismiss the fact that a profound amount of people have been forced into unemployment and can negotiate if it gets there.
- bagels 3y ago
  Yes, it's mostly a critique of those threads as a source. There's no forum to discuss them though. Not here, and the threads discourage feedback about the format. I think folks know there's downward pressure, but I'd rather not anyone have to waste time engaging with employers offering minimum wage for software engineers.
  - 3y ago

nico 3y ago

One of the most challenging aspects is having a proper interface for prompt management. Right now there are 5 different elements that are combined together to form the prompt in a specific format. The prompt elements are stored in the .env file, are are read from there when running the app Would love to get some tips or ideas on how to allow the user to manage the prompt, to make changes and test it (without braking it), from the command line

john-radio 3y ago
Well, you could throw in something like this - https://github.com/nicobrenner/commandjobs/compare/master...BenQuigley:commandjobs:master https://github.com/nicobrenner/commandjobs/compare/master...... You can also go further by writing code that prompts the user for these values, one by one, and stores them in a config file. But for a CLI application, you might reasonably wonder if the juice is worth the squeeze there. CLI users tend to be comfortable editing a config file to hold a secret. Edit: or if some of those values are actually optional, then you would maybe exclude them from this check.
- nico 3y ago
  Thank you so much Excellent contribution, definitely need a ton of validations! Also pending is getting some tests in there, especially to test consistent LLM/GPT-dependent behavior Totally right about the difference in expectations from a cli user pov. I just want to make sure this tool is helpful to more people than just myself

toomuchtodo 3y ago

Are you looking for contributors?

nico 3y ago
For sure, would love any contributions. Feel free to create a ticket, or a PR on Github Let me know if you have any issues getting it up and running

thomasfromcdnjs 3y ago

Hey Nico, this looks awesome. I built out pretty much the exact same thing for JSON Resume (far uglier) but uses vector embeddings to rank relevant jobs -> https://registry.jsonresume.org/thomasdavis/jobs https://registry.jsonresume.org/thomasdavis/jobs My prompt when running it through GPT-3.5 sucks, doesn't always get location and salary. GPT-4 is way better but $$$$. I've seen about 3-4 projects doing this with HN WhoIsHiring projects now, I wonder if we should all just share an API (of GPT parsed HN job posts) and pool some $$$. (In my version, I also created a JSON Schema that represents a Job Description) scripts: https://github.com/jsonresume/jsonresume.org/tree/master/apps/registry/scripts/jobs https://github.com/jsonresume/jsonresume.org/tree/master/app... api code: https://github.com/jsonresume/jsonresume.org/blob/master/apps/registry/pages/api/jobs.js https://github.com/jsonresume/jsonresume.org/blob/master/app...

nico 3y ago
Amazing, thank you! Very cool Did you do any sort of benchmark between embeddings and direct inference? How did you tune the distance for relevance, how did you decide on chunking and how many results would you include? I want to try filtering using the distance of embeddings RAG style, but I wonder about how that changes the results The current method the app uses feels pretty good, it does get you the best matches, and sometimes it also includes a few “extra” suggestions that typical filters would have left out, but that end up being potentially interesting matches
- thomasfromcdnjs 3y ago
  I'm not exactly sure what you mean by direct inference? Do you mean that you pass the resume and job posting to the initial GPT prompt? and rely on the returned "fit_for_resume" property? (I'm curious if you tried using OpenAI function outputs too) If so I didn't try that. And thinking about it, I like your style. (There's no reason why you couldn't do both anyway) Another question; Did you try asking for a "fit_rating"? Here is the prompt/call I used for GPT'ing the HN posts -> https://github.com/jsonresume/jsonresume.org/blob/master/apps/registry/scripts/jobs/gpted.js https://github.com/jsonresume/jsonresume.org/blob/master/app... As for the distance of the vector similarity search. I am just using Supabase's pg_vector plugin support, with 3704 dimensions using the default search (cosine).

ab_testing 3y ago

Excellent project. Is there a way to scrape and pull jobs from workday portals. I see lots of companies using workday and usually it is a pain to go through their job postings?

nico 3y ago
Thank you! It’s been fun building it and I’m looking forward to keep improving it I definitely want to add scrapers for more sources Haven't tried workday, is it popular for tech jobs (ie. with people that use the command line)? Do you have a sample url with listings you could share?
- ydant 3y ago
  Here's some with tech jobs listed: https://workday.wd5.myworkdayjobs.com/Workday https://workday.wd5.myworkdayjobs.com/Workday https://shipt.wd1.myworkdayjobs.com/en-US/Shipt_External https://shipt.wd1.myworkdayjobs.com/en-US/Shipt_External It doesn't seem like there's a global listing across myworkdayjobs.com sites - each company using it has its own sub-domain. Which sort of makes sense - Workday is an internal HR/employee portal tool licensed to companies.
  - nico 3y ago
    Excellent, thank you so much for the links, it's super interesting It should be pretty straightforward to create a scraper for one of those portals If you feel like taking a crack at it, checkout the code of the Ask HN who's hiring scraper here: https://github.com/nicobrenner/commandjobs/blob/47b5c89402a37ea2929a6481301434a224f86b10/src/hn_scraping.py https://github.com/nicobrenner/commandjobs/blob/47b5c89402a3... I added a comment under the Add Sources issue, to support Workday's career portals: https://github.com/nicobrenner/commandjobs/issues/23#issuecomment-1995738457 https://github.com/nicobrenner/commandjobs/issues/23#issueco...

3abiton 3y ago

One of the best application of LLM!

RyanShook 3y ago

This is awesome. I've been looking for a way to scrape Who's Hiring for my site https://200kjobs.com https://200kjobs.com, thanks for sharing.

nico 3y ago
Thank you for posting your site! Will definitely explore it. Great and simple concept Might try scraping it too ;) Btw, random observation that surprised me when scraping Ask HN, most jobs (100+) are posted the first day of the month, but there are about 15-25 additional jobs posted everyday. This means it’s worth it as a user to get those extra daily listings, but it’s a pain to search through them
- RyanShook 3y ago
  I came across this site which scrapes and shares job postings. The challenge is finding salary ranges: https://hnhiring.com/ https://hnhiring.com/ Feel free to scrape 200kjobs: https://200kjobs.com/feed https://200kjobs.com/feed
  - nico 3y ago
    Thank you! Will definitely go through the feed and try to add it to Command Jobs :) As an aggregator it’s true that finding salary ranges is quite the challenge As an applicant, I’ve settled for just only applying to the positions that have a range If Im really curious about a company without ranges in their listings, GPT/Google sometimes give decent estimates But also, after seeing 100+ listings, I’ve noticed that for the positions I’m looking for, there’s a pretty standard range, which doesn’t vary too much, unless it’s a FAANG or super hot company In any case, using Command Jobs I only get 5-20 job matches, which are also pretty easy to check manually, and it’s not too annoying, given that usually the matches are very interesting and also it’s not 100s of jobs to r
kpandit 3y ago
Your site looks really nice and addresses a real problem. Are you targeting US only remote roles? If not then including any domicile or timezone restrictions in job snippets may be helpful. Also thanks for providing the feed.

cheesehead 3y ago

Nice work! I did something similar with a personal project a few months ago using an open source llm. Also, not sure if you know, there is an api you can use. https://github.com/HackerNews/API https://github.com/HackerNews/API

nico 3y ago
Awesome, did not know about the HN api, nice! What open source LLM did you use? Anything that would run on a CPU of a 5 year old MacBook Air (which I’m using as my home server) Do you have the code for your project? Or maybe some features/ideas that you could share? Thank you
- cheesehead 3y ago
  I'm using https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and currently mistral 7b (on a m1 macbook pro). I'm sure with some prompt examples you can get pretty good results on a smaller model. At the moment I don't have it open sourced due to it being part of a larger project that I'm working on that contains tailwindui licensed components. A cool feature that I'm working on is creating a firefox plugin so you can save/index job postings from other sites and extract out meta information via an LLM. Very similar to this chrome plugin. https://chromewebstore.google.com/detail/huntr-job-search-tracker/mihdfbecejheednfigjpdacgeilhlmnf https://chromewebstore.google.com/detail/huntr-job-search-tr...
  - nico 3y ago
    Love the extension idea, definitely useful for tracking opportunities Thank you for the perspective on running local LLMs Do you think that saas or local LLMs will be the most popular in the next 5 years?
    - cheesehead 3y ago
      Hoping it's local :)

kpandit 3y ago

Congrats Nico. Came here to tell you about the HN API (https://github.com/HackerNews/API https://github.com/HackerNews/API) but I see cheesehead beat me to it. Using the API it is much easier to pick up the latest thread each month and setup a cron job.

nico 3y ago
That’s an excellent tip, thank you very much. It will definitely come in handy soon when implementing the cron :) Are you using the API on some project?
- kpandit 3y ago
  Yes, I am building an application tracking system(ATS) that also includes a job board. Among other sources, I used HN to populate the job board. My project is not open source, but I am happy to share the HN scraping code with you if you like. My email is in my profile.

hn2424 3y ago

[dead]

azophy_2 3y ago

would be interesting to turn this into something like `npx commandjobs` or `ssh commandjobs.com` so you wouldn't even need to install new software

nico 3y ago
Love the ideas, especially the ssh one, it's something that I had been fantasizing about My first year of college, the school had a Telnet-based application to request classes, it would be amazing to build something like that Any pointers on how to build the npx installer?
cookiemonsieur 3y ago
I've always wondered how to build SSH based applications. If anyone has any pointers I'd be extremely grateful.
- nico 3y ago
  My uninformed understanding, is that you allow people to login into an actual user account within the system, but then you replace their shell (usually you change something like /bin/sh to /usr/bin/ssh_application in the entry for that account within the /etc/passwd file), so when someone logs into that account, instead of running an "open" shell, they get the application instead, and when the application exits, the session ends So essentially you should be able to create an ssh application with just a shell script
thomasfromcdnjs 3y ago
There is something similar https://github.com/alexpovel/ancv/ https://github.com/alexpovel/ancv/
- nico 3y ago
  This looks really cool, thank you for sharing!

mfiro 3y ago

Very nice project. Thanks for sharing it here. I saw you used beautifulsoup to scrape, but HN has a public API which works great and it doesn't need any token or auth. I recently wrote a minimal wrapper for that: https://github.com/mfiro/hnconnector https://github.com/mfiro/hnconnector

nico 3y ago
Really appreciate it Super cool wrapper, love the methods that make it super easy to get the content What would be the best way to for example, get the latest Ask HN: who's hiring page and all it's top level comments? Thank you
- mfiro 3y ago
  Here's a quick rundown, you can pull a list of up to 200 "Ask HN" stories using this endpoint: ask_stories = client.get_askstories() then you have the ids and you can sort them reversed to catch the latest stories. client.get_item(<id>) will give you the story's details, in which you'll find item['kids'], which are the IDs for the top-level comments. You can then loop through these comment IDs to get more info on each comment. Another option is to use Algolia's API for HN. But that's not yet supported in my wrapper.

hughdbrown 3y ago

I have not read all the code closely but it looks like you are scraping the HTML directly. True? It would be easier to use the JSON feed for a couple of reasons: - the data is segregated into types of things that you otherwise have to back out from the HTML/CSS - you don't need logic for parsing multiple pages Have a look at code I wrote on this: https://github.com/hughdbrown/who-is-hiring https://github.com/hughdbrown/who-is-hiring The code to get a month of Who Is Hiring posts just hits `https://hn.algolia.com/api/v1/items/{article_id} https://hn.algolia.com/api/v1/items/{article_id}` with a GET request for a particular article_id. Finding the monthly article_ids is the only manual maintenance. The code is split into multiple commandline tools that extract JSON from HN, preprocess the JSON, and filter on what I was interested in (rust job postings). You should be able to replace downstream parts like preprocessing and filtering. See if any of this is helpful.

nico 3y ago
Cool, thank you Yes, scraping the html directly was relatively easy Will try Algolia and the api when doing a cron job for it