12 ms·
I analyzed the history of “Who is Hiring?” threads
- deleted 7y ago[deleted]
- philipkiely 7y agoAuthor here. I scraped 8 years of data on Who is Hiring? threads. The data is available through the article and on GitHub. Spoiler alert: people want to hire full time software engineers in San Francisco. Happy to answer any questions.
- ulucs 7y agoIf not intended as a learning exercise, why not use the existent API?
- cushychicken 7y agoThe author is still an undergraduate, so yeah, I'd file under "learning exercise".
- philipkiely 7y agoI hope to engage in learning exercises long after I graduate!
- filoleg 7y agoHearing this was pretty refreshing and encouraging. Thanks for posting that, and I mean it.
- philipkiely 7y agoThe article that I was hired to write was a web scraping tutorial, I did all of this to make it interesting.
- fapjacks 7y agoI've always found it really "interesting" how much resistance the industry shows to hiring part time and/or remote, the two changes in hiring which would really ease their supposed labor issue. I refuse to believe the dogma about labor shortages until I see companies actually going out and hiring all the available qualified employees. All the nitpicking over what constitutes "qualified" which isn't about being able to do the work is just an excuse to exert butts-in-seats dominion over workers.
- someguydave 7y agoThe number of electrical engineers employed in the US has been going down for years. One wonders what will happen when an entire generation is raised to think that “technology”=web pages and games. Who will build new computers?
- minimaxir 7y agoHTML scraping should be the last option you consider to get data after all else fails. Even though the Hacker News API (https://github.com/HackerNews/API https://github.com/HackerNews/API) is somewhat old, it's a much more kosher way of getting data. Even better is to use the public data dump in BigQuery (https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news https://console.cloud.google.com/marketplace/details/y-combi...). Quick query to get all top-level comments in posts by whoishiring: #standardSQL WITH whoishiring_posts AS ( SELECT id from `bigquery-public-data.hacker_news.full` WHERE `by`="whoishiring" AND type="story" ) SELECT text FROM `bigquery-public-data.hacker_news.full` WHERE type="comment" AND parent IN (SELECT id from whoishiring_posts)
- plibither8 7y agoThe API that https://hn.algolia.com https://hn.algolia.com provides can also be used! In my experience, it's more verbose and the queries are more customisable.
- saagarjha 7y agoThis is great! I've been wondering for a while how to efficiently load a comment tree without spamming the Hacker News API with hundreds of requests, and this seems like it would work nicely.
- ChristianBundy 7y agoExample Node.js library that I wrote a few years ago: https://github.com/fraction/node-hacker-news-api https://github.com/fraction/node-hacker-news-api
- minimaxir 7y agoAgreed, unfortunately.
- Naac 7y agoBut the official Hacker News API is really painful to use. Specifically because each comment is its own API call (?! ). I know the HN staff mentioned they're working on a V2 API, but until then the HN HTML tree is so clean, I think this is an exceptional case where scraping the HTML yields a cleaner result that's also less load on the HN servers. EDIT: Like the other comment on this thread mentioned, using https://hn.algolia.com/api https://hn.algolia.com/api seems ideal here.
- nwsm 7y agoNeat. I'd be interested in seeing how many job posts are reposted each month, and which companies post the most. I have noticed many jobs/companies that are present almost every month. The rainbow coloring of the graphs is distracting.
- ryandrake 7y agoI had “plans” a while back to put together a cynically-named “Who’s Not Hiring” script that did just that. Look at which companies post the same or extremely similar jobs month after month. Life got in the way and I abandoned the project early, but it would be interesting to see the results of such an effort.
- jotux 7y agoI'm on a team at my current job with ~10 other engineers, and at my previous job I was on a team of ~12. With growth, retirements, and turnover at both jobs we are always hiring. My rough approximation from both jobs (spanning about 10 years) is that we're able to hire one person every 6 months. I'm not a hot tech geographic region, and the field I work in is somewhat niche, but there are many non-cynical reasons a job might be posted indefinitely.
- unleashit 7y agoIndefinitely posting while hiring only once every six months is actually the poster child of the problem. If it really takes six months to find someone to fit the role, you might want to take a look at your hiring process because as you must be rejecting (or ignoring the applications) a lot of great, qualified people. IMHO if someone did take the time to publish the metrics of companies who perpetually post the same positions, it would be a fair counter balance.
- jofer 7y agoI'd argue six months is about average for the jobs I've seen and had over the years. Not every role needs an off-the-shelf front end dev. A lot of things really do require significant domain and industry specific knowledge. For those roles, it takes a very long time to turn up even one minimally qualified candidate. It's easy to say "find someone bright who can learn on the job", but that's not practical in the cases I've seen. Many things need either specialized education or experience in a similar role to be effective within a 2-3 year timeframe. When on-the-job training would take multiple years, it's worth spending six months to a year to find someone who can fill the role. At least in the industries I've worked in, the roles that require extensive and rare training/experience outnumber the roles that don't.
- lazyant 7y agoThe last section about when not to scrap warms my heart.
- garysahota93 7y agoSame!! Like with most tools, web scraping can be used for nefarious reasons very easily. I'm glad he provided that caveat. Signs of someone truly trying to create good work and not just clickbait.
- philipkiely 7y agoThanks! It was important to me to include that in an article intended to teach the skill.
- hadsed 7y agoA little off topic... but I love this because it gives me an idea. I've been looking at LinkedIn with some immensely negative feelings for some time now. I've always wished someone would just disrupt the hell out of them, because they don't deserve the near-monopolistic position they're in. They just don't care about their users! And before someone says, "well [pushes up glasses] their real users are recruiters that they sell $20k/year seats to their search tool". Yeah, as a hiring manager I've tried using it. Awful! :) And jobs boards are nothing new, I guess, but something feels really nice about how much bigger these "Who is Hiring?" threads are getting on HN. Maybe we really do need a jobs-centric "social network" for industry verticals. Beyond terrible UX and general disrespect, LinkedIn also just doesn't do enough with their data to make it useable. Maybe that's because it's just too hard to do for every job and industry out there. And maybe that next-gen professionals' social network can be the first open and free one? Or maybe I'm just wishing too hard for something that can't/won't be done.
- minimaxir 7y ago> Beyond terrible UX and general disrespect, LinkedIn also just doesn't do enough with their data to make it useable. From LinkedIn's perspective, that's a feature, not a bug.
- gota 7y agoLinkedIn has become plagued by self-help-like stories and other self promoting, vague and meaningless write ups, in my experience. It wasn't always like this. I used to have a reasonably cool feed centered towards technologies and news that are at least partly relevant to me and my industry. No more. It's all 'I gave a homeless person lunch and today they are the #1 duck breeder in Brazil' or some other nonsense. I don't know when this started or if it is localized, but it renders the 'place to find useful information' aspect of social media completely void. There's no amount of reasonable curating effort that will save a news feed like that. I try not to be too negative about things but without some serious change I'll use a static LinkedIn profile and visit it for a fraction of a second every week - to check for messages - and nothing more.
- sharadov 7y ago
- danbrooks 7y agoNice work! One comment - the plots show the fraction (not percentage) of postings for interns and remote workers.
- mfbx9da4 7y agoWhy didn't you just use the Algolia API? I used the Algolia API to build a 24 / 7 live stream of randomly selected highly voted HN vídeos https://www.crowdform.co.uk/hntv https://www.crowdform.co.uk/hntv saved me a lot of time
- guessmyname 7y agoWhat happened on December 2011 and June 2013? Why is there no data?
- philipkiely 7y agoA couple of months were missing for each data set, if you page through the "whoishiring" user submissions you'll notice them missing.
- rsweeney21 7y agoThe data on "Freelancer? Seeking freelancer?" is interesting and I think it can be misleading. The disparity between freelancers seeking work and people seeking freelancers on HN is huge - way more people looking for work, than people looking to hire. That makes it feel like there is an oversupply of freelancers, which would drive down the pay rates. However, I think it's more of a reflection of the HN audience than the broader market. I.E. HN readers don't hire as many contract/freelance employees. This aligns with what we've seen in practice as well. If software is the company's core competency, they will be less likely to hire contract or outsource. On the other hand, there is a huge need for non-software companies to hire senior devs and they hire a lot of contract/freelance. Demand is high and rates are good ($120-200/hr)...at the right companies. So, if you are a contractor looking for work, I would target companies where software isn't their primary business. Source: I run a company that helps senior software engineers find contract work. (Shameless plug: www.facetdev.com)
- felideon 7y ago> I run a company that helps FAANG-rite-of-passage-waving software engineers find contract work. FTFY.
- whatamidoingyo 7y agoFixed indeed. Brace yourself for the downvotes. :o
- deleted 7y ago[deleted]
- dang 7y agoPlease don't do this here.
- mattsfrey 7y agoI get the comment is a bit snarky, but if you dig into this guys site it's required that you have worked 3+ years at a "top tier company" with a rotating ticker of FAANG and other giant SV behemoths on it's front page ad, which IMO is fairly pretentious and warrants a bit of a call out.
- QUICdownstrokes 7y agoAh ha! Just as I had suspected! The new way to marginalize disposable team members is to replace "interns" with "remote" workers. The savings are likely shifted over to cutting costs for the on-site office footprint, instead of optimizing for freebie slave labor from the lesser skilled, green college newbies. This also probably carries a quality over quantity trade off, in terms of hours spent working (college kids sweating 70 hour burn-out weeks, because paying dues or whatever). Tech is mature enough, that there are enough veterans in the talent pool shrewdly compromising the level of their engagements, and there is less novelty to attract wide-eyed, coffee fetching would-be script kiddies anymore.
- vipbababd 7y agoAuthor here. I scraped 8 years of data on Who is Hiring? threads. The data is available through the article and on GitHub. Spoiler alert: people want to hire full time software engineers in San Francisco. Happy to answer any questions.
- depressedpanda 7y agoPeople who are saying that the author should have used an API are missing the point; sometimes the only way to gather data is by resorting to scraping, and this article details how one could go about doing that, by using an interesting target: HN. To the author: Good job, I'm sure many people will find your article useful! I wish I would've had this article at hand a couple of years ago when I needed to aggregate data from several websites that provided no API.
- minimaxir 7y ago> sometimes the only way to gather data is by resorting to scraping Nowadays, web scraping has a nonzero risk of hitting legal issues for sites when "the only way to gather data is by resulting to scraping", especially when an API already exists with a ToS/Guidelines on how the data should be obtained and used. And even moreso if the data itself has monetary value.
- philipkiely 7y agoThank you!!
- tsumnia 7y agoIf you're still feeling froggy with your data, I'd love a follow up with some NLP term frequency used in the posts. Did intern frequency drop or did the sheer increase in posts shrink their overall percentage? The concern I have there is the idea of becoming too "top heavy" where no one is interested in helping interns grow and only want to recruit experienced people.
- philipkiely 7y agoYou raise a great question. I also wonder if Hacker News as a platform tends towards more senior people, I know that I usually feel like one of the least experienced people in most comment sections.
- eurticket 7y agolove this, beautiful.
- philipkiely 7y agoThanks you!
- m3at 7y agoWhile the article seem intended as a web scraping tutorial, it's good to remember that there is an official HN api [1] in case you really want to do data science on the site. [1] https://github.com/HackerNews/API https://github.com/HackerNews/API
- killjoywashere 7y agoSince we're in the middle of the Who is Hiring cycle, what's the best way to post a couple of jobs?
- dloxvic123 7y agoI was recommended to this professional hacker, He helped me to expose all my partner’s secret that kept my marriage intact, who my spouse was cheating with and all…, Contact him on: alimohammedprohacker @ g m a i l . com contact whatsapp phone : + 1 6 1 9 6 3 2 5 9 2 6 his services include hacking (hint: mobile phones, Instagram, Facebook, gmail,twitter, whatsapp, kik, bank account, iphones, MeetMe, Snapchat, WeChat, hike etc.), tracking, cloning ,upgrading result,preventing you from been hacked or tracked,Adding any important account to your account without account owner knowing he can also help you to spy on your spouse so that you'll know whether he or she is cheating. he can also teach hacking at a very affordable price.
- manubeauty 7y agoHave you ever searched for a way to create a side income? If so, then you want to check out this free training how to make 6 figure income in 2019 https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J
- powellbu 7y agoI know a professional hacker SPARRO who has worked for me once this month. He offers legit services such as clearing of bad records online without being traced back to you, He clone phones, hack Facebook account, instagram, WhatsApp, emails, Twitter, bank accounts, Skype, FIXES CREDIT REPORTs, track calls,clear criminal records. He also help retrieve accounts that have been taking by hackers. His charges are affordable, reliable and 100% safe. Contact him via address below... Email..SPARROHACKER (AT) GMAILCOM Text no.. +1 (9 7 8) 4 3 4 0 9 5 3 Whatsapp: +9 7 1 5 8 6 2 8 4 4 9 8