6 ms·

I think there are some flaws in this analysis that can be confirmed when the code is released. I think the reason Scala seems surprisingly popular is that is w

I think there are some flaws in this analysis that can be confirmed when the code is released. I think the reason Scala seems surprisingly popular is that is was simple string matching so a post that said "We are looking for someone who can build scalable infrastructure" would count as one mention of "scala". Could be wrong though. Also see: https://news.ycombinator.com/reply?id=18725655&goto=item%3Fid%3D18725425%2318725655 https://news.ycombinator.com/reply?id=18725655&goto=item%3Fi...

JshWright 8y ago
Yeah, similarly with "Excel". Are there really that many posts looking for people with deep spreadsheet experience? Seems more likely that it's a lot of false positives for "excellent" and "excel at".
- osoba 8y ago
  Excel is probably mislabelled, it should say HTML instead. Look at the list above
mchaver 8y ago
Probably the same for rust, looking at the October 2018 data set with https://kennytilton.github.io/whoishiring/ https://kennytilton.github.io/whoishiring/ rust gives 50 results, trust gives 21, \brust gives 26. Similar results for scala.
alehul 8y ago
Would there be an issue with just changing the string to be matched to "scala " or " rust"?
- jgtrosh 8y ago
  The correct method is probably something like "\<$lang\>"
- meritt 8y ago
  There are numerous ways to fix the glaring flaws in this analysis but a very simple approach would be using regex word boundaries, that way it'll work not only for the space but for commas, periods, end-of-line, etc.
chefandy 8y ago
VaLUAble. tRUST. I would be pretty surprised to see these two so much more commonly requested than Javascript.
AznHisoka 8y ago
This is why 90% of data science is the messy, grunt work of just simply cleaning/normalizing the data. The analysis is the easy part.
- screye 8y ago
  > grunt work Honestly, it is an insult to call good data preprocessing skills grunt work. It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.
  - minimaxir 8y ago
    That's the point of the comment. Modern data science tutorials assume the source data is a nice and tidy CSV, when the reality is that data is very often a mess, and it's a constraint often not reflected in the glamorization of the field. (it's also a reason that I recommend that code always be published when making posts such as these, as it's very easy to do wrong!)
    - alexhutcheson 8y ago
      Hadley Wickham's book "R for Data Science"[1] does a good job emphasizing the data cleaning and reshaping steps in the analysis process. [1] https://r4ds.had.co.nz/ https://r4ds.had.co.nz/
    - tfehring 8y ago
      >Modern data science tutorials assume the source data is a nice and tidy CSV For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science. As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.
fredbo22 8y ago
Yup. JavaScript, Node.js, Node, npm, React, Angular, Vue, Express and a few more words mean pretty the same thing and are fishing for the same pool of developers (Yes, I know there's a difference between Angular and React.)
- jenscow 8y ago
  > Yes, I know there's a difference between Angular and React. When can you start?
lettergram 8y ago
You’re very likely correct. See the compare by volume: https://hnprofile.com/compare?search=scala,python,rust https://hnprofile.com/compare?search=scala,python,rust Not the only issue I see, for instance my company posted for “machine learning engineers”, yet that isn’t in the results at all. I think they are doing regex search in the strings, as opposed to identifying the words and doing comparisons
- fulafel 8y ago
  Here's a link with more languages in case anyone else got curious: https://hnprofile.com/compare?search=scala,python,rust,clojure,java,ruby,javascript,haskell https://hnprofile.com/compare?search=scala,python,rust,cloju...
ashelmire 8y ago
Gotta add some word boundaries to that regex!
NicoJuicy 8y ago
I wonder what's included for the regex of .net. Eg: asp.net core, dotnet, c#, VB.net, f#, .net, xamarin,... Because there are a multitude of possibilities.
- emmanueloga_ 8y ago
  Agreed. It looks like the author may be looking for something like a /.net/ regexp? I don't think .net is popular at all in HN job postings. A better regex would be pretty much what you suggest: /asp\.net|dotnet|c#|VB.net|f#|xamarin/i
danieka 8y ago
It appears that the same problem occurs with locations. For example it would appear that the small town of Visby (Sweden) which is 1/100 of the size of Stockholm has 1/10 the jobs of Stockholm. Or more likely the location matches the company name Visby of startup visby.io.
- TheAdamist 8y ago
  Pittsburgh (120) seems to be beating Pittsburg (6) as well, so theres a lot of data cleanup to be done.
NorthOf33rd 8y ago
Same with product manager. Many descriptions list working with product management as a requirement. There have not been 425 product management jobs listed this year. Trust me, I've applied to most of them.
adrianmonk 8y ago
I've noticed that in the hiring threads, a particular company is always hiring, and the email contact username is "austin". It would be easy to pick this up as a location even though it's a person's name. This analysis has the Austin as one of the top 10 locations, and I'm curious whether or not that is skewing the numbers. There are a steady stream of jobs actually in Austin, so either seems possible.
PhasmaFelis 8y ago
For the record, the original post has been edited: your suspicion was correct, and they've fixed it now.
rhizome 8y ago
I think there's also a bug where the numbers don't account for "among those who mentioned [category]."

JshWright 8y ago

Yeah, similarly with "Excel". Are there really that many posts looking for people with deep spreadsheet experience? Seems more likely that it's a lot of false positives for "excellent" and "excel at".

osoba 8y ago
Excel is probably mislabelled, it should say HTML instead. Look at the list above

mchaver 8y ago

Probably the same for rust, looking at the October 2018 data set with https://kennytilton.github.io/whoishiring/ https://kennytilton.github.io/whoishiring/ rust gives 50 results, trust gives 21, \brust gives 26. Similar results for scala.

alehul 8y ago

Would there be an issue with just changing the string to be matched to "scala " or " rust"?

jgtrosh 8y ago
The correct method is probably something like "\<$lang\>"
meritt 8y ago
There are numerous ways to fix the glaring flaws in this analysis but a very simple approach would be using regex word boundaries, that way it'll work not only for the space but for commas, periods, end-of-line, etc.

chefandy 8y ago

VaLUAble. tRUST. I would be pretty surprised to see these two so much more commonly requested than Javascript.

AznHisoka 8y ago

This is why 90% of data science is the messy, grunt work of just simply cleaning/normalizing the data. The analysis is the easy part.

screye 8y ago
> grunt work Honestly, it is an insult to call good data preprocessing skills grunt work. It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.
- minimaxir 8y ago
  That's the point of the comment. Modern data science tutorials assume the source data is a nice and tidy CSV, when the reality is that data is very often a mess, and it's a constraint often not reflected in the glamorization of the field. (it's also a reason that I recommend that code always be published when making posts such as these, as it's very easy to do wrong!)
  - alexhutcheson 8y ago
    Hadley Wickham's book "R for Data Science"[1] does a good job emphasizing the data cleaning and reshaping steps in the analysis process. [1] https://r4ds.had.co.nz/ https://r4ds.had.co.nz/
  - tfehring 8y ago
    >Modern data science tutorials assume the source data is a nice and tidy CSV For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science. As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.

fredbo22 8y ago

Yup. JavaScript, Node.js, Node, npm, React, Angular, Vue, Express and a few more words mean pretty the same thing and are fishing for the same pool of developers (Yes, I know there's a difference between Angular and React.)

jenscow 8y ago
> Yes, I know there's a difference between Angular and React. When can you start?

lettergram 8y ago

You’re very likely correct. See the compare by volume: https://hnprofile.com/compare?search=scala,python,rust https://hnprofile.com/compare?search=scala,python,rust Not the only issue I see, for instance my company posted for “machine learning engineers”, yet that isn’t in the results at all. I think they are doing regex search in the strings, as opposed to identifying the words and doing comparisons

fulafel 8y ago
Here's a link with more languages in case anyone else got curious: https://hnprofile.com/compare?search=scala,python,rust,clojure,java,ruby,javascript,haskell https://hnprofile.com/compare?search=scala,python,rust,cloju...

ashelmire 8y ago

Gotta add some word boundaries to that regex!

NicoJuicy 8y ago

I wonder what's included for the regex of .net. Eg: asp.net core, dotnet, c#, VB.net, f#, .net, xamarin,... Because there are a multitude of possibilities.

emmanueloga_ 8y ago
Agreed. It looks like the author may be looking for something like a /.net/ regexp? I don't think .net is popular at all in HN job postings. A better regex would be pretty much what you suggest: /asp\.net|dotnet|c#|VB.net|f#|xamarin/i

danieka 8y ago

It appears that the same problem occurs with locations. For example it would appear that the small town of Visby (Sweden) which is 1/100 of the size of Stockholm has 1/10 the jobs of Stockholm. Or more likely the location matches the company name Visby of startup visby.io.

TheAdamist 8y ago
Pittsburgh (120) seems to be beating Pittsburg (6) as well, so theres a lot of data cleanup to be done.

NorthOf33rd 8y ago

Same with product manager. Many descriptions list working with product management as a requirement. There have not been 425 product management jobs listed this year. Trust me, I've applied to most of them.

adrianmonk 8y ago

I've noticed that in the hiring threads, a particular company is always hiring, and the email contact username is "austin". It would be easy to pick this up as a location even though it's a person's name. This analysis has the Austin as one of the top 10 locations, and I'm curious whether or not that is skewing the numbers. There are a steady stream of jobs actually in Austin, so either seems possible.

PhasmaFelis 8y ago

For the record, the original post has been edited: your suspicion was correct, and they've fixed it now.

rhizome 8y ago

I think there's also a bug where the numbers don't account for "among those who mentioned [category]."