4 ms·

Am I the only person getting progressively more creeped out by the series of bizarre, unsourced, pseudo-scientific ads TripleByte has been running? On Reddit r

Am I the only person getting progressively more creeped out by the series of bizarre, unsourced, pseudo-scientific ads TripleByte has been running? On Reddit right now they're running this weird faux-linear-algebra thing where they imply that they can build a vector of your skills vs. a vector of job requirements and get a meaningful answer via the dot product. Which is a bit like saying you can predict the weather by taking the dot product of a vector of ocean and air. What even are the units? What does any of this even mean? Hiring is a challenging, multi-dimensional thing. It involves a high-risk and ideally informed decision by multiple parties. Doing it effectively is hard. Doing it effectively and respectfully is harder still. And yet TripleByte comes in and says, "We sound vaguely like machine learning. We got this." Honestly, they make HackerRank, which was another extremely sketchy organization making a lot of very questionable decisions, look reasonable by comparison.

kofejnik 8y ago
Vector thing actually does make some sense, you might want to read about embedding. Or just watch fast.ai lesson 5. Ironically, this was the topic of my final (and failed) conversation with them
- sidlls 8y ago
  How does embedding apply here? What's the justification?
  - kofejnik 8y ago
    It works very nicely for predicting what movie you'd like to watch, so it could potentially work with jobs, too. Collecting enough data could be challenging, though
    - sidlls 8y ago
      That's an incredibe over-simplification of the utility and usage of embedding. For an embedding to work there must be some legitimate (as opposed to arbitrary or even non-existent) relationship to be teased out.
      - kchronic8 8y ago
        > For an embedding to work there must be some legitimate relationship to be teased out. There is most definitely a relationship between candidate skills, job requirements, and interview result/job performance. The point is to get rid of all these bullshit subjective excuses when people fail an interview/get fired. The answer is simple: they likely weren't good enough. But no one likes hearing that.
        KirinDave 8y ago
        > There is most definitely a relationship between candidate skills, job requirements, and interview result/job performance. This is actually not at all an obvious fact. It's continuously offered as a ground truth, but many people dispute it and a lot of successful organizations do not recruit weighing these factors as heavily as you're suggesting. > The answer is simple: they likely weren't good enough. But no one likes hearing that. Possibly, or possibly they were plenty good but so obnoxious or outrageous that they wouldn't be welcome. I've certainly done that more than once in my time building tech organizations. I still remember the guy who effusively praised the beauty of all the women he saw and congratulated me on "the haul". Even touched a woman's hair to compliment it. Too bad he was such a sleazebag, he seemed smart. But even from a cold economic standpoint the cost to the company for an inevitable sexual harassment lawsuit would always eclipse any value he could provide.
      - kofejnik 8y ago
        >> a vector of your skills vs. a vector of job requirements Of course it's an over-simplification, but you don't think there's a legitimate relationship between the two?
        sidlls 8y ago
        Maybe, maybe not. It isn't obvious in any case, and reducing the problem to an embedding in the way suggested requires more validation.
- KirinDave 8y ago
  No. Sorry, you're not forming a coherent vector space between both skills and demands without published research on the subject. You do not simply say, "Gosh it seems like I have said words related to ML and therefore its application is plausible." What's more, the notion that recruiting is in fact a skills demand model is itself fundmentally misleading. Many of the skills you want are domain specific and in fact cannot be expected to be acquired anywhere but on the job. Given how many shops subtly permute "react" or "Golang" to mean a lot of skills in a utility cluster, any space you form is going to be incredibly specific to the employer and difficult to map anywhere else.
  - itronitron 8y ago
    >> subtly permute "react" or "Golang" to mean a lot of skills in a utility cluster This, and the tendency for shops to evaluate candidates' skillset against the shops' most mission critical parts and processes which for whatever reason have not been operationally hardened and locked down.
mlthoughts2018 8y ago
The vector of skills vs. vector of job properties thing could be a coherent idea. I'd argue it's not a good idea (and I have created joint-embedding models for multi-modal text and image nearest neighbor search systems). But it could be a coherent idea. In deep learning anyway, it's a very similar idea in spirit to something like word2vec. In word2vec you learn how to create some arbitrary vector of real numbers, let's say a 256-component vector, that represents words. The components of the vector for each word are learned by forcing the model to solve a prediction task, like predicting whether two words were truly seen in the same context in a given document or not. The model updates the 256-vector of each word such that this prediction task (which can be based on the inner product of those 256-vectors for sets of words) gets better and better. By the end of this, you can in some sense claim a word's mapping into its 256-vector "represents" some "context" intrinsic to that word. And that 256-dimensional "space of context" where the vectors reside is known as an embedding space. So the 256-vector of a word is often called that word's embedding. For more generic problem, you aren't learning just one type of embedding, but possibly many. In TripleByte's case, I'm guessing they are inputting text characteristics of candidate resumes, self-descriptions, GitHub repos, Stack Overflow profiles, LinkedIn pages, etc. etc., and treating a candidate like a bag-of-words from all those combined sources of data. And maybe they also have other data too, like performance on skills assessments, favorite craft beer, and whether you think dog-friendly offices are unprofessional or you think they are super cool. On the other side they input a bunch of data about jobs: bag-of-words text from job ads, tech blogs from that company, the company's GitHub repos if any are public facing, profiles of their existing team members, what gourmet coffee they keep in the office, and whether or not they believe in "unlimited" vacation. If they have labeled training data, like say a bunch of people who work in their client companies that offered to give them resumes and personal data, and then use the existing job those people occupy as a ground truth label for a "positive" matching between a candidate and a job, then they could train a model that learns how to map all that candidate data and also all that job-specific data both into 256-component embedding vectors that are constrained to reside in the same space. And if they really want for the dot product to be the key way to describe applicant-to-job similarity, they could make the loss function work based on cosine distance between the learned embedding vectors of candidates and the learned embedding vectors of their ground truth actual jobs. I have no idea if this approach, or something like it, is what TripleByte actually does. But it sounds like they want to market themselves as really fancy machine learning engineers by at least talking about this. Frankly, in deep learning this is a fairly cookie cutter approach and is often the first thing someone tries. In fact, I just Googled for "machine learning visual searches" and one of the first links was this post from the Squarespace engineering team (I do not work at or have any affiliation with Squarespace): < https://engineering.squarespace.com/blog/2017/using-machine-learning-for-visual-searches https://engineering.squarespace.com/blog/2017/using-machine-... > From the blog post it looks like this was just a sort of fun side project, with a goal of developing it into an actual search service for Squarespace's search engine at some point (maybe they already did?). The reason why TripleByte's attempts to use this in advertisements fall flat is because it's a commonplace idea -- so common that here we even have a website hosting and content platform demoing how the basic idea works for image searching as some random blog post. My guess is that other experienced ML engineers would see TripleByte's choice with this ad as at best a little self-aggrandizing and silly, or at worst maybe even intellectually dishonest about the real importance or effectiveness of this type of embedding-based similarity approach.
booleandilemma 8y ago
Don’t speak ill of them, they’ll lower your TripleByte score.
deleted 8y ago
[deleted]

kofejnik 8y ago

Vector thing actually does make some sense, you might want to read about embedding. Or just watch fast.ai lesson 5. Ironically, this was the topic of my final (and failed) conversation with them

sidlls 8y ago
How does embedding apply here? What's the justification?
- kofejnik 8y ago
  It works very nicely for predicting what movie you'd like to watch, so it could potentially work with jobs, too. Collecting enough data could be challenging, though
  - sidlls 8y ago
    That's an incredibe over-simplification of the utility and usage of embedding. For an embedding to work there must be some legitimate (as opposed to arbitrary or even non-existent) relationship to be teased out.
    - kchronic8 8y ago
      > For an embedding to work there must be some legitimate relationship to be teased out. There is most definitely a relationship between candidate skills, job requirements, and interview result/job performance. The point is to get rid of all these bullshit subjective excuses when people fail an interview/get fired. The answer is simple: they likely weren't good enough. But no one likes hearing that.
      - KirinDave 8y ago
        > There is most definitely a relationship between candidate skills, job requirements, and interview result/job performance. This is actually not at all an obvious fact. It's continuously offered as a ground truth, but many people dispute it and a lot of successful organizations do not recruit weighing these factors as heavily as you're suggesting. > The answer is simple: they likely weren't good enough. But no one likes hearing that. Possibly, or possibly they were plenty good but so obnoxious or outrageous that they wouldn't be welcome. I've certainly done that more than once in my time building tech organizations. I still remember the guy who effusively praised the beauty of all the women he saw and congratulated me on "the haul". Even touched a woman's hair to compliment it. Too bad he was such a sleazebag, he seemed smart. But even from a cold economic standpoint the cost to the company for an inevitable sexual harassment lawsuit would always eclipse any value he could provide.
    - kofejnik 8y ago
      >> a vector of your skills vs. a vector of job requirements Of course it's an over-simplification, but you don't think there's a legitimate relationship between the two?
      - sidlls 8y ago
        Maybe, maybe not. It isn't obvious in any case, and reducing the problem to an embedding in the way suggested requires more validation.
KirinDave 8y ago
No. Sorry, you're not forming a coherent vector space between both skills and demands without published research on the subject. You do not simply say, "Gosh it seems like I have said words related to ML and therefore its application is plausible." What's more, the notion that recruiting is in fact a skills demand model is itself fundmentally misleading. Many of the skills you want are domain specific and in fact cannot be expected to be acquired anywhere but on the job. Given how many shops subtly permute "react" or "Golang" to mean a lot of skills in a utility cluster, any space you form is going to be incredibly specific to the employer and difficult to map anywhere else.
- itronitron 8y ago
  >> subtly permute "react" or "Golang" to mean a lot of skills in a utility cluster This, and the tendency for shops to evaluate candidates' skillset against the shops' most mission critical parts and processes which for whatever reason have not been operationally hardened and locked down.

mlthoughts2018 8y ago

The vector of skills vs. vector of job properties thing could be a coherent idea. I'd argue it's not a good idea (and I have created joint-embedding models for multi-modal text and image nearest neighbor search systems). But it could be a coherent idea. In deep learning anyway, it's a very similar idea in spirit to something like word2vec. In word2vec you learn how to create some arbitrary vector of real numbers, let's say a 256-component vector, that represents words. The components of the vector for each word are learned by forcing the model to solve a prediction task, like predicting whether two words were truly seen in the same context in a given document or not. The model updates the 256-vector of each word such that this prediction task (which can be based on the inner product of those 256-vectors for sets of words) gets better and better. By the end of this, you can in some sense claim a word's mapping into its 256-vector "represents" some "context" intrinsic to that word. And that 256-dimensional "space of context" where the vectors reside is known as an embedding space. So the 256-vector of a word is often called that word's embedding. For more generic problem, you aren't learning just one type of embedding, but possibly many. In TripleByte's case, I'm guessing they are inputting text characteristics of candidate resumes, self-descriptions, GitHub repos, Stack Overflow profiles, LinkedIn pages, etc. etc., and treating a candidate like a bag-of-words from all those combined sources of data. And maybe they also have other data too, like performance on skills assessments, favorite craft beer, and whether you think dog-friendly offices are unprofessional or you think they are super cool. On the other side they input a bunch of data about jobs: bag-of-words text from job ads, tech blogs from that company, the company's GitHub repos if any are public facing, profiles of their existing team members, what gourmet coffee they keep in the office, and whether or not they believe in "unlimited" vacation. If they have labeled training data, like say a bunch of people who work in their client companies that offered to give them resumes and personal data, and then use the existing job those people occupy as a ground truth label for a "positive" matching between a candidate and a job, then they could train a model that learns how to map all that candidate data and also all that job-specific data both into 256-component embedding vectors that are constrained to reside in the same space. And if they really want for the dot product to be the key way to describe applicant-to-job similarity, they could make the loss function work based on cosine distance between the learned embedding vectors of candidates and the learned embedding vectors of their ground truth actual jobs. I have no idea if this approach, or something like it, is what TripleByte actually does. But it sounds like they want to market themselves as really fancy machine learning engineers by at least talking about this. Frankly, in deep learning this is a fairly cookie cutter approach and is often the first thing someone tries. In fact, I just Googled for "machine learning visual searches" and one of the first links was this post from the Squarespace engineering team (I do not work at or have any affiliation with Squarespace): < https://engineering.squarespace.com/blog/2017/using-machine-learning-for-visual-searches https://engineering.squarespace.com/blog/2017/using-machine-... > From the blog post it looks like this was just a sort of fun side project, with a goal of developing it into an actual search service for Squarespace's search engine at some point (maybe they already did?). The reason why TripleByte's attempts to use this in advertisements fall flat is because it's a commonplace idea -- so common that here we even have a website hosting and content platform demoing how the basic idea works for image searching as some random blog post. My guess is that other experienced ML engineers would see TripleByte's choice with this ad as at best a little self-aggrandizing and silly, or at worst maybe even intellectually dishonest about the real importance or effectiveness of this type of embedding-based similarity approach.

booleandilemma 8y ago

Don’t speak ill of them, they’ll lower your TripleByte score.

deleted 8y ago

[deleted]