4 ms·

I am surprised to hear that. Would you be willing to share more details with me?

mlthoughts2018 8y ago
I actually mentioned it in a Hacker News thread a few years ago, and someone named Jack contacted me from Asana to try to provide feedback by email and help clarify what happened. While I appreciated the extra feedback, and the tone of the communication was reasonably polite, the actual commentary of the feedback seemed unreasonable and totally subjective to me. One of the key issues was that the code test on one hand encouraged someone not to spend too long, and to choose a simple statistical model and work on explaining how it works, but on the other hand the reviewer rejected it because it wasn't "results-oriented" or "focused enough on final conclusions." It seemed obvious that given the nature of the take-home test, nobody was looking for a model that definitively solved the problem or had amazing accuracy. Just something small-scale, but with a meaningful discussion of the trade-offs and basic discussion of how you fit a model, diagnostics, items you would work on if given more time, etc. But the feedback I got was that my approach needed to be more "results focused" and that in my write-up I was overly cautious to say not to draw any significant conclusions from a simple model on the toy data set or to read too much into a model that needed to be completed in a couple of hours. They said someone "couldn't thrive" in the Asana environment if they weren't able to work in a more results-driven and conclusion-driven way. I felt the criticisms were extremely subjective and political. As a candidate, I can't read someone's mind about how "results oriented" a vague take-home modeling assignment needs to be, let alone that I have limited personal time which I can devote to it anyway. I sank a lot of time into writing code for a reasonable "first attempt" kind of model, and providing some diagnostic charts and discussion. It felt like Asana really wasted my time on it, and that in the end the feedback reflected some unspecified and vague properties that couldn't be inferred or understood from the test instructions or talking with any of the engineers. It was just random. My same solution for the same test at another company might have been viewed as a great one, even from a "results-oriented" point of view because it would be unreasonable to expect that from a short take-home assignment. I actually got a job as a deep learning engineer not long after that, and then got my most recent job as a machine learning team manager. My experiences working on machine learning models in production and hiring engineers for my team has only reinforced to me that such a manner of take-home data science project is really a bad way to do things.
- erinakinci 8y ago
  I appreciate your honest feedback here! I’m currently the hiring manager for this position, and I agree that this doesn’t sound like a good experience. I disagree that a take-home assignment is inherently a bad idea - I actually have found a lot of value in seeing the work people produce there and feel it’s the part of the interview that’s closest to what the team actually does day-to-day - but I also don’t want candidates to feel like their efforts are being graded in a way that’s unfair or lacks transparency. Because of that, in the time since you interviewed, we’ve updated the rubric to be less subjective and better set up to reward the diverse ways someone could show data science skill, updated the text of the assignment to more clearly indicate what we’ll be evaluating, and made it standard practice to share with candidates right away where they did well and where they lost points. If I'm reading your concerns correctly, I think those changes are likely to address many of them.
  - mlthoughts2018 8y ago
    Sounds like you’re doing a lot of good work on the process since the time of my experience. As long as the full set of evaluation criteria and expectations are made clear, and they sincerely fit into the time limit given, a take-home test can be perfectly fine. The trouble is two-fold - the test creators usually drastically underestimate how long the data canonicalization / normalization process takes just to get up & running with a toy data set. This stuff plus cursory data exploration always takes an hour or two — so code tests asking you to just hack together a model plus a write-up in a couple of hours are just not realistic. - tests are often subverted for political reasons. Someone doesn’t want to hire a candidate who is more skilled than they are; someone wants to recommend their college friend for the role; someone just prefers to work with extraverts but the candidate comes across as an introvert on the phone — whatever it is. Then arbitrary and secondary aspects of a test solution get amplified for debate to steer the decision in a political direction. Nobody wants to believe their teammates would do this, and many times people do it without realizing. But I’ve seen my own team doing it in the past during interview evaluations, and it’s particularly bad when debating over how to assess a take-home test submission. As a candidate I experienced it a few times too, notably with Asana as mentioned before. The other sad part of that Asana interview was that nobody in any part of the Asana pipeline ever asked me about my work experience. Every interview was tech questions, programming questions, and the take-home test. Not a single question about my work history. In the test feedback, the reviewer suggested I might not work well in a product-oriented environment — yet nobody had asked about my work experience in a product-oriented environment at an education technology company. That’s just one example, but there were others. It saddens me a lot that the tech industry is like this. It’s so myopic to believe I could assess someone’s subjective disposition towards being results-oriented or customer-focused or product-minded based on some random 3-hour test, and never even so much as ask them about their actual work history, where they actually solved problems and delivered products. It doesn’t feel good to be told, “based on this ambiguous code test, you’re probably not good at thinking about product development,” when you’ve got plenty of product-development items on your resume you’d be happy to share yet are never even asked about.

mlthoughts2018 8y ago

I actually mentioned it in a Hacker News thread a few years ago, and someone named Jack contacted me from Asana to try to provide feedback by email and help clarify what happened. While I appreciated the extra feedback, and the tone of the communication was reasonably polite, the actual commentary of the feedback seemed unreasonable and totally subjective to me. One of the key issues was that the code test on one hand encouraged someone not to spend too long, and to choose a simple statistical model and work on explaining how it works, but on the other hand the reviewer rejected it because it wasn't "results-oriented" or "focused enough on final conclusions." It seemed obvious that given the nature of the take-home test, nobody was looking for a model that definitively solved the problem or had amazing accuracy. Just something small-scale, but with a meaningful discussion of the trade-offs and basic discussion of how you fit a model, diagnostics, items you would work on if given more time, etc. But the feedback I got was that my approach needed to be more "results focused" and that in my write-up I was overly cautious to say not to draw any significant conclusions from a simple model on the toy data set or to read too much into a model that needed to be completed in a couple of hours. They said someone "couldn't thrive" in the Asana environment if they weren't able to work in a more results-driven and conclusion-driven way. I felt the criticisms were extremely subjective and political. As a candidate, I can't read someone's mind about how "results oriented" a vague take-home modeling assignment needs to be, let alone that I have limited personal time which I can devote to it anyway. I sank a lot of time into writing code for a reasonable "first attempt" kind of model, and providing some diagnostic charts and discussion. It felt like Asana really wasted my time on it, and that in the end the feedback reflected some unspecified and vague properties that couldn't be inferred or understood from the test instructions or talking with any of the engineers. It was just random. My same solution for the same test at another company might have been viewed as a great one, even from a "results-oriented" point of view because it would be unreasonable to expect that from a short take-home assignment. I actually got a job as a deep learning engineer not long after that, and then got my most recent job as a machine learning team manager. My experiences working on machine learning models in production and hiring engineers for my team has only reinforced to me that such a manner of take-home data science project is really a bad way to do things.

erinakinci 8y ago
I appreciate your honest feedback here! I’m currently the hiring manager for this position, and I agree that this doesn’t sound like a good experience. I disagree that a take-home assignment is inherently a bad idea - I actually have found a lot of value in seeing the work people produce there and feel it’s the part of the interview that’s closest to what the team actually does day-to-day - but I also don’t want candidates to feel like their efforts are being graded in a way that’s unfair or lacks transparency. Because of that, in the time since you interviewed, we’ve updated the rubric to be less subjective and better set up to reward the diverse ways someone could show data science skill, updated the text of the assignment to more clearly indicate what we’ll be evaluating, and made it standard practice to share with candidates right away where they did well and where they lost points. If I'm reading your concerns correctly, I think those changes are likely to address many of them.
- mlthoughts2018 8y ago
  Sounds like you’re doing a lot of good work on the process since the time of my experience. As long as the full set of evaluation criteria and expectations are made clear, and they sincerely fit into the time limit given, a take-home test can be perfectly fine. The trouble is two-fold - the test creators usually drastically underestimate how long the data canonicalization / normalization process takes just to get up & running with a toy data set. This stuff plus cursory data exploration always takes an hour or two — so code tests asking you to just hack together a model plus a write-up in a couple of hours are just not realistic. - tests are often subverted for political reasons. Someone doesn’t want to hire a candidate who is more skilled than they are; someone wants to recommend their college friend for the role; someone just prefers to work with extraverts but the candidate comes across as an introvert on the phone — whatever it is. Then arbitrary and secondary aspects of a test solution get amplified for debate to steer the decision in a political direction. Nobody wants to believe their teammates would do this, and many times people do it without realizing. But I’ve seen my own team doing it in the past during interview evaluations, and it’s particularly bad when debating over how to assess a take-home test submission. As a candidate I experienced it a few times too, notably with Asana as mentioned before. The other sad part of that Asana interview was that nobody in any part of the Asana pipeline ever asked me about my work experience. Every interview was tech questions, programming questions, and the take-home test. Not a single question about my work history. In the test feedback, the reviewer suggested I might not work well in a product-oriented environment — yet nobody had asked about my work experience in a product-oriented environment at an education technology company. That’s just one example, but there were others. It saddens me a lot that the tech industry is like this. It’s so myopic to believe I could assess someone’s subjective disposition towards being results-oriented or customer-focused or product-minded based on some random 3-hour test, and never even so much as ask them about their actual work history, where they actually solved problems and delivered products. It doesn’t feel good to be told, “based on this ambiguous code test, you’re probably not good at thinking about product development,” when you’ve got plenty of product-development items on your resume you’d be happy to share yet are never even asked about.