4 ms·

The problem is that Asana seems to have preconceived ideas of what answers are supposed to look like, but the questions are vague and open-ended enough that eve

by p4wnc6 11y ago

The problem is that Asana seems to have preconceived ideas of what answers are supposed to look like, but the questions are vague and open-ended enough that even very good candidates can give good answers that simply diverge from the manner of thinking that the Asana question designer had in mind. If the question has a well-defined answer, then this is good. If you are asking questions that the candidate "doesn't know the answer to" merely as a parochial side effect of the question's vagueness and the vast space of possible good approaches, then you're really only screening for people who happen to think about things the same way you already do, and thus are good at guessing what you want to hear. I'd argue that this is very bad in terms of diversity of thought.

pklausler 11y ago
I use problems with multiple acceptable solutions and easy tests of correctness -- i.e., it's obvious whether or not the approach is sound and the code works. For example, write a function that determines whether two rectangles on a display overlap, given the coordinates of their corners. Solving it requires the ability to analyze the problem and describe the conditions, the ability to code a simple Boolean predicate, and some aptitude for checking boundary cases and writing unit tests. I have others; what they have in common is that they're really easy problems for great programmers, and well within the limits of a 45-minute technical interview for competent coders, and good at exposing candidates who lack some basic skills.
jacklionheart 11y ago
Hi there! That sounds bad. Could you let me know what's giving you the impression that that's Asana's goal? We really try to explicitly avoid that kind of interview evaluation, and put strong emphasis on Communication, User empathy, and Learning, per the guide. But definitely "did they say the right answer?" is an easy trap to fall into so its entirely possible that some interviewers are doing it -- would love any concrete feedback you could give to help us avoid it in the future.
- p4wnc6 11y ago
  The more concrete feedback is in the previous post (you commented on that one too). But I will elaborate here: On 8/4/15, I received a message from an Asana representative via Stack Overflow Careers. After a few back-and-forths discussing specifics about whether I was a good fit, I then agree to do the take-home test. I received the take home test on 8/7/15. I emailed back on 8/8/15 with a verifiable error in a timestamp column of an AWS-hosted Postgres database that was configured with some test data (data resembling some certain kinds of user data for modeling what an "engaged" user is, and what factors predict when a user will be "engaged"). I included code which could be run to demonstrate the erroneous data. On 8/9/15 I wrote in again stating that I had created a workaround for the erroneous timestamp data, since the only way I could complete the exam was to do it on those particular days (I couldn't wait around for the exam data to be fixed). I submitted my solution on 8/10/15. I got an email on 8/11/15 with a note from someone in engineering apologizing for the timestamp data error, but also saying it shouldn't affect my solution. The recruiter who sent that email also said that my submission was currently being evaluated. Then I waited a week before hearing anything. On 8/18/15 I got a response apologizing for the delay and saying that I was invited for the next round interview. The email said I would be speaking with a person named Jack (maybe you?) a few days later (8/21/15 at 4:00 pm EST). But that call actually included two data scientists over one conference call line. Both interviewers effectively made no comment on my software choices. In fact, when I spoke about how I cared about software design and best practices, even when working on a rapid prototype, it actually sounded as if the interviewers thought this was a bad thing and there was some awkward silence. I brought up examples of how I had worked in a very fast paced environment before (quant finance) and my team had learned some hard lessons that the naive approach of letting data scientists just write scripts for ad hoc modeling, assuming that engineering will "productionize" it later, tends to lead to some bad failure cases, and that it's far more efficient over time to simply require data scientists to construct rapid prototypes, even from the first line of exploratory, ad-hoc code, that adhere to many best practices and are only a short distance from "production" even in their first version. Then they asked me some vague and hard-to-decipher-over-a-conference-call questions about how to measure the accuracy for the random forest model I had fitted, and about my conclusions regarding which predictors were most efficacious. Despite having used random forests for published research results and even having taught part of a course on random forests when I was a grad student, I could not make heads nor tails of what exact accuracy metric the interviewer was fishing for. I am sure that if such a question was asked in-person, with some pen and paper, or something where it's not just the vague descriptions of some stranger I've never met before, then we could have sorted it out. But the communication was just too poor over the conference call for this kind of question. Given that I had to solve the problem in my spare time while conducting all the other stuff in my life, I felt it was quite reasonable for me to take a very conservative approach: fit a very simple model, don't waste time chasing down parameter optimizations (like depth of the trees or some huge set of features). Just walk through some very straightforward analysis, use some straightforward features, produce some common diagnostic curves (which showed a huge bias in the data labels -- almost all example users were not "engaged" ), and draw only conservative conclusions (e.g. with such severe bias in the toy data set, it's pretty hard to conclusively say anything, so don't pretend you can say more than you really can, don't recommend strong conclusions, and leave it at that). The point is to show basically a mock-up of the end-to-end workflow, from querying the data, to constructing the features, to fitting the model, to assessing the fit. The point is clearly not to do that in some kind of insanely accurate way when you're talking about a busy adult doing it in their spare time over a few days. The two interviewers had some gripes with the specifics of my accuracy measures (which seemed really misplaced, again for an analysis done in a busy adult's spare time using erroneous toy data).

by p4wnc6 11y ago

pklausler 11y ago

I use problems with multiple acceptable solutions and easy tests of correctness -- i.e., it's obvious whether or not the approach is sound and the code works. For example, write a function that determines whether two rectangles on a display overlap, given the coordinates of their corners. Solving it requires the ability to analyze the problem and describe the conditions, the ability to code a simple Boolean predicate, and some aptitude for checking boundary cases and writing unit tests. I have others; what they have in common is that they're really easy problems for great programmers, and well within the limits of a 45-minute technical interview for competent coders, and good at exposing candidates who lack some basic skills.

jacklionheart 11y ago

Hi there! That sounds bad. Could you let me know what's giving you the impression that that's Asana's goal? We really try to explicitly avoid that kind of interview evaluation, and put strong emphasis on Communication, User empathy, and Learning, per the guide. But definitely "did they say the right answer?" is an easy trap to fall into so its entirely possible that some interviewers are doing it -- would love any concrete feedback you could give to help us avoid it in the future.

p4wnc6 11y ago
The more concrete feedback is in the previous post (you commented on that one too). But I will elaborate here: On 8/4/15, I received a message from an Asana representative via Stack Overflow Careers. After a few back-and-forths discussing specifics about whether I was a good fit, I then agree to do the take-home test. I received the take home test on 8/7/15. I emailed back on 8/8/15 with a verifiable error in a timestamp column of an AWS-hosted Postgres database that was configured with some test data (data resembling some certain kinds of user data for modeling what an "engaged" user is, and what factors predict when a user will be "engaged"). I included code which could be run to demonstrate the erroneous data. On 8/9/15 I wrote in again stating that I had created a workaround for the erroneous timestamp data, since the only way I could complete the exam was to do it on those particular days (I couldn't wait around for the exam data to be fixed). I submitted my solution on 8/10/15. I got an email on 8/11/15 with a note from someone in engineering apologizing for the timestamp data error, but also saying it shouldn't affect my solution. The recruiter who sent that email also said that my submission was currently being evaluated. Then I waited a week before hearing anything. On 8/18/15 I got a response apologizing for the delay and saying that I was invited for the next round interview. The email said I would be speaking with a person named Jack (maybe you?) a few days later (8/21/15 at 4:00 pm EST). But that call actually included two data scientists over one conference call line. Both interviewers effectively made no comment on my software choices. In fact, when I spoke about how I cared about software design and best practices, even when working on a rapid prototype, it actually sounded as if the interviewers thought this was a bad thing and there was some awkward silence. I brought up examples of how I had worked in a very fast paced environment before (quant finance) and my team had learned some hard lessons that the naive approach of letting data scientists just write scripts for ad hoc modeling, assuming that engineering will "productionize" it later, tends to lead to some bad failure cases, and that it's far more efficient over time to simply require data scientists to construct rapid prototypes, even from the first line of exploratory, ad-hoc code, that adhere to many best practices and are only a short distance from "production" even in their first version. Then they asked me some vague and hard-to-decipher-over-a-conference-call questions about how to measure the accuracy for the random forest model I had fitted, and about my conclusions regarding which predictors were most efficacious. Despite having used random forests for published research results and even having taught part of a course on random forests when I was a grad student, I could not make heads nor tails of what exact accuracy metric the interviewer was fishing for. I am sure that if such a question was asked in-person, with some pen and paper, or something where it's not just the vague descriptions of some stranger I've never met before, then we could have sorted it out. But the communication was just too poor over the conference call for this kind of question. Given that I had to solve the problem in my spare time while conducting all the other stuff in my life, I felt it was quite reasonable for me to take a very conservative approach: fit a very simple model, don't waste time chasing down parameter optimizations (like depth of the trees or some huge set of features). Just walk through some very straightforward analysis, use some straightforward features, produce some common diagnostic curves (which showed a huge bias in the data labels -- almost all example users were not "engaged" ), and draw only conservative conclusions (e.g. with such severe bias in the toy data set, it's pretty hard to conclusively say anything, so don't pretend you can say more than you really can, don't recommend strong conclusions, and leave it at that). The point is to show basically a mock-up of the end-to-end workflow, from querying the data, to constructing the features, to fitting the model, to assessing the fit. The point is clearly not to do that in some kind of insanely accurate way when you're talking about a busy adult doing it in their spare time over a few days. The two interviewers had some gripes with the specifics of my accuracy measures (which seemed really misplaced, again for an analysis done in a busy adult's spare time using erroneous toy data).