5 ms·

For what it's worth, we run our interviews in the language of the engineer's choosing (including clojure), and get people jobs at Clojure and Haskel and C++ com

by ammon 9y ago

For what it's worth, we run our interviews in the language of the engineer's choosing (including clojure), and get people jobs at Clojure and Haskel and C++ companies. We went with Python for this (toy) quiz because it's the most common language we see among applicants. I tried to add comments to explain anything python specific, and make the question about the underlying logic. But it seems it's still easier for Python folks.

kefka 9y ago
Sorry, not buying it. I caught 3 "Whoops sorry you missed a weird bug in a specific programming language that you weren't explicitly being tested on". Sure, if I was doing a Python3 quiz, I'd expect those. But this wasn't what the quiz claimed to be. Turns out, the "fill in the blank" code was easy, got all of them. Along with all the 1st year theory questions.
- horsawlarway 9y ago
  I agree. Coming in with a LOT of experience in JS and a good damn chunk in c++, but none in python... this is really bad test. You ask dev about aspects of the language that are essentially memorization: Does matrix.append(x) in python append by reference or by value? Both are equally likely without knowing anything about the language. the question is also pretty easy to answer correct IF I know that, but I have no reason to know that. You're not testing skill in any sense of the word. You're purely testing familiarity with this specific language (and worse, this language's standard library) That's bad.
  - kbp 9y ago
    > Does matrix.append(x) in python append by reference or by value? Both are equally likely without knowing anything about the language. I don't think they're equally likely. The question would have the same answer in Javascript, Ruby, Lisp, Perl 6, Java, and many other languages. You can see from the code samples that Python is a high level, dynamic, imperative language, so I think it's fair to assume you would expect that behaviour from it. There's a bit of guessing, but you can make an educated guess.
    - horsawlarway 9y ago
      Way late, but your response sort of proves the point. You're having to argue about what I can infer about the language. What JS/Ruby/Lisp/Perl/Java do is irrelevant. Hey, C/Go/D/C++ would all be more likely to append by value, but neither of us care a whit because it's asking about Python, which I know nothing about. So (and this is really hard) try to forget that you've got experience with the language, and approach it as a random pseudo language: this question has no clear answer, because it depends on the memorized implementation details of the standard library of a language... That's bad. --- Finally: JS doesn't have matrix as a base type (boolean/number/string/null/undefined/symbol is it, baby), so for this same question, I'd strongly argue that you're simply wrong about bucketing JS with those other languages. Again, for JS the question would revolve around the implementation details of the matrix object and its prototype. Trust me, I can write a useful matrix implementation that appends by value in JS.
      - kbp 9y ago
        > What JS/Ruby/Lisp/Perl/Java do is irrelevant. Hey, C/Go/D/C++ would all be more likely to append by value, but neither of us care a whit because it's asking about Python, which I know nothing about. I guess my point was that you should be able to infer that Python would be more likely to behave like JS/Ruby/Lisp/Perl/Java than C/Go/D/C++. If you know or can tell by looking at it that it's much more in the first camp then the second, then you should be able to guess. You're probably right that I overstated how easy that is to tell just from the syntax if you don't know anything at all about Python, but it's also pretty short of memorizing implementation details. > Finally: JS doesn't have matrix as a base type (boolean/number/string/null/undefined/symbol is it, baby), so for this same question, I'd strongly argue that you're simply wrong about bucketing JS with those other languages. Again, for JS the question would revolve around the implementation details of the matrix object and its prototype. The 'matrix' in the question wasn't a special matrix type, it was just a list of lists. Word-for-word translated into Javascript it would look like: let matrix = [] let row = [0, 0] for (let i = 0; i < 2; ++i) matrix.push(row) matrix[1][1] = 1
        kefka 9y ago
- eropple 9y ago
  Agreed - this is a really Python-heavy thing that shouldn't be. I got 13/15 largely through guessing the syntax and semantic weirdness of a language I only rarely touch and never for long (like, the matrix question, by-value or by-reference? I had a 50/50 chance to get it right). Not a big fan. Feels really weird.
cperciva 9y ago
Those comments definitely helped. But there were a couple questions which came down to "does Python copy this data structure or simply pass a reference", and all I had to go on there was "if I was designing the language, how would I do it?" In one case, I apparently think that Python got it wrong. :-)
VLM 9y ago
Yes I didn't see those comments. Possibly, the "Is Python call by reference or call by value" question could be rewritten in a language independent form by providing correct syntax Python, the correct python output, and asking verbally "Given either this evidence OR your Python coding experience, is Python call by reference or call by value?" I've never used Python. I breezed thru the theory and regex questions, presumably I could be a halfway decent Python programmer in a week or two, it looks reasonably non-tricky. Thats another issue, some jobs merely require syntactically correct code, others require good algos, and its a lot easier on the job to learn syntax than to learn algos.
- ammon 9y ago
  I like that idea for the question!
fenomas 9y ago
If you wanted to make a generalist quiz, why not ask the questions in psuedocode? Asking the user to choose between 'queue.extend(node)' and 'queue.extend([node])' is purely a Python syntax question; it has nothing to do with programming per se.
- Yen 9y ago
  Neither of those are the correct answer, though. The correct answer is node.children, which isn't syntax-specific, it's about the pseudocode of whether you want to visit the node, or the node's children.
- ammon 9y ago
  Using pseudocode is a great idea. We've talked about that. We've not done it because we still need to decide what style of syntax to use (and that will still preference certain programmers). But maybe we should think about it again!
  - estomagordo 9y ago
    Maybe start off with a The Elder Scrolls style background quiz, where people are asked things such as "Ever since childhood you've been having terrible nightmares about (a) curly braces (b) dynamic typing (c) memory management. And then, after 20 background questions, you could know with high certainty what style of pseudo code (or even language!) a person prefers. Yeah, I think this is a great idea.
Bartweiss 9y ago
There's certainly nothing objectionable about choosing a common language for a toy quiz like this. It did feel jarring to have the early questions be language-specific, and the UTF question was sort of unknowable, but I got it by guessing what the most likely meaning of the syntax was. I was frustrated that language features felt trickier than actual coding knowledge in the early questions, but given that it's nothing with actual stakes there's obviously no harm in it.
speby 9y ago
Although it would be a fairly decent extra amount of work ... make the quiz available in the top 20 languages and have the person choose which language to view the quiz in.

by ammon 9y ago

kefka 9y ago

Sorry, not buying it. I caught 3 "Whoops sorry you missed a weird bug in a specific programming language that you weren't explicitly being tested on". Sure, if I was doing a Python3 quiz, I'd expect those. But this wasn't what the quiz claimed to be. Turns out, the "fill in the blank" code was easy, got all of them. Along with all the 1st year theory questions.

horsawlarway 9y ago
I agree. Coming in with a LOT of experience in JS and a good damn chunk in c++, but none in python... this is really bad test. You ask dev about aspects of the language that are essentially memorization: Does matrix.append(x) in python append by reference or by value? Both are equally likely without knowing anything about the language. the question is also pretty easy to answer correct IF I know that, but I have no reason to know that. You're not testing skill in any sense of the word. You're purely testing familiarity with this specific language (and worse, this language's standard library) That's bad.
- kbp 9y ago
  > Does matrix.append(x) in python append by reference or by value? Both are equally likely without knowing anything about the language. I don't think they're equally likely. The question would have the same answer in Javascript, Ruby, Lisp, Perl 6, Java, and many other languages. You can see from the code samples that Python is a high level, dynamic, imperative language, so I think it's fair to assume you would expect that behaviour from it. There's a bit of guessing, but you can make an educated guess.
  - horsawlarway 9y ago
    Way late, but your response sort of proves the point. You're having to argue about what I can infer about the language. What JS/Ruby/Lisp/Perl/Java do is irrelevant. Hey, C/Go/D/C++ would all be more likely to append by value, but neither of us care a whit because it's asking about Python, which I know nothing about. So (and this is really hard) try to forget that you've got experience with the language, and approach it as a random pseudo language: this question has no clear answer, because it depends on the memorized implementation details of the standard library of a language... That's bad. --- Finally: JS doesn't have matrix as a base type (boolean/number/string/null/undefined/symbol is it, baby), so for this same question, I'd strongly argue that you're simply wrong about bucketing JS with those other languages. Again, for JS the question would revolve around the implementation details of the matrix object and its prototype. Trust me, I can write a useful matrix implementation that appends by value in JS.
    - kbp 9y ago
      > What JS/Ruby/Lisp/Perl/Java do is irrelevant. Hey, C/Go/D/C++ would all be more likely to append by value, but neither of us care a whit because it's asking about Python, which I know nothing about. I guess my point was that you should be able to infer that Python would be more likely to behave like JS/Ruby/Lisp/Perl/Java than C/Go/D/C++. If you know or can tell by looking at it that it's much more in the first camp then the second, then you should be able to guess. You're probably right that I overstated how easy that is to tell just from the syntax if you don't know anything at all about Python, but it's also pretty short of memorizing implementation details. > Finally: JS doesn't have matrix as a base type (boolean/number/string/null/undefined/symbol is it, baby), so for this same question, I'd strongly argue that you're simply wrong about bucketing JS with those other languages. Again, for JS the question would revolve around the implementation details of the matrix object and its prototype. The 'matrix' in the question wasn't a special matrix type, it was just a list of lists. Word-for-word translated into Javascript it would look like: let matrix = [] let row = [0, 0] for (let i = 0; i < 2; ++i) matrix.push(row) matrix[1][1] = 1
      - kefka 9y ago
eropple 9y ago
Agreed - this is a really Python-heavy thing that shouldn't be. I got 13/15 largely through guessing the syntax and semantic weirdness of a language I only rarely touch and never for long (like, the matrix question, by-value or by-reference? I had a 50/50 chance to get it right). Not a big fan. Feels really weird.

cperciva 9y ago

Those comments definitely helped. But there were a couple questions which came down to "does Python copy this data structure or simply pass a reference", and all I had to go on there was "if I was designing the language, how would I do it?" In one case, I apparently think that Python got it wrong. :-)

VLM 9y ago

Yes I didn't see those comments. Possibly, the "Is Python call by reference or call by value" question could be rewritten in a language independent form by providing correct syntax Python, the correct python output, and asking verbally "Given either this evidence OR your Python coding experience, is Python call by reference or call by value?" I've never used Python. I breezed thru the theory and regex questions, presumably I could be a halfway decent Python programmer in a week or two, it looks reasonably non-tricky. Thats another issue, some jobs merely require syntactically correct code, others require good algos, and its a lot easier on the job to learn syntax than to learn algos.

ammon 9y ago
I like that idea for the question!

fenomas 9y ago

If you wanted to make a generalist quiz, why not ask the questions in psuedocode? Asking the user to choose between 'queue.extend(node)' and 'queue.extend([node])' is purely a Python syntax question; it has nothing to do with programming per se.

Yen 9y ago
Neither of those are the correct answer, though. The correct answer is node.children, which isn't syntax-specific, it's about the pseudocode of whether you want to visit the node, or the node's children.
ammon 9y ago
Using pseudocode is a great idea. We've talked about that. We've not done it because we still need to decide what style of syntax to use (and that will still preference certain programmers). But maybe we should think about it again!
- estomagordo 9y ago
  Maybe start off with a The Elder Scrolls style background quiz, where people are asked things such as "Ever since childhood you've been having terrible nightmares about (a) curly braces (b) dynamic typing (c) memory management. And then, after 20 background questions, you could know with high certainty what style of pseudo code (or even language!) a person prefers. Yeah, I think this is a great idea.

Bartweiss 9y ago

There's certainly nothing objectionable about choosing a common language for a toy quiz like this. It did feel jarring to have the early questions be language-specific, and the UTF question was sort of unknowable, but I got it by guessing what the most likely meaning of the syntax was. I was frustrated that language features felt trickier than actual coding knowledge in the early questions, but given that it's nothing with actual stakes there's obviously no harm in it.

speby 9y ago

Although it would be a fairly decent extra amount of work ... make the quiz available in the top 20 languages and have the person choose which language to view the quiz in.