6 ms·

We've been using a quiz as part of our hiring process (at Triplebyte) for the last two years. However, a lot of people do our quiz for fun. So we decided to cre

• chatmasta 9y ago
  If you're going to pick one language for the quiz (python) it seems a little unfair to non-python developers to focus the questions on idiosyncrasies of python. Whereas if all questions were of the more general algorithm variety (like you see toward the end), non-python developers could infer the answer without any esoteric python knowledge.
  • ammon 9y ago
    We talked a bunch about this. Python and JS are the two most common languages we see on our site, and Python has slightly more compact syntax, which works well in the questions. The idea, however, was not to require any python-specific knowledge. Which question do you think is looking at syntax?
    • bussierem 9y ago
      Hey, not the GP, but just glancing through the quiz - Question 3 focuses on the fact that Python passes things by reference to functions. This is definitely specific to the two languages you guys use (as is your prerogative), but if you're expanding this for others, you should definitely explain that these focus on the JS and Python languages.
      • ammon 9y ago
        Good point. I think that objects are passed by reference in a high enough % of language in common use (PHP and C++ are the two exceptions I can think of) that this is a fair question. But if enough people think it's bad, we'll remove it.
        • macintux 9y ago
          Every functional programmer in the world is looking at you askance.
        • marcosdumay 9y ago
          You got me on that too. A high enough % of the languages pass stuff by value (C/C++), copy on assignment (C++, D, I'm sure there are more), or do not have the concept of references or changes to start with (Prolog, Haskell). Anyway, if you want to bias it into Python, go for it. It's a fun quiz.
          • Bartweiss 9y ago
            Copy on assignment was what got me, and I didn't feel terribly bad for missing it. No harm done though, seeing as nothing was riding on the quiz.
      • rodovich 9y ago
        Additionally, question 2 amounts to "what is the usage of python's queue class", question 4 requires knowing whether array indexes start at 0 or 1 in python, and question 8 amounts to "what is the usage of python's bytearray class".
        • philh 9y ago
          I'm not sure that's fair about question 2. It's hard to say because I know python (but not the deque class), but none of the incorrect answers seem like they'd be correct in any sensible language. Like, whatever language you're using, you need to push the children of the current node on to the queue. Only one answer looks like it could reasonably do that.
          • Bartweiss 9y ago
            Yeah, agreed. Question 2 as asked demanded Python knowledge, but once you checked against the answers provided it seemed much clearer that 3 options were invalid.
      • freyr 9y ago
        Also, strictly speaking, Python is not "pass by reference" as the term is used for other languages. It's more accurately "pass by object" or "pass by object reference." Somewhat of a pedantic distinction, and the quiz question is clear regardless. [1] http://robertheaton.com/2014/02/09/pythons-pass-by-object-reference-as-explained-by-philip-k-dick/ http://robertheaton.com/2014/02/09/pythons-pass-by-object-re... [2] http://foobarnbaz.com/2012/07/08/understanding-python-variables/ http://foobarnbaz.com/2012/07/08/understanding-python-variab... [3] https://jeffknupp.com/blog/2012/11/13/is-python-callbyvalue-or-callbyreference-neither/ https://jeffknupp.com/blog/2012/11/13/is-python-callbyvalue-...
    • vitno 9y ago
      they didn't say syntax, they said Python idiosyncrasies. Knowing move/copy rules in python counts as that. It's something that isn't intuitive unless you know python, ex: the matrix question. An experienced dev who isn't a python person would pick the intuitive choice, see the matrix is malformed, then Google how to clone objects in python.
      • safek 9y ago
        On the other hand, how many languages do you know where arrays are passed by value?
        • macintux 9y ago
          My primary language (Erlang) does. I assume all functional programming languages do.
          • safek 9y ago
            That's true. I suppose my next argument is that seeing matrix.append(row) on its own line signals strongly that this is not a functional language.
          • thedufer 9y ago
            Is that true? I thought most functional programming languages did call-by-reference. I mean, if you have immutable data you can't tell the difference (outside of performance).
            • macintux 9y ago
              Under the hood it's probably by reference, but the semantics are by value. By reference indicates that the original value can be changed in the called function. I think. Not a language lawyer.
      • echevil 9y ago
        lol, the same mistake can be made in a lot more languages~
    • acconrad 9y ago
      > The idea, however, was not to require any python-specific knowledge But the quiz specifically mentions on multiple answers that "In Python, unlike language X, Y, Z" OR "In Python, like [similar language] A, B, C"...I mean this says nothing about my programming ability, this says whether I know the quirks of the language in the same way that I can tell you all the quirks of JavaScript but not in Python. This to me is not a programming quiz, but a Python quiz designed to determine how much someone knows about what can go wrong in Python.
      • ammon 9y ago
        Well, the questions get into other things too.(Character encoding, timing attacks, graph algorithms.) Writing a good single-paragraph description of iterative deepening depth-first search was a challenge!
        • acconrad 9y ago
          Yeah except that character encoding question assumes I know how Python handles bytearrays. That trick answer of "it will break" threw me for a loop because I needed language-specific knowledge about the types that Python could handle for that input. And then with timing attacks - I mean SQL Injection is a fair web security question to ask someone, but the timing is like, very fringe knowledge. Most devs let their framework handle all of those kinds of things. Not sure why I would need to know this unless I was specifically in the sec world. The last 3 questions were actually general CS because they went through search trees and shortest paths algos, which are language-agnostic.
    • deleted 9y ago
      [deleted]
    • bdcravens 9y ago
      popleft is shift in many other languages. Not all languages manipulate arrays in the same way re ref/value. Not all languages have the concept of range() - you could have used a simple array to be more language agnostic. (I'm sure I could have found more, but I'm Python isn't a language I spend much time in)
      • echevil 9y ago
        Just to be fair, i think the comments in the source code has explained it well
        • bdcravens 9y ago
          Yes, but finding answers required to understand what was set by ref, and what was set by value, which wasn't in any comments.
    • d0100 9y ago
      I only did the first 3 , but both 2 and 3 focus on python-specific names (extend(), queue) and pass-by(reference/value).
    • tobyhinloopen 9y ago
      I used many programming languages - C, C++, Java, Ruby, Elixir, C#, Swift, Objective-C & Javascript. I've never used Python.
    • deleted 9y ago
      [deleted]
    • GrinningFool 9y ago
      Q2: THis isn't a syntax thing, but it does require knowledge of python queue objects to get right Q3: This requires knowledge around the semantics of argument passing, which vary from language to language. Q4: this again requires some understanding of how values are stored. Some languages will copy, while others will use a reference to the originally inserted value. Q5: python closure syntax. python scoping rules. Q8: unless you know what "bytearray.decode" does, the best you can do is make an educated guess from the context.
  • deleted 9y ago
    [deleted]
• ibdknox 9y ago
  Question 9 appears to have two answers that are correct, but only one is accepted.
  • StavrosK 9y ago
    Yep, I just came here to post that: In [2]: re.search("b[l].e", "babel") In [3]: re.search("b[l].e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030> It looks like they meant "match" as in "re.match", i.e. "^b[l].*e$", which is not the usual definition of the word.
    • ibdknox 9y ago
      That would also make this question specific to python, which wasn't at all what I was thinking about. It's a question about regular expressions, it should be as agnostic as possible to implementation.
      • StavrosK 9y ago
        Yeah, agreed, although there are different RE flavors, so you can't get too agnostic.
  • ammon 9y ago
    Hmm. Just checked again. Two of the expressions match neither word, one matches both, and one matches only one. EDIT: I see. The confusion is over partial match vs whole match. I thought whole matches were standard when talking about regular expression in the abstract, but looks like I was wrong. Just changed the problem to include the anchors. Thanks for pointing this out.
    • ibdknox 9y ago
      That's not the standard meaning in my experience. Regex's are often used for extracting information from larger strings, forcing a full match by default would be pretty counter to their purpose.
• Herald_MJ 9y ago
  It feels like a lot of this quiz is just trying to "catch me out" on subtle behaviours of Python I didn't know or might have forgot about. I guess this is fine if you're making a little trivia toy to distribute to Python developers on the internet, but I don't really think it achieves much as a hiring tool. Getting questions on this test wrong would probably demoralise a lot of great candidates.
  • StavrosK 9y ago
    I got them all right, and I agree that there were many gotchas. For example, the ones about the matrix, the array modification, etc. The only thing worse would have been something like: https://www.pastery.net/mbeghh/ https://www.pastery.net/mbeghh/ (it'll print 3)
    • ammon 9y ago
      Hmm. We included those issues, because we see a lot of programmers get stuck on exactly those two bugs! (In all sorts of languages, not just Python). I guess maybe that's the definition of a gotcha -- a thing that a lot of programmers get stuck on? In any case, I don't think it's an arcane gotcha! Being clear about references and shallow / deep copies is important.
      • thedufer 9y ago
        Being clear on reference vs value is certainly important, but you lose the ability to honestly call it a programming quiz. Those questions make it a Python quiz - which is fine! But not what you've sold it as.
      • StavrosK 9y ago
        In that case, by far the biggest Python gotcha is: def do_stuff(mylist=[]): mylist.append("thing") Which will lead to the list containing N items after N invocations (i.e. the second function call won't start with an empty list).
        • 9y ago
• Silhouette 9y ago
  Fun exercise. Thanks for sharing. I'd suggest fine-tuning the question about hashes. My first impression from reading the code, specifically the name user_submitted_hash, was that we were looking at some sort of client-server system where the hash was being calculated remotely and then submitted instead of the original credentials, and that this was a question about not trusting the client. Even without that, in a client-server system the attack you give as the mostly likely wouldn't make much sense given randomness in network latency, so the misunderstanding about the context still breaks the question. Maybe a little more context about how this hypothetical function would be used would help here? In terms of the presentation here, if you want to show the answers immediately for each question, you might also like to provide some sort of confirmation button. It's frustratingly easy to hit the answer next to the one you intended, particularly on mobile devices. If you're also keeping statistics based on earlier attempts by other people, this is probably distorting your numbers as well.
  • ammon 9y ago
    Timing attacks like this still work over a network, you just have to make an LOT of duplicate requests, and average the times. The lucky thirteen attack (https://en.wikipedia.org/wiki/Lucky_Thirteen_attack https://en.wikipedia.org/wiki/Lucky_Thirteen_attack) is a real example. Good point about the accidental answers. I'll look at that.
    • Silhouette 9y ago
      "A lot" is going to be a mighty big number in this case, though. In some networks it will surely be prohibitively large. But even if the latency is predictable enough for that not to be the case, what real world system is going to let someone make that many failed authentication requests without doing something about it? You might be better just trying to brute force it. The trouble with hypothetical security questions is that it's so difficult to conceive a scenario where you have a deliberate vulnerability that you want someone to see, yet no other implausible aspects that will throw off anyone capable of seeing it. I am reminded of a murder mystery party I once went to with a group of mathematicians, programmers, and other logical people. When we got to the end of the evening and the true murderer was revealed, I think the majority of the participants had long ago ruled out that suspect on the basis of one of several subtle deductions from the clues provided, none of which had apparently been intended or considered by the authors of the scenario. Instead we were supposed to have ignored all the minor inconsistencies in clues and ambiguities in phrasing, and just gone for the person with the flashing neon sign over their head in the first place...
• mikekchar 9y ago
  To be honest, this quiz makes me feel that I'm learning more about you than you are about me. I especially find the reactions to the complaints interesting. This response could be read in a negative way, which is not as I intend it. Hopefully it can provide a window to view things from a different perspective. It is not possible to really determine the ability of a programmer in an interview -- the scope is too large. It takes time and effort to understand how well a person will perform. In an interview we try to find indicators of how well a person will perform. So, a quiz that tries to unearth those indicators is interesting. Your quiz makes me feel that technical details are foremost in your mind when you asses performance. Will the candidate make technical mistakes that you have seen many times before? Similarly, there appears to be considerable bias evident in your selection of questions. All of these questions asses whether or not a candidate deeply understands the functioning of the code snippet, or whether they understand it shallowly. For the candidate who admits a shallow understanding, their only recourse is to guess (and potentially fail). From that I infer that your cultural makeup is one where you have a bar that you expect all candidates to surpass. This bar is single dimensional, though, and especially your responses to the complaints imply a lack of realisation that you are narrowly optimising for a single useful ability. Finally, the quiz is set up in a "run the guantlet" fashion. Can you avoid the pitfalls that others have fallen into? This tells me that you fear candidates who do not statically measure up to your bar more than you are excited to find out what a candidate brings to the table. While you clearly will have other interview techniques for other aspects that you value, this particular quiz would worry me as a candidate. It's essentially the same question 15 times in a different context. When people complain that the quiz is too Python specific, the answer is essentially "But it covers concepts that are similar for many languages and you should be able to figure it out". It's a kind of "Well, I'm sure I could do it pretty easily, so you should be able to too". It makes me worried that there is a lack of empathy on team and that there will be problems if I don't think exactly like the leaders on the team. On the other hand, this may be fine if that's what you want in your interview process. Personally, I probably would not apply to your company having seen this quiz. This may also be a good thing from your perspective :-). However, if you ever get to a point where, despite having excellent people on staff, you can't seem to get to the next level, I would concentrate on examining potential biases of what is "good" on your team. Good luck!