17 ms·

Show HN: 15-question programming quiz with answers

227 points by ammon 9y ago

ammon 9y ago
We've been using a quiz as part of our hiring process (at Triplebyte) for the last two years. However, a lot of people do our quiz for fun. So we decided to create a version optimized for this (and hopefully a little education). The problems come from the most common mistakes / misunderstandings we see when giving interviews, and ramp up in difficulty. We put a lot of effort into writing interesting answers. Let me know what you think!
- chatmasta 9y ago
  If you're going to pick one language for the quiz (python) it seems a little unfair to non-python developers to focus the questions on idiosyncrasies of python. Whereas if all questions were of the more general algorithm variety (like you see toward the end), non-python developers could infer the answer without any esoteric python knowledge.
  - ammon 9y ago
    We talked a bunch about this. Python and JS are the two most common languages we see on our site, and Python has slightly more compact syntax, which works well in the questions. The idea, however, was not to require any python-specific knowledge. Which question do you think is looking at syntax?
    - bussierem 9y ago
      Hey, not the GP, but just glancing through the quiz - Question 3 focuses on the fact that Python passes things by reference to functions. This is definitely specific to the two languages you guys use (as is your prerogative), but if you're expanding this for others, you should definitely explain that these focus on the JS and Python languages.
      - ammon 9y ago
        Good point. I think that objects are passed by reference in a high enough % of language in common use (PHP and C++ are the two exceptions I can think of) that this is a fair question. But if enough people think it's bad, we'll remove it.
        macintux 9y ago
        Every functional programmer in the world is looking at you askance.
        marcosdumay 9y ago
        You got me on that too. A high enough % of the languages pass stuff by value (C/C++), copy on assignment (C++, D, I'm sure there are more), or do not have the concept of references or changes to start with (Prolog, Haskell). Anyway, if you want to bias it into Python, go for it. It's a fun quiz.
        Bartweiss 9y ago
        Copy on assignment was what got me, and I didn't feel terribly bad for missing it. No harm done though, seeing as nothing was riding on the quiz.
      - rodovich 9y ago
        Additionally, question 2 amounts to "what is the usage of python's queue class", question 4 requires knowing whether array indexes start at 0 or 1 in python, and question 8 amounts to "what is the usage of python's bytearray class".
        philh 9y ago
        I'm not sure that's fair about question 2. It's hard to say because I know python (but not the deque class), but none of the incorrect answers seem like they'd be correct in any sensible language. Like, whatever language you're using, you need to push the children of the current node on to the queue. Only one answer looks like it could reasonably do that.
    - vitno 9y ago
      they didn't say syntax, they said Python idiosyncrasies. Knowing move/copy rules in python counts as that. It's something that isn't intuitive unless you know python, ex: the matrix question. An experienced dev who isn't a python person would pick the intuitive choice, see the matrix is malformed, then Google how to clone objects in python.
      - safek 9y ago
        On the other hand, how many languages do you know where arrays are passed by value?
        macintux 9y ago
        My primary language (Erlang) does. I assume all functional programming languages do.
        safek 9y ago
        That's true. I suppose my next argument is that seeing matrix.append(row) on its own line signals strongly that this is not a functional language.
        thedufer 9y ago
        Is that true? I thought most functional programming languages did call-by-reference. I mean, if you have immutable data you can't tell the difference (outside of performance).
        macintux 9y ago
        Under the hood it's probably by reference, but the semantics are by value. By reference indicates that the original value can be changed in the called function. I think. Not a language lawyer.
      - echevil 9y ago
        lol, the same mistake can be made in a lot more languages~
    - acconrad 9y ago
      > The idea, however, was not to require any python-specific knowledge But the quiz specifically mentions on multiple answers that "In Python, unlike language X, Y, Z" OR "In Python, like [similar language] A, B, C"...I mean this says nothing about my programming ability, this says whether I know the quirks of the language in the same way that I can tell you all the quirks of JavaScript but not in Python. This to me is not a programming quiz, but a Python quiz designed to determine how much someone knows about what can go wrong in Python.
      - ammon 9y ago
        Well, the questions get into other things too.(Character encoding, timing attacks, graph algorithms.) Writing a good single-paragraph description of iterative deepening depth-first search was a challenge!
        acconrad 9y ago
        Yeah except that character encoding question assumes I know how Python handles bytearrays. That trick answer of "it will break" threw me for a loop because I needed language-specific knowledge about the types that Python could handle for that input. And then with timing attacks - I mean SQL Injection is a fair web security question to ask someone, but the timing is like, very fringe knowledge. Most devs let their framework handle all of those kinds of things. Not sure why I would need to know this unless I was specifically in the sec world. The last 3 questions were actually general CS because they went through search trees and shortest paths algos, which are language-agnostic.
    - deleted 9y ago
      [deleted]
    - bdcravens 9y ago
      popleft is shift in many other languages. Not all languages manipulate arrays in the same way re ref/value. Not all languages have the concept of range() - you could have used a simple array to be more language agnostic. (I'm sure I could have found more, but I'm Python isn't a language I spend much time in)
      - echevil 9y ago
        Just to be fair, i think the comments in the source code has explained it well
        bdcravens 9y ago
        Yes, but finding answers required to understand what was set by ref, and what was set by value, which wasn't in any comments.
    - d0100 9y ago
      I only did the first 3 , but both 2 and 3 focus on python-specific names (extend(), queue) and pass-by(reference/value).
    - tobyhinloopen 9y ago
      I used many programming languages - C, C++, Java, Ruby, Elixir, C#, Swift, Objective-C & Javascript. I've never used Python.
    - deleted 9y ago
      [deleted]
    - GrinningFool 9y ago
      Q2: THis isn't a syntax thing, but it does require knowledge of python queue objects to get right Q3: This requires knowledge around the semantics of argument passing, which vary from language to language. Q4: this again requires some understanding of how values are stored. Some languages will copy, while others will use a reference to the originally inserted value. Q5: python closure syntax. python scoping rules. Q8: unless you know what "bytearray.decode" does, the best you can do is make an educated guess from the context.
  - deleted 9y ago
    [deleted]
- ibdknox 9y ago
  Question 9 appears to have two answers that are correct, but only one is accepted.
  - StavrosK 9y ago
    Yep, I just came here to post that: In [2]: re.search("b[l].e", "babel") In [3]: re.search("b[l].e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030> It looks like they meant "match" as in "re.match", i.e. "^b[l].*e$", which is not the usual definition of the word.
    - ibdknox 9y ago
      That would also make this question specific to python, which wasn't at all what I was thinking about. It's a question about regular expressions, it should be as agnostic as possible to implementation.
      - StavrosK 9y ago
        Yeah, agreed, although there are different RE flavors, so you can't get too agnostic.
  - ammon 9y ago
    Hmm. Just checked again. Two of the expressions match neither word, one matches both, and one matches only one. EDIT: I see. The confusion is over partial match vs whole match. I thought whole matches were standard when talking about regular expression in the abstract, but looks like I was wrong. Just changed the problem to include the anchors. Thanks for pointing this out.
    - ibdknox 9y ago
      That's not the standard meaning in my experience. Regex's are often used for extracting information from larger strings, forcing a full match by default would be pretty counter to their purpose.
- Herald_MJ 9y ago
  It feels like a lot of this quiz is just trying to "catch me out" on subtle behaviours of Python I didn't know or might have forgot about. I guess this is fine if you're making a little trivia toy to distribute to Python developers on the internet, but I don't really think it achieves much as a hiring tool. Getting questions on this test wrong would probably demoralise a lot of great candidates.
  - StavrosK 9y ago
    I got them all right, and I agree that there were many gotchas. For example, the ones about the matrix, the array modification, etc. The only thing worse would have been something like: https://www.pastery.net/mbeghh/ https://www.pastery.net/mbeghh/ (it'll print 3)
    - ammon 9y ago
      Hmm. We included those issues, because we see a lot of programmers get stuck on exactly those two bugs! (In all sorts of languages, not just Python). I guess maybe that's the definition of a gotcha -- a thing that a lot of programmers get stuck on? In any case, I don't think it's an arcane gotcha! Being clear about references and shallow / deep copies is important.
      - thedufer 9y ago
        Being clear on reference vs value is certainly important, but you lose the ability to honestly call it a programming quiz. Those questions make it a Python quiz - which is fine! But not what you've sold it as.
      - StavrosK 9y ago
        In that case, by far the biggest Python gotcha is: def do_stuff(mylist=[]): mylist.append("thing") Which will lead to the list containing N items after N invocations (i.e. the second function call won't start with an empty list).
        9y ago
- Silhouette 9y ago
  Fun exercise. Thanks for sharing. I'd suggest fine-tuning the question about hashes. My first impression from reading the code, specifically the name user_submitted_hash, was that we were looking at some sort of client-server system where the hash was being calculated remotely and then submitted instead of the original credentials, and that this was a question about not trusting the client. Even without that, in a client-server system the attack you give as the mostly likely wouldn't make much sense given randomness in network latency, so the misunderstanding about the context still breaks the question. Maybe a little more context about how this hypothetical function would be used would help here? In terms of the presentation here, if you want to show the answers immediately for each question, you might also like to provide some sort of confirmation button. It's frustratingly easy to hit the answer next to the one you intended, particularly on mobile devices. If you're also keeping statistics based on earlier attempts by other people, this is probably distorting your numbers as well.
  - ammon 9y ago
    Timing attacks like this still work over a network, you just have to make an LOT of duplicate requests, and average the times. The lucky thirteen attack (https://en.wikipedia.org/wiki/Lucky_Thirteen_attack https://en.wikipedia.org/wiki/Lucky_Thirteen_attack) is a real example. Good point about the accidental answers. I'll look at that.
    - Silhouette 9y ago
      "A lot" is going to be a mighty big number in this case, though. In some networks it will surely be prohibitively large. But even if the latency is predictable enough for that not to be the case, what real world system is going to let someone make that many failed authentication requests without doing something about it? You might be better just trying to brute force it. The trouble with hypothetical security questions is that it's so difficult to conceive a scenario where you have a deliberate vulnerability that you want someone to see, yet no other implausible aspects that will throw off anyone capable of seeing it. I am reminded of a murder mystery party I once went to with a group of mathematicians, programmers, and other logical people. When we got to the end of the evening and the true murderer was revealed, I think the majority of the participants had long ago ruled out that suspect on the basis of one of several subtle deductions from the clues provided, none of which had apparently been intended or considered by the authors of the scenario. Instead we were supposed to have ignored all the minor inconsistencies in clues and ambiguities in phrasing, and just gone for the person with the flashing neon sign over their head in the first place...
- mikekchar 9y ago
  To be honest, this quiz makes me feel that I'm learning more about you than you are about me. I especially find the reactions to the complaints interesting. This response could be read in a negative way, which is not as I intend it. Hopefully it can provide a window to view things from a different perspective. It is not possible to really determine the ability of a programmer in an interview -- the scope is too large. It takes time and effort to understand how well a person will perform. In an interview we try to find indicators of how well a person will perform. So, a quiz that tries to unearth those indicators is interesting. Your quiz makes me feel that technical details are foremost in your mind when you asses performance. Will the candidate make technical mistakes that you have seen many times before? Similarly, there appears to be considerable bias evident in your selection of questions. All of these questions asses whether or not a candidate deeply understands the functioning of the code snippet, or whether they understand it shallowly. For the candidate who admits a shallow understanding, their only recourse is to guess (and potentially fail). From that I infer that your cultural makeup is one where you have a bar that you expect all candidates to surpass. This bar is single dimensional, though, and especially your responses to the complaints imply a lack of realisation that you are narrowly optimising for a single useful ability. Finally, the quiz is set up in a "run the guantlet" fashion. Can you avoid the pitfalls that others have fallen into? This tells me that you fear candidates who do not statically measure up to your bar more than you are excited to find out what a candidate brings to the table. While you clearly will have other interview techniques for other aspects that you value, this particular quiz would worry me as a candidate. It's essentially the same question 15 times in a different context. When people complain that the quiz is too Python specific, the answer is essentially "But it covers concepts that are similar for many languages and you should be able to figure it out". It's a kind of "Well, I'm sure I could do it pretty easily, so you should be able to too". It makes me worried that there is a lack of empathy on team and that there will be problems if I don't think exactly like the leaders on the team. On the other hand, this may be fine if that's what you want in your interview process. Personally, I probably would not apply to your company having seen this quiz. This may also be a good thing from your perspective :-). However, if you ever get to a point where, despite having excellent people on staff, you can't seem to get to the next level, I would concentrate on examining potential biases of what is "good" on your team. Good luck!
JackFr 9y ago
Reminded me why I dislike Python...
StefanKarpinski 9y ago
Cute quiz. Question 9 is incorrect, however – or at least poorly worded. It asks about regex matches and only considers the last one to match, while the second one also matches, albeit incompletely. The last regex is the only one that matches the entire input string, but that's not usually how regexes work – e.g. in Perl or Ruby. This verbiage seems to be specific to Python, which uses `re.match(str)` to mean "match the entire string" as compared to `re.search(str)` which means "search for any matching substring". Arguably a decent distinction, but very Python-specific. [Update: this is now fixed in the quiz.]
- tobyhinloopen 9y ago
  The given regular expression does not match `blabber`, it matches `blabbe` (minus the `r`). I got that one wrong as well. The question should have been "which regular expression FULLY MATCHES X and not Y"
  - StavrosK 9y ago
    Nope: In [3]: re.search("b[l].*e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030> "Matches" means "matches". If they wanted "fully matches", they should have specified "^$". That's why anchors exist.
    - tobyhinloopen 9y ago
      Yes, I agree. I just explained where it went wrong and that "Match" can mean multiple things. It matches `blabber`, but the match result is `blabbe`.
    - deleted 9y ago
      [deleted]
- Bartweiss 9y ago
  Thanks, I had to check the Python docs to be sure I wasn't crazy! I went A->D, stopped at B because it 'worked', and got it wrong. D also works, obviously, but I'm glad the question is cleaned up now.
FalconNL 9y ago
The wording on question 9 (the regex one) is a little unclear: both b[l].e and ([^b]b[^b]*){3} will match blabber but not babel. Presumably the intended meaning is that the entire string must be matched, but the question does not make this explicit.
- ammon 9y ago
  Just added anchors to the expressions. Thanks for pointing this out!
- acconrad 9y ago
  Glad someone pointed that out. I chose b[l].e and got it wrong and was scratching my head...
- aaronblohowiak 9y ago
  I got the same one wrong, for the same reason. I picked the first one that worked instead of looking at all and picking the best one.
- krallja 9y ago
  I'm ashamed that the only one I missed was this regexp question, even after the anchors were added.
philh 9y ago
Mildly annoyed at myself for getting question 8 wrong: the true answer was what I thought that code would be doing, but the bytes didn't look like valid UTF-8. I had a brainfart and thought that continuation characters were always valid ASCII, so a string containing only characters >= 0x80 wouldn't be valid. The (much saner) reality is that continuation characters are never valid ASCII. 10 I admit I got through elimination: three answers were obviously wrong, one could have gone either way.
quiz_dude 9y ago
Question for ammon : What's the time frame for expanding to remote jobs? Question for other HN readers : How much did you score on the quiz? I scored 10/15 and it shows that I'm better than 50% of engineers i.e I'm somewhere right in the middle. Not very encouraging, but if I brushed up on the graph algorithms, I could have done better.
- panda88888 9y ago
  I scored top 3% at 14/15. I had experience with Python so that helped. Honestly I don't think I am in top 3%, since I don't work as sw engineer in real life.
- ammon 9y ago
  The truth is that there's just a lot less demand for remote engineers, and among companies that do hire remote, it's usually for very specialized roles. (There are obviously exceptions.) Looking at our stats, about 5% of the companies we work with are interested in remote engineers. We're not going to expand to remote jobs until we have that number up to a point where we can provide a good experince. I don't have a time-frame, but not in the next 3 months (over the next 3 months we're focused on expanding to other cities)
  - ammon 9y ago
    Re the % scores, they're currently set based on the people we had test the quiz yesterday. May well not have been a representative group! We'll reset them later today.
  - quiz_dude2 9y ago
    Follow up question : In your faq, you mentioned that you do not interview candidates who need visa sponsorship. Any plans to change that part?
    - ammon 9y ago
      Unfortunately probably not :( I would love to, but the jobs are just not there. H1Bs are just too hard to get. What I do hope to do is expand out of the US, and get around the issue that way.
dxbydt 9y ago
imo, this is the right way to do interviewing. I'm currently interviewing with several companies, and one thing that stresses me out is the expectation that 1. candidate must discover algorithm to solve unseen problem 2. candidate must code up algorithm discovered in #1 3. candidate's code must work 4. steps 1,2,3 must happen under 1 hour I think this is a bit too much, and completely misrepresents the current working state of professional programmers. If I am asked to do #1, I usually do a literature survey - I read papers before rolling my own algo from scratch. 99% of the time, there's a good enough algo already out there. If #2 - ok. If #3 - not ok. Typically you need a debugger & 1-2 passes before you get it right. If #4 - definitely not ok. Step 1 itself can take you forever, and you don't even know if its correct. Then you've got #2,#3...adding time constraint to this mix is crazy. So yeah, this quiz has two highlights: showing a block of code & asking to fill in the blanks - much better, highly recommended. Also, looking at a block of code & figuring out what it does - definitely part of programmer's everyday work. I hope more companies adapt this technique.
- j_s 9y ago
  A live example, from Uptake (Chicago): https://twitter.com/ch1gg1ns/status/867043938779222016 https://twitter.com/ch1gg1ns/status/867043938779222016
  - zootam 9y ago
    i thought those were just for free tacos?
    - j_s 9y ago
      Hard to tell if you're joking or serious. (Either way: "Why not both?") Here is the official tweet: https://twitter.com/Uptake/status/867105586336456705 https://twitter.com/Uptake/status/867105586336456705
- Bartweiss 9y ago
  My worst ever interview was for a mid-college internship. The task was a string-matching question, and I was explicitly failed for not using the Rabin-Karp algorithm. Mind, I had never heard of the thing, and I produced a valid answer with slower runtime, but since I didn't know it or invent a publication-worthy algorithm on the spot we were done. Most interviews are a bit better than that, but I'm always disturbed when people ask for algorithms that aren't either common sense or common knowledge. If it was worthy of publication and isn't relevant to the job, it probably shouldn't be tested on the fly. This system (and a lot of others) beat the heck out of that.
cperciva 9y ago
This was an interesting quiz, but perhaps not the way the authors intended. The second half was straightforward first-year computer science, but the first half challenged me to speculate about the syntax of a novel scripting language.
- lj3 9y ago
  Oh good, I wasn't the only one. The second question had me wondering if it was testing my knowledge of tree searches or python queues. I don't know python at all.
  - iamNumber4 9y ago
    I can attest that you are not the only one. Don't know python either, Nor should you need too. "There should be a warning label on this thing!!!"
  - fgonzag 9y ago
    Exactly, I know what a breadth first search is. I don't know the specifics of how queues work in python.
- arohner 9y ago
  Agreed. I've been a professional developer for nearly 20 years. I've shipped kernel C, windows C++, java,python, ruby, clojure to production, but the last time I wrote python was nearly a decade ago. Since then, I've been firmly in the Clojure community, with no intention of leaving. I know I'm a good programmer, and I got nearly every python-specific question wrong. I understand closures and call-by-reference tyvm, but I don't remember or care about the quirks of python. Triplebyte in particular seems to not care about the difference between "good at programming" and "good at programming in python". The skillset needed for C, python, clojure and JS are wildly different. They do a disservice to their users by pretending otherwise.
  - ammon 9y ago
    For what it's worth, we run our interviews in the language of the engineer's choosing (including clojure), and get people jobs at Clojure and Haskel and C++ companies. We went with Python for this (toy) quiz because it's the most common language we see among applicants. I tried to add comments to explain anything python specific, and make the question about the underlying logic. But it seems it's still easier for Python folks.
pfarnsworth 9y ago
I took the Triplebyte quiz on Facebook and after finishing, they didn't even bother giving the score. It was very outputting, and felt like a trap, asking for an email address instead of giving the score. I won't ever be doing business with them.
- ammon 9y ago
  Ah, you're talking about going through our regular process and looking for a job? In that case, we try to make it clear that what you're doing is applying for a job (and we require an account creation at the beginning of the process). This quiz posted here is something we created because some people (like you?) are applying to our process just to answer the questions. So we created this version (which does not require an account creation or ask for an email) and totally gives you a score and shows what we think the answers are.
  - unusximmortalis 9y ago
    I took the 35 question quiz. At the end it told me I passed and I shoul wait for feedback on how to move next but after a day I got an email informing me I failed.
isomorph 9y ago
Got a few easy ones that were about language implementation details wrong because I've never done Python. Only realised it was Python and not some kind of beautiful pseudocode when I kept getting answers wrong
nsaslideface 9y ago
If you only fail the regex, unicode, and hash security question you ought to get a badge of innocence you can print and cut out
cousin_it 9y ago
15/15. Some questions felt like they were trying to trap me. Or maybe I'm just tired, I've had a long day of work. Congratulations on making this! It's less fun for me than algorithm puzzles, but I can see it working well for hiring purposes. I'd love to see some software company use it and report the results.
golergka 9y ago
Failed two questions in the first part because I don't understand Python tree syntax and don't know what does it pass by reference and what by value. Would be great if authors translated it to at least a couple of other languages.
deleted 9y ago
[deleted]
scott00 9y ago
Is this intended to be taken with access to references and a python interpreter, or closed book?
- ammon 9y ago
  Either way. It's just supposed to be fun / interesting
ezekg 9y ago
Not entirely applicable to this particular quiz (most of the questions were relatively simple until the final few), but the interview questions like the ones that Hired and Triplebyte focus on honestly make me feel like an imposter. I've been a software developer for over 4 years but I don't have a formal CS education. I've gone through CLRS along with MIT's open-courseware--but I rarely use any of that directly in the workplace and so have forgotten a lot of it. And I think I'm a pretty good software developer, yet in the past when I've tried to get into Hired or Triplebyte, I always leave feeling like I don't belong in this profession because I can't tell you the big-O complexity of DFS off the top of my head, much less write the algorithm on a scratch pad. I've been doing technical interviews for a few weeks now (I was recently laid off) and I'm honestly just a little tired of the brain teasers and non-applicable algo questions--and more often than not, the interviewers flexing their "knowledge" muscles. Case in point: I'm in final stages with multiple companies for a Rails position, yet only one of those had even mentioned Rails in the technical interviews--the others decided to quiz my CS knowledge and one of them even quizzed my FP knowledge (for a Ruby position…). I don't understand it.
- ammon 9y ago
  Hello Ezekg, I am sorry that our process makes you feel bad :( The simple truth is that there are a lot of different ways to be a skilled programmer. One ways is being really good at CS (and to people who want skills in CS, talking about DFS is a reasonable low-bar question). However, another way is writing great, well-structured rails code. We work with people all the time who don't have CS backgrounds (we help them get jobs at companies that don't care about CS). The intent of our process is to measure both CS and practical skills (we actually break it down on more axes, but you get the idea). This does mean that if you go through our process, you will be asked CS questions. But you'll also be asked to code. And not knowing the CS questions is not an obstacle to passing.
  - mattmanser 9y ago
    Do you truly believe that being really good at CS will magically make you a skilled programmer? Are they not different skill sets? One is a practical skill, the other an academic one. I've met "programmers" who had depth of theoretical knowledge but couldn't code their way out of a paper bag. I've watched online lectures where Doctors of Computer Science lecture about OO but clearly have no understanding of what works and what doesn't. I will concede that being good at CS will give programmers an advantage in solving problems in a subset of domains. I suspect I could add the word small to advantage and subset and that statement would still be true.
dollaholla 9y ago
Is there a JS equivalent of something like this out there?
- ammon 9y ago
  If people like this, we'll be creating more version (in different languages, and drilling into different areas)
  - dotancohen 9y ago
    Judging by other people's comments, this would be a good quiz if it was written in their favorite language. The quiz doesn't seem too hard to port (ha, I've heard "it won't be hard to write" before). I think that a good approach might be to let the user see the code in multiple languages via tabs or an interactive select element, like the MSDN docs do.
MichaelBurge 9y ago
14/15 - I missed the one where Python doesn't copy the row before appending it to the matrix.
bensummers 9y ago
On the last two questions, I'm curious why the ability to memorise which algorithm to use is a useful skill for a professional software engineer, let alone a useful interview question. Isn't the best answer "I would look up the best algorithm by ..."?
- ammon 9y ago
  If you understand the graph traversal algorithms in question, there's nothing to memorize. You can just think through the problems. Is it important to understand graph traversal algos? I don't think it is for everyone. But a lot of people who like CS find it interesting. And if you work on, say, a web crawler or a game AI, then you'll need them.
  - bensummers 9y ago
    The last two questions are little more than "what is the name of the algorithm I should use to solve this problem?". This doesn't test understanding in any way. How is it useful to determine whether someone can work on the examples you give, let alone the average software engineering job?
    - ammon 9y ago
      The goal of this quiz is to be a little challenging and fun and maybe educational. It's not to judge whether someone can do a software engineering job! The answers to the graph problems are indeed fairly straight forward. But asking about why one might use a BST in place of a sorted list prompts interesting discussion. And going into Dijkstra's vs A* and iterative deepening depth first search is (I think) pretty interesting!
    - estomagordo 9y ago
      It's highly unlikely that you would be good at solving the single source shortest path problem on a weighted graph, and not be familiar with Dijkstra. And I'd say it's unheard of that you would know the first thing about the general, unit weight, problem, and not be familiar with BFS.
arohner 9y ago
The quiz wouldn't pass security review, FWIW. The answers are shipped with the javascript, and validation is done client side, so they're just an 'inspect element' away.
- ammon 9y ago
  :-) Yup. This is just for fun
- perfmode 9y ago
  I'd cap scores at 5/15 for applicants who automate their responses. I especially enjoyed the post-response explanation of the time and memory bounds of iterative deepening. It would be a shame to miss the opportunity to test one's skills on such a well put together experience.
  - arohner 9y ago
    In a quiz with multiple security questions, I'd give bonus points to an applicant who found security problems.
    - perfmode 9y ago
      Good point.
bstamour 9y ago
I would have done better had I known how python does parameter passing...
chris54721 9y ago
In the 3rd question, if you give the wrong answer, it says Java is pass-by-reference as well, though IIRC it is always pass-by-value. Also JS passes primitives by val as well if I'm not mistaken.
- ammon 9y ago
  Well, objects in Java are only accessible via references. So the result is behavior identical to Python and JS (primitives are passed by value, objects are passed by reference)
- Tharkun 9y ago
  Java is indeed pass-by-value, but it behaves like pass-by-reference when dealing with objects/arrays (because object references are passed by value).
nottorp 9y ago
Someone please change the title to "15-question python programming quiz". Or python and js? I stopped at the 2nd python "trick" question. Also, remove the "generalist" reference. Generalist those questions aren't.
- Tharkun 9y ago
  A bit harsh to make a generalist judgement if you quit after two questions. I don't know any python and still did pretty well.
  - nottorp 9y ago
    It was their 'generalist' statement that made me quit.
RodericDay 9y ago
Expected to dislike it, but ended up liking it quite a bit.
lerie 9y ago
its a python quiz.
smcuiuc 9y ago
I'm sorry but I just can't take a quiz/website/company seriously that assesses programmer ability, and yet they put the freaking answer key in Local Storage. Seriously, quizState? I got a 100%. ;) {"questions":[{"name":"max_function","answerIndex":2,"question":"Fill in the missing line of code:","code": ...
- ammon 9y ago
  This quiz is just supposed to be fun :) We opted for speed over security, on the idea that anyone who cared enough to pull out the answers deserves a 100!
  - smcuiuc 9y ago
    Gotcha. Just know it could give a bad first impression about the security of other assessments and competence of the team. While I find these questions to be quite flawed for the stated purpose, I really do wish you and others in this space luck!
- yellowapple 9y ago
  I actually appreciated that, since my phone has a spotty network connection.
- bbcbasic 9y ago
  Yeah plus with allowing full access to my own browser I could write a HTML doc saying 15/15 well done. How unsecure!
  - deleted 9y ago
    [deleted]
- Bartweiss 9y ago
  Eh - there's nothing to be won by getting the right answers! At best, you could share a fake 15/15 on Facebook, but could do that regardless by taking the quiz twice. Given that Triplebyte doesn't store answers that way for their job-finding quizzes, the criticism feels a bit hollow.
psyc 9y ago
Seems extremely language-semantics specific.
mhkool 9y ago
This quiz is not a programming quiz but a Python quiz.
yellowapple 9y ago
> claims this is a generalist programming quiz > first half focuses entirely on weird Python quirks. Wew. On the plus side, I still somehow managed to get 9 out of 15 right, which apparently puts me above 50% of all programmers.
graphememes 9y ago
This is very python specific. While I got a 15/15 and can easily answer these, the reason for that lies with the fact that I am very fluent in Python, had I not been I would have not gotten at least 6-8 of these wrong. This isn't a generalist quiz, nor is it a good baseline for interviewing candidates for the hiring process, this wouldn't tell me anything about the application of their skillset.
- cha5m 9y ago
  Particularly the question(s?) about list mutability. That is such a language specific property.
- squeaky-clean 9y ago
  I took a quiz at the Facebook booth at PyCon this past weekend. It was Python specific, but had a few annoying gotchas. There was one question in there twice, but with 2 different values for the input (which is what tipped me off that it was a trick). It basically relied on this fact... x = 256 x is 256 # True y = 257 y is 257 # False Which for one, you shouldn't use `is` to compare integers in Python. I wish I could have just written that as my answer. Secondly, you need to be aware the CPython uses an integer cache which causes the answers to be different. You'll actually get different answers based on whether you run it as a file, enter it line-by-line into the interpreter, or use a different implementation such as PyPy. https://stackoverflow.com/questions/15171695/whats-with-the-integer-cache-inside-python https://stackoverflow.com/questions/15171695/whats-with-the-... There wasn't any point to the quiz, so it's okay that they put in silly questions like that. But it's clearly there just to try and trick you, and not quiz you on anything relevant. Most of the other questions were tricks of the sort, or or Python 3 specific gotcha's.
  - Bartweiss 9y ago
    That's kind of nasty. It's certainly fun to play with pathological language features, but it's not exactly a useful thing to test on when the answer is "don't do that". I'd enjoy a JavaScript quiz on some of the "wat?" features, but I wouldn't want to be evaluated on my ability to guess them.
s3nnyy 9y ago
I admire triplebyte's effort and the content they produce. Their narrative is really data-driven common sense: some engineers fit to some companies but not to others. Especially enjoyed: http://blog.triplebyte.com/who-y-combinator-companies-want http://blog.triplebyte.com/who-y-combinator-companies-want I am a programmer and I run http://www.coderfit.com http://www.coderfit.com, a tiny recruiting agency in Zurich / Switzerland (reach out to me via e-mail address in my HN-handle if you look for a tech job in Zurich). Recruiting is really, really hard and I am happy to see platforms like http://interviewing.io http://interviewing.io and http://triplebyte http://triplebyte popping up that give programmers without classic CS background a chance.
mikejmoffitt 9y ago
This is a quiz about python gotchas.
jimmaswell 9y ago
What's the deal with the warp searching question? There are both depth and breadth first algorithms that can handle cycles. Bit annoying since the example I had in mind was Djikstra's but it turned out that was classified as a BFS instead of DFS like I thought it was. Finding the shortest path from A to D in 4 steps for example seemed more "depth" to me, but maybe the algorithm as a whole doesn't neatly fit into either.
- WoodenChair 9y ago
  I think you're misunderstanding the point of the question. BFS is guaranteed to find you a shortest path in terms of "# of hops" in an unweighted graph. DFS is not.
greatNespresso 9y ago
Thanks a lot for that short quizz !
deleted 9y ago
[deleted]
steven777400 9y ago
I'm not very familiar with Python but I liked this quiz. I got two wrong (13/15), once for a stupid mistake and once for not knowing the algorithms well enough. In a couple cases I didn't know the right answer, but was able to determine that all other answers were wrong, which worked. Overall, I think this is a good level of knowledge that isn't too nitpicky but has enough meat to really test if someone knows something.
pmorici 9y ago
Where do the percentiles come from that are displayed at the end of the quiz? Are these based on a previous test sample group or are they being updated as more people take the quiz?
ifree 9y ago
Too many language specific aspects.
keithnz 9y ago
"True programming generalists are rare. This quiz covers basic programming, web development, algorithms and low-level systems. The questions start easy, but get harder fast. Only 3% of programmers can get all the way through. Can you?" This test doesn't cover any of that, makes me worry if they think it does. it's more :- do you understand python syntax? do you understand a few algorithms? do you understand a few security issues? not sure how you'd go from this test to working out if a person is a programming generalist. In fact I'd be worried that someone passing this test would be fresh graduated from college where they used python.
sowbug 9y ago
This was fun. It's been a while since I took a multiple-choice test, and it called to mind the Cluss Test <https://www.recompile.se/~teddy/cluss-test> https://www.recompile.se/~teddy/cluss-test>. This test was fairly cluss-resistant, as I don't recall making any inferences from the test's format or language rather than its content.
Confusion 9y ago
Concerning 12: timing attacks are not a relevant threat for hash function computations. See https://github.com/codahale/bcrypt-ruby/pull/43#issuecomment-15797932 https://github.com/codahale/bcrypt-ruby/pull/43#issuecomment.... Supported by our own tptacek here https://twitter.com/tqbf/status/866115075110825984 https://twitter.com/tqbf/status/866115075110825984
- tptacek 9y ago
  Whoah. Careful! Timing attacks aren't relevant to password hashes, because the user doesn't control the hash, only the input to the hash, whose output is unpredictable. But cases like the TripleByte quiz, in which a user is directly submitting a hash to check (typically: an HMAC on a message) are totally relevant; in fact, they're the most common kind of legitimate timing attack.
  - Confusion 9y ago
    Yes, sorry, my comment is sloppily phrased. I should have been specific. And then I may have realized I was wrong, because it wasn't about password hashes. (It seems that tweet and the link were fresh in my memory: I automatically interpreted the question as being about password hashes, although the code actually does not imply that at all. And the linked thread is clear about the difference between password hashes and other hashes, but it seems the 'simple' message was more available to me than the deeper truth)
mrkgnao 9y ago
I like the idea, but my first reaction was that this is a test of how well you know Python's quirks (like the problems with not deep-copying things). But then I realised that that's not true. For example, call-by-value and call-by-reference are pretty pervasive across languages: many of those examples would've worked the same in Java, say. It's not familiarity with Python, but rather familiarity with a certain kind of language (which admittedly constitutes the majority of those used in industry ATM) that the test demands, at a time when even Java is getting nice FP-like abstractions and the JavaScript community is cooking up all sorts of ways to bring the benefits of that kind of thinking to their own work (from React/Redux all the way to Ramda and the like). If you're more used to languages that "get out of the way" by eliding what could very well be termed "implementation details" (e.g. is there any reason[0] to use call-by-reference in the absence of global mutable state given a "sufficiently smart optimizing compiler"?), I'd predict (n = 1) that you'll be more likely to make mistakes. I would've done far better on this quiz back when I was playing with matplotlib as an eighth- or ninth-grader: apart from the token map implementation, the quiz really favors knowledge of counterintuitive behavior common in "mainstream" imperative languages over an understanding of "composability" that users of modern functional languages are used to. (And no, that doesn't only mean Haskell or "ultra-Haskells" like Idris, but also languages like OCaml, F#, and so on.) It would be much nicer if the quiz could be less about the candidate coaxing her preferred meaning out of a language implementation and more about testing the meanings a candidate knows how to implement with the help of the language. [0]: For instance, the standard reaction to things like sum (map (\x -> x ** 2) (map (\y -> sin (y * pi / 10)) [0..9])) is that a single for-loop is better than this for "real-world purposes". That is very false: the Haskell vector package is usually capable of efficiently fusing the code above into exactly what one would've written by hand. https://www.schoolofhaskell.com/user/commercial/content/vector https://www.schoolofhaskell.com/user/commercial/content/vect... At the risk of going all peak-HN, these things are more a problem with implicit, nonuniform behavior which well-designed languages solve with abstractions like the Copy trait in Rust or even C++ move semantics.
estomagordo 9y ago
I first found out about Ammon and Triplebyte through this YC video interview. I thought it was interesting, so I'll share: https://www.youtube.com/watch?v=haxZDcTWdOo https://www.youtube.com/watch?v=haxZDcTWdOo
sharemywin 9y ago
I only got 50 percentile which I thought wasn't bad for not knowing python

227 points by ammon 9y ago

We've been using a quiz as part of our hiring process (at Triplebyte) for the last two years. However, a lot of people do our quiz for fun. So we decided to create a version optimized for this (and hopefully a little education). The problems come from the most common mistakes / misunderstandings we see when giving interviews, and ramp up in difficulty. We put a lot of effort into writing interesting answers. Let me know what you think!

chatmasta 9y ago
If you're going to pick one language for the quiz (python) it seems a little unfair to non-python developers to focus the questions on idiosyncrasies of python. Whereas if all questions were of the more general algorithm variety (like you see toward the end), non-python developers could infer the answer without any esoteric python knowledge.
- ammon 9y ago
  We talked a bunch about this. Python and JS are the two most common languages we see on our site, and Python has slightly more compact syntax, which works well in the questions. The idea, however, was not to require any python-specific knowledge. Which question do you think is looking at syntax?
  - bussierem 9y ago
    Hey, not the GP, but just glancing through the quiz - Question 3 focuses on the fact that Python passes things by reference to functions. This is definitely specific to the two languages you guys use (as is your prerogative), but if you're expanding this for others, you should definitely explain that these focus on the JS and Python languages.
    - ammon 9y ago
      Good point. I think that objects are passed by reference in a high enough % of language in common use (PHP and C++ are the two exceptions I can think of) that this is a fair question. But if enough people think it's bad, we'll remove it.
      - macintux 9y ago
        Every functional programmer in the world is looking at you askance.
      - marcosdumay 9y ago
        You got me on that too. A high enough % of the languages pass stuff by value (C/C++), copy on assignment (C++, D, I'm sure there are more), or do not have the concept of references or changes to start with (Prolog, Haskell). Anyway, if you want to bias it into Python, go for it. It's a fun quiz.
        Bartweiss 9y ago
        Copy on assignment was what got me, and I didn't feel terribly bad for missing it. No harm done though, seeing as nothing was riding on the quiz.
    - rodovich 9y ago
      Additionally, question 2 amounts to "what is the usage of python's queue class", question 4 requires knowing whether array indexes start at 0 or 1 in python, and question 8 amounts to "what is the usage of python's bytearray class".
      - philh 9y ago
        I'm not sure that's fair about question 2. It's hard to say because I know python (but not the deque class), but none of the incorrect answers seem like they'd be correct in any sensible language. Like, whatever language you're using, you need to push the children of the current node on to the queue. Only one answer looks like it could reasonably do that.
  - vitno 9y ago
    they didn't say syntax, they said Python idiosyncrasies. Knowing move/copy rules in python counts as that. It's something that isn't intuitive unless you know python, ex: the matrix question. An experienced dev who isn't a python person would pick the intuitive choice, see the matrix is malformed, then Google how to clone objects in python.
    - safek 9y ago
      On the other hand, how many languages do you know where arrays are passed by value?
      - macintux 9y ago
        My primary language (Erlang) does. I assume all functional programming languages do.
        safek 9y ago
        That's true. I suppose my next argument is that seeing matrix.append(row) on its own line signals strongly that this is not a functional language.
        thedufer 9y ago
        Is that true? I thought most functional programming languages did call-by-reference. I mean, if you have immutable data you can't tell the difference (outside of performance).
        macintux 9y ago
        Under the hood it's probably by reference, but the semantics are by value. By reference indicates that the original value can be changed in the called function. I think. Not a language lawyer.
    - echevil 9y ago
      lol, the same mistake can be made in a lot more languages~
  - acconrad 9y ago
    > The idea, however, was not to require any python-specific knowledge But the quiz specifically mentions on multiple answers that "In Python, unlike language X, Y, Z" OR "In Python, like [similar language] A, B, C"...I mean this says nothing about my programming ability, this says whether I know the quirks of the language in the same way that I can tell you all the quirks of JavaScript but not in Python. This to me is not a programming quiz, but a Python quiz designed to determine how much someone knows about what can go wrong in Python.
    - ammon 9y ago
      Well, the questions get into other things too.(Character encoding, timing attacks, graph algorithms.) Writing a good single-paragraph description of iterative deepening depth-first search was a challenge!
      - acconrad 9y ago
        Yeah except that character encoding question assumes I know how Python handles bytearrays. That trick answer of "it will break" threw me for a loop because I needed language-specific knowledge about the types that Python could handle for that input. And then with timing attacks - I mean SQL Injection is a fair web security question to ask someone, but the timing is like, very fringe knowledge. Most devs let their framework handle all of those kinds of things. Not sure why I would need to know this unless I was specifically in the sec world. The last 3 questions were actually general CS because they went through search trees and shortest paths algos, which are language-agnostic.
  - deleted 9y ago
    [deleted]
  - bdcravens 9y ago
    popleft is shift in many other languages. Not all languages manipulate arrays in the same way re ref/value. Not all languages have the concept of range() - you could have used a simple array to be more language agnostic. (I'm sure I could have found more, but I'm Python isn't a language I spend much time in)
    - echevil 9y ago
      Just to be fair, i think the comments in the source code has explained it well
      - bdcravens 9y ago
        Yes, but finding answers required to understand what was set by ref, and what was set by value, which wasn't in any comments.
  - d0100 9y ago
    I only did the first 3 , but both 2 and 3 focus on python-specific names (extend(), queue) and pass-by(reference/value).
  - tobyhinloopen 9y ago
    I used many programming languages - C, C++, Java, Ruby, Elixir, C#, Swift, Objective-C & Javascript. I've never used Python.
  - deleted 9y ago
    [deleted]
  - GrinningFool 9y ago
    Q2: THis isn't a syntax thing, but it does require knowledge of python queue objects to get right Q3: This requires knowledge around the semantics of argument passing, which vary from language to language. Q4: this again requires some understanding of how values are stored. Some languages will copy, while others will use a reference to the originally inserted value. Q5: python closure syntax. python scoping rules. Q8: unless you know what "bytearray.decode" does, the best you can do is make an educated guess from the context.
- deleted 9y ago
  [deleted]
ibdknox 9y ago
Question 9 appears to have two answers that are correct, but only one is accepted.
- StavrosK 9y ago
  Yep, I just came here to post that: In [2]: re.search("b[l].e", "babel") In [3]: re.search("b[l].e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030> It looks like they meant "match" as in "re.match", i.e. "^b[l].*e$", which is not the usual definition of the word.
  - ibdknox 9y ago
    That would also make this question specific to python, which wasn't at all what I was thinking about. It's a question about regular expressions, it should be as agnostic as possible to implementation.
    - StavrosK 9y ago
      Yeah, agreed, although there are different RE flavors, so you can't get too agnostic.
- ammon 9y ago
  Hmm. Just checked again. Two of the expressions match neither word, one matches both, and one matches only one. EDIT: I see. The confusion is over partial match vs whole match. I thought whole matches were standard when talking about regular expression in the abstract, but looks like I was wrong. Just changed the problem to include the anchors. Thanks for pointing this out.
  - ibdknox 9y ago
    That's not the standard meaning in my experience. Regex's are often used for extracting information from larger strings, forcing a full match by default would be pretty counter to their purpose.
Herald_MJ 9y ago
It feels like a lot of this quiz is just trying to "catch me out" on subtle behaviours of Python I didn't know or might have forgot about. I guess this is fine if you're making a little trivia toy to distribute to Python developers on the internet, but I don't really think it achieves much as a hiring tool. Getting questions on this test wrong would probably demoralise a lot of great candidates.
- StavrosK 9y ago
  I got them all right, and I agree that there were many gotchas. For example, the ones about the matrix, the array modification, etc. The only thing worse would have been something like: https://www.pastery.net/mbeghh/ https://www.pastery.net/mbeghh/ (it'll print 3)
  - ammon 9y ago
    Hmm. We included those issues, because we see a lot of programmers get stuck on exactly those two bugs! (In all sorts of languages, not just Python). I guess maybe that's the definition of a gotcha -- a thing that a lot of programmers get stuck on? In any case, I don't think it's an arcane gotcha! Being clear about references and shallow / deep copies is important.
    - thedufer 9y ago
      Being clear on reference vs value is certainly important, but you lose the ability to honestly call it a programming quiz. Those questions make it a Python quiz - which is fine! But not what you've sold it as.
    - StavrosK 9y ago
      In that case, by far the biggest Python gotcha is: def do_stuff(mylist=[]): mylist.append("thing") Which will lead to the list containing N items after N invocations (i.e. the second function call won't start with an empty list).
      - 9y ago
Silhouette 9y ago
Fun exercise. Thanks for sharing. I'd suggest fine-tuning the question about hashes. My first impression from reading the code, specifically the name user_submitted_hash, was that we were looking at some sort of client-server system where the hash was being calculated remotely and then submitted instead of the original credentials, and that this was a question about not trusting the client. Even without that, in a client-server system the attack you give as the mostly likely wouldn't make much sense given randomness in network latency, so the misunderstanding about the context still breaks the question. Maybe a little more context about how this hypothetical function would be used would help here? In terms of the presentation here, if you want to show the answers immediately for each question, you might also like to provide some sort of confirmation button. It's frustratingly easy to hit the answer next to the one you intended, particularly on mobile devices. If you're also keeping statistics based on earlier attempts by other people, this is probably distorting your numbers as well.
- ammon 9y ago
  Timing attacks like this still work over a network, you just have to make an LOT of duplicate requests, and average the times. The lucky thirteen attack (https://en.wikipedia.org/wiki/Lucky_Thirteen_attack https://en.wikipedia.org/wiki/Lucky_Thirteen_attack) is a real example. Good point about the accidental answers. I'll look at that.
  - Silhouette 9y ago
    "A lot" is going to be a mighty big number in this case, though. In some networks it will surely be prohibitively large. But even if the latency is predictable enough for that not to be the case, what real world system is going to let someone make that many failed authentication requests without doing something about it? You might be better just trying to brute force it. The trouble with hypothetical security questions is that it's so difficult to conceive a scenario where you have a deliberate vulnerability that you want someone to see, yet no other implausible aspects that will throw off anyone capable of seeing it. I am reminded of a murder mystery party I once went to with a group of mathematicians, programmers, and other logical people. When we got to the end of the evening and the true murderer was revealed, I think the majority of the participants had long ago ruled out that suspect on the basis of one of several subtle deductions from the clues provided, none of which had apparently been intended or considered by the authors of the scenario. Instead we were supposed to have ignored all the minor inconsistencies in clues and ambiguities in phrasing, and just gone for the person with the flashing neon sign over their head in the first place...
mikekchar 9y ago
To be honest, this quiz makes me feel that I'm learning more about you than you are about me. I especially find the reactions to the complaints interesting. This response could be read in a negative way, which is not as I intend it. Hopefully it can provide a window to view things from a different perspective. It is not possible to really determine the ability of a programmer in an interview -- the scope is too large. It takes time and effort to understand how well a person will perform. In an interview we try to find indicators of how well a person will perform. So, a quiz that tries to unearth those indicators is interesting. Your quiz makes me feel that technical details are foremost in your mind when you asses performance. Will the candidate make technical mistakes that you have seen many times before? Similarly, there appears to be considerable bias evident in your selection of questions. All of these questions asses whether or not a candidate deeply understands the functioning of the code snippet, or whether they understand it shallowly. For the candidate who admits a shallow understanding, their only recourse is to guess (and potentially fail). From that I infer that your cultural makeup is one where you have a bar that you expect all candidates to surpass. This bar is single dimensional, though, and especially your responses to the complaints imply a lack of realisation that you are narrowly optimising for a single useful ability. Finally, the quiz is set up in a "run the guantlet" fashion. Can you avoid the pitfalls that others have fallen into? This tells me that you fear candidates who do not statically measure up to your bar more than you are excited to find out what a candidate brings to the table. While you clearly will have other interview techniques for other aspects that you value, this particular quiz would worry me as a candidate. It's essentially the same question 15 times in a different context. When people complain that the quiz is too Python specific, the answer is essentially "But it covers concepts that are similar for many languages and you should be able to figure it out". It's a kind of "Well, I'm sure I could do it pretty easily, so you should be able to too". It makes me worried that there is a lack of empathy on team and that there will be problems if I don't think exactly like the leaders on the team. On the other hand, this may be fine if that's what you want in your interview process. Personally, I probably would not apply to your company having seen this quiz. This may also be a good thing from your perspective :-). However, if you ever get to a point where, despite having excellent people on staff, you can't seem to get to the next level, I would concentrate on examining potential biases of what is "good" on your team. Good luck!

JackFr 9y ago

Reminded me why I dislike Python...

StefanKarpinski 9y ago

Cute quiz. Question 9 is incorrect, however – or at least poorly worded. It asks about regex matches and only considers the last one to match, while the second one also matches, albeit incompletely. The last regex is the only one that matches the entire input string, but that's not usually how regexes work – e.g. in Perl or Ruby. This verbiage seems to be specific to Python, which uses `re.match(str)` to mean "match the entire string" as compared to `re.search(str)` which means "search for any matching substring". Arguably a decent distinction, but very Python-specific. [Update: this is now fixed in the quiz.]

tobyhinloopen 9y ago
The given regular expression does not match `blabber`, it matches `blabbe` (minus the `r`). I got that one wrong as well. The question should have been "which regular expression FULLY MATCHES X and not Y"
- StavrosK 9y ago
  Nope: In [3]: re.search("b[l].*e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030> "Matches" means "matches". If they wanted "fully matches", they should have specified "^$". That's why anchors exist.
  - tobyhinloopen 9y ago
    Yes, I agree. I just explained where it went wrong and that "Match" can mean multiple things. It matches `blabber`, but the match result is `blabbe`.
  - deleted 9y ago
    [deleted]
Bartweiss 9y ago
Thanks, I had to check the Python docs to be sure I wasn't crazy! I went A->D, stopped at B because it 'worked', and got it wrong. D also works, obviously, but I'm glad the question is cleaned up now.

FalconNL 9y ago

The wording on question 9 (the regex one) is a little unclear: both b[l].e and ([^b]b[^b]*){3} will match blabber but not babel. Presumably the intended meaning is that the entire string must be matched, but the question does not make this explicit.

ammon 9y ago
Just added anchors to the expressions. Thanks for pointing this out!
acconrad 9y ago
Glad someone pointed that out. I chose b[l].e and got it wrong and was scratching my head...
aaronblohowiak 9y ago
I got the same one wrong, for the same reason. I picked the first one that worked instead of looking at all and picking the best one.
krallja 9y ago
I'm ashamed that the only one I missed was this regexp question, even after the anchors were added.

philh 9y ago

Mildly annoyed at myself for getting question 8 wrong: the true answer was what I thought that code would be doing, but the bytes didn't look like valid UTF-8. I had a brainfart and thought that continuation characters were always valid ASCII, so a string containing only characters >= 0x80 wouldn't be valid. The (much saner) reality is that continuation characters are never valid ASCII. 10 I admit I got through elimination: three answers were obviously wrong, one could have gone either way.

quiz_dude 9y ago

Question for ammon : What's the time frame for expanding to remote jobs? Question for other HN readers : How much did you score on the quiz? I scored 10/15 and it shows that I'm better than 50% of engineers i.e I'm somewhere right in the middle. Not very encouraging, but if I brushed up on the graph algorithms, I could have done better.

panda88888 9y ago
I scored top 3% at 14/15. I had experience with Python so that helped. Honestly I don't think I am in top 3%, since I don't work as sw engineer in real life.
ammon 9y ago
The truth is that there's just a lot less demand for remote engineers, and among companies that do hire remote, it's usually for very specialized roles. (There are obviously exceptions.) Looking at our stats, about 5% of the companies we work with are interested in remote engineers. We're not going to expand to remote jobs until we have that number up to a point where we can provide a good experince. I don't have a time-frame, but not in the next 3 months (over the next 3 months we're focused on expanding to other cities)
- ammon 9y ago
  Re the % scores, they're currently set based on the people we had test the quiz yesterday. May well not have been a representative group! We'll reset them later today.
- quiz_dude2 9y ago
  Follow up question : In your faq, you mentioned that you do not interview candidates who need visa sponsorship. Any plans to change that part?
  - ammon 9y ago
    Unfortunately probably not :( I would love to, but the jobs are just not there. H1Bs are just too hard to get. What I do hope to do is expand out of the US, and get around the issue that way.

dxbydt 9y ago

imo, this is the right way to do interviewing. I'm currently interviewing with several companies, and one thing that stresses me out is the expectation that 1. candidate must discover algorithm to solve unseen problem 2. candidate must code up algorithm discovered in #1 3. candidate's code must work 4. steps 1,2,3 must happen under 1 hour I think this is a bit too much, and completely misrepresents the current working state of professional programmers. If I am asked to do #1, I usually do a literature survey - I read papers before rolling my own algo from scratch. 99% of the time, there's a good enough algo already out there. If #2 - ok. If #3 - not ok. Typically you need a debugger & 1-2 passes before you get it right. If #4 - definitely not ok. Step 1 itself can take you forever, and you don't even know if its correct. Then you've got #2,#3...adding time constraint to this mix is crazy. So yeah, this quiz has two highlights: showing a block of code & asking to fill in the blanks - much better, highly recommended. Also, looking at a block of code & figuring out what it does - definitely part of programmer's everyday work. I hope more companies adapt this technique.

j_s 9y ago
A live example, from Uptake (Chicago): https://twitter.com/ch1gg1ns/status/867043938779222016 https://twitter.com/ch1gg1ns/status/867043938779222016
- zootam 9y ago
  i thought those were just for free tacos?
  - j_s 9y ago
    Hard to tell if you're joking or serious. (Either way: "Why not both?") Here is the official tweet: https://twitter.com/Uptake/status/867105586336456705 https://twitter.com/Uptake/status/867105586336456705
Bartweiss 9y ago
My worst ever interview was for a mid-college internship. The task was a string-matching question, and I was explicitly failed for not using the Rabin-Karp algorithm. Mind, I had never heard of the thing, and I produced a valid answer with slower runtime, but since I didn't know it or invent a publication-worthy algorithm on the spot we were done. Most interviews are a bit better than that, but I'm always disturbed when people ask for algorithms that aren't either common sense or common knowledge. If it was worthy of publication and isn't relevant to the job, it probably shouldn't be tested on the fly. This system (and a lot of others) beat the heck out of that.

cperciva 9y ago

This was an interesting quiz, but perhaps not the way the authors intended. The second half was straightforward first-year computer science, but the first half challenged me to speculate about the syntax of a novel scripting language.

lj3 9y ago
Oh good, I wasn't the only one. The second question had me wondering if it was testing my knowledge of tree searches or python queues. I don't know python at all.
- iamNumber4 9y ago
  I can attest that you are not the only one. Don't know python either, Nor should you need too. "There should be a warning label on this thing!!!"
- fgonzag 9y ago
  Exactly, I know what a breadth first search is. I don't know the specifics of how queues work in python.
arohner 9y ago
Agreed. I've been a professional developer for nearly 20 years. I've shipped kernel C, windows C++, java,python, ruby, clojure to production, but the last time I wrote python was nearly a decade ago. Since then, I've been firmly in the Clojure community, with no intention of leaving. I know I'm a good programmer, and I got nearly every python-specific question wrong. I understand closures and call-by-reference tyvm, but I don't remember or care about the quirks of python. Triplebyte in particular seems to not care about the difference between "good at programming" and "good at programming in python". The skillset needed for C, python, clojure and JS are wildly different. They do a disservice to their users by pretending otherwise.
- ammon 9y ago
  For what it's worth, we run our interviews in the language of the engineer's choosing (including clojure), and get people jobs at Clojure and Haskel and C++ companies. We went with Python for this (toy) quiz because it's the most common language we see among applicants. I tried to add comments to explain anything python specific, and make the question about the underlying logic. But it seems it's still easier for Python folks.

pfarnsworth 9y ago

I took the Triplebyte quiz on Facebook and after finishing, they didn't even bother giving the score. It was very outputting, and felt like a trap, asking for an email address instead of giving the score. I won't ever be doing business with them.

ammon 9y ago
Ah, you're talking about going through our regular process and looking for a job? In that case, we try to make it clear that what you're doing is applying for a job (and we require an account creation at the beginning of the process). This quiz posted here is something we created because some people (like you?) are applying to our process just to answer the questions. So we created this version (which does not require an account creation or ask for an email) and totally gives you a score and shows what we think the answers are.
- unusximmortalis 9y ago
  I took the 35 question quiz. At the end it told me I passed and I shoul wait for feedback on how to move next but after a day I got an email informing me I failed.

isomorph 9y ago

Got a few easy ones that were about language implementation details wrong because I've never done Python. Only realised it was Python and not some kind of beautiful pseudocode when I kept getting answers wrong

nsaslideface 9y ago

If you only fail the regex, unicode, and hash security question you ought to get a badge of innocence you can print and cut out

cousin_it 9y ago

15/15. Some questions felt like they were trying to trap me. Or maybe I'm just tired, I've had a long day of work. Congratulations on making this! It's less fun for me than algorithm puzzles, but I can see it working well for hiring purposes. I'd love to see some software company use it and report the results.

golergka 9y ago

Failed two questions in the first part because I don't understand Python tree syntax and don't know what does it pass by reference and what by value. Would be great if authors translated it to at least a couple of other languages.

deleted 9y ago

[deleted]

scott00 9y ago

Is this intended to be taken with access to references and a python interpreter, or closed book?

ammon 9y ago
Either way. It's just supposed to be fun / interesting

ezekg 9y ago

Not entirely applicable to this particular quiz (most of the questions were relatively simple until the final few), but the interview questions like the ones that Hired and Triplebyte focus on honestly make me feel like an imposter. I've been a software developer for over 4 years but I don't have a formal CS education. I've gone through CLRS along with MIT's open-courseware--but I rarely use any of that directly in the workplace and so have forgotten a lot of it. And I think I'm a pretty good software developer, yet in the past when I've tried to get into Hired or Triplebyte, I always leave feeling like I don't belong in this profession because I can't tell you the big-O complexity of DFS off the top of my head, much less write the algorithm on a scratch pad. I've been doing technical interviews for a few weeks now (I was recently laid off) and I'm honestly just a little tired of the brain teasers and non-applicable algo questions--and more often than not, the interviewers flexing their "knowledge" muscles. Case in point: I'm in final stages with multiple companies for a Rails position, yet only one of those had even mentioned Rails in the technical interviews--the others decided to quiz my CS knowledge and one of them even quizzed my FP knowledge (for a Ruby position…). I don't understand it.

ammon 9y ago
Hello Ezekg, I am sorry that our process makes you feel bad :( The simple truth is that there are a lot of different ways to be a skilled programmer. One ways is being really good at CS (and to people who want skills in CS, talking about DFS is a reasonable low-bar question). However, another way is writing great, well-structured rails code. We work with people all the time who don't have CS backgrounds (we help them get jobs at companies that don't care about CS). The intent of our process is to measure both CS and practical skills (we actually break it down on more axes, but you get the idea). This does mean that if you go through our process, you will be asked CS questions. But you'll also be asked to code. And not knowing the CS questions is not an obstacle to passing.
- mattmanser 9y ago
  Do you truly believe that being really good at CS will magically make you a skilled programmer? Are they not different skill sets? One is a practical skill, the other an academic one. I've met "programmers" who had depth of theoretical knowledge but couldn't code their way out of a paper bag. I've watched online lectures where Doctors of Computer Science lecture about OO but clearly have no understanding of what works and what doesn't. I will concede that being good at CS will give programmers an advantage in solving problems in a subset of domains. I suspect I could add the word small to advantage and subset and that statement would still be true.

dollaholla 9y ago

Is there a JS equivalent of something like this out there?

ammon 9y ago
If people like this, we'll be creating more version (in different languages, and drilling into different areas)
- dotancohen 9y ago
  Judging by other people's comments, this would be a good quiz if it was written in their favorite language. The quiz doesn't seem too hard to port (ha, I've heard "it won't be hard to write" before). I think that a good approach might be to let the user see the code in multiple languages via tabs or an interactive select element, like the MSDN docs do.

MichaelBurge 9y ago

14/15 - I missed the one where Python doesn't copy the row before appending it to the matrix.

bensummers 9y ago

On the last two questions, I'm curious why the ability to memorise which algorithm to use is a useful skill for a professional software engineer, let alone a useful interview question. Isn't the best answer "I would look up the best algorithm by ..."?

ammon 9y ago
If you understand the graph traversal algorithms in question, there's nothing to memorize. You can just think through the problems. Is it important to understand graph traversal algos? I don't think it is for everyone. But a lot of people who like CS find it interesting. And if you work on, say, a web crawler or a game AI, then you'll need them.
- bensummers 9y ago
  The last two questions are little more than "what is the name of the algorithm I should use to solve this problem?". This doesn't test understanding in any way. How is it useful to determine whether someone can work on the examples you give, let alone the average software engineering job?
  - ammon 9y ago
    The goal of this quiz is to be a little challenging and fun and maybe educational. It's not to judge whether someone can do a software engineering job! The answers to the graph problems are indeed fairly straight forward. But asking about why one might use a BST in place of a sorted list prompts interesting discussion. And going into Dijkstra's vs A* and iterative deepening depth first search is (I think) pretty interesting!
  - estomagordo 9y ago
    It's highly unlikely that you would be good at solving the single source shortest path problem on a weighted graph, and not be familiar with Dijkstra. And I'd say it's unheard of that you would know the first thing about the general, unit weight, problem, and not be familiar with BFS.

arohner 9y ago

The quiz wouldn't pass security review, FWIW. The answers are shipped with the javascript, and validation is done client side, so they're just an 'inspect element' away.

ammon 9y ago
:-) Yup. This is just for fun
perfmode 9y ago
I'd cap scores at 5/15 for applicants who automate their responses. I especially enjoyed the post-response explanation of the time and memory bounds of iterative deepening. It would be a shame to miss the opportunity to test one's skills on such a well put together experience.
- arohner 9y ago
  In a quiz with multiple security questions, I'd give bonus points to an applicant who found security problems.
  - perfmode 9y ago
    Good point.

bstamour 9y ago

I would have done better had I known how python does parameter passing...

chris54721 9y ago

In the 3rd question, if you give the wrong answer, it says Java is pass-by-reference as well, though IIRC it is always pass-by-value. Also JS passes primitives by val as well if I'm not mistaken.

ammon 9y ago
Well, objects in Java are only accessible via references. So the result is behavior identical to Python and JS (primitives are passed by value, objects are passed by reference)
Tharkun 9y ago
Java is indeed pass-by-value, but it behaves like pass-by-reference when dealing with objects/arrays (because object references are passed by value).

nottorp 9y ago

Someone please change the title to "15-question python programming quiz". Or python and js? I stopped at the 2nd python "trick" question. Also, remove the "generalist" reference. Generalist those questions aren't.

Tharkun 9y ago
A bit harsh to make a generalist judgement if you quit after two questions. I don't know any python and still did pretty well.
- nottorp 9y ago
  It was their 'generalist' statement that made me quit.

RodericDay 9y ago

Expected to dislike it, but ended up liking it quite a bit.

lerie 9y ago

its a python quiz.

smcuiuc 9y ago

I'm sorry but I just can't take a quiz/website/company seriously that assesses programmer ability, and yet they put the freaking answer key in Local Storage. Seriously, quizState? I got a 100%. ;) {"questions":[{"name":"max_function","answerIndex":2,"question":"Fill in the missing line of code:","code": ...

ammon 9y ago
This quiz is just supposed to be fun :) We opted for speed over security, on the idea that anyone who cared enough to pull out the answers deserves a 100!
- smcuiuc 9y ago
  Gotcha. Just know it could give a bad first impression about the security of other assessments and competence of the team. While I find these questions to be quite flawed for the stated purpose, I really do wish you and others in this space luck!
yellowapple 9y ago
I actually appreciated that, since my phone has a spotty network connection.
bbcbasic 9y ago
Yeah plus with allowing full access to my own browser I could write a HTML doc saying 15/15 well done. How unsecure!
- deleted 9y ago
  [deleted]
Bartweiss 9y ago
Eh - there's nothing to be won by getting the right answers! At best, you could share a fake 15/15 on Facebook, but could do that regardless by taking the quiz twice. Given that Triplebyte doesn't store answers that way for their job-finding quizzes, the criticism feels a bit hollow.

psyc 9y ago

Seems extremely language-semantics specific.

mhkool 9y ago

This quiz is not a programming quiz but a Python quiz.

yellowapple 9y ago

> claims this is a generalist programming quiz > first half focuses entirely on weird Python quirks. Wew. On the plus side, I still somehow managed to get 9 out of 15 right, which apparently puts me above 50% of all programmers.

graphememes 9y ago

This is very python specific. While I got a 15/15 and can easily answer these, the reason for that lies with the fact that I am very fluent in Python, had I not been I would have not gotten at least 6-8 of these wrong. This isn't a generalist quiz, nor is it a good baseline for interviewing candidates for the hiring process, this wouldn't tell me anything about the application of their skillset.

cha5m 9y ago
Particularly the question(s?) about list mutability. That is such a language specific property.
squeaky-clean 9y ago
I took a quiz at the Facebook booth at PyCon this past weekend. It was Python specific, but had a few annoying gotchas. There was one question in there twice, but with 2 different values for the input (which is what tipped me off that it was a trick). It basically relied on this fact... x = 256 x is 256 # True y = 257 y is 257 # False Which for one, you shouldn't use `is` to compare integers in Python. I wish I could have just written that as my answer. Secondly, you need to be aware the CPython uses an integer cache which causes the answers to be different. You'll actually get different answers based on whether you run it as a file, enter it line-by-line into the interpreter, or use a different implementation such as PyPy. https://stackoverflow.com/questions/15171695/whats-with-the-integer-cache-inside-python https://stackoverflow.com/questions/15171695/whats-with-the-... There wasn't any point to the quiz, so it's okay that they put in silly questions like that. But it's clearly there just to try and trick you, and not quiz you on anything relevant. Most of the other questions were tricks of the sort, or or Python 3 specific gotcha's.
- Bartweiss 9y ago
  That's kind of nasty. It's certainly fun to play with pathological language features, but it's not exactly a useful thing to test on when the answer is "don't do that". I'd enjoy a JavaScript quiz on some of the "wat?" features, but I wouldn't want to be evaluated on my ability to guess them.

s3nnyy 9y ago

I admire triplebyte's effort and the content they produce. Their narrative is really data-driven common sense: some engineers fit to some companies but not to others. Especially enjoyed: http://blog.triplebyte.com/who-y-combinator-companies-want http://blog.triplebyte.com/who-y-combinator-companies-want I am a programmer and I run http://www.coderfit.com http://www.coderfit.com, a tiny recruiting agency in Zurich / Switzerland (reach out to me via e-mail address in my HN-handle if you look for a tech job in Zurich). Recruiting is really, really hard and I am happy to see platforms like http://interviewing.io http://interviewing.io and http://triplebyte http://triplebyte popping up that give programmers without classic CS background a chance.

mikejmoffitt 9y ago

This is a quiz about python gotchas.

jimmaswell 9y ago

What's the deal with the warp searching question? There are both depth and breadth first algorithms that can handle cycles. Bit annoying since the example I had in mind was Djikstra's but it turned out that was classified as a BFS instead of DFS like I thought it was. Finding the shortest path from A to D in 4 steps for example seemed more "depth" to me, but maybe the algorithm as a whole doesn't neatly fit into either.

WoodenChair 9y ago
I think you're misunderstanding the point of the question. BFS is guaranteed to find you a shortest path in terms of "# of hops" in an unweighted graph. DFS is not.

greatNespresso 9y ago

Thanks a lot for that short quizz !

deleted 9y ago

[deleted]

steven777400 9y ago

I'm not very familiar with Python but I liked this quiz. I got two wrong (13/15), once for a stupid mistake and once for not knowing the algorithms well enough. In a couple cases I didn't know the right answer, but was able to determine that all other answers were wrong, which worked. Overall, I think this is a good level of knowledge that isn't too nitpicky but has enough meat to really test if someone knows something.

pmorici 9y ago

Where do the percentiles come from that are displayed at the end of the quiz? Are these based on a previous test sample group or are they being updated as more people take the quiz?

ifree 9y ago

Too many language specific aspects.

keithnz 9y ago

"True programming generalists are rare. This quiz covers basic programming, web development, algorithms and low-level systems. The questions start easy, but get harder fast. Only 3% of programmers can get all the way through. Can you?" This test doesn't cover any of that, makes me worry if they think it does. it's more :- do you understand python syntax? do you understand a few algorithms? do you understand a few security issues? not sure how you'd go from this test to working out if a person is a programming generalist. In fact I'd be worried that someone passing this test would be fresh graduated from college where they used python.

sowbug 9y ago

This was fun. It's been a while since I took a multiple-choice test, and it called to mind the Cluss Test <https://www.recompile.se/~teddy/cluss-test> https://www.recompile.se/~teddy/cluss-test>. This test was fairly cluss-resistant, as I don't recall making any inferences from the test's format or language rather than its content.

Confusion 9y ago

Concerning 12: timing attacks are not a relevant threat for hash function computations. See https://github.com/codahale/bcrypt-ruby/pull/43#issuecomment-15797932 https://github.com/codahale/bcrypt-ruby/pull/43#issuecomment.... Supported by our own tptacek here https://twitter.com/tqbf/status/866115075110825984 https://twitter.com/tqbf/status/866115075110825984

tptacek 9y ago
Whoah. Careful! Timing attacks aren't relevant to password hashes, because the user doesn't control the hash, only the input to the hash, whose output is unpredictable. But cases like the TripleByte quiz, in which a user is directly submitting a hash to check (typically: an HMAC on a message) are totally relevant; in fact, they're the most common kind of legitimate timing attack.
- Confusion 9y ago
  Yes, sorry, my comment is sloppily phrased. I should have been specific. And then I may have realized I was wrong, because it wasn't about password hashes. (It seems that tweet and the link were fresh in my memory: I automatically interpreted the question as being about password hashes, although the code actually does not imply that at all. And the linked thread is clear about the difference between password hashes and other hashes, but it seems the 'simple' message was more available to me than the deeper truth)

mrkgnao 9y ago

I like the idea, but my first reaction was that this is a test of how well you know Python's quirks (like the problems with not deep-copying things). But then I realised that that's not true. For example, call-by-value and call-by-reference are pretty pervasive across languages: many of those examples would've worked the same in Java, say. It's not familiarity with Python, but rather familiarity with a certain kind of language (which admittedly constitutes the majority of those used in industry ATM) that the test demands, at a time when even Java is getting nice FP-like abstractions and the JavaScript community is cooking up all sorts of ways to bring the benefits of that kind of thinking to their own work (from React/Redux all the way to Ramda and the like). If you're more used to languages that "get out of the way" by eliding what could very well be termed "implementation details" (e.g. is there any reason[0] to use call-by-reference in the absence of global mutable state given a "sufficiently smart optimizing compiler"?), I'd predict (n = 1) that you'll be more likely to make mistakes. I would've done far better on this quiz back when I was playing with matplotlib as an eighth- or ninth-grader: apart from the token map implementation, the quiz really favors knowledge of counterintuitive behavior common in "mainstream" imperative languages over an understanding of "composability" that users of modern functional languages are used to. (And no, that doesn't only mean Haskell or "ultra-Haskells" like Idris, but also languages like OCaml, F#, and so on.) It would be much nicer if the quiz could be less about the candidate coaxing her preferred meaning out of a language implementation and more about testing the meanings a candidate knows how to implement with the help of the language. [0]: For instance, the standard reaction to things like sum (map (\x -> x ** 2) (map (\y -> sin (y * pi / 10)) [0..9])) is that a single for-loop is better than this for "real-world purposes". That is very false: the Haskell vector package is usually capable of efficiently fusing the code above into exactly what one would've written by hand. https://www.schoolofhaskell.com/user/commercial/content/vector https://www.schoolofhaskell.com/user/commercial/content/vect... At the risk of going all peak-HN, these things are more a problem with implicit, nonuniform behavior which well-designed languages solve with abstractions like the Copy trait in Rust or even C++ move semantics.

estomagordo 9y ago

I first found out about Ammon and Triplebyte through this YC video interview. I thought it was interesting, so I'll share: https://www.youtube.com/watch?v=haxZDcTWdOo https://www.youtube.com/watch?v=haxZDcTWdOo

sharemywin 9y ago

I only got 50 percentile which I thought wasn't bad for not knowing python