4 ms·

How are you planning on combating cheating? For sure people will post solutions online to your CTFs.

gknoy 12y ago
I bet that Thomas and Patrick could do something like have dynamically created tests, where there are multiple versions of an exploit, so that they test the same skill but in different ways. You might have a challenge that tests whether you can do SQL injection, but the tables may differ. Overrunning a buffer (as Patrick mentioned in his post about Microcorruption) could depend on different payloads. I naiively believe that things could be varied enough that you would need to understand the concepts rather than paste answers, in which case "cheating" would mean "learning", and is basically something they say they want you to do. I'm not sure how hard it would be to get to that point rather than being vulnerable to Bob the Super Coder posting walkthrough screencasts, but I trust that the founders are already savvy enough to have been thinking about that.
- psquid 12y ago
  Unfortunately, I suspect you overestimate the obfuscating power of dynamic content when the number of users is sufficiently large. A certain MMO I play recently had a limited-time event built around figuring out the meaning of different clues (locations to go to for the actual meat of the event), and despite a fairly large number of variations, people had collectively figured out just about every possible clue->location mapping within a matter of hours. That's not to say you can't prevent cheating, but that even with relatively little incentive (that whole clues thing gave only a single cosmetic item, and anecdotally I've seen very few people actually use theirs) users can and most likely will outpace any attempt to prevent it by means of varying the problem.
Pyxl101 12y ago
This is an important problem. As part of interviewing processes, I have several times tried offering "take home work challenges". I had to stop after, having run the experiment several times, I detected plagiarism in about 30-40% of the cases. The risk is of plagiarism is real for any problem that's well known enough that the problem and solution can appear online, and detecting plagiarism is not always easy.
- tptacek 12y ago
  It's less important than it sounds for a bunch of reasons having both to do with the actual "game" we are building and with our business model. It's not like we spit out a number for every player, and everyone with a number better than X gets a job offer.
- areyousure 12y ago
  Wait, why'd you stop? Isn't this a great way to filter out those 30--40% of people?
  - omegaham 12y ago
    I like this response. I think that one of the best things about a take-home challenge is that during the interview, you can then say, "The way you implemented this function was interesting. Let's talk about why you did it that way." If he knows what he's doing (or made a weird solution), you get a great glimpse into his thought process. If he plagiarized, then he's garbage. English teachers do the same thing for detecting plagiarism. Bring the kid into your office and talk about the essay. If he's completely clueless, someone else wrote it for him.
    - Pyxl101 12y ago
      It's a neat idea, but ultimately, worrying about cheating or whether someone has cheated feels like a distraction. It suggests a test that isn't repeatable. I'm only confident in my ability to detect obvious plagiarism. Subtle plagiarism can exist in varying forms, such as reading an analysis of the problem and solutions. Some people who plagiarize will pass a thorough Q&A about their code, because they fully understood the explanation of the solution, but are getting an unfair advantage over candidates who worked out a solution themselves from scratch. Overall, I get more value from tests that are constructed so that I can learn positive things about the candidate - as many opportunities as possible for the candidate to distinguish themselves. If the only reason for a particular approach was to provide the opportunity for immoral candidates to weed themselves out by committing obvious plagiarism (negative data), then there's probably a better approach that tells me more about the candidate per unit time. If I was going to continue, I would use a problem that is (1) more representative of the actual work being done by the team; less of a puzzle (2) custom designed for the team or company; not a preexisting or well known problem. (Even candidates who don't cheat can have an unfair advantage on well-known problems if they have coincidentally encountered it before! Another reason to use unique questions.)

gknoy 12y ago

I bet that Thomas and Patrick could do something like have dynamically created tests, where there are multiple versions of an exploit, so that they test the same skill but in different ways. You might have a challenge that tests whether you can do SQL injection, but the tables may differ. Overrunning a buffer (as Patrick mentioned in his post about Microcorruption) could depend on different payloads. I naiively believe that things could be varied enough that you would need to understand the concepts rather than paste answers, in which case "cheating" would mean "learning", and is basically something they say they want you to do. I'm not sure how hard it would be to get to that point rather than being vulnerable to Bob the Super Coder posting walkthrough screencasts, but I trust that the founders are already savvy enough to have been thinking about that.

psquid 12y ago
Unfortunately, I suspect you overestimate the obfuscating power of dynamic content when the number of users is sufficiently large. A certain MMO I play recently had a limited-time event built around figuring out the meaning of different clues (locations to go to for the actual meat of the event), and despite a fairly large number of variations, people had collectively figured out just about every possible clue->location mapping within a matter of hours. That's not to say you can't prevent cheating, but that even with relatively little incentive (that whole clues thing gave only a single cosmetic item, and anecdotally I've seen very few people actually use theirs) users can and most likely will outpace any attempt to prevent it by means of varying the problem.

Pyxl101 12y ago

This is an important problem. As part of interviewing processes, I have several times tried offering "take home work challenges". I had to stop after, having run the experiment several times, I detected plagiarism in about 30-40% of the cases. The risk is of plagiarism is real for any problem that's well known enough that the problem and solution can appear online, and detecting plagiarism is not always easy.

tptacek 12y ago
It's less important than it sounds for a bunch of reasons having both to do with the actual "game" we are building and with our business model. It's not like we spit out a number for every player, and everyone with a number better than X gets a job offer.
areyousure 12y ago
Wait, why'd you stop? Isn't this a great way to filter out those 30--40% of people?
- omegaham 12y ago
  I like this response. I think that one of the best things about a take-home challenge is that during the interview, you can then say, "The way you implemented this function was interesting. Let's talk about why you did it that way." If he knows what he's doing (or made a weird solution), you get a great glimpse into his thought process. If he plagiarized, then he's garbage. English teachers do the same thing for detecting plagiarism. Bring the kid into your office and talk about the essay. If he's completely clueless, someone else wrote it for him.
  - Pyxl101 12y ago
    It's a neat idea, but ultimately, worrying about cheating or whether someone has cheated feels like a distraction. It suggests a test that isn't repeatable. I'm only confident in my ability to detect obvious plagiarism. Subtle plagiarism can exist in varying forms, such as reading an analysis of the problem and solutions. Some people who plagiarize will pass a thorough Q&A about their code, because they fully understood the explanation of the solution, but are getting an unfair advantage over candidates who worked out a solution themselves from scratch. Overall, I get more value from tests that are constructed so that I can learn positive things about the candidate - as many opportunities as possible for the candidate to distinguish themselves. If the only reason for a particular approach was to provide the opportunity for immoral candidates to weed themselves out by committing obvious plagiarism (negative data), then there's probably a better approach that tells me more about the candidate per unit time. If I was going to continue, I would use a problem that is (1) more representative of the actual work being done by the team; less of a puzzle (2) custom designed for the team or company; not a preexisting or well known problem. (Even candidates who don't cheat can have an unfair advantage on well-known problems if they have coincidentally encountered it before! Another reason to use unique questions.)