7 ms·

I took a look at the sample questions for the test, having never heard of it. It's 66 multiple choice questions, and many of them seem like “gotcha” questions w

by bfrydl 8y ago

I took a look at the sample questions for the test, having never heard of it. It's 66 multiple choice questions, and many of them seem like “gotcha” questions where one of the answers is an off-by-one error or something like that. And how about this lovely question: > A personal identification number that opens a certain lock consists of a sequence of 3 DIFFERENT digits from 0 though 9, inclusive. How many possible PINs are there? Putting aside the value of this question in the first place, the correct answer is 720 but it also includes the answer 1000 in case you didn't notice the emphasis on “different”. Doesn't seem like a great test to me unless it's about your reading comprehension and ability to decipher code with single-letter variable names.

povertyworld 8y ago
Isn't this just the sort of stuff students do in a first year discrete math class? This question is hardly scandalous.
- ggggtez 8y ago
  I agree. Looking at the same sample questions, these are not difficult questions if you know the relevant field of study. The sample questions cover amortization, memory allocation, finite state machines, formal logic... If you just studied enough to pass a whiteboard interview, I'd hazard that you probably can't answer half of these questions without looking up what the symbols mean. Sure, most people here may be able to figure out the PIN question, but it's not about whether any single question is easy or hard, but whether the test can sort the students according to ability. A good student may misread one or two, but a bad student is going to miss a lot more.
  - rightbyte 8y ago
    There is a general description of some nomenclature and symbols in the appendix which I assume the student get. What this really measures is corricilum. And pressumbly it's more stuff like this in the US then Russia?
- bfrydl 8y ago
  I didn't mean to say this question is scandalous. It is a needlessly confusing question. No PIN I've ever heard of requires unique digits, and the question does not actually have anything to do with PINs. If a CS student taking this test does not get “tricked” by this question, they should definitely be able to answer it.
newman8r 8y ago
Funny. Yeah I totally would have missed the word "different"... does such a lock even exist in the real world?
- deleted 8y ago
  [deleted]
- killaken2000 8y ago
  I would prefer it be more explicit and say, "each digit can only be used once". Different can be interpreted in multiple ways. Reminds me of a physics test where the question is something like: Susie is almost at her house and comes to a stop in 4 seconds. What was her acceleration.
  - edflsafoiewq 8y ago
    The only other interpretation that is obvious to me is "not all of the digits can be the same", but 990 is not an available answer.
- ggggtez 8y ago
  Yes, such locks exist, and are actually sold in the real world. They are often used in "outdoor key storage", which will find you images of what they look like. Though I can't guarantee any particular model has this particular PIN flaw (some are even worse, such as only allowing a single number from each row).
  - zrobotics 8y ago
    Which is a limitation from the mechanics of the mechanism used. Including such a restriction on an electronic PIN is stupid, but I've seen enough idiotic password policies that I don't doubt someone has done it. For example, my bank (at least 2 months ago, haven't tested recently) makes no distinction between case of letters, so 'aaa' = 'aAa' = 'AAA'.
    - bfrydl 8y ago
mnbvkhgvmj 8y ago
That question first and foremost tests English reading abilities. Many mathematically gifted non-native speakers (and indeed native speakers) would miss-understand the "different".
- avmich 8y ago
  "The exam was translated for the students in China and Russia."
  - djtriptych 8y ago
    Would love to see results compared to a Chinese or Russian-first test later translated to English.
  - mnbvkhgvmj 8y ago
    I wonder if the translator fully understood the questions.
    - ggggtez 8y ago
      There is no reason to doubt the translator just because you don't like the study's result.
      - zazen 8y ago
        No, but there might be reason to doubt the translation on the basis that the intersection of the skillsets "fluent in Russian" and "total command of the relevant computer science material" might not have many people. For a fair comparison, the translator would have to fully appreciate the relevant connotation of every significant word like "different" in every question, and successfully convey the same in Russian. You couldn't just go to a regular translation service and get a good result.
        avmich 8y ago
        A lot of computer-related words in Russian are borrowed with small modifications from English (or from where English took them). Nobody says "electronnaya vychislitel'naya mashina" anymore, they say "computer". Same for file, register, class, object, compiler (kompilyator), algorithm... It happens that tutorials for some computer-related subjects - especially some niche ones - are available only in English. It sometimes help that programming languages use English language base for keywords, so there is less confusion with Russian words.
sbov 8y ago
The funny thing is the question is actively harmful to real world problem solving. If I had to write a feature where passwords could not repeat any character the spec would spend a lot more than 1 word explaining that (and why there is that weird restriction in the first place). Assuming they wanted such a system based upon a single word would be a bizarre conclusion to jump to.
- bfrydl 8y ago
  Yeah if this were a Software Engineering test the correct answer to the question would be: ”First I would ask the client why they needed this. If they didn't have a good answer I would advise them not to do it. If they still want it anyway: 720.”
ketzu 8y ago
Personally I really dislike multiple choice question format, as they often end in "gotcha" type of questions or overthinking of the question. In the exams of my university (for computer science) they are rare (most exams have at most 1/10 as multiple choice). Does anyone have an assessment of reliability/validity/fairness of multiple choice comparing to other test types? (afair objectivity is very good in multiple choice tests)
kristopolous 8y ago
Asking "how many 3 digit numbers are there where each digit of the number only appears once?" is nearly a mathematically identical problem but without the fake backstory. Being able to go from some constructed story to a math problem is a skill that can be tested elsewhere (and really, should be assumed if someone clearly has proficient enough numeracy to do combinatorics) Questions should be testing one thing as isolated as possible, not the same things repeatedly.
- HarryHirsch 8y ago
  Transforming the backstory into math is the important skill that it's all about, and no, you can't assume that a student who knows how to manipulate numbers knows how to apply them. I can see this in my students in my introductory class. You can ask them to calculate a concentration of a solution and they will reply just fine. But when you ask them to go into the lab and actually prepare a solution of a certain concentration you draw blanks in 9 cases out of 10. Students are trained in AP Chemistry to spot problems according to keywords and apply the appropiate formula, but they have never thought about the underlying meaning of the things they do, and they cannot connect different subjects to save their lives.
  - kristopolous 8y ago
    Those are dramatically different skills. The mechanical kinesthetics of combining things is not the same skillset as the mathematical process of computing it. Lab work is its own craft. Based on what you're saying, I'm assuming you already put your questions in the "Story time" format and are getting that 1/10 success rate because the story problems are yet a Third skill - this time one about reading comprehension and language mastery. Reading comprehension isn't kinesthetics and that's not computation. One may partially inform the other, but each skill is fairly independent.
salsadip 8y ago
i am lost. how can a sequence of 3 digits be 4 digits long?
- deleted 8y ago
  [deleted]
- alexsb92 8y ago
  Where does it say it has to be 4 digits long? Most PINs are I guess, but you could have 3 digit pins just as easily.
- kevinwang 8y ago
  They're saying that 1000 was also a choice on the multiple choice test
  - salsadip 8y ago
    i can't read. thank you
    - deleted 8y ago
      [deleted]
- wlamond 8y ago
  He's saying there are 1000 possible PINs (000-999) if you don't have the unique digit constraint.
- kiallmacinnes 8y ago
  It asked how many possible choices are there, 000 through 999 is 1000 choices, just as there are 10 numbers between 0 and 9 inclusive. (Yes, the Q also said each digit must be unique, hence the real answer is 720). Anyway - IMO this is a terrible question.
mywrathacademia 8y ago
Thinking of the problem as a nested for loop it's intuitive to see how you get 1000 combinations without the numbers being unique but I dont see how to get 720 combinations as the answer when each digit is different. Can you explain where 720 combinations comes from in the context I mentioned?
- mooreds 8y ago
  10 * 9 * 8 The first time you pick, you have 10 different numbers to choose from, the second time, 9, the third time 8.
  - somehnreader 8y ago
    Or if you are like me (and a bit stupid): 1000 = all possible combinations from 000 to 999. less 10 combinations with all numbers identical is 990. now there are three patterns for numbers to be identical left that need to be taken out: 00x 0x0 x00. The x can be replaced with all digits != 0 in this example, so thats 3 (patterns) x 9 ('x' digits) = 27. That times 10 digits that can have double patterns is 270. 1000 - 10 - 270 = 720.
    - mooreds 8y ago
      Hmmm. I wouldn't have approached it that way. Thanks for sharing! Lots of ways to approach any problem.
- deleted 8y ago
  [deleted]
- the_jeremy 8y ago
  for i = 1-10 for j = 1-10 if i == j break for k = 1-10 if i == j or i == k break log(i,j,k)
- the_jeremy 8y ago
  for i = 1-10 for j = 1-10 if i == j break for k = 1-10 if i == j or i == k break log(i,j,k) (I don't understand HN formatting still)
ajross 8y ago
> Doesn't seem like a great test to me unless it's about your reading comprehension and ability to decipher code with single-letter variable names. Really? I mean, to first approximation, all software engineering work is debugging. And to first approximation, all debugging is about carefully reading results and the code that generated them. That sounds pretty spot-on to my eyes. And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code. I mean, obviously people do rely on those things for understanding, so when writing code it's important to have good semantically and contextually clear names for stuff (which certainly can include single letter names for local symbols repeated often!). But for debugging, you really have to train yourself to read the code, not the names.
- bfrydl 8y ago
  This is a Computer Science test, not a Software Engineering test. I think it's even worse as a test for the latter. I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points. In my experience most debugging presents as trial-and-error (a terrible test-taking strategy) or seeing a bug and reasoning at a high level which component of the software is responsible and gradually narrowing it down. Carefully reading through lines of code is usually the last step, unless I've got an especially elusive bug. > And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code. I'm not sure what sorts of code you're reading but this opinion seems extreme. Do you really just wholly ignore comments and variable names? I have encountered inaccurate examples of both but I have never thought to distrust them by default. As far as I recall this has not caused me serious issues while debugging.
  - rizzom5000 8y ago
    > I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points. I don't think I agree. You're talking about heuristics for debugging code and I'd argue that many of us apply similar heuristics to reading comprehension. Computer Science is not the same thing as reading comprehension, but I have to argue that if you're really good at one of these; you're probably potentially good at the other.
deleted 8y ago
[deleted]
nickm12 8y ago
Totally agree with what you say about multiple choice programming tests. I have a PhD in computer science and have worked as a programmer for over a decade. For fun, I took a practice AP CS exam with my son, which had similar multiple choice questions. I did not get 100%. I actually thought the exam was pretty good, but it tested something like 1% of the skills I use as a working programmer. Skills like API design, software architecture, documenting code, and yes even naming variables, which I think are important to writing quality code.
- enedil 8y ago
  Maybe that's because computer science is not software development?
djaouen 8y ago
> A personal identification number that opens a certain lock consists of a sequence of 3 DIFFERENT digits from 0 though 9, inclusive. How many possible PINs are there? Do people really struggle with problems like these?

by bfrydl 8y ago

povertyworld 8y ago

Isn't this just the sort of stuff students do in a first year discrete math class? This question is hardly scandalous.

ggggtez 8y ago
I agree. Looking at the same sample questions, these are not difficult questions if you know the relevant field of study. The sample questions cover amortization, memory allocation, finite state machines, formal logic... If you just studied enough to pass a whiteboard interview, I'd hazard that you probably can't answer half of these questions without looking up what the symbols mean. Sure, most people here may be able to figure out the PIN question, but it's not about whether any single question is easy or hard, but whether the test can sort the students according to ability. A good student may misread one or two, but a bad student is going to miss a lot more.
- rightbyte 8y ago
  There is a general description of some nomenclature and symbols in the appendix which I assume the student get. What this really measures is corricilum. And pressumbly it's more stuff like this in the US then Russia?
bfrydl 8y ago
I didn't mean to say this question is scandalous. It is a needlessly confusing question. No PIN I've ever heard of requires unique digits, and the question does not actually have anything to do with PINs. If a CS student taking this test does not get “tricked” by this question, they should definitely be able to answer it.

newman8r 8y ago

Funny. Yeah I totally would have missed the word "different"... does such a lock even exist in the real world?

deleted 8y ago
[deleted]
killaken2000 8y ago
I would prefer it be more explicit and say, "each digit can only be used once". Different can be interpreted in multiple ways. Reminds me of a physics test where the question is something like: Susie is almost at her house and comes to a stop in 4 seconds. What was her acceleration.
- edflsafoiewq 8y ago
  The only other interpretation that is obvious to me is "not all of the digits can be the same", but 990 is not an available answer.
ggggtez 8y ago
Yes, such locks exist, and are actually sold in the real world. They are often used in "outdoor key storage", which will find you images of what they look like. Though I can't guarantee any particular model has this particular PIN flaw (some are even worse, such as only allowing a single number from each row).
- zrobotics 8y ago
  Which is a limitation from the mechanics of the mechanism used. Including such a restriction on an electronic PIN is stupid, but I've seen enough idiotic password policies that I don't doubt someone has done it. For example, my bank (at least 2 months ago, haven't tested recently) makes no distinction between case of letters, so 'aaa' = 'aAa' = 'AAA'.
  - bfrydl 8y ago

mnbvkhgvmj 8y ago

That question first and foremost tests English reading abilities. Many mathematically gifted non-native speakers (and indeed native speakers) would miss-understand the "different".

avmich 8y ago
"The exam was translated for the students in China and Russia."
- djtriptych 8y ago
  Would love to see results compared to a Chinese or Russian-first test later translated to English.
- mnbvkhgvmj 8y ago
  I wonder if the translator fully understood the questions.
  - ggggtez 8y ago
    There is no reason to doubt the translator just because you don't like the study's result.
    - zazen 8y ago
      No, but there might be reason to doubt the translation on the basis that the intersection of the skillsets "fluent in Russian" and "total command of the relevant computer science material" might not have many people. For a fair comparison, the translator would have to fully appreciate the relevant connotation of every significant word like "different" in every question, and successfully convey the same in Russian. You couldn't just go to a regular translation service and get a good result.
      - avmich 8y ago
        A lot of computer-related words in Russian are borrowed with small modifications from English (or from where English took them). Nobody says "electronnaya vychislitel'naya mashina" anymore, they say "computer". Same for file, register, class, object, compiler (kompilyator), algorithm... It happens that tutorials for some computer-related subjects - especially some niche ones - are available only in English. It sometimes help that programming languages use English language base for keywords, so there is less confusion with Russian words.

sbov 8y ago

The funny thing is the question is actively harmful to real world problem solving. If I had to write a feature where passwords could not repeat any character the spec would spend a lot more than 1 word explaining that (and why there is that weird restriction in the first place). Assuming they wanted such a system based upon a single word would be a bizarre conclusion to jump to.

bfrydl 8y ago
Yeah if this were a Software Engineering test the correct answer to the question would be: ”First I would ask the client why they needed this. If they didn't have a good answer I would advise them not to do it. If they still want it anyway: 720.”

ketzu 8y ago

Personally I really dislike multiple choice question format, as they often end in "gotcha" type of questions or overthinking of the question. In the exams of my university (for computer science) they are rare (most exams have at most 1/10 as multiple choice). Does anyone have an assessment of reliability/validity/fairness of multiple choice comparing to other test types? (afair objectivity is very good in multiple choice tests)

kristopolous 8y ago

Asking "how many 3 digit numbers are there where each digit of the number only appears once?" is nearly a mathematically identical problem but without the fake backstory. Being able to go from some constructed story to a math problem is a skill that can be tested elsewhere (and really, should be assumed if someone clearly has proficient enough numeracy to do combinatorics) Questions should be testing one thing as isolated as possible, not the same things repeatedly.

HarryHirsch 8y ago
Transforming the backstory into math is the important skill that it's all about, and no, you can't assume that a student who knows how to manipulate numbers knows how to apply them. I can see this in my students in my introductory class. You can ask them to calculate a concentration of a solution and they will reply just fine. But when you ask them to go into the lab and actually prepare a solution of a certain concentration you draw blanks in 9 cases out of 10. Students are trained in AP Chemistry to spot problems according to keywords and apply the appropiate formula, but they have never thought about the underlying meaning of the things they do, and they cannot connect different subjects to save their lives.
- kristopolous 8y ago
  Those are dramatically different skills. The mechanical kinesthetics of combining things is not the same skillset as the mathematical process of computing it. Lab work is its own craft. Based on what you're saying, I'm assuming you already put your questions in the "Story time" format and are getting that 1/10 success rate because the story problems are yet a Third skill - this time one about reading comprehension and language mastery. Reading comprehension isn't kinesthetics and that's not computation. One may partially inform the other, but each skill is fairly independent.

salsadip 8y ago

i am lost. how can a sequence of 3 digits be 4 digits long?

deleted 8y ago
[deleted]
alexsb92 8y ago
Where does it say it has to be 4 digits long? Most PINs are I guess, but you could have 3 digit pins just as easily.
kevinwang 8y ago
They're saying that 1000 was also a choice on the multiple choice test
- salsadip 8y ago
  i can't read. thank you
  - deleted 8y ago
    [deleted]
wlamond 8y ago
He's saying there are 1000 possible PINs (000-999) if you don't have the unique digit constraint.
kiallmacinnes 8y ago
It asked how many possible choices are there, 000 through 999 is 1000 choices, just as there are 10 numbers between 0 and 9 inclusive. (Yes, the Q also said each digit must be unique, hence the real answer is 720). Anyway - IMO this is a terrible question.

mywrathacademia 8y ago

Thinking of the problem as a nested for loop it's intuitive to see how you get 1000 combinations without the numbers being unique but I dont see how to get 720 combinations as the answer when each digit is different. Can you explain where 720 combinations comes from in the context I mentioned?

mooreds 8y ago
10 * 9 * 8 The first time you pick, you have 10 different numbers to choose from, the second time, 9, the third time 8.
- somehnreader 8y ago
  Or if you are like me (and a bit stupid): 1000 = all possible combinations from 000 to 999. less 10 combinations with all numbers identical is 990. now there are three patterns for numbers to be identical left that need to be taken out: 00x 0x0 x00. The x can be replaced with all digits != 0 in this example, so thats 3 (patterns) x 9 ('x' digits) = 27. That times 10 digits that can have double patterns is 270. 1000 - 10 - 270 = 720.
  - mooreds 8y ago
    Hmmm. I wouldn't have approached it that way. Thanks for sharing! Lots of ways to approach any problem.
deleted 8y ago
[deleted]
the_jeremy 8y ago
for i = 1-10 for j = 1-10 if i == j break for k = 1-10 if i == j or i == k break log(i,j,k)
the_jeremy 8y ago
for i = 1-10 for j = 1-10 if i == j break for k = 1-10 if i == j or i == k break log(i,j,k) (I don't understand HN formatting still)

ajross 8y ago

> Doesn't seem like a great test to me unless it's about your reading comprehension and ability to decipher code with single-letter variable names. Really? I mean, to first approximation, all software engineering work is debugging. And to first approximation, all debugging is about carefully reading results and the code that generated them. That sounds pretty spot-on to my eyes. And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code. I mean, obviously people do rely on those things for understanding, so when writing code it's important to have good semantically and contextually clear names for stuff (which certainly can include single letter names for local symbols repeated often!). But for debugging, you really have to train yourself to read the code, not the names.

bfrydl 8y ago
This is a Computer Science test, not a Software Engineering test. I think it's even worse as a test for the latter. I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points. In my experience most debugging presents as trial-and-error (a terrible test-taking strategy) or seeing a bug and reasoning at a high level which component of the software is responsible and gradually narrowing it down. Carefully reading through lines of code is usually the last step, unless I've got an especially elusive bug. > And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code. I'm not sure what sorts of code you're reading but this opinion seems extreme. Do you really just wholly ignore comments and variable names? I have encountered inaccurate examples of both but I have never thought to distrust them by default. As far as I recall this has not caused me serious issues while debugging.
- rizzom5000 8y ago
  > I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points. I don't think I agree. You're talking about heuristics for debugging code and I'd argue that many of us apply similar heuristics to reading comprehension. Computer Science is not the same thing as reading comprehension, but I have to argue that if you're really good at one of these; you're probably potentially good at the other.

deleted 8y ago

[deleted]

nickm12 8y ago

Totally agree with what you say about multiple choice programming tests. I have a PhD in computer science and have worked as a programmer for over a decade. For fun, I took a practice AP CS exam with my son, which had similar multiple choice questions. I did not get 100%. I actually thought the exam was pretty good, but it tested something like 1% of the skills I use as a working programmer. Skills like API design, software architecture, documenting code, and yes even naming variables, which I think are important to writing quality code.

enedil 8y ago
Maybe that's because computer science is not software development?

djaouen 8y ago

> A personal identification number that opens a certain lock consists of a sequence of 3 DIFFERENT digits from 0 though 9, inclusive. How many possible PINs are there? Do people really struggle with problems like these?