4 ms·

"A comparative usability study on a language is very simple to perform: you basically print out some code samples, let people with no experience in the language

by sedachv 16y ago

"A comparative usability study on a language is very simple to perform: you basically print out some code samples, let people with no experience in the language (but general programming experience) read them for a set time period, and then evaluate their comprehension." That's just stupid. If the programmer doesn't understand English and the program is written with English identifiers, does that fail the test? What about APL and Perl? "Because the people who "get" Lisp are almost always theoreticians who know what they want to code before they code it." And you seem to be a theoretician who has never had to maintain any Lisp code.

derefr 16y ago
> That's just stupid. If the programmer doesn't understand English and the program is written with English identifiers, does that fail the test? What about APL and Perl? It's called a comparative usability study. The same person looks at the same code sample written in two different languages, A and B (in random order), given time T to absorb each one (where T is much less than the time it would actually take to fully absorb the code), and then the references are removed and they are given tests on the properties of sample_A and sample_B (again in random order.) The score is not a measure of either A or B individually, but rather the ratio A:B, derived by the ratio of correct questions in sample_A:sample_B. A:B is then handed to a Bayesian neural network as a confidence score. Thus, if the test-taker doesn't know English, they get the same (low) score on both tests—to the BNN, that is the same as "no new evidence." Also, you wouldn't be comparing, say, Java to APL. You'd be comparing Java to Java + "an unless statement", or APL to APL + "enforced whitespace." You want to see the relative worth of language features, not the relative worth of our current languages, which have mostly been slapped together with no concern for UX. The result of such a study would be a set of UX suggestions, of language features that comparatively increase, or comparatively decrease comprehension. > And you seem to be a theoretician who has never had to maintain any Lisp code. For the last eight months, I've been writing Clojure for a living. I've written quite a bit of CL before that. I still find it harder to come back to any Lisp project after a month or two than to come back to code in any other language.
- sedachv 16y ago
  "Also, you wouldn't be comparing, say, Java to APL. You'd be comparing Java to Java + "an unless statement", or APL to APL + "enforced whitespace." You want to see the relative worth of language features, not the relative worth of our current languages, which have mostly been slapped together with no concern for UX. The result of such a study would be a set of UX suggestions, of language features that comparatively increase, or comparatively decrease comprehension." This is still meaningless if you're testing it on someone who doesn't know APL. Having an "unless" statement in your example is exactly like testing for English comprehension. This technique would make sense if you're testing proposed design choices for changes to APL on APL programmers (here's 60 seconds to read this program; what do you think it means?), it would make sense if you have equivalent programs in different languages being tested on users of those languages ("users of Perl understood program X 10% better than users of APL given 60 seconds to examine it"), but it is meaningless in the context you provide.
  - derefr 16y ago
    You ignored the most important part of what I said. > Thus, if the test-taker doesn't know English, they get the same (low) score on both tests—to the BNN, that is the same as "no new evidence." So, of course, you wouldn't test APL vs. APL+whitespace on someone who doesn't know APL, because your test results would always, predictably, be 1:1. But you wouldn't be testing APL vs. anything—because APL is a set of design choices that have already been made, and anyone who knows APL would already have taken the time to absorb those design choices. It would be most scientifically rigorous, as I said, to start with a made-up programming language (a random combination of language features that has no overarching design principle which users could recognize to reduce cognitive load), teach it to the users, and then do the tests on that language, vs. that language w/ modifications.
    - silentbicycle 16y ago
      > because APL is a set of design choices that have already been made Compare J, Q, and Dyalog. Three modern APL dialects, very different in feel. You might protest that J or Q aren't APLs (they only use ASCII! Q has ragged arrays!), but that's like arguing Scheme isn't a Lisp.

by sedachv 16y ago

derefr 16y ago

> That's just stupid. If the programmer doesn't understand English and the program is written with English identifiers, does that fail the test? What about APL and Perl? It's called a comparative usability study. The same person looks at the same code sample written in two different languages, A and B (in random order), given time T to absorb each one (where T is much less than the time it would actually take to fully absorb the code), and then the references are removed and they are given tests on the properties of sample_A and sample_B (again in random order.) The score is not a measure of either A or B individually, but rather the ratio A:B, derived by the ratio of correct questions in sample_A:sample_B. A:B is then handed to a Bayesian neural network as a confidence score. Thus, if the test-taker doesn't know English, they get the same (low) score on both tests—to the BNN, that is the same as "no new evidence." Also, you wouldn't be comparing, say, Java to APL. You'd be comparing Java to Java + "an unless statement", or APL to APL + "enforced whitespace." You want to see the relative worth of language features, not the relative worth of our current languages, which have mostly been slapped together with no concern for UX. The result of such a study would be a set of UX suggestions, of language features that comparatively increase, or comparatively decrease comprehension. > And you seem to be a theoretician who has never had to maintain any Lisp code. For the last eight months, I've been writing Clojure for a living. I've written quite a bit of CL before that. I still find it harder to come back to any Lisp project after a month or two than to come back to code in any other language.

sedachv 16y ago
"Also, you wouldn't be comparing, say, Java to APL. You'd be comparing Java to Java + "an unless statement", or APL to APL + "enforced whitespace." You want to see the relative worth of language features, not the relative worth of our current languages, which have mostly been slapped together with no concern for UX. The result of such a study would be a set of UX suggestions, of language features that comparatively increase, or comparatively decrease comprehension." This is still meaningless if you're testing it on someone who doesn't know APL. Having an "unless" statement in your example is exactly like testing for English comprehension. This technique would make sense if you're testing proposed design choices for changes to APL on APL programmers (here's 60 seconds to read this program; what do you think it means?), it would make sense if you have equivalent programs in different languages being tested on users of those languages ("users of Perl understood program X 10% better than users of APL given 60 seconds to examine it"), but it is meaningless in the context you provide.
- derefr 16y ago
  You ignored the most important part of what I said. > Thus, if the test-taker doesn't know English, they get the same (low) score on both tests—to the BNN, that is the same as "no new evidence." So, of course, you wouldn't test APL vs. APL+whitespace on someone who doesn't know APL, because your test results would always, predictably, be 1:1. But you wouldn't be testing APL vs. anything—because APL is a set of design choices that have already been made, and anyone who knows APL would already have taken the time to absorb those design choices. It would be most scientifically rigorous, as I said, to start with a made-up programming language (a random combination of language features that has no overarching design principle which users could recognize to reduce cognitive load), teach it to the users, and then do the tests on that language, vs. that language w/ modifications.
  - silentbicycle 16y ago
    > because APL is a set of design choices that have already been made Compare J, Q, and Dyalog. Three modern APL dialects, very different in feel. You might protest that J or Q aren't APLs (they only use ASCII! Q has ragged arrays!), but that's like arguing Scheme isn't a Lisp.