7 ms·

Perl is no more accurate than randomly-generated language

131 points by j_baker 15y ago

wccrawford 15y ago
Suddenly I'd like to know how other languages fare. I have a feeling they might not be much better.
- edsrzf 15y ago
  You may be in luck. The paper says: "Given this interesting first result, we plan to test a number of additional languages using the same procedures." Hopefully they'll use larger sample sizes too. This one involved 18 people across 3 groups.
  - Mithaldu 15y ago
    If they use the same procedure the result will be completely useless.
- jerf 15y ago
  Of the current suite of top 20 languages at TIOBE (which for all its faults is as good as any other top-20 list), the ones that I know of that have heavy pedagogical backgrounds (that is, designed to teach students, not merely used to teach students because they were already popular) would be Python, Visual Basic, and Pascal, with Python's background probably being the most recent with its ABC predecessor, which some Googling suggests dates to the late 1980s, at which point the other two were already heading down the "industrial strength" path. Those would be the most interesting to test. (If you're in a hurry to mention Scheme, that's #27. And on the list of "pedagogical languages" it would still be beaten by Logo at #23, which, frankly, is a sign we probably shouldn't trust the list at all below about #20....)
Miky 15y ago
The language was not designed by chance. Some of the syntax was designed by chance. “Some of the syntax” is a very tiny part of a language. While this is interesting, presenting it as a “randomly-generated language” is extremely misleading.
- m0th87 15y ago
  Fair point. They can't, for example, randomly generate language semantics (how the hell would that work? :) All in all, though, this paper goes much further in trying to derive language usability than any I've seen. And it does seem disturbing that randomly selected characters (even for a subset of the language) should fare as well as Perl's hand-picked syntax.
bradleybuda 15y ago
N = 18
celoyd 15y ago
I’ll point out what a dozen others are probably typing right now: what works for beginners is not necessarily what’s best for experts. An analogy: I don’t know how to read sheet music with any proficiency. It would be easier for me if note duration were shown by horizontal length instead of the system of tails and open/filled circles. However, this wouldn’t be as good for experts, who care more about fingering than about being able to see the melody in the most literal way. I don’t happen to like perl’s syntax, but this study by itself isn’t proof that it’s objectively bad for the people who use it heavily.
- jerf 15y ago
  "what works for beginners is not necessarily what’s best for experts.... this study by itself isn’t proof that it’s objectively bad for the people who use it heavily." More than that. Three languages were chosen. One was based on what previous studies had shown works, and one sounds like it was basically a ROT13'ed perl (only with symbols), and from what some were saying on Reddit it was an overcomplicated dialect of perl (though IMHO that would change little). Then, in a study in which they pitted Perl against a language designed to do well in studies, the language designed to do well in studies prevailed. Even though I do question whether there may be a wee whiff of tautology about this whole exercise, this is interesting because it shows that studies may be able to distinguish between the features of two languages w.r.t. how easily they can be taught and learned. This is valuable stuff, millions upon millions of dollars to society's worth of valuable stuff. What it says about Perl is simply that it was not designed to do well in such studies, quelle surprise, and in particular it doesn't say anything at all about Perl relative to any other real language. $YOUR_FAVORITE_LANGUAGE may well have fared worse, and $YOUR_MOST_HATED_LANGUAGE may well have fared much better. It is a true shame that the underlying valuable message of the study is being missed by so many people in a tearing hurry to go "hurr hurr, perl t3h suckz, hurr hurr".
  - chromatic 15y ago
    What it says about Perl is simply that it was not designed to do well in such studies... I disagree. If it says anything, it's that if you give bad code to people who don't know anything about programming, they'll have trouble writing good code.
chromatic 15y ago
What an awful title. What a silly thing to test in isolation; does the syntax of a programming language present a greater barrier to a novice than the semantics of programming? A better study would have controlled for the ability of the eighteen participants to describe their algorithms in pseudocode.
- colanderman 15y ago
  It's not silly if it's significant.
  - chromatic 15y ago
    If you can't isolate that sole characteristic, your findings aren't empirically significant.
dbbo 15y ago
Unless I missed something, I don't think being easy for novices to use was one of Larry's design goals for Perl. Nevertheless, I found Perl pretty easy to learn, and it's the only language I ever "stuck with" long enough to build complicated programs.
raldi 15y ago
Misleading title. s/no more accurate/no easier for beginnners to learn/
- RickHull 15y ago
  Yes, upon reading the obvious-flamebait title I asked myself, "What does it mean for a language to be accurate?", and "How could something randomly generated possibly be more accurate than something designed?". I did not expect my questions to be answered by the article, and in that I was not disappointed.
  - j_baker 15y ago
    I didn't intend the title to be flame bait. I only wanted to get to the gist of the matter as tersely as possible, which isn't easy. What would you have titled this? EDIT: And why the downvote? I would actually be interested to hear a better title.
    - smokestack 15y ago
      "Perl's syntax is as intuitive to beginners as that of another language whose tokens and keywords were randomly-generated" is verbose, but accurate. The structure of Randomo was entirely based on Quorum, with certain tokens and keywords randomized. The study really only speaks to the "intuitiveness" of the most superficial parts of those languages' syntax, which is something that Quorum happens to focus on: action z(integer a, integer b, integer c) returns number number d = 0.0 number e = 0.0 integer i = a repeat b - a times if i mod c = 0 then d = d + 1 end else then e = e + 1 end i = i + 1 end edit: I realize the title I gave isn't short enough to post on HN, but I wanted to reply just to make a point about the study, not your title.
      - wazoox 15y ago
        Actually Quorum looks much more verbose than Perl. That may be the main distinctive feature helping absolute newcomers to grasp the code. In that vein, Java probably would fare way better than most terser languages, too. This would be something to test for (on a significant number of guinea pigs preferably).
hippich 15y ago
"While they spend their time explaining how Perl can't be used for productive work, we do commit even more code to CPAN and make our sh$t just work by the end of the day."
wedesoft 15y ago
In other words: When a beginner writes code, the programming language does not matter.
deleted 15y ago
[deleted]
m0shen 15y ago
The provided perl is a little dated: $x = &z(1, 100, 3); sub z{ $a = $_[0]; $b = $_[1]; $c = $_[2]; $d = 0.0; $e = 0.0; for ($i = $a; $i <= $b; $i++){ if($i % $c == 0) { $d = $d + 1; } else { $e = $e + 1; } } if($d > $e) { $d; } else { $e; } } Fixed? $x = z(1, 100, 3); sub z{ my( $a, $b, $c ) = @_; my $d = 0; my $e = 0; for ($a..$b){ if($_ % $c == 0) { $d++; } else { $e++; } } if($d > $e) { return $d; } else { return $e; } }
- kokey 15y ago
  This also clearly reminds me of one of the uses of perl. It's nice to use for prototyping something that can be implemented in ANSI C when necessary.
darrikmazey 15y ago
I have spent years upon years programming in perl, and I love it dearly, but I laughed pretty hard at this. No offense to Larry Wall, but if you've spent any time writing perl code, surely you've had at least one moment where you wondered if someone hadn't gotten one over on you. I don't knw about relative ease of learning, but perl can be obscure. Of course, some think that adds to the fun. ;)
chc 15y ago
This title is misleading to the point of being nonsensical. Attributing the inaccuracy to Perl itself is just plain weird. It's the people programming in Perl who failed to be accurate in their code — it's not like they wrote correct code and Perl (or Randomo for that matter) moved things around on them. But that doesn't get as much of a guffaw out of Perl-haters, I suppose.
aneth 15y ago
Shall we compare new users of TextEdit to new users of vi?
Mithaldu 15y ago
This study is entirely useless because they invalidated themselves from the start by providing the worst possible Perl examples they could. I quote the salient parts: "participants were given a code sample worksheet for the particular language group they were in. The general idea of the experiment is to give novice users code samples similarly to if a participant was learning to program from home on the Internet. This code shows one of the code samples provided to participants." $x = &z(1, 100, 3); sub z{ $a = $ [0]; $b = $ [1]; $c = $ [2]; $d = 0.0; $e = 0.0; for ($i = $a; $i <= $b; $i++){ if ($i % $c == 0) { $d = $d + 1; } else { $e = $e + 1; } } if ($d > $e) { $d; } else { $e; } } This ridiculous thing was supposed to help their test subjects learn Perl. If all their perl samples looked like that it should not be a surprise to anyone that their test subjects were entirely confounded. What is missing in that paper is a description of how they actually sourced their code samples, as well as a list of all code samples provided.
acqq 15y ago
The whole experiment is based on learning from other code samples (see http://www.cs.siue.edu/~astefik/papers/StefikPlateau2011.pdf http://www.cs.siue.edu/~astefik/papers/StefikPlateau2011.pdf). That's what they measured -- how easy is for the beginner to learn from the code samples. If you don't explain the logic behind the special characters, of course it's easier to learn what the code is about from the notation which has less special characters and less semantic behind the special characters, like their better faring language: action z(integer a, integer b, integer c) returns number number d = 0.0 number e = 0.0 integer i = a repeat b - a times if i mod c = 0 then I'd say the students were talented if they managed to make reasonable results in any language, from my perspective, their "better" language is still way too hard for absolute beginners.
- sasha-dv 15y ago
  You're absolutely right. I blame the title for turning this into Perl sucks vs. no it doesn't discussion. Not going into the problems with their study (N=18, poor Perl samples,...), all they prove is that a novice will much better understand pseudo code designed to be easy to understand than a real programming language. I'm a Perl hacker and I would be the last to claim that it has an easy to understand syntax. Perl is full of shortcuts and magic that is hard for beginners, but it makes my life as a programmer much more easier. If they compare their pseudo language with Perl, Ruby, and Python, Perl would be at the bottom of the list.
danssig 15y ago
This shouldn't be surprising. Even experts get surprised by what perl's syntax actually does [1]. [1] http://blog.plover.com//prs/accidental-syntax.html http://blog.plover.com//prs/accidental-syntax.html
cstross 15y ago
Perl was designed around two slogans: "the UNIX swiss-army chainsaw" and "there's more than one way to do it" (for any value of "it"). I first met Perl after spending some years working in the UNIX command line environment, and it made intuitive sense ... to someone who primarily used awk, sed, and bash (and able to get stuff done in C). I was up to speed and productive in it within a couple of days. (This was in the days of Perl 4 -- the OO stuff in Perl 5 came years later.) At that level, Perl was basically every UNIX text-handling tool rolled into one handy bundle oriented around script execution rather than interactive shell work, with a bunch of POSIX extras and things like the socket() API thrown in to take it beyond the toy language level. The point is, Perl isn't a beginner's language. Rather, it's an elaboration of a pre-existing toolset that assumes you're already familiar with the UNIX way before you get started. Pitching it against a teaching language isn't even an apples/oranges comparison: it's like comparing a circus merry-go-round to a helicopter (they're simply not intended for the same purpose or the same user demographic).
- chromatic 15y ago
  "Perl is, in intent, a cleaned up and summarized version of that wonderful semi-natural language known as 'Unix'." Larry Wall, http://groups.google.com/group/comp.lang.misc/msg/1ebc89bbc0a28c3b http://groups.google.com/group/comp.lang.misc/msg/1ebc89bbc0...
wazoox 15y ago
There seems to be a number of "studies" with a ridiculously low number of participants (here, 18), from which "scientists" (actually, students) try to draw conclusion from quasi-random, statistically insignificant results. Thousands of people learned programming using perl. This is much more significant than this ridiculous study (randomo language? quorum? give me a break would you?). Fortunately, at least this poor-piss stuff won't make the headlines, like too many such ridiculous studies run on 30 students do (remember lately "women wearing make-up appear more competent", augh).

131 points by j_baker 15y ago

wccrawford 15y ago

Suddenly I'd like to know how other languages fare. I have a feeling they might not be much better.

edsrzf 15y ago
You may be in luck. The paper says: "Given this interesting first result, we plan to test a number of additional languages using the same procedures." Hopefully they'll use larger sample sizes too. This one involved 18 people across 3 groups.
- Mithaldu 15y ago
  If they use the same procedure the result will be completely useless.
jerf 15y ago
Of the current suite of top 20 languages at TIOBE (which for all its faults is as good as any other top-20 list), the ones that I know of that have heavy pedagogical backgrounds (that is, designed to teach students, not merely used to teach students because they were already popular) would be Python, Visual Basic, and Pascal, with Python's background probably being the most recent with its ABC predecessor, which some Googling suggests dates to the late 1980s, at which point the other two were already heading down the "industrial strength" path. Those would be the most interesting to test. (If you're in a hurry to mention Scheme, that's #27. And on the list of "pedagogical languages" it would still be beaten by Logo at #23, which, frankly, is a sign we probably shouldn't trust the list at all below about #20....)

Miky 15y ago

The language was not designed by chance. Some of the syntax was designed by chance. “Some of the syntax” is a very tiny part of a language. While this is interesting, presenting it as a “randomly-generated language” is extremely misleading.

m0th87 15y ago
Fair point. They can't, for example, randomly generate language semantics (how the hell would that work? :) All in all, though, this paper goes much further in trying to derive language usability than any I've seen. And it does seem disturbing that randomly selected characters (even for a subset of the language) should fare as well as Perl's hand-picked syntax.

bradleybuda 15y ago

N = 18

celoyd 15y ago

I’ll point out what a dozen others are probably typing right now: what works for beginners is not necessarily what’s best for experts. An analogy: I don’t know how to read sheet music with any proficiency. It would be easier for me if note duration were shown by horizontal length instead of the system of tails and open/filled circles. However, this wouldn’t be as good for experts, who care more about fingering than about being able to see the melody in the most literal way. I don’t happen to like perl’s syntax, but this study by itself isn’t proof that it’s objectively bad for the people who use it heavily.

jerf 15y ago
"what works for beginners is not necessarily what’s best for experts.... this study by itself isn’t proof that it’s objectively bad for the people who use it heavily." More than that. Three languages were chosen. One was based on what previous studies had shown works, and one sounds like it was basically a ROT13'ed perl (only with symbols), and from what some were saying on Reddit it was an overcomplicated dialect of perl (though IMHO that would change little). Then, in a study in which they pitted Perl against a language designed to do well in studies, the language designed to do well in studies prevailed. Even though I do question whether there may be a wee whiff of tautology about this whole exercise, this is interesting because it shows that studies may be able to distinguish between the features of two languages w.r.t. how easily they can be taught and learned. This is valuable stuff, millions upon millions of dollars to society's worth of valuable stuff. What it says about Perl is simply that it was not designed to do well in such studies, quelle surprise, and in particular it doesn't say anything at all about Perl relative to any other real language. $YOUR_FAVORITE_LANGUAGE may well have fared worse, and $YOUR_MOST_HATED_LANGUAGE may well have fared much better. It is a true shame that the underlying valuable message of the study is being missed by so many people in a tearing hurry to go "hurr hurr, perl t3h suckz, hurr hurr".
- chromatic 15y ago
  What it says about Perl is simply that it was not designed to do well in such studies... I disagree. If it says anything, it's that if you give bad code to people who don't know anything about programming, they'll have trouble writing good code.

chromatic 15y ago

What an awful title. What a silly thing to test in isolation; does the syntax of a programming language present a greater barrier to a novice than the semantics of programming? A better study would have controlled for the ability of the eighteen participants to describe their algorithms in pseudocode.

colanderman 15y ago
It's not silly if it's significant.
- chromatic 15y ago
  If you can't isolate that sole characteristic, your findings aren't empirically significant.

dbbo 15y ago

Unless I missed something, I don't think being easy for novices to use was one of Larry's design goals for Perl. Nevertheless, I found Perl pretty easy to learn, and it's the only language I ever "stuck with" long enough to build complicated programs.

raldi 15y ago

Misleading title. s/no more accurate/no easier for beginnners to learn/

RickHull 15y ago
Yes, upon reading the obvious-flamebait title I asked myself, "What does it mean for a language to be accurate?", and "How could something randomly generated possibly be more accurate than something designed?". I did not expect my questions to be answered by the article, and in that I was not disappointed.
- j_baker 15y ago
  I didn't intend the title to be flame bait. I only wanted to get to the gist of the matter as tersely as possible, which isn't easy. What would you have titled this? EDIT: And why the downvote? I would actually be interested to hear a better title.
  - smokestack 15y ago
    "Perl's syntax is as intuitive to beginners as that of another language whose tokens and keywords were randomly-generated" is verbose, but accurate. The structure of Randomo was entirely based on Quorum, with certain tokens and keywords randomized. The study really only speaks to the "intuitiveness" of the most superficial parts of those languages' syntax, which is something that Quorum happens to focus on: action z(integer a, integer b, integer c) returns number number d = 0.0 number e = 0.0 integer i = a repeat b - a times if i mod c = 0 then d = d + 1 end else then e = e + 1 end i = i + 1 end edit: I realize the title I gave isn't short enough to post on HN, but I wanted to reply just to make a point about the study, not your title.
    - wazoox 15y ago
      Actually Quorum looks much more verbose than Perl. That may be the main distinctive feature helping absolute newcomers to grasp the code. In that vein, Java probably would fare way better than most terser languages, too. This would be something to test for (on a significant number of guinea pigs preferably).

hippich 15y ago

"While they spend their time explaining how Perl can't be used for productive work, we do commit even more code to CPAN and make our sh$t just work by the end of the day."

wedesoft 15y ago

In other words: When a beginner writes code, the programming language does not matter.

deleted 15y ago

[deleted]

m0shen 15y ago

The provided perl is a little dated: $x = &z(1, 100, 3); sub z{ $a = $_[0]; $b = $_[1]; $c = $_[2]; $d = 0.0; $e = 0.0; for ($i = $a; $i <= $b; $i++){ if($i % $c == 0) { $d = $d + 1; } else { $e = $e + 1; } } if($d > $e) { $d; } else { $e; } } Fixed? $x = z(1, 100, 3); sub z{ my( $a, $b, $c ) = @_; my $d = 0; my $e = 0; for ($a..$b){ if($_ % $c == 0) { $d++; } else { $e++; } } if($d > $e) { return $d; } else { return $e; } }

kokey 15y ago
This also clearly reminds me of one of the uses of perl. It's nice to use for prototyping something that can be implemented in ANSI C when necessary.

darrikmazey 15y ago

I have spent years upon years programming in perl, and I love it dearly, but I laughed pretty hard at this. No offense to Larry Wall, but if you've spent any time writing perl code, surely you've had at least one moment where you wondered if someone hadn't gotten one over on you. I don't knw about relative ease of learning, but perl can be obscure. Of course, some think that adds to the fun. ;)

chc 15y ago

This title is misleading to the point of being nonsensical. Attributing the inaccuracy to Perl itself is just plain weird. It's the people programming in Perl who failed to be accurate in their code — it's not like they wrote correct code and Perl (or Randomo for that matter) moved things around on them. But that doesn't get as much of a guffaw out of Perl-haters, I suppose.

aneth 15y ago

Shall we compare new users of TextEdit to new users of vi?

Mithaldu 15y ago

This study is entirely useless because they invalidated themselves from the start by providing the worst possible Perl examples they could. I quote the salient parts: "participants were given a code sample worksheet for the particular language group they were in. The general idea of the experiment is to give novice users code samples similarly to if a participant was learning to program from home on the Internet. This code shows one of the code samples provided to participants." $x = &z(1, 100, 3); sub z{ $a = $ [0]; $b = $ [1]; $c = $ [2]; $d = 0.0; $e = 0.0; for ($i = $a; $i <= $b; $i++){ if ($i % $c == 0) { $d = $d + 1; } else { $e = $e + 1; } } if ($d > $e) { $d; } else { $e; } } This ridiculous thing was supposed to help their test subjects learn Perl. If all their perl samples looked like that it should not be a surprise to anyone that their test subjects were entirely confounded. What is missing in that paper is a description of how they actually sourced their code samples, as well as a list of all code samples provided.

acqq 15y ago

The whole experiment is based on learning from other code samples (see http://www.cs.siue.edu/~astefik/papers/StefikPlateau2011.pdf http://www.cs.siue.edu/~astefik/papers/StefikPlateau2011.pdf). That's what they measured -- how easy is for the beginner to learn from the code samples. If you don't explain the logic behind the special characters, of course it's easier to learn what the code is about from the notation which has less special characters and less semantic behind the special characters, like their better faring language: action z(integer a, integer b, integer c) returns number number d = 0.0 number e = 0.0 integer i = a repeat b - a times if i mod c = 0 then I'd say the students were talented if they managed to make reasonable results in any language, from my perspective, their "better" language is still way too hard for absolute beginners.

sasha-dv 15y ago
You're absolutely right. I blame the title for turning this into Perl sucks vs. no it doesn't discussion. Not going into the problems with their study (N=18, poor Perl samples,...), all they prove is that a novice will much better understand pseudo code designed to be easy to understand than a real programming language. I'm a Perl hacker and I would be the last to claim that it has an easy to understand syntax. Perl is full of shortcuts and magic that is hard for beginners, but it makes my life as a programmer much more easier. If they compare their pseudo language with Perl, Ruby, and Python, Perl would be at the bottom of the list.

danssig 15y ago

This shouldn't be surprising. Even experts get surprised by what perl's syntax actually does [1]. [1] http://blog.plover.com//prs/accidental-syntax.html http://blog.plover.com//prs/accidental-syntax.html

cstross 15y ago

Perl was designed around two slogans: "the UNIX swiss-army chainsaw" and "there's more than one way to do it" (for any value of "it"). I first met Perl after spending some years working in the UNIX command line environment, and it made intuitive sense ... to someone who primarily used awk, sed, and bash (and able to get stuff done in C). I was up to speed and productive in it within a couple of days. (This was in the days of Perl 4 -- the OO stuff in Perl 5 came years later.) At that level, Perl was basically every UNIX text-handling tool rolled into one handy bundle oriented around script execution rather than interactive shell work, with a bunch of POSIX extras and things like the socket() API thrown in to take it beyond the toy language level. The point is, Perl isn't a beginner's language. Rather, it's an elaboration of a pre-existing toolset that assumes you're already familiar with the UNIX way before you get started. Pitching it against a teaching language isn't even an apples/oranges comparison: it's like comparing a circus merry-go-round to a helicopter (they're simply not intended for the same purpose or the same user demographic).

chromatic 15y ago
"Perl is, in intent, a cleaned up and summarized version of that wonderful semi-natural language known as 'Unix'." Larry Wall, http://groups.google.com/group/comp.lang.misc/msg/1ebc89bbc0a28c3b http://groups.google.com/group/comp.lang.misc/msg/1ebc89bbc0...

wazoox 15y ago

There seems to be a number of "studies" with a ridiculously low number of participants (here, 18), from which "scientists" (actually, students) try to draw conclusion from quasi-random, statistically insignificant results. Thousands of people learned programming using perl. This is much more significant than this ridiculous study (randomo language? quorum? give me a break would you?). Fortunately, at least this poor-piss stuff won't make the headlines, like too many such ridiculous studies run on 30 students do (remember lately "women wearing make-up appear more competent", augh).