5 ms·

NLP in Python vs other Programming Languages

32 points by rayvega 16y ago

Jun8 16y ago
I agree with everyone that Perl syntax can look random gibberish, however, their particular Perl example seems quite easy to interpret.
- lars512 16y ago
  Sure, it's roughly readable, but even here Perl makes several aspects of the loop implicit and buried in syntax (the line variable, stdin as source of the lines).
  - pyre 16y ago
    Perl doesn't make it implicit, the programmer does. It could easily be rewritten as: foreach my $line (<STDIN>) { foreach my $word (split($line)) { print "$word\n" if $word =~ /ing$/; } } The programmer made a conscious decision to read STDIN implicitly in a while loop and then call split with implicit usage of $_, then went on to complain that too much was hidden away. I could make the python example equally obtuse: import sys, re for a in sys.fopen(sys.stdin.fileno(),'r',1): for b in re.split(a,r''): if re.search(b,r'.*ing$') print b Compressed forms: # PERL map { print "$_\n" } grep { /ing$/ } map { split($_) } <STDIN>; # PYTHON import sys for a in [ a for a in [ b for a in sys.stdin for b in a.split() ] if a.endswith('ing') ]: print a # PYTHON import sys print "\n".join(filter(lambda x: x.endswith('ing'), [ var2 for var1 in sys.stdin for var2 in var1.split() ]))
    - lars512 16y ago
      Individual programmer style does matter, but languages also guide our style. I'm often asked to convert Perl scripts to Python scripts, and the originals don't exhibit the "optional readability" you're advertising, and which Python encourages by default. An entirely different angle on all of this is that NLTK is aimed at novice programmers (linguists) as well as expert programmers (CS students). Perhaps readability is a different concept for these two user groups...
      - phaylon 16y ago
        Well, since Perl was first designed by a linguist, one might think they aren't all of one mindset :)
    - l0nwlf 16y ago
      "There should be one-- and preferably only one --obvious way to do it." and your code mylord does not passes the bill. Also I do agree with OP, python support readability and it make sense to choose it over other languages where clarity matters. Every language have their own use-case, python suits here quite well.
      - kscaldef 16y ago
        Well, of course, when you assume that the design criteria of Python are the correct ones you reach the conclusion that Python is the preferred language to use. However, even Python fails here. I would say it's "obvious" that you should use filter rather than the for/if combination. Perhaps to someone else it's "obvious" that you should use a list comprehension.
      - sigzero 16y ago
        If find that once you get beyond the "easy" in code there is rarely only "one obvious way" to do it.
- pyre 16y ago
  Yea. I don't agree with them ragging on the '$' in the Perl example. I could have made the Python code just as obtuse by using the regex library rather than word.endswith(). Ragging on a language because a beginner doesn't understand regular expressions seems a bit misleading. They are also inconsistent with their coding style. In the while() loop, they make use of $_ implicitly (even in the split statement in the foreach loop), but in the foreach loop, they don't use $_, but instead define $word. Then they go off on split being 'difficult to guess what it represents.' > Having used Perl ourselves in research and teaching > since the 1980s, we have found that Perl programs of > any size are inordinately difficult to maintain and > re-use. Says more about the programmer than the language. You can write maintainable code in any language. "Too many choices" doesn't cause code to be unmaintainable. Lack of discipline in your programming practices does (as well as lack of documentation). Saying that a language is better in this respect is just to say that "X language took the ability to choose away from me so that I'm forced to do things a certain way, whether I like it or not."
  - sigzero 16y ago
    I agree with you. Bad code isn't the programming languages fault. Bad code is the programmer's fault. Perl itself and the Perl culture has changed enormously since the "1980s".
jules 16y ago
His Ruby for line in ARGF for word in line.split if word.match(/ing$/) then puts word end end end I'd write as for line in ARGF puts line.split.grep(/ing$/) end Or puts ARGF.map{|line| line.split.grep(/ing$/)}
- epochwolf 16y ago
  Or ARGF.each {|line| line.scan(/\b\w+ing\b/).each(&:puts) } You know, why are we breaking this by line? :P ARGF.read.scan(/\b\w+ing\b/).each(&:puts) Probably requires ActiveSupport for some versions of ruby :) The &:symbol trick isn't in all versions.
- petercooper 16y ago
  The Python implementation doesn't use ARGF and so appears clearer. Luckily this is no problem in Ruby too: while line = gets puts line.split.select { |word| word =~ /ing$/ } end
  - swolchok 16y ago
    This is clearly not as bad as Perl, but it's still raving a bit. My key complaints are the complete lack of clarity as to what kind of thing split is, as well as what exactly is going on with select. (I'm assuming it's getting passed a predicate in the form of a lambda or block or whatever the cool kids are calling it these days, and it's obvious to the Perl programmer what the block does after that.) The equivalent Python is something like import sys for line in sys.stdin: print '\n'.join(filter(line.split(), lambda word: word.endswith('ing')) which I would argue is more readable to a layperson because function calls look like function calls. (By the way, does Ruby's puts automatically do the join with newlines, or what?)
    - petercooper 16y ago
      My key complaints are the complete lack of clarity as to what kind of thing split is The post seems to be about language independent readability. Rather than "functions should be called with () - that is clearly not a function - can not compute!!", the question is "can this code be understood even ignoring unfamiliar syntax?" Most developers familiar with languages like Java, JavaScript, Python, Perl, C# or Ruby could correctly guess what "split" refers to and that it is unlikely to be used as a variable name. Further, if a developer is familiar with the idea of method calls, it will be inferred that "whatever.split.whatever" is a sequence of method calls.. much as if it were, in some fictional languages, whatever->split->whatever or whatever:split:whatever. Similarly, a(b(c())) and a[b[c[]]] could both be easily inferred to be a set of nested function calls. "import sys" is not exactly clear either, but again, the meaning of this could be accurately inferred by most developers. print '\n'.join(filter(line.split(), lambda word: word.endswith('ing')) I want to join the array's elements together, not join the string. Joining together an array's elements by calling a method on the delimiter is no less raving than any Ruby I may cook up. Your example is both longer and has more syntax.
emef 16y ago
The entire time I was reading, I hoped that a Haskell solution would be there (knowing it would be much simpler), I got my wish :) +1 to haskell
- megaman821 16y ago
  Except for Prolog, I am familiar with all of the languages presented. Python and Haskell were the easiest for me to parse, not that any of them were overly hard.
  - eru 16y ago
    Perhaps more idiomatic: import Data.List main = putStr . unlines . filter ("ing" `isSuffixOf`) . words =<< getContents (To be read from right to left.) A Forth example would be interesting.
    - Avshalom 16y ago
      Here is a Factor version: : print-ing ( filepath encoding -- ) [ [ " " split [ dup "ing" tail? [ print ] [ drop ] if ] each ] each-line ] with-file-reader ; It's kind of ugly because it's designed to print out the values without leaving anything on the stack. It can be prettier if it produces an array of matching words : collect-ing ( filepath encoding -- seq ) { } -rot [ [ " " split [ "ing" tail? ] filter append ] each-line ] with-file-reader ; and then just prints that out : print-ing ( filepath encoding -- ) collect-ing [ print ] each ;
      - eru 16y ago
        Thanks!
    - deleted 16y ago
      [deleted]
mark_l_watson 16y ago
I use Ruby, and not Python. That said I still bought a print copy of this book a few years ago: nice book and the NLTK package has a lot of grate tools built in. Definitely "batteries included."
- mark_l_watson 16y ago
  The comments on why "my language is better than yours" could have been toned down however. The book is available at the same site: http://nltk.googlecode.com/svn/trunk/doc/book/book.html http://nltk.googlecode.com/svn/trunk/doc/book/book.html really nice work on both the book and the NLTK library.
zephyrfalcon 16y ago
For what it's worth, this could be written in one line of Io: File standardInput readToEnd split select(endsWithSeq("ing")) foreach(println) (Given that I don't actually use Io a lot, there might be shorter ways to do this.) Anyway, as usual, code samples prove very little. =)
lars512 16y ago
There's many dimensions on which we can evaluate programming languages, but the NLTK folk are only really interested in one: readability. Their page implicitly argues that high-level languages with good string processing are the most readable, and that amongst those Python is more readable than the alternatives (for both non-programmers and experts). NLTK is supposed to be an educational toolkit. It's used by linguists taking their first steps in programming, and by CS students taking their first steps in complexity and mess of human language. They're not looking for the shortest code, the fastest code, or the most <quality attribute X> code, just the most readable, insofar as readability can be supported and encouraged by a language.
- bradleyland 16y ago
  In that case, Python and Ruby must be front runners. I'm not sure why they put Ruby so close to the bottom, or why they used the obscure ARGF mechanism instead of the more readable examples elsewhere in this discussion. It seems as though the author may have an interest in promoting Python as the choice.
  - chromatic 16y ago
    It seems as though the author may have an interest in promoting Python as the choice. Clearly the author has made a strong assumption that the peculiarities of Python syntax and semantics (`import`, `sys.stdin`, `for` ... `in`) are somehow clear to Python novices.
tspiteri 16y ago
The C version has a buffer overflow if more than 1024 consecutive alphanumeric characters are input. And a much less serious point, isalnum(c) looks much better than (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
- pgbovine 16y ago
  yup, agreed, and if word.endswith('ing') looks much better than if word[-3] == 'i' and word[-2] == 'n' and word[-1] == 'g' clearly the authors of that article weren't proficient in the C standard library :)
10ren 16y ago
lua (a better version is welcome): for line in io.lines() do for word in line:gfind("[^%s]+") do if word:find("ing$") then print( word ) end end end
ekiru 16y ago
The C and Prolog examples solve a different problem than the others. The others split on either any whitespace or on only spaces. The Prolog example splits on whitespace and punctuation. The C example splits on anything that isn't alphanumeric.
tspiteri 16y ago
A C++ version: #include <iostream> #include <string> int main() { std::string s; while (std::cin >> s) { if (s.size() >= 3 && s.match(s.size()-3, 3, "ing") == 0) { std::cout << s << '\n'; } } }
- pgbovine 16y ago
  were you trying to make an argument as to how this compares to other languages? (sorry, i don't mean to sound trollish, i'm actually curious as to your intent)
  - tspiteri 16y ago
    If anything, this particular comment is to show that C and C++ are quite different although they have quite a bit in common.
eru 16y ago
"LISP is a so-called functional programming language, in which all objects are lists, and all operations are performed by (nested) functions of the form (function arg1 arg2 ...). " Reading this hurts.
pgbovine 16y ago
i love python more than anything else in the world (well, almost), but i think that this example is quite superficial ... this line alone gives python its enormous 'readability edge' over other languages: if word.endswith('ing') of course, it's great standard library design to have a string method called endswith() rather than making people use a regexp ending in '$', since finding suffixes is a common operation. but such a simple operation is hardly indicative of hardcore NLP (which would mostly be hidden in special-purpose library code anyways)
kroger 16y ago
The lisp code has a few problems. It's using a regex library that's only available in clisp (I think), it's not using the standard input like the other examples, and having two functions named has-suffix and has_suffix is no good. Also, it'll return an error if the string is shorter than the suffix. In the following example I'm using the portable http://www.cliki.net/SPLIT-SEQUENCE http://www.cliki.net/SPLIT-SEQUENCE to split the words: (defun endswith (string suffix) (let ((size (- (length string) (length suffix)))) (unless (minusp size) (equalp (subseq string size) suffix)))) (loop for line = (read-line *standard-input* nil) while line do (loop for word in (split-sequence #\Space line) do (if (endswith word "ing") (write-line word)))) It's still wordier than python, though.
urza 16y ago
C# Console.ReadLine().Split().Where( word => word.EndsWith("ing")) .ForEach( word => Console.WriteLine(word));
- deleted 16y ago
  [deleted]

32 points by rayvega 16y ago

Jun8 16y ago

I agree with everyone that Perl syntax can look random gibberish, however, their particular Perl example seems quite easy to interpret.

lars512 16y ago
Sure, it's roughly readable, but even here Perl makes several aspects of the loop implicit and buried in syntax (the line variable, stdin as source of the lines).
- pyre 16y ago
  Perl doesn't make it implicit, the programmer does. It could easily be rewritten as: foreach my $line (<STDIN>) { foreach my $word (split($line)) { print "$word\n" if $word =~ /ing$/; } } The programmer made a conscious decision to read STDIN implicitly in a while loop and then call split with implicit usage of $_, then went on to complain that too much was hidden away. I could make the python example equally obtuse: import sys, re for a in sys.fopen(sys.stdin.fileno(),'r',1): for b in re.split(a,r''): if re.search(b,r'.*ing$') print b Compressed forms: # PERL map { print "$_\n" } grep { /ing$/ } map { split($_) } <STDIN>; # PYTHON import sys for a in [ a for a in [ b for a in sys.stdin for b in a.split() ] if a.endswith('ing') ]: print a # PYTHON import sys print "\n".join(filter(lambda x: x.endswith('ing'), [ var2 for var1 in sys.stdin for var2 in var1.split() ]))
  - lars512 16y ago
    Individual programmer style does matter, but languages also guide our style. I'm often asked to convert Perl scripts to Python scripts, and the originals don't exhibit the "optional readability" you're advertising, and which Python encourages by default. An entirely different angle on all of this is that NLTK is aimed at novice programmers (linguists) as well as expert programmers (CS students). Perhaps readability is a different concept for these two user groups...
    - phaylon 16y ago
      Well, since Perl was first designed by a linguist, one might think they aren't all of one mindset :)
  - l0nwlf 16y ago
    "There should be one-- and preferably only one --obvious way to do it." and your code mylord does not passes the bill. Also I do agree with OP, python support readability and it make sense to choose it over other languages where clarity matters. Every language have their own use-case, python suits here quite well.
    - kscaldef 16y ago
      Well, of course, when you assume that the design criteria of Python are the correct ones you reach the conclusion that Python is the preferred language to use. However, even Python fails here. I would say it's "obvious" that you should use filter rather than the for/if combination. Perhaps to someone else it's "obvious" that you should use a list comprehension.
    - sigzero 16y ago
      If find that once you get beyond the "easy" in code there is rarely only "one obvious way" to do it.
pyre 16y ago
Yea. I don't agree with them ragging on the '$' in the Perl example. I could have made the Python code just as obtuse by using the regex library rather than word.endswith(). Ragging on a language because a beginner doesn't understand regular expressions seems a bit misleading. They are also inconsistent with their coding style. In the while() loop, they make use of $_ implicitly (even in the split statement in the foreach loop), but in the foreach loop, they don't use $_, but instead define $word. Then they go off on split being 'difficult to guess what it represents.' > Having used Perl ourselves in research and teaching > since the 1980s, we have found that Perl programs of > any size are inordinately difficult to maintain and > re-use. Says more about the programmer than the language. You can write maintainable code in any language. "Too many choices" doesn't cause code to be unmaintainable. Lack of discipline in your programming practices does (as well as lack of documentation). Saying that a language is better in this respect is just to say that "X language took the ability to choose away from me so that I'm forced to do things a certain way, whether I like it or not."
- sigzero 16y ago
  I agree with you. Bad code isn't the programming languages fault. Bad code is the programmer's fault. Perl itself and the Perl culture has changed enormously since the "1980s".

jules 16y ago

His Ruby for line in ARGF for word in line.split if word.match(/ing$/) then puts word end end end I'd write as for line in ARGF puts line.split.grep(/ing$/) end Or puts ARGF.map{|line| line.split.grep(/ing$/)}

epochwolf 16y ago
Or ARGF.each {|line| line.scan(/\b\w+ing\b/).each(&:puts) } You know, why are we breaking this by line? :P ARGF.read.scan(/\b\w+ing\b/).each(&:puts) Probably requires ActiveSupport for some versions of ruby :) The &:symbol trick isn't in all versions.
petercooper 16y ago
The Python implementation doesn't use ARGF and so appears clearer. Luckily this is no problem in Ruby too: while line = gets puts line.split.select { |word| word =~ /ing$/ } end
- swolchok 16y ago
  This is clearly not as bad as Perl, but it's still raving a bit. My key complaints are the complete lack of clarity as to what kind of thing split is, as well as what exactly is going on with select. (I'm assuming it's getting passed a predicate in the form of a lambda or block or whatever the cool kids are calling it these days, and it's obvious to the Perl programmer what the block does after that.) The equivalent Python is something like import sys for line in sys.stdin: print '\n'.join(filter(line.split(), lambda word: word.endswith('ing')) which I would argue is more readable to a layperson because function calls look like function calls. (By the way, does Ruby's puts automatically do the join with newlines, or what?)
  - petercooper 16y ago
    My key complaints are the complete lack of clarity as to what kind of thing split is The post seems to be about language independent readability. Rather than "functions should be called with () - that is clearly not a function - can not compute!!", the question is "can this code be understood even ignoring unfamiliar syntax?" Most developers familiar with languages like Java, JavaScript, Python, Perl, C# or Ruby could correctly guess what "split" refers to and that it is unlikely to be used as a variable name. Further, if a developer is familiar with the idea of method calls, it will be inferred that "whatever.split.whatever" is a sequence of method calls.. much as if it were, in some fictional languages, whatever->split->whatever or whatever:split:whatever. Similarly, a(b(c())) and a[b[c[]]] could both be easily inferred to be a set of nested function calls. "import sys" is not exactly clear either, but again, the meaning of this could be accurately inferred by most developers. print '\n'.join(filter(line.split(), lambda word: word.endswith('ing')) I want to join the array's elements together, not join the string. Joining together an array's elements by calling a method on the delimiter is no less raving than any Ruby I may cook up. Your example is both longer and has more syntax.

emef 16y ago

The entire time I was reading, I hoped that a Haskell solution would be there (knowing it would be much simpler), I got my wish :) +1 to haskell

megaman821 16y ago
Except for Prolog, I am familiar with all of the languages presented. Python and Haskell were the easiest for me to parse, not that any of them were overly hard.
- eru 16y ago
  Perhaps more idiomatic: import Data.List main = putStr . unlines . filter ("ing" `isSuffixOf`) . words =<< getContents (To be read from right to left.) A Forth example would be interesting.
  - Avshalom 16y ago
    Here is a Factor version: : print-ing ( filepath encoding -- ) [ [ " " split [ dup "ing" tail? [ print ] [ drop ] if ] each ] each-line ] with-file-reader ; It's kind of ugly because it's designed to print out the values without leaving anything on the stack. It can be prettier if it produces an array of matching words : collect-ing ( filepath encoding -- seq ) { } -rot [ [ " " split [ "ing" tail? ] filter append ] each-line ] with-file-reader ; and then just prints that out : print-ing ( filepath encoding -- ) collect-ing [ print ] each ;
    - eru 16y ago
      Thanks!
  - deleted 16y ago
    [deleted]

mark_l_watson 16y ago

I use Ruby, and not Python. That said I still bought a print copy of this book a few years ago: nice book and the NLTK package has a lot of grate tools built in. Definitely "batteries included."

mark_l_watson 16y ago
The comments on why "my language is better than yours" could have been toned down however. The book is available at the same site: http://nltk.googlecode.com/svn/trunk/doc/book/book.html http://nltk.googlecode.com/svn/trunk/doc/book/book.html really nice work on both the book and the NLTK library.

zephyrfalcon 16y ago

For what it's worth, this could be written in one line of Io: File standardInput readToEnd split select(endsWithSeq("ing")) foreach(println) (Given that I don't actually use Io a lot, there might be shorter ways to do this.) Anyway, as usual, code samples prove very little. =)

lars512 16y ago

There's many dimensions on which we can evaluate programming languages, but the NLTK folk are only really interested in one: readability. Their page implicitly argues that high-level languages with good string processing are the most readable, and that amongst those Python is more readable than the alternatives (for both non-programmers and experts). NLTK is supposed to be an educational toolkit. It's used by linguists taking their first steps in programming, and by CS students taking their first steps in complexity and mess of human language. They're not looking for the shortest code, the fastest code, or the most <quality attribute X> code, just the most readable, insofar as readability can be supported and encouraged by a language.

bradleyland 16y ago
In that case, Python and Ruby must be front runners. I'm not sure why they put Ruby so close to the bottom, or why they used the obscure ARGF mechanism instead of the more readable examples elsewhere in this discussion. It seems as though the author may have an interest in promoting Python as the choice.
- chromatic 16y ago
  It seems as though the author may have an interest in promoting Python as the choice. Clearly the author has made a strong assumption that the peculiarities of Python syntax and semantics (`import`, `sys.stdin`, `for` ... `in`) are somehow clear to Python novices.

tspiteri 16y ago

The C version has a buffer overflow if more than 1024 consecutive alphanumeric characters are input. And a much less serious point, isalnum(c) looks much better than (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')

pgbovine 16y ago
yup, agreed, and if word.endswith('ing') looks much better than if word[-3] == 'i' and word[-2] == 'n' and word[-1] == 'g' clearly the authors of that article weren't proficient in the C standard library :)

10ren 16y ago

lua (a better version is welcome): for line in io.lines() do for word in line:gfind("[^%s]+") do if word:find("ing$") then print( word ) end end end

ekiru 16y ago

The C and Prolog examples solve a different problem than the others. The others split on either any whitespace or on only spaces. The Prolog example splits on whitespace and punctuation. The C example splits on anything that isn't alphanumeric.

tspiteri 16y ago

A C++ version: #include <iostream> #include <string> int main() { std::string s; while (std::cin >> s) { if (s.size() >= 3 && s.match(s.size()-3, 3, "ing") == 0) { std::cout << s << '\n'; } } }

pgbovine 16y ago
were you trying to make an argument as to how this compares to other languages? (sorry, i don't mean to sound trollish, i'm actually curious as to your intent)
- tspiteri 16y ago
  If anything, this particular comment is to show that C and C++ are quite different although they have quite a bit in common.

eru 16y ago

"LISP is a so-called functional programming language, in which all objects are lists, and all operations are performed by (nested) functions of the form (function arg1 arg2 ...). " Reading this hurts.

pgbovine 16y ago

i love python more than anything else in the world (well, almost), but i think that this example is quite superficial ... this line alone gives python its enormous 'readability edge' over other languages: if word.endswith('ing') of course, it's great standard library design to have a string method called endswith() rather than making people use a regexp ending in '$', since finding suffixes is a common operation. but such a simple operation is hardly indicative of hardcore NLP (which would mostly be hidden in special-purpose library code anyways)

kroger 16y ago

The lisp code has a few problems. It's using a regex library that's only available in clisp (I think), it's not using the standard input like the other examples, and having two functions named has-suffix and has_suffix is no good. Also, it'll return an error if the string is shorter than the suffix. In the following example I'm using the portable http://www.cliki.net/SPLIT-SEQUENCE http://www.cliki.net/SPLIT-SEQUENCE to split the words: (defun endswith (string suffix) (let ((size (- (length string) (length suffix)))) (unless (minusp size) (equalp (subseq string size) suffix)))) (loop for line = (read-line *standard-input* nil) while line do (loop for word in (split-sequence #\Space line) do (if (endswith word "ing") (write-line word)))) It's still wordier than python, though.

urza 16y ago

C# Console.ReadLine().Split().Where( word => word.EndsWith("ing")) .ForEach( word => Console.WriteLine(word));

deleted 16y ago
[deleted]