4 ms·

The scope of index variables in Python's for loops

21 points by zoidb 12y ago

acadien 12y ago
Yes another super useful trick you can use with this scope trick is to get the length of a file (or its last line) just do: for i,v in enumerate(open(f)): pass now 'i' is the number of lines in your file and 'v' is the last line. This is also blazingly fast, especially useful when you have ~several GB large log files to parse. Great article!
- potash 12y ago
  Note that your code still forces python to read the entire file! If all you want is the last line of a file, use the UNIX tail: tail -n 1 $f Tail seeks backwards so it will only read the one line. Of course, this won't give you a line count. EDIT: I haven't tested it but you might be interested in this implementation of tail in python: http://stackoverflow.com/a/136368 http://stackoverflow.com/a/136368
  - acadien 12y ago
    I've done timing tests and it is typically on par with tail & wc. The bulk of the time is wasted reading your file into ram, the time it takes to count the lines is essentially 0. Edit: Of course I misspoke, yes tail is much faster for getting the last line of the file! I meant for getting the line count the loop methods is typically just ~5% slower than wc on sufficiently large files.
- cnvogel 12y ago
  It's a nice trick, but it's not too "blazingly" fast. In [73]: cProfile.run('for i,v in enumerate(open(fn)): pass\nprint(i)') 15496799 406215 function calls in 10.998 seconds If you really don't want to do anything with the content of the file, better not spend time splitting lines, just read in the data in nice, big chunks. def file_block_count_lf(f) : while True : k = f.read(1<<20) if k == '' : break yield k.count('\n') In [77]: cProfile.run('print(sum(file_block_count_lf(open(fn))))') 15496800 10977 function calls in 6.479 seconds On the other hand, I think sum(1 for l in open(fn)) is a little more pythonesque, and can also be written in one single expression, if you must. It's a little slower, though. In [70]: cProfile.run('sum(1 for l in open(fn))') 15903016 function calls in 14.409 seconds Besides cProfile, also "timeit" is a useful python module for this kind of micro benchmarks. (the long output of the profiler is a little more interesting, I've put it in a pastebin http://pastebin.com/WXX93sS8 http://pastebin.com/WXX93sS8 ) EDIT/ADD: There's an awful lot time spent in decoding the characters to unicode strings! def file_block_count_lf(f) : while True : k = f.read(1<<20) if not k : break yield k.count(b'\n') In [48]: cProfile.run("sum(file_block_count_lf(open('more_logs.txt','rb')))") 4768 function calls in 2.969 seconds
  - acadien 12y ago
    When you do your timing tests make sure to do it on different files to make sure you aren't taking advantage of caching your system may be doing. That may explain some of your timing discrepancies. And when I say blazingly fast I mean that these pyscripts are nearly as fast as wc/tail. Edit; I just tested your code and my own and against a set of files of uniform size. time wc -l GIANTFILE1 -> 16.7 seconds cProfile.run("for i,v in enumerate(open("GIANTFILE2")):pass") -> 18.1 seconds cProfile.run("sum(file_block_count_lf(open('GIANTFILE3','rb')))") -> 22.9 seconds
    - cnvogel 12y ago
      Obviously, when comparing different methods of computation with negligible cache/memory footprint, I'd run it on hot caches to maximize the visible effect on the implementation. File here was about 1.5 GB on a 4GB RAM machine, and I repeated each test to have a stable duration for each. Would I have tested, say, different patterns of data access (reading files sequentially, parallel read from multiple disks/partitions), of course I would have made sure that either caches are cleared (/proc/sys/vm/drop_caches), datasets will exceed memory, and actual CPU load of the algorithm is according to the real-world-usage scenario (pure read() throughput is meaningless when CPU is bogged, and L1-3 caches swamped by actual algorithmic work). Re: your additional tests: Interesting, what python-version and size of files are you running it on?
      - acadien 12y ago
        I'm trapped using 2.7! On this machine it looks like I'm running 2.7.8. Files are roughly 1GB each. They're on a relatively slow file system, just a single HD on this machine... no raid, no ssd.
nhaehnle 12y ago
Somewhat tangentially related, the for-else-statement of Python is something that I occasionally miss in every other language that I seriously use. And sometimes I really wish that the scope of variables inside a do-while-loop in C-ish languages would extend into the loop condition of the final while...
Animats 12y ago
One problem with implicit declaration is that it's sometimes hard to decide whether some construct should create a new variable or not. Python's basic rule, that everything has function scope, was at least simple. I'm surprised that was changed for list comprehensions. Go has worse problems. The ":=" operator in Go indicates the creation of a new variable. Except when there are multiple variables on the left side of the :=. a, b := 1,2 can declare a, b, or a and b, but not neither of a and b. (See https://play.golang.org/p/zE7AIMhleo https://play.golang.org/p/zE7AIMhleo). This can create some hard to find shadowing bugs. Declaration-free language are hard. It's impressive how far Python got without declarations. Now there's an advisory typing proposal for Python. It's not a very good proposal; some of the type info goes in comments because it didn't fit the syntax.
- aroberge 12y ago
  > It's not a very good proposal; some of the type info goes in comments because it didn't fit the syntax. Actually, the reason why it is put in the comments is that it is implemented as an experimental feature so that its usefulness can be evaluated without having to (possibly only temporarily) change the syntax.
JeremyBanks 12y ago
The Python 2 list comprehension behaviour could be used to assign local variables within lambda expressions. Disgusting abuse, but fun: http://stackoverflow.com/a/14617232/1114 http://stackoverflow.com/a/14617232/1114
cnvogel 12y ago
Probably this stems from the typical C-way of error-reporting out of a for-loop, which is to terminate earlier than the "regular loop condition", and hence a second test for the loop-condition can easily be used to check for abnormal termination... int i; for (i=0; i<length; i++) { if (is_broken(element[i])) break; process(element[i]); } /* if everything went fine, i == length, else we broke out early */ if (i < length) { fprintf(stderr,"Something was broken."); return -1; } Whereas the Python version obviously doesn't make sense in a pythonesque program iterating over sets of elements. for i in [1,2,3] : do_something(i) # ..now i is 3, not 4... obviously
leoh 12y ago
I found this example most surprising: >>> def foo(): ... lst = [] ... for i in range(4): ... lst.append(lambda: i) ... print([f() for f in lst]) ... >>> foo() [3, 3, 3, 3] ...So I looked into this a bit more. It appears that the lambda creates a closure for i, but i is defined in the outer scope and we capture the reference for i, rather than i's value. So we can get around this: >>> def foo(): ... lst = [] ... for i in range(4): ... lst.append((lambda a: lambda: a)(i)) ... print([f() for f in lst]) ... >>> foo() [0, 1, 2, 3] It appears that i is dereferenced when passed to the first lambda constructor, which provides i's dereferenced value as a in a scope provided for the inner lambda. Please correct me if I am wrong, anyone?! --------Edit 2-------- Out of curiosity, I looked at this in Clojure and the way it creates lambdas is similar to Python (captures reference as opposed to value). user=> (def a 1) #'user/a user=> (def f (fn [] a)) #'user/f user=> (f) 1 user=> (def a 2) #'user/a user=> (f) 2 You could do something like this: user=> (def f ((fn [x] (fn [] x)) a)) #'user/f user=> (def a 1) #'user/a user=> (def f ((fn [x] (fn [] x)) a)) #'user/f user=> (f) 1 user=> (def a 10) #'user/a user=> (f) 1 But perhaps there is another way as well.
- rhizome31 12y ago
  You can capture the index with a keyword argument default value: def foo(): lst = [] for i in range(4): lst.append(lambda i=i: i) print([f() for f in lst]) FWIW I played around with this issue in various languages in this blog post: https://my.smeuh.org/al/blog/lost-in-scope https://my.smeuh.org/al/blog/lost-in-scope Once you get the difference between block scope and function scope, it's quite easy to see what's happening.
- SeanLuke 12y ago
  Common Lisp: (setf *a* nil) (dotimes (x 3) (push (lambda () x) *a*)) (mapcar #'funcall *a*) prints (3 3 3 3) as well. I think the author is mixing up variable capture with scoping.

21 points by zoidb 12y ago

Yes another super useful trick you can use with this scope trick is to get the length of a file (or its last line) just do: for i,v in enumerate(open(f)): pass now 'i' is the number of lines in your file and 'v' is the last line. This is also blazingly fast, especially useful when you have ~several GB large log files to parse. Great article!

potash 12y ago
Note that your code still forces python to read the entire file! If all you want is the last line of a file, use the UNIX tail: tail -n 1 $f Tail seeks backwards so it will only read the one line. Of course, this won't give you a line count. EDIT: I haven't tested it but you might be interested in this implementation of tail in python: http://stackoverflow.com/a/136368 http://stackoverflow.com/a/136368
- acadien 12y ago
  I've done timing tests and it is typically on par with tail & wc. The bulk of the time is wasted reading your file into ram, the time it takes to count the lines is essentially 0. Edit: Of course I misspoke, yes tail is much faster for getting the last line of the file! I meant for getting the line count the loop methods is typically just ~5% slower than wc on sufficiently large files.
cnvogel 12y ago
It's a nice trick, but it's not too "blazingly" fast. In [73]: cProfile.run('for i,v in enumerate(open(fn)): pass\nprint(i)') 15496799 406215 function calls in 10.998 seconds If you really don't want to do anything with the content of the file, better not spend time splitting lines, just read in the data in nice, big chunks. def file_block_count_lf(f) : while True : k = f.read(1<<20) if k == '' : break yield k.count('\n') In [77]: cProfile.run('print(sum(file_block_count_lf(open(fn))))') 15496800 10977 function calls in 6.479 seconds On the other hand, I think sum(1 for l in open(fn)) is a little more pythonesque, and can also be written in one single expression, if you must. It's a little slower, though. In [70]: cProfile.run('sum(1 for l in open(fn))') 15903016 function calls in 14.409 seconds Besides cProfile, also "timeit" is a useful python module for this kind of micro benchmarks. (the long output of the profiler is a little more interesting, I've put it in a pastebin http://pastebin.com/WXX93sS8 http://pastebin.com/WXX93sS8 ) EDIT/ADD: There's an awful lot time spent in decoding the characters to unicode strings! def file_block_count_lf(f) : while True : k = f.read(1<<20) if not k : break yield k.count(b'\n') In [48]: cProfile.run("sum(file_block_count_lf(open('more_logs.txt','rb')))") 4768 function calls in 2.969 seconds
- acadien 12y ago
  When you do your timing tests make sure to do it on different files to make sure you aren't taking advantage of caching your system may be doing. That may explain some of your timing discrepancies. And when I say blazingly fast I mean that these pyscripts are nearly as fast as wc/tail. Edit; I just tested your code and my own and against a set of files of uniform size. time wc -l GIANTFILE1 -> 16.7 seconds cProfile.run("for i,v in enumerate(open("GIANTFILE2")):pass") -> 18.1 seconds cProfile.run("sum(file_block_count_lf(open('GIANTFILE3','rb')))") -> 22.9 seconds
  - cnvogel 12y ago
    Obviously, when comparing different methods of computation with negligible cache/memory footprint, I'd run it on hot caches to maximize the visible effect on the implementation. File here was about 1.5 GB on a 4GB RAM machine, and I repeated each test to have a stable duration for each. Would I have tested, say, different patterns of data access (reading files sequentially, parallel read from multiple disks/partitions), of course I would have made sure that either caches are cleared (/proc/sys/vm/drop_caches), datasets will exceed memory, and actual CPU load of the algorithm is according to the real-world-usage scenario (pure read() throughput is meaningless when CPU is bogged, and L1-3 caches swamped by actual algorithmic work). Re: your additional tests: Interesting, what python-version and size of files are you running it on?
    - acadien 12y ago
      I'm trapped using 2.7! On this machine it looks like I'm running 2.7.8. Files are roughly 1GB each. They're on a relatively slow file system, just a single HD on this machine... no raid, no ssd.

nhaehnle 12y ago

Somewhat tangentially related, the for-else-statement of Python is something that I occasionally miss in every other language that I seriously use. And sometimes I really wish that the scope of variables inside a do-while-loop in C-ish languages would extend into the loop condition of the final while...

Animats 12y ago

One problem with implicit declaration is that it's sometimes hard to decide whether some construct should create a new variable or not. Python's basic rule, that everything has function scope, was at least simple. I'm surprised that was changed for list comprehensions. Go has worse problems. The ":=" operator in Go indicates the creation of a new variable. Except when there are multiple variables on the left side of the :=. a, b := 1,2 can declare a, b, or a and b, but not neither of a and b. (See https://play.golang.org/p/zE7AIMhleo https://play.golang.org/p/zE7AIMhleo). This can create some hard to find shadowing bugs. Declaration-free language are hard. It's impressive how far Python got without declarations. Now there's an advisory typing proposal for Python. It's not a very good proposal; some of the type info goes in comments because it didn't fit the syntax.

aroberge 12y ago
> It's not a very good proposal; some of the type info goes in comments because it didn't fit the syntax. Actually, the reason why it is put in the comments is that it is implemented as an experimental feature so that its usefulness can be evaluated without having to (possibly only temporarily) change the syntax.

JeremyBanks 12y ago

The Python 2 list comprehension behaviour could be used to assign local variables within lambda expressions. Disgusting abuse, but fun: http://stackoverflow.com/a/14617232/1114 http://stackoverflow.com/a/14617232/1114

cnvogel 12y ago

Probably this stems from the typical C-way of error-reporting out of a for-loop, which is to terminate earlier than the "regular loop condition", and hence a second test for the loop-condition can easily be used to check for abnormal termination... int i; for (i=0; i<length; i++) { if (is_broken(element[i])) break; process(element[i]); } /* if everything went fine, i == length, else we broke out early */ if (i < length) { fprintf(stderr,"Something was broken."); return -1; } Whereas the Python version obviously doesn't make sense in a pythonesque program iterating over sets of elements. for i in [1,2,3] : do_something(i) # ..now i is 3, not 4... obviously

leoh 12y ago

I found this example most surprising: >>> def foo(): ... lst = [] ... for i in range(4): ... lst.append(lambda: i) ... print([f() for f in lst]) ... >>> foo() [3, 3, 3, 3] ...So I looked into this a bit more. It appears that the lambda creates a closure for i, but i is defined in the outer scope and we capture the reference for i, rather than i's value. So we can get around this: >>> def foo(): ... lst = [] ... for i in range(4): ... lst.append((lambda a: lambda: a)(i)) ... print([f() for f in lst]) ... >>> foo() [0, 1, 2, 3] It appears that i is dereferenced when passed to the first lambda constructor, which provides i's dereferenced value as a in a scope provided for the inner lambda. Please correct me if I am wrong, anyone?! --------Edit 2-------- Out of curiosity, I looked at this in Clojure and the way it creates lambdas is similar to Python (captures reference as opposed to value). user=> (def a 1) #'user/a user=> (def f (fn [] a)) #'user/f user=> (f) 1 user=> (def a 2) #'user/a user=> (f) 2 You could do something like this: user=> (def f ((fn [x] (fn [] x)) a)) #'user/f user=> (def a 1) #'user/a user=> (def f ((fn [x] (fn [] x)) a)) #'user/f user=> (f) 1 user=> (def a 10) #'user/a user=> (f) 1 But perhaps there is another way as well.

rhizome31 12y ago
You can capture the index with a keyword argument default value: def foo(): lst = [] for i in range(4): lst.append(lambda i=i: i) print([f() for f in lst]) FWIW I played around with this issue in various languages in this blog post: https://my.smeuh.org/al/blog/lost-in-scope https://my.smeuh.org/al/blog/lost-in-scope Once you get the difference between block scope and function scope, it's quite easy to see what's happening.
SeanLuke 12y ago
Common Lisp: (setf *a* nil) (dotimes (x 3) (push (lambda () x) *a*)) (mapcar #'funcall *a*) prints (3 3 3 3) as well. I think the author is mixing up variable capture with scoping.