4 ms·
It's a nice trick, but it's not too "blazingly" fast. In [73]: cProfile.run('for i,v in enumerate(open(fn)): pass\nprint(i)') 15496799 406215 funct
by cnvogel 12y ago
It's a nice trick, but it's not too "blazingly" fast.
In [73]: cProfile.run('for i,v in enumerate(open(fn)): pass\nprint(i)')
15496799
406215 function calls in 10.998 seconds
If you really don't want to do anything with the content of the file, better not spend time splitting lines, just read in the data in nice, big chunks.
def file_block_count_lf(f) :
while True :
k = f.read(1<<20)
if k == '' :
break
yield k.count('\n')
In [77]: cProfile.run('print(sum(file_block_count_lf(open(fn))))')
15496800
10977 function calls in 6.479 seconds
On the other hand, I think sum(1 for l in open(fn)) is a little more pythonesque, and can also be written in one single expression, if you must. It's a little slower, though.
In [70]: cProfile.run('sum(1 for l in open(fn))')
15903016 function calls in 14.409 seconds
Besides cProfile, also "timeit" is a useful python module for this kind of micro benchmarks.
(the long output of the profiler is a little more interesting, I've put it in a pastebin http://pastebin.com/WXX93sS8 http://pastebin.com/WXX93sS8 )
EDIT/ADD: There's an awful lot time spent in decoding the characters to unicode strings!
def file_block_count_lf(f) :
while True :
k = f.read(1<<20)
if not k :
break
yield k.count(b'\n')
In [48]: cProfile.run("sum(file_block_count_lf(open('more_logs.txt','rb')))")
4768 function calls in 2.969 seconds
- acadien 12y agoWhen you do your timing tests make sure to do it on different files to make sure you aren't taking advantage of caching your system may be doing. That may explain some of your timing discrepancies. And when I say blazingly fast I mean that these pyscripts are nearly as fast as wc/tail. Edit; I just tested your code and my own and against a set of files of uniform size. time wc -l GIANTFILE1 -> 16.7 seconds cProfile.run("for i,v in enumerate(open("GIANTFILE2")):pass") -> 18.1 seconds cProfile.run("sum(file_block_count_lf(open('GIANTFILE3','rb')))") -> 22.9 seconds
- cnvogel 12y agoObviously, when comparing different methods of computation with negligible cache/memory footprint, I'd run it on hot caches to maximize the visible effect on the implementation. File here was about 1.5 GB on a 4GB RAM machine, and I repeated each test to have a stable duration for each. Would I have tested, say, different patterns of data access (reading files sequentially, parallel read from multiple disks/partitions), of course I would have made sure that either caches are cleared (/proc/sys/vm/drop_caches), datasets will exceed memory, and actual CPU load of the algorithm is according to the real-world-usage scenario (pure read() throughput is meaningless when CPU is bogged, and L1-3 caches swamped by actual algorithmic work). Re: your additional tests: Interesting, what python-version and size of files are you running it on?
- acadien 12y agoI'm trapped using 2.7! On this machine it looks like I'm running 2.7.8. Files are roughly 1GB each. They're on a relatively slow file system, just a single HD on this machine... no raid, no ssd.