3 ms·
When you do your timing tests make sure to do it on different files to make sure you aren't taking advantage of caching your system may be doing. That may expla
by acadien 12y ago
When you do your timing tests make sure to do it on different files to make sure you aren't taking advantage of caching your system may be doing. That may explain some of your timing discrepancies. And when I say blazingly fast I mean that these pyscripts are nearly as fast as wc/tail.
Edit; I just tested your code and my own and against a set of files of uniform size.
time wc -l GIANTFILE1 -> 16.7 seconds
cProfile.run("for i,v in enumerate(open("GIANTFILE2")):pass") -> 18.1 seconds
cProfile.run("sum(file_block_count_lf(open('GIANTFILE3','rb')))") -> 22.9 seconds
- cnvogel 12y agoObviously, when comparing different methods of computation with negligible cache/memory footprint, I'd run it on hot caches to maximize the visible effect on the implementation. File here was about 1.5 GB on a 4GB RAM machine, and I repeated each test to have a stable duration for each. Would I have tested, say, different patterns of data access (reading files sequentially, parallel read from multiple disks/partitions), of course I would have made sure that either caches are cleared (/proc/sys/vm/drop_caches), datasets will exceed memory, and actual CPU load of the algorithm is according to the real-world-usage scenario (pure read() throughput is meaningless when CPU is bogged, and L1-3 caches swamped by actual algorithmic work). Re: your additional tests: Interesting, what python-version and size of files are you running it on?
- acadien 12y agoI'm trapped using 2.7! On this machine it looks like I'm running 2.7.8. Files are roughly 1GB each. They're on a relatively slow file system, just a single HD on this machine... no raid, no ssd.