4 ms·
However, most Python programmers tend to just read the entire file into one huge string and process it that way. I feel funny doing that. Having used machin
by jeremya 15y ago
However, most Python programmers tend to just read the
entire file into one huge string and process it that way.
I feel funny doing that. Having used machines with very
limited amounts of RAM, I tend to try to keep my file
processing to a single line at a time. However, that
method is going the way of the dodo.
contents = file('filename.txt').read()
all_input = sys.stdin.read()
This is not the correct way to do things. The first example given is better.
for line in file('filename.txt'):
print line
Generators are a beautiful thing [1][2]
[1] http://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python http://stackoverflow.com/questions/519633/lazy-method-for-re...
[2] http://www.dabeaz.com/generators/ http://www.dabeaz.com/generators/
- ajross 15y agoNo doubt. It must be noted, though, that for a very-well-thought-of, modern language: python has a staggeringly large number of "incorrect" ways to do things.
- celer 15y agoPython has a design ethic that most other languages don't have. There are very few, if any, good synonyms for 'Pythonic.'
- eblume 15y agoThough the results may have changed since, I once had to parse a very, very large file (about 5 GB) line-by-line. I tested a number of different methods, and ultimately discovered that reading the entire file using .readlines() was faster by a significant margin. I vaguely recall it was more than four times faster, actually - and that was a really big win when you're talking about a data set that large. Granted, our machine had a perhaps-unusual design in that it had multiple RAID 6 data banks, a very powerful dedicated IO controller, and about 128 GB of RAM. So your mileage may vary. Unfortunately I don't have access to that machine any more so I can't perform the same benchmark, but I have a nagging feeling that if I ran the tests again with Python 3.2 I'd get different results.
- lucian1900 15y agoThat's most likely because of the overhead of all those syscalls for each line. A nicer way to do such things is to open the file with mmap, which does one system call and then just bumps a pointer. mmaps expose both a file and a list-like API.
- deleted 15y ago[deleted]