Yes that's a classic tradeoff, a proficient programmer will have to pick one. Personally I always read entire files into memory first unless I have reason to b

Yes that's a classic tradeoff, a proficient programmer will have to pick one. Personally I always read entire files into memory first unless I have reason to believe memory will be an issue or need to program defensively against malicious/careless input. The code is always much cleaner and easier to read and if you need to do a second pass on the data you don't need to re-read it from disk.