I've noticed this in some code I've been refactoring at work recently (a feed generation system). It's easy in Perl to suck in a large amount of data into one a

I've noticed this in some code I've been refactoring at work recently (a feed generation system). It's easy in Perl to suck in a large amount of data into one array and map/grep it into another array of hashes and output filtered/formatted data in a few lines. The problem of course is that your business'/project's own growth overwhelms the algorithm as the data structure balloons. This is further compounded by libraries which are designed to accept an array as input (reinforcing the idiom causing the first problem) since you end up with thousands of additional iterations. I refactored the code to lazy load each item in the collection and map/filter/generate the XML at the end of each node's iteration. The end result is a nice flat memory space (save for the odd leak here and there) that's about 1/10th of the original program's, at the 'expense' of only a few more lines of code.