I have run into this problem more than once. Usually I am not creating the initial data, but am importing it. Usually I have ignored optimizations initially a

I have run into this problem more than once. Usually I am not creating the initial data, but am importing it. Usually I have ignored optimizations initially and just loaded and processed the data. Then I go back and hack on the subroutines loading the data and splitting it into records. I was the only programmer working on both projects though. Once I did it with a large RDF (XML) that was a list of books. Initially I loaded and processed it in Perl with XML::Simple, but XML::Twig let me deal with things one record at a time. The other time I had a large Java linked list holding objects, where I grabbed the head, processed it, and then shrunk the list (removeFirst()) as I proceeded (I probably could have optimized the loading even more). As a one-person project, punting on the big load optimization worked for me, as I had a working program even before optimizing. If it was a larger project, what time would be best to optimize might have differed.