3 ms·

The flip side is that sometimes unnecessary optimizations can slow progress towards the actual goal, especially if the data transformations are sophisticated ma

The flip side is that sometimes unnecessary optimizations can slow progress towards the actual goal, especially if the data transformations are sophisticated math.

Ologn 5y ago
I have run into this problem more than once. Usually I am not creating the initial data, but am importing it. Usually I have ignored optimizations initially and just loaded and processed the data. Then I go back and hack on the subroutines loading the data and splitting it into records. I was the only programmer working on both projects though. Once I did it with a large RDF (XML) that was a list of books. Initially I loaded and processed it in Perl with XML::Simple, but XML::Twig let me deal with things one record at a time. The other time I had a large Java linked list holding objects, where I grabbed the head, processed it, and then shrunk the list (removeFirst()) as I proceeded (I probably could have optimized the loading even more). As a one-person project, punting on the big load optimization worked for me, as I had a working program even before optimizing. If it was a larger project, what time would be best to optimize might have differed.
jka 5y ago
In the context of high-performance, revenue-sensitive data transformations, it'd be interesting to see how reusable free/open-source components would fare, especially with regard to licensing. I can see MIT/permissive licenses being less attractive than usual in an environment like that, because the author might perceive reduced chance of reaping rewards (versus aiding competitors) from publishing components. If improvements led to greater-than-zero reward for publishers and consumers alike, though (as enforced by copyleft approaches), then perhaps the results would be different. All conjecture, really.

3 ms·

The flip side is that sometimes unnecessary optimizations can slow progress towards the actual goal, especially if the data transformations are sophisticated ma

by alpineidyll3 5y ago

The flip side is that sometimes unnecessary optimizations can slow progress towards the actual goal, especially if the data transformations are sophisticated math.

Ologn 5y ago
I have run into this problem more than once. Usually I am not creating the initial data, but am importing it. Usually I have ignored optimizations initially and just loaded and processed the data. Then I go back and hack on the subroutines loading the data and splitting it into records. I was the only programmer working on both projects though. Once I did it with a large RDF (XML) that was a list of books. Initially I loaded and processed it in Perl with XML::Simple, but XML::Twig let me deal with things one record at a time. The other time I had a large Java linked list holding objects, where I grabbed the head, processed it, and then shrunk the list (removeFirst()) as I proceeded (I probably could have optimized the loading even more). As a one-person project, punting on the big load optimization worked for me, as I had a working program even before optimizing. If it was a larger project, what time would be best to optimize might have differed.
jka 5y ago
In the context of high-performance, revenue-sensitive data transformations, it'd be interesting to see how reusable free/open-source components would fare, especially with regard to licensing. I can see MIT/permissive licenses being less attractive than usual in an environment like that, because the author might perceive reduced chance of reaping rewards (versus aiding competitors) from publishing components. If improvements led to greater-than-zero reward for publishers and consumers alike, though (as enforced by copyleft approaches), then perhaps the results would be different. All conjecture, really.