3 ms·

I wrote the article right after I fixed a huge inefficiency problem in a function within my own project. I neglected to give the specifics in the article, but h

I wrote the article right after I fixed a huge inefficiency problem in a function within my own project. I neglected to give the specifics in the article, but here they are since you asked. My Didgets tool lets you create pivot tables against relational database tables, even very large ones. For the pivot values, you can choose to just count the occurrence of each value or if it is a number type you can add them up. You can also add up the values in a separate number column. Here is a quick demo video: https://www.youtube.com/watch?v=2ScBd-71OLQ https://www.youtube.com/watch?v=2ScBd-71OLQ When adding up numbers in a separate column, I had just a few lines of unnecessary code that ended up being called exponentially. For smaller tables it was barely noticeable, but for tables with 30 million+ rows it really bogged down. A simple fix to the affected lines caused a certain test against a large table to go from over 10 minutes down to under 20 seconds. The effects of just a few lines of code when applied to a big enough data set can really impact performance. It is the old Einstein equation E=mc2 in effect which is discussed here: https://didgets.substack.com/p/musings-from-an-old-programmer https://didgets.substack.com/p/musings-from-an-old-programme...

shurane 4y ago
I guess there is a lost art of writing for optimal code/memory/execution time, especially as our resources increase. I think the idea here is to write code quickly that's inefficient, and re-write it to be efficient if the performance is required down the line. For companies where there's bigger fish to fry, i.e. customer acquisition, it's more useful to pump out more features (even at the expense of bugs) because that draws customers. But in places where performance is important, you do see developers squeeze out more cycles/memory. I.e. kernel/OS development, database servers, video games. It's just that most developers aren't in those areas of specialty anymore. Btw, have you heard of https://handmade.network/ https://handmade.network/ and https://en.wikipedia.org/wiki/Demoscene https://en.wikipedia.org/wiki/Demoscene ? Wondering what your thoughts are in those areas. There are probably more communities like the ones I mentioned, where developers are interested in writing the kind of code that you are talking about.

shurane 4y ago

I guess there is a lost art of writing for optimal code/memory/execution time, especially as our resources increase. I think the idea here is to write code quickly that's inefficient, and re-write it to be efficient if the performance is required down the line. For companies where there's bigger fish to fry, i.e. customer acquisition, it's more useful to pump out more features (even at the expense of bugs) because that draws customers. But in places where performance is important, you do see developers squeeze out more cycles/memory. I.e. kernel/OS development, database servers, video games. It's just that most developers aren't in those areas of specialty anymore. Btw, have you heard of https://handmade.network/ https://handmade.network/ and https://en.wikipedia.org/wiki/Demoscene https://en.wikipedia.org/wiki/Demoscene ? Wondering what your thoughts are in those areas. There are probably more communities like the ones I mentioned, where developers are interested in writing the kind of code that you are talking about.