3 ms·
I learned to program at the dawn of the PC age when memory was measured in KB and CPUs in MHz; so I learned to write fast, efficient code because that was a bas
by didgetmaster 3y ago
I learned to program at the dawn of the PC age when memory was measured in KB and CPUs in MHz; so I learned to write fast, efficient code because that was a basic requirement. Modern CPUs, super fast memory, and Gen 5 SSDs can certainly mask inefficiencies in poorly written code; but data can always expand enough to bring even supercomputers to their knees.
My expertise was centered around data management; particularly file systems. So when hard drives got big enough to hold more than 100 million files; I realized that the decades-old file system architectures could no longer handle them efficiently enough for my taste. So I set out to build a better type of file system (a data object store).
I spent weeks (sometimes months) fine tuning algorithms and code paths to make it as efficient as possible. I regularly ran my code on old, outdated hardware to make sure it ran fast in low-resource environments. I tried to take advantage of parallel processing and every other trick I could think of.
But it seemed that no matter how much faster I could make the code run; I had trouble getting any potential users to care about that aspect of the code. I could make it 10x faster than their existing solution; but if it was missing even one feature their old one supported, they would reject it even if that feature was not that important.
The code was especially efficient at handling unstructured data (since that was its primary purpose), but the metadata management features I built in turned out to be extremely efficient at managing structured data as well. When I started benchmarking it against major databases it was performing very well (e.g. see it compared to SQLite: https://www.youtube.com/watch?v=Va5ZqfwQXWI https://www.youtube.com/watch?v=Va5ZqfwQXWI). Still, it has been very slow to attract attention in spite of its speed.
We can complain about slow, buggy software all day; but we will be stuck with bloated, inefficient code if we don't put our money where our mouth is and support code bases that focus on small, fast code.