3 ms·
TAOCP was invaluable for a couple of jobs I had. One job involved indexing arbitrarily large books (gigabyte+ was not uncommon) on an embedded CPU with less th
by rented_mule 4y ago
TAOCP was invaluable for a couple of jobs I had.
One job involved indexing arbitrarily large books (gigabyte+ was not uncommon) on an embedded CPU with less than 1MB of RAM available to the indexer (this is 15+ years ago, before smart phones). It also involved searching hundreds of those books in a few seconds with ~8MB of RAM available. Indexing was taking place on a single core CPU while the user was using the device, so the indexer had to be able to stop its work within a few milliseconds of any user action and resume later without losing progress. On the surface, Knuth's concepts felt old fashioned (abstract assembly language and talk of multiple tape drives for intermediate storage). But that mapped very well onto this indexing problem. Books and indexes resided on an SD card and individual files on the SD card could act very much like Knuth's individual tape drives. Doing append only operations on those files (as you would want to do on a tape drive) proved fast and reliable. My 1MB of RAM would have been an embarrassment of riches when the original volumes of TAOCP were released. I don't remember if any of the algorithms that shipped as part of that device were straight out of TAOCP, but the thinking involved was heavily influenced by those books. Whenever I was stuck, it was back to Knuth.
My next job was focused on distributed systems. We were rebuilding an ads system because the older one couldn't handle the scale needed. That involved lots of MapReduce and lots of stream processing (before there were good open source stream processing packages to build on top of, this started in 2010). Those same tape drive oriented algorithms came right to the surface again. When processing TB or PB of data, you want single pass algorithms wherever you can come up with them - merely linear isn't good enough. Same as tape drives - there's a huge benefit if you can avoid having to rewind. And so many stream processing primitives (windowed joins, groupings, etc.) are exactly what people were doing with tape drives 50 years ago. Now we were using many GBs of RAM spread across many machines, but relative to the amount of data being processed, it was still miniscule.
This all required spotting the similarities between the world Knuth drew his examples from and the constraints I was working under. But once I did, his concepts were quite useful. And the framework for thinking about things was more valuable to me than any specific algorithm or data structure. It was much more useful to me as a narrative to read rather than a reference to pull something out of.
- sn9 4y agoHow on earth do you even find work like this? I would love to have a career working on problems like this but can't imagine how I'd get a job doing anything but commonplace back-end work.
- rented_mule 4y agoThose jobs sort of found me... My path involved working at a series of tiny startups through the first 15 years of my career - the 1990s and early 2000s. At the smallest, I was the only engineer. At the largest, there were 8. At that scale, you learn to do everything (seriously: front end, back end, databases, assembling servers, networking the office, setting literal rat traps, cleaning the cat litter after "upgrading" the rat traps, ...). You also get an opportunity to get to know people very well and build a lot of mutual trust with some of them. In 2005, the head of engineering from the "big" startup was helping to start a new subsidiary of an established tech company and needed someone to develop indexing and search for the device mentioned above. He asked me to join. In 2010, the head of marketing at one of those startups was now an executive at a soon-to-IPO company who needed to scale their ad delivery system. She asked me to join. Both of those relationships had turned into friendships long before they asked me to join their newer endeavors. They knew what I was good at and what motivated me. That resulting in me being deeply trusted the day I joined and being given a lot of freedom to create solutions. So, I guess one way to get this kind of work is experience, relationship building, and some luck.
- sn9 4y agoThat's quite the journey! Thank you for sharing it. It was actually probably more insightful and useful than I think you may have realized.