8 ms·
This reminds me of the first internship I did. It was at an AI based anamoly detection startup. Their pipeline for the biggest client used to run in just under
by alex_smart 3y ago
This reminds me of the first internship I did. It was at an AI based anamoly detection startup. Their pipeline for the biggest client used to run in just under 3 hours, most of which was apparently spent extracting meaningful information from sensor data from a proprietary binary format using a shared library provided by the client. Ran it under perf and found that most of the time was spent calculating a bunch of sines and cosines. Used the LD_PRELOAD trick to log the sine and cosine calls and turns out that we were just calculating the sines and cosines of the same set of angles over and over again literally thousands of times. Wrote a wrapper to cache the values of sines and cosines in a table and return the cached value if it is already calculated and brought the running time of the pipeline to under 15 minutes. Which meant that the data science team could run their experiments and iterate on their model much faster and cheaper. Fun times.
- ditsuke 3y agoWow, that sounds like a fun place to work at!
- ahoka 3y agoAlmost any workplace is like that if you can, want and allowed to create value.
- alex_smart 3y agoIt was, and great colleagues too! Although this wasn't "assigned" work. Just me being curious/bored.
- bdavbdav 3y agoIt always amazes me when people miss this stuff first time around. “Dynamic programming” was always one of the common optimisations throughout my CS degree.
- daymanstep 3y agoThat sounds more like memoization than dynamic programming.
- bdavbdav 3y agoAIUI, DP is around memorising the results of static functions for a particular input. “Dynamic Programming is mainly an optimization over plain recursion. Wherever we see a recursive solution that has repeated calls for same inputs, we can optimize it using Dynamic Programming. The idea is to simply store the results of subproblems, so that we do not have to re-compute them when needed later”