3 ms·
> My point was not that algorithms are irrelevant but they are table stakes You're wrong. A previous team built a simple data processing pipeline for a big cus
by hintymad 5y ago
> My point was not that algorithms are irrelevant but they are table stakes
You're wrong. A previous team built a simple data processing pipeline for a big customer. The throughput is single digit per second. The customer was furious as they would have wait for days to get their data in. The reason is the team was not aware that their graph building and traversal algorithms are O(n^2), and they didn't understand how to distribute their processing logic. My team came in, fixed the algorithms with merely dozens of lines, and improved the throughput by more than 200 times in a matter of days. If you build large backend for data intensive systems, which pretty much every tech company needs, you will need algorithms and data structures. Probabilistic data structures? Check. Manually optimizing popular machine learning algorithms? Check. Rolling out our own implementation of graphs, trees, vector similarity search algorithms and indices, parallelization algorithms, cache oblivious data structures? Check, check, and check. You don't have to use those techniques, but your value to your projects will increase exponentially if you push the boundary.
- throwaway2016a 5y agoWhat part of my statement is wrong? I said algorithms are table stakes (a poker term for something you need to know just to play at the table). To continue the analogy, sounds like whoever designed that system didn't have enough to buy into the table. > If you build large backend for data intensive systems I have. Many.
- hintymad 5y agoouch, my bad. Somehow I read “algorithm is irrelevant”, and you’re right, I didn’t get your analogy.
- Aeolun 5y ago200x single digit/second still seems lowish? I’m not sure if I’d call that a large backend for a data intensive system. Depends on how extensive your processing is of course.