13 ms·

Dynamic Programming vs. Divide-and-Conquer (2018)

214 points by trekhleb 5y ago

water8 5y ago
The ratio of importance placed on these algorithm design in interviews vs the amount of times they actually come up in real world problems seems skewed IMO.
- mightymercado 5y ago
  I would argue that it's a pretty good measure of programming and CS problem solving skill with weak alternatives.
  - water8 5y ago
    Most of these types of algorithms already have tons of research available online as people try to figure out what the lower bound of optimization is. It's far more telling to just talk about previous projects the person has worked on to gauge their level of competence. Asking them to explain why they made the choice they did vs trying to see how much they can memorize tests two different skill sets. The person who makes better choices is the one you want to hire.
    - fakename11 5y ago
      How does someone talking tell you if they can actually do basic programming... I think you would be surprised at the number of people that apply for software engineering jobs but barely know how to program.
      - saagarjha 5y ago
        To be fair, I know a fair number of people who are good at competitive programming but are absolutely awful at writing maintainable code.
        Flex247A 5y ago
        Yeah, competitive programming forces you to use short variable names and write makeshift code which is fast enough to pass all the test cases ...
        qsort 5y ago
        Yeah, those competitions are not really representative of actual skill. I was into competitive math as a teenager and was somewhat successful, but I actually kind of suck at math. Similarly, I'm a professional developer but I'm really bad at competitive programming: what usually happens is that I know how to solve the problems but the time limit is too low (for me, at least). I'd say success in competitions is a good indicator of dedication and perseverance, but not sufficient to spot someone who's good at the job.
      - pjmlp 5y ago
        By the same way we have doctors do surgery in place, construction workers do a toy house, teachers give a class for free, cooks spend one day serving meals for free,...
      - water8 5y ago
        Because you can determine if they organize and test things in a repeatable and maintainable way or do they have trouble organizing structures and make questionable performance decisions. Are they clear on the hasA vs isA. Do they know what a mutex or static scope is? These are the things that will cause huge debugging nightmares. Syntax issues are no where close to as problematic so why use whiteboards vs an actual computer? In my experience of interviewing, questions about Security and Threading(performance / micro-opts) are good for separating the wheat from the chaff.
    - beforeolives 5y ago
      Are these mutually exclusive? I've never been through an interview that didn't ask questions about previous work regardless of how the technical test was structured.
  - tekkk 5y ago
    You'd think so, but from my experience the folk that are very interested in algorithmic design and so forth produce highly abstract and hard to understand solutions to simple problems, which, in the end, are the majority in the regular dev work. Sure if you're applying for a job that really demands algorithmic design skills it should be a great asset but in general the most valuable skills any programmer has is producing simple and robust code that works and others can continue building on. I don't deny that knowing algorithmic design skills well helps a lot but it does seem to feed the egos of the programmers to produce overly complicated solutions.
    - teachingassist 5y ago
      This is a nice point - I'd answer this type of algorithm question in truth by identifying the relevant library wherever possible, not by coding it myself, and I'd strongly expect anyone I was working with to do the same.
      - water8 5y ago
        "I always code my own AES libraries because I'm an expert" - said no expert ever.
  - wst_ 5y ago
    It's much better to give a candidate a simplified version of your typical daily task. Give them enough time so they can google and learn if needed. That usually means very simple task that you could solve in hour or two at your leisure at home. Now, I do get that there's a lot of people who don't like spending time at home for interview tasks but when you think about AND it's not skewed to extreme (say, big task 8 working hours worth) then, in terms of time wasted, it's not such a big difference. Interviewer can then see the code quality, can talk about it with candidate, clarify some missing pieces or pitfalls found, etc. IMHO most important is not if the candidate knows how to solve some hard or even medium problem when I speak to them and they are stressed enough already. What is important is if they're willing to learn, if they know how to search for stuff they may not know and if they can produce performant enough, but excellent to read, code.
    - globular-toast 5y ago
      It depends what you're looking for. If you want someone who can turn the handle on your typical daily task then, sure, test them on your typical daily task. But if you want someone capable of developing solutions to brand new problems then it's not so easy and testing fundamental computer science theory is important.
      - slumpt_ 5y ago
        It’s not. Theory can be referenced. People do not work in a vacuum. Interviewing in eng is broken, but afaict its a “worst solution save all others” kind of scenario. But let us not begin to deem these intrinsically important. Some of the most creative and productive coworkers I’ve had struggled with leetcode style interviews. They’re a bad tool for anyone who isnt a new grad, and even then.
        globular-toast 5y ago
        When you apply for jobs do you simply look for "engineering" positions? Why am I always applying for software engineering and not electrical engineering? It's all engineering, and theory can be referenced, right? In fact, why doesn't everyone just buy a book and become a top engineer? The point is not (or shouldn't be) to recite a textbook. The point is you can navigate your way around the textbooks. I've got both The Art of Computer Programming and The Art of Electronics on my shelf. I could find the sections to help sorting a list in seconds. As for the latter, I have no idea why the majority of that book even exists. I can't call myself an electrical engineer, even though all the theory I need is within arm's reach. I assume you're arguing against the "recite the textbook" approach. I would agree that this is not the way to do things. But equally, "throw the textbooks out" is not the right way either. We need to evaluate a high-level grasp of the literature/theory but don't punish for forgetting minutiae. I might ask a candidate to talk about choice of sorting algorithms. There is, of course, no perfect answer, but what I'll be expecting is general evaluation of algorithms: time/memory tradeoffs, probing for more domain knowledge (e.g. does the data often come in sorted or random), platform constraints etc. I won't even expect a name drop of an actual sorting algorithm as that's not really the point. What they're telling me is they know why Knuth has a whole chapter on sorting. That's the important thing.
  - paraph1n 5y ago
    It's a much better measure of how many leetcode DP problems you solved while grinding interview prep. I'd argue it's a pretty poor measure of software engineering ability.
- waynesonfire 5y ago
  i believe the claim is the ability to understand these algorithms vs understand business problems is not.
- aliceryhl 5y ago
  Yeah, interviews should really be asking more questions about graph algorithms instead. Those are so much more useful.
- deleted 5y ago
  [deleted]
- tarsinge 5y ago
  I do mostly not too sophisticated web apps and binary search and minimum edit distance do come up regularly. I agree you don't have to remember the implementation details, but you have to know they exist and which problems they solve.
  - philips4350 5y ago
    But that doesn't mean you need to be able to implement them though. Shouldn't having a high level knowledge about these problem be enough if all you are doing is building apps.
- Bancakes 5y ago
  It's not just the algorithm, but the frame of mind to consider an optimisation. I guess it's a rare sight in the age of electron apps and cloud startups.
  - DaiPlusPlus 5y ago
    Oh c'mon... Electron apps aren't slow, bloated, and awkward because the new junior SWE on the team used a O(n^2) tree-walking algorithm for the app's search feature - they're like that because it's inherent in using a general-purpose web-browser engine for your desktop GUI. Micro-optimizing application software programs by implementing different algorithms is completely detached from the big engineering choices made at the very start of a project where the application's substrate and platform are chosen - and those decisions are made not with a view towards program computational efficiency, but primarily towards developer-productivity. Thanks to Electron someone who grew-up making websites as a teenager with little to no exposure to the horrendously unproductive and beginner-hostile world of MFC, GTK, and Qt can make an engaging and appropriate cross-platform desktop UI in under a day. ----- If we want to see the Electron "problem" fixed, then the best solution is for the Electron team to figure out how to cut down their build of Chromium to remove all of the features unnecessary for trusted desktop applications (no, we don't need process isolation!). I'd love to see a build of Electron+Chromium with all of the JavaScript removed, so that it's a bare-bones HTML+CSS layout and rendering system, and have it wired-up to some OOP application binary (be it Java, .NET, C/C++, etc) which manipulates the DOM - I don't see why that should need more than a few dozen MB RAM and run in a single process.
    - jorl17 5y ago
      People in these arguments always talk about the algorithms, but I think that entirely misses the point. It rarely is about the algorithms themselves, but, rather, it's about the data structures. One is obviously tied to the other, but what I mean is that many slow apps are slow because the people just used the wrong data structure. Sometimes it's as simple and silly as using a list and constantly iterating over it instead of using a dictionary/KV-map. I think the idea with having people know about "algorithms" is to get them in a state of mind where they will automatically pick more appropriate data structures. I really don't remember how to implement RB-trees or AVL-trees, nor do I really know their pros and cons against each other at this moment (I have a very very faint idea), but I know they exist and I certainly have a better idea of when to use a tree versus a list, versus whatever. Would I fail these interviews? Probably, unless I studied a bit, but do I think that the concepts that they ask about are pointless? No, not at all. I've looked at my fair share of legacy code bases built by subpar developers and the one thing that always pops up is the bad data structures chosen. Once we fix that, usually everything else automatically falls into place. EDIT: To be clear, what I mean is that in most cases, just picking the right data structure, among the most basic and elementary data structures given to you by the language, is more than enough. Only in rare cases does one then have to go beyond that and carefully engineer a more precise algorithm. The data structures are way more than half the problem, in nearly every application.
- kryptiskt 5y ago
  I disagree. There are so much code out there that nest loops and become accidentally quadratic, where a little knowledge would have helped make it perform well at scale. Knowledge about the time complexity of algorithms isn't valuable only for people implementing libraries. Every single time you iterate over things or partition things by a predicate you are using the building blocks of algorithms to make a new custom algorithm, and knowledge of the theory will help you avoid bad performance. All nested loops are harbingers of algorithmic doom, and should be treated as such, and they come up all the time in real code.
  - lifthrasiir 5y ago
    There are only a few useful parts of the algorithm theory in practice. The time complexity is surely one of them, but it is still overvalued in a sense that the actual performance on the real hardware and realistic input distribution is more important. And when you actually need algorithms, you always have a luxury of existing literatures and implementations. Real-time algorithm design in the interview is very unrealistic and in most cases only exists to detect interviewee's signaling.
    - EvilEy3 5y ago
      > And when you actually need algorithms, you always have a luxury of existing literatures and implementations. And how well that works in practice? How will candidate even know where to look at if he has no idea what he needs to find?
      - lifthrasiir 5y ago
        > How will candidate even know where to look at if he has no idea what he needs to find? That happens all the time, not just for algorithms. I don't expect candidates to know every possible algorithm (as I surely don't), I expect candidates to identify and learn what's required for the task. A knowledge of the specific algorithm is not of much value. The ability to learn and possibly implement algorithms is.
- thrower123 5y ago
  Software interviewing tends to be selecting for neurosurgeons, and then the job is putting them in a rickety ambulance with some steristrips and a shot of narcan.
Labo333 5y ago
I think that what the author calls "divide and conquer" is actually "recursion". Also the complexity of naive fibonacci is exactly the fibonacci sequence so O(2^n) is correct but less precise than O(phi^n)
- slver 5y ago
  "Recursion" is part of the "how" of the "why" of "divide and conquer" :P
- aliceryhl 5y ago
  Recursion is not the same as divide and conquer. Divide and conquer is a category of algorithms that you would often implement with recursion, but you don't have to. For example, consider the bottom-up implementation of merge sort [1]. This implementation is not recursive, but merge sort uses divide and conquer regardless of whether or not you implement it top-down or bottom-up. On the other hand, the naive fibonacci implementation that runs in exponential type is recursive, but it does not use divide and conquer. [1]: https://en.wikipedia.org/wiki/Merge_sort#Bottom-up_implementation https://en.wikipedia.org/wiki/Merge_sort#Bottom-up_implement...
- neonological 5y ago
  All recursion can be translated into a loop and a stack. In the tail recursive case you don’t even need a stack, a loop would suffice. That means all divide and conquer algorithms can be implemented without recursion.
- limoce 5y ago
  Seems you got down-voted. I guess you're doing competitive programming? People that are good at CP never call "recursion with memoization" as "divide and conquer". Yeah, they just call it recursion. "Divide and conquer" in CP world seems to be specific to those problems whose subproblems are not overlapping (therefore completely "divided"), e.g. merge sort, segment trees. Considering the classic problem "Tower of Hanoi", is it "divide and conquer"? No to CP people, and even Wikipedia [0] does not explicitly regard it as "divide and conquer". [0]: https://en.wikipedia.org/wiki/Tower_of_Hanoi https://en.wikipedia.org/wiki/Tower_of_Hanoi
Veen 5y ago
The diagrams in this article are excellent. Does anyone know what the author used to make them?
- lasfter 5y ago
  Powerpoint I'm guessing?
- trekhleb 5y ago
  I made them in draw.io
  - tomduncalf 5y ago
    Awesome, I had no idea this existed and was free! Recently had to do a few diagrams and Google Drawing is just too basic so ended up using Lucidchart, but for the tiny amount of diagramming I do, it’s too pricey. This looks perfect so thanks for sharing. Also I thoroughly enjoyed your post, well done on explaining a potentially complex area so clearly - I’ve signed up for future posts!
    - trekhleb 5y ago
      Cool! I’m glad that the link was useful! Another alternative that I’ve been using and that I liked is https://sketch.io/sketchpad/ https://sketch.io/sketchpad/. Also pretty good tool (online and free)
      - tomduncalf 5y ago
        Thanks for that! :)
      - the_arun 5y ago
        Also https://whimsical.com/ https://whimsical.com/ and https://miro.com/ https://miro.com/ are great for beautiful diagrams!
  - specialist 5y ago
codetrotter 5y ago
This is really good. Much more understandable explanation than anything I have read, or seen, about DP before.
tracyhenry 5y ago
Merge sort seems to be a more classic example for divide-and-conquer which involves merging the results from two subproblems.
- strikelaserclaw 5y ago
  i personally like quick sort because it leads to understanding other divide and conquer algorithms like finding the kth order statistic.
chalst 5y ago
The paradigmatic example of a DP algorithm for me is the simple DP solution of the 0-1 knapsack problem, which, although because of NP-hardness of the general case, becomes infeasible on many easily constructed cases, actually performs pretty well on many non-artificial examples. It's an example of what the article calls bottom-up dynamic programming, but I think it is a poor example of divide-and-conquer, because it naturally fits the following purely functional form: h(foldr f a weights) where weights is the problem, expressed as a list of (item, weight) pairs, and f, h and a are subfunctions. This is a pretty paradigmatic non-divide-and-conquer form in functional programming: it's a one-at-a-time iteration through the list expressing the problem So while I think this is good article with plenty of food for thought, I reject the central claim.
- thealig 5y ago
  The example given in the article for bottom-up DP is edit distance, unless you're referring to something I missed?
  - chalst 5y ago
    I was commenting on the high-level claims of the article, not the specific algorithms the article describes. When you move from the article's example of a DP algorithm to the simple implementation I sketch of the 0-1 knapsack problem, the claim that DP is a kind of divide-and-conquer looks harder to sustain.
    - thealig 5y ago
      Yes, agreed. The examples seem fine to me as far as DP is concerned, but the claim of divide and conquer is a bit weird
da39a3ee 5y ago
Looks great, bookmarked. A big problem with explaining/learning this area is the name. Names are important but unfortunately the name "dynamic programming" is, according to the people responsible for choosing the name, just BS that they made up one day for their employer.
- ketzu 5y ago
  > unfortunately the name "dynamic programming" is, according to the people responsible for choosing the name, just BS that they made up one day for their employer. For people interested, Richard Bellman who apparently came up with the name, put down the story in his autobiography which is cited on wikipedia: https://en.wikipedia.org/wiki/Dynamic_programming#History https://en.wikipedia.org/wiki/Dynamic_programming#History "I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes. An interesting question is, "Where did the name, dynamic programming, come from?" The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word "research". I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word "programming". I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying. I thought, let's kill two birds with one stone. Let's take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that is it's impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It's impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities."
goldenkey 5y ago
vs Pure Reason Reason gets you a closed form for F(n), the nth Fibonacci number. The author's naive fib with memoization is O(n). The closed form is O(1) if you consider exponentiation to be a constant time operation. https://en.m.wikipedia.org/wiki/Fibonacci_number#Closed-form_expression https://en.m.wikipedia.org/wiki/Fibonacci_number#Closed-form... Quite a thing to overlook in an article about efficiency of algorithms... why am I not surprised?
- ketzu 5y ago
  My understanding why this is usually not discussed is the following: The fibonacci sequence is usually used as an illustrative example or motivating problem for a given topic, not as a problem in itself. I believe a side note might most likely detract from the overall point for a technicality. For the same reason you only wrote "if you consider exponentiation to be a constant time operation" instead of including a side analysis of how everything changes once you can't do that anymore, possible problems with accuracy of floating point representations of phi and it's exponentiation and everything else one has to consider once we leave the comfortable home of architecture native integers. It is usually very valid to do so.
- bidirectional 5y ago
  Why are you assuming exponentiation is constant time? For machine integers it near-enough is, but there are fewer than 100 Fibonacci numbers that fit in a 64-bit integer so you may as well use a lookup table. For arbitrary precision integers it definitely isn't.
lokesh1729 5y ago
A good history about how bellman came up with this name https://en.wikipedia.org/wiki/Dynamic_programming#History https://en.wikipedia.org/wiki/Dynamic_programming#History
- agumonkey 5y ago
  Btw people should read his books, they'll be surprised.
lokesh1729 5y ago
DP and DC there's lot left to learn but feels like you learnt so much already...
anon_tor_12345 5y ago
i'm late to the comments but hopefully this helps someone: i struggled with DP as much as anyone. i read all of the standard resources (CLRS, vazirani, kleinberg, etc), watch all the youtube videos, did all of the practice problems in the books, and still couldn't solve the kinds that are asked on interviews. i even went as far as emailing kleinberg for help. what made it basically unconsciously fluent for me (i.e. i can read a problem statement and sketch out the recursion and subproblems in about 60s and then just perform fixup) was doing hordes of them on leetcode in preparation for a FB interview. it got to the point where i could solve hard ones in about 5 minutes using either bottom-up or top-down (i.e. memoization). so if you're struggling with DP for interviews my suggestion (which is basically the standard suggestion) is to just grind the problems on leetcode. and contrary to popular belief they do come up outside of interviews - i had to solve a circuit synthesis problem last week and it turned out to be basically DP substring counting problem. took me all of 5 minutes.
- dominotw 5y ago
  brute force recursive solution + @lru_cache annotation. works for me every time on leetcode.
  - mettamage 5y ago
    I've solved some DP problems (I call it: "find the right table/array and then decide whether you feel recursive or iterative"). I know about lru caches. Yet, I'm not fully understanding what your implying with your comment.
    - jvanderbot 5y ago
      Op's saying (I think) that the difference between brute force recursion and DP ( top down ) is memoization, and that the language / library construct lru_cache will perform that memoization for you. Its not a fascinating insight. If you can express a problem as a recursive brute force solution and there's a least a little recalculation involved, you're one step from DP. The lru_cache just does that step. Just google lru_cache edit: "op"
andreygrehov 5y ago
If anyone is interested, I recorded a beginners course about Dynamic Programming during the lock-downs: https://www.youtube.com/playlist?list=PLVrpF4r7WIhTT1hJqZmjP10nxsmrbRvlf https://www.youtube.com/playlist?list=PLVrpF4r7WIhTT1hJqZmjP...

214 points by trekhleb 5y ago

The ratio of importance placed on these algorithm design in interviews vs the amount of times they actually come up in real world problems seems skewed IMO.

mightymercado 5y ago
I would argue that it's a pretty good measure of programming and CS problem solving skill with weak alternatives.
- water8 5y ago
  Most of these types of algorithms already have tons of research available online as people try to figure out what the lower bound of optimization is. It's far more telling to just talk about previous projects the person has worked on to gauge their level of competence. Asking them to explain why they made the choice they did vs trying to see how much they can memorize tests two different skill sets. The person who makes better choices is the one you want to hire.
  - fakename11 5y ago
    How does someone talking tell you if they can actually do basic programming... I think you would be surprised at the number of people that apply for software engineering jobs but barely know how to program.
    - saagarjha 5y ago
      To be fair, I know a fair number of people who are good at competitive programming but are absolutely awful at writing maintainable code.
      - Flex247A 5y ago
        Yeah, competitive programming forces you to use short variable names and write makeshift code which is fast enough to pass all the test cases ...
      - qsort 5y ago
        Yeah, those competitions are not really representative of actual skill. I was into competitive math as a teenager and was somewhat successful, but I actually kind of suck at math. Similarly, I'm a professional developer but I'm really bad at competitive programming: what usually happens is that I know how to solve the problems but the time limit is too low (for me, at least). I'd say success in competitions is a good indicator of dedication and perseverance, but not sufficient to spot someone who's good at the job.
    - pjmlp 5y ago
      By the same way we have doctors do surgery in place, construction workers do a toy house, teachers give a class for free, cooks spend one day serving meals for free,...
    - water8 5y ago
      Because you can determine if they organize and test things in a repeatable and maintainable way or do they have trouble organizing structures and make questionable performance decisions. Are they clear on the hasA vs isA. Do they know what a mutex or static scope is? These are the things that will cause huge debugging nightmares. Syntax issues are no where close to as problematic so why use whiteboards vs an actual computer? In my experience of interviewing, questions about Security and Threading(performance / micro-opts) are good for separating the wheat from the chaff.
  - beforeolives 5y ago
    Are these mutually exclusive? I've never been through an interview that didn't ask questions about previous work regardless of how the technical test was structured.
- tekkk 5y ago
  You'd think so, but from my experience the folk that are very interested in algorithmic design and so forth produce highly abstract and hard to understand solutions to simple problems, which, in the end, are the majority in the regular dev work. Sure if you're applying for a job that really demands algorithmic design skills it should be a great asset but in general the most valuable skills any programmer has is producing simple and robust code that works and others can continue building on. I don't deny that knowing algorithmic design skills well helps a lot but it does seem to feed the egos of the programmers to produce overly complicated solutions.
  - teachingassist 5y ago
    This is a nice point - I'd answer this type of algorithm question in truth by identifying the relevant library wherever possible, not by coding it myself, and I'd strongly expect anyone I was working with to do the same.
    - water8 5y ago
      "I always code my own AES libraries because I'm an expert" - said no expert ever.
- wst_ 5y ago
  It's much better to give a candidate a simplified version of your typical daily task. Give them enough time so they can google and learn if needed. That usually means very simple task that you could solve in hour or two at your leisure at home. Now, I do get that there's a lot of people who don't like spending time at home for interview tasks but when you think about AND it's not skewed to extreme (say, big task 8 working hours worth) then, in terms of time wasted, it's not such a big difference. Interviewer can then see the code quality, can talk about it with candidate, clarify some missing pieces or pitfalls found, etc. IMHO most important is not if the candidate knows how to solve some hard or even medium problem when I speak to them and they are stressed enough already. What is important is if they're willing to learn, if they know how to search for stuff they may not know and if they can produce performant enough, but excellent to read, code.
  - globular-toast 5y ago
    It depends what you're looking for. If you want someone who can turn the handle on your typical daily task then, sure, test them on your typical daily task. But if you want someone capable of developing solutions to brand new problems then it's not so easy and testing fundamental computer science theory is important.
    - slumpt_ 5y ago
      It’s not. Theory can be referenced. People do not work in a vacuum. Interviewing in eng is broken, but afaict its a “worst solution save all others” kind of scenario. But let us not begin to deem these intrinsically important. Some of the most creative and productive coworkers I’ve had struggled with leetcode style interviews. They’re a bad tool for anyone who isnt a new grad, and even then.
      - globular-toast 5y ago
        When you apply for jobs do you simply look for "engineering" positions? Why am I always applying for software engineering and not electrical engineering? It's all engineering, and theory can be referenced, right? In fact, why doesn't everyone just buy a book and become a top engineer? The point is not (or shouldn't be) to recite a textbook. The point is you can navigate your way around the textbooks. I've got both The Art of Computer Programming and The Art of Electronics on my shelf. I could find the sections to help sorting a list in seconds. As for the latter, I have no idea why the majority of that book even exists. I can't call myself an electrical engineer, even though all the theory I need is within arm's reach. I assume you're arguing against the "recite the textbook" approach. I would agree that this is not the way to do things. But equally, "throw the textbooks out" is not the right way either. We need to evaluate a high-level grasp of the literature/theory but don't punish for forgetting minutiae. I might ask a candidate to talk about choice of sorting algorithms. There is, of course, no perfect answer, but what I'll be expecting is general evaluation of algorithms: time/memory tradeoffs, probing for more domain knowledge (e.g. does the data often come in sorted or random), platform constraints etc. I won't even expect a name drop of an actual sorting algorithm as that's not really the point. What they're telling me is they know why Knuth has a whole chapter on sorting. That's the important thing.
- paraph1n 5y ago
  It's a much better measure of how many leetcode DP problems you solved while grinding interview prep. I'd argue it's a pretty poor measure of software engineering ability.
waynesonfire 5y ago
i believe the claim is the ability to understand these algorithms vs understand business problems is not.
aliceryhl 5y ago
Yeah, interviews should really be asking more questions about graph algorithms instead. Those are so much more useful.
deleted 5y ago
[deleted]
tarsinge 5y ago
I do mostly not too sophisticated web apps and binary search and minimum edit distance do come up regularly. I agree you don't have to remember the implementation details, but you have to know they exist and which problems they solve.
- philips4350 5y ago
  But that doesn't mean you need to be able to implement them though. Shouldn't having a high level knowledge about these problem be enough if all you are doing is building apps.
Bancakes 5y ago
It's not just the algorithm, but the frame of mind to consider an optimisation. I guess it's a rare sight in the age of electron apps and cloud startups.
- DaiPlusPlus 5y ago
  Oh c'mon... Electron apps aren't slow, bloated, and awkward because the new junior SWE on the team used a O(n^2) tree-walking algorithm for the app's search feature - they're like that because it's inherent in using a general-purpose web-browser engine for your desktop GUI. Micro-optimizing application software programs by implementing different algorithms is completely detached from the big engineering choices made at the very start of a project where the application's substrate and platform are chosen - and those decisions are made not with a view towards program computational efficiency, but primarily towards developer-productivity. Thanks to Electron someone who grew-up making websites as a teenager with little to no exposure to the horrendously unproductive and beginner-hostile world of MFC, GTK, and Qt can make an engaging and appropriate cross-platform desktop UI in under a day. ----- If we want to see the Electron "problem" fixed, then the best solution is for the Electron team to figure out how to cut down their build of Chromium to remove all of the features unnecessary for trusted desktop applications (no, we don't need process isolation!). I'd love to see a build of Electron+Chromium with all of the JavaScript removed, so that it's a bare-bones HTML+CSS layout and rendering system, and have it wired-up to some OOP application binary (be it Java, .NET, C/C++, etc) which manipulates the DOM - I don't see why that should need more than a few dozen MB RAM and run in a single process.
  - jorl17 5y ago
    People in these arguments always talk about the algorithms, but I think that entirely misses the point. It rarely is about the algorithms themselves, but, rather, it's about the data structures. One is obviously tied to the other, but what I mean is that many slow apps are slow because the people just used the wrong data structure. Sometimes it's as simple and silly as using a list and constantly iterating over it instead of using a dictionary/KV-map. I think the idea with having people know about "algorithms" is to get them in a state of mind where they will automatically pick more appropriate data structures. I really don't remember how to implement RB-trees or AVL-trees, nor do I really know their pros and cons against each other at this moment (I have a very very faint idea), but I know they exist and I certainly have a better idea of when to use a tree versus a list, versus whatever. Would I fail these interviews? Probably, unless I studied a bit, but do I think that the concepts that they ask about are pointless? No, not at all. I've looked at my fair share of legacy code bases built by subpar developers and the one thing that always pops up is the bad data structures chosen. Once we fix that, usually everything else automatically falls into place. EDIT: To be clear, what I mean is that in most cases, just picking the right data structure, among the most basic and elementary data structures given to you by the language, is more than enough. Only in rare cases does one then have to go beyond that and carefully engineer a more precise algorithm. The data structures are way more than half the problem, in nearly every application.
kryptiskt 5y ago
I disagree. There are so much code out there that nest loops and become accidentally quadratic, where a little knowledge would have helped make it perform well at scale. Knowledge about the time complexity of algorithms isn't valuable only for people implementing libraries. Every single time you iterate over things or partition things by a predicate you are using the building blocks of algorithms to make a new custom algorithm, and knowledge of the theory will help you avoid bad performance. All nested loops are harbingers of algorithmic doom, and should be treated as such, and they come up all the time in real code.
- lifthrasiir 5y ago
  There are only a few useful parts of the algorithm theory in practice. The time complexity is surely one of them, but it is still overvalued in a sense that the actual performance on the real hardware and realistic input distribution is more important. And when you actually need algorithms, you always have a luxury of existing literatures and implementations. Real-time algorithm design in the interview is very unrealistic and in most cases only exists to detect interviewee's signaling.
  - EvilEy3 5y ago
    > And when you actually need algorithms, you always have a luxury of existing literatures and implementations. And how well that works in practice? How will candidate even know where to look at if he has no idea what he needs to find?
    - lifthrasiir 5y ago
      > How will candidate even know where to look at if he has no idea what he needs to find? That happens all the time, not just for algorithms. I don't expect candidates to know every possible algorithm (as I surely don't), I expect candidates to identify and learn what's required for the task. A knowledge of the specific algorithm is not of much value. The ability to learn and possibly implement algorithms is.
thrower123 5y ago
Software interviewing tends to be selecting for neurosurgeons, and then the job is putting them in a rickety ambulance with some steristrips and a shot of narcan.

Labo333 5y ago

I think that what the author calls "divide and conquer" is actually "recursion". Also the complexity of naive fibonacci is exactly the fibonacci sequence so O(2^n) is correct but less precise than O(phi^n)

slver 5y ago
"Recursion" is part of the "how" of the "why" of "divide and conquer" :P
aliceryhl 5y ago
Recursion is not the same as divide and conquer. Divide and conquer is a category of algorithms that you would often implement with recursion, but you don't have to. For example, consider the bottom-up implementation of merge sort [1]. This implementation is not recursive, but merge sort uses divide and conquer regardless of whether or not you implement it top-down or bottom-up. On the other hand, the naive fibonacci implementation that runs in exponential type is recursive, but it does not use divide and conquer. [1]: https://en.wikipedia.org/wiki/Merge_sort#Bottom-up_implementation https://en.wikipedia.org/wiki/Merge_sort#Bottom-up_implement...
neonological 5y ago
All recursion can be translated into a loop and a stack. In the tail recursive case you don’t even need a stack, a loop would suffice. That means all divide and conquer algorithms can be implemented without recursion.
limoce 5y ago
Seems you got down-voted. I guess you're doing competitive programming? People that are good at CP never call "recursion with memoization" as "divide and conquer". Yeah, they just call it recursion. "Divide and conquer" in CP world seems to be specific to those problems whose subproblems are not overlapping (therefore completely "divided"), e.g. merge sort, segment trees. Considering the classic problem "Tower of Hanoi", is it "divide and conquer"? No to CP people, and even Wikipedia [0] does not explicitly regard it as "divide and conquer". [0]: https://en.wikipedia.org/wiki/Tower_of_Hanoi https://en.wikipedia.org/wiki/Tower_of_Hanoi

Veen 5y ago

The diagrams in this article are excellent. Does anyone know what the author used to make them?

lasfter 5y ago
Powerpoint I'm guessing?
trekhleb 5y ago
I made them in draw.io
- tomduncalf 5y ago
  Awesome, I had no idea this existed and was free! Recently had to do a few diagrams and Google Drawing is just too basic so ended up using Lucidchart, but for the tiny amount of diagramming I do, it’s too pricey. This looks perfect so thanks for sharing. Also I thoroughly enjoyed your post, well done on explaining a potentially complex area so clearly - I’ve signed up for future posts!
  - trekhleb 5y ago
    Cool! I’m glad that the link was useful! Another alternative that I’ve been using and that I liked is https://sketch.io/sketchpad/ https://sketch.io/sketchpad/. Also pretty good tool (online and free)
    - tomduncalf 5y ago
      Thanks for that! :)
    - the_arun 5y ago
      Also https://whimsical.com/ https://whimsical.com/ and https://miro.com/ https://miro.com/ are great for beautiful diagrams!
- specialist 5y ago

codetrotter 5y ago

This is really good. Much more understandable explanation than anything I have read, or seen, about DP before.

tracyhenry 5y ago

Merge sort seems to be a more classic example for divide-and-conquer which involves merging the results from two subproblems.

strikelaserclaw 5y ago
i personally like quick sort because it leads to understanding other divide and conquer algorithms like finding the kth order statistic.

chalst 5y ago

The paradigmatic example of a DP algorithm for me is the simple DP solution of the 0-1 knapsack problem, which, although because of NP-hardness of the general case, becomes infeasible on many easily constructed cases, actually performs pretty well on many non-artificial examples. It's an example of what the article calls bottom-up dynamic programming, but I think it is a poor example of divide-and-conquer, because it naturally fits the following purely functional form: h(foldr f a weights) where weights is the problem, expressed as a list of (item, weight) pairs, and f, h and a are subfunctions. This is a pretty paradigmatic non-divide-and-conquer form in functional programming: it's a one-at-a-time iteration through the list expressing the problem So while I think this is good article with plenty of food for thought, I reject the central claim.

thealig 5y ago
The example given in the article for bottom-up DP is edit distance, unless you're referring to something I missed?
- chalst 5y ago
  I was commenting on the high-level claims of the article, not the specific algorithms the article describes. When you move from the article's example of a DP algorithm to the simple implementation I sketch of the 0-1 knapsack problem, the claim that DP is a kind of divide-and-conquer looks harder to sustain.
  - thealig 5y ago
    Yes, agreed. The examples seem fine to me as far as DP is concerned, but the claim of divide and conquer is a bit weird

da39a3ee 5y ago

Looks great, bookmarked. A big problem with explaining/learning this area is the name. Names are important but unfortunately the name "dynamic programming" is, according to the people responsible for choosing the name, just BS that they made up one day for their employer.

ketzu 5y ago
> unfortunately the name "dynamic programming" is, according to the people responsible for choosing the name, just BS that they made up one day for their employer. For people interested, Richard Bellman who apparently came up with the name, put down the story in his autobiography which is cited on wikipedia: https://en.wikipedia.org/wiki/Dynamic_programming#History https://en.wikipedia.org/wiki/Dynamic_programming#History "I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes. An interesting question is, "Where did the name, dynamic programming, come from?" The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word "research". I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word "programming". I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying. I thought, let's kill two birds with one stone. Let's take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that is it's impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It's impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities."

goldenkey 5y ago

vs Pure Reason Reason gets you a closed form for F(n), the nth Fibonacci number. The author's naive fib with memoization is O(n). The closed form is O(1) if you consider exponentiation to be a constant time operation. https://en.m.wikipedia.org/wiki/Fibonacci_number#Closed-form_expression https://en.m.wikipedia.org/wiki/Fibonacci_number#Closed-form... Quite a thing to overlook in an article about efficiency of algorithms... why am I not surprised?

ketzu 5y ago
My understanding why this is usually not discussed is the following: The fibonacci sequence is usually used as an illustrative example or motivating problem for a given topic, not as a problem in itself. I believe a side note might most likely detract from the overall point for a technicality. For the same reason you only wrote "if you consider exponentiation to be a constant time operation" instead of including a side analysis of how everything changes once you can't do that anymore, possible problems with accuracy of floating point representations of phi and it's exponentiation and everything else one has to consider once we leave the comfortable home of architecture native integers. It is usually very valid to do so.
bidirectional 5y ago
Why are you assuming exponentiation is constant time? For machine integers it near-enough is, but there are fewer than 100 Fibonacci numbers that fit in a 64-bit integer so you may as well use a lookup table. For arbitrary precision integers it definitely isn't.

lokesh1729 5y ago

A good history about how bellman came up with this name https://en.wikipedia.org/wiki/Dynamic_programming#History https://en.wikipedia.org/wiki/Dynamic_programming#History

agumonkey 5y ago
Btw people should read his books, they'll be surprised.

lokesh1729 5y ago

DP and DC there's lot left to learn but feels like you learnt so much already...

anon_tor_12345 5y ago

i'm late to the comments but hopefully this helps someone: i struggled with DP as much as anyone. i read all of the standard resources (CLRS, vazirani, kleinberg, etc), watch all the youtube videos, did all of the practice problems in the books, and still couldn't solve the kinds that are asked on interviews. i even went as far as emailing kleinberg for help. what made it basically unconsciously fluent for me (i.e. i can read a problem statement and sketch out the recursion and subproblems in about 60s and then just perform fixup) was doing hordes of them on leetcode in preparation for a FB interview. it got to the point where i could solve hard ones in about 5 minutes using either bottom-up or top-down (i.e. memoization). so if you're struggling with DP for interviews my suggestion (which is basically the standard suggestion) is to just grind the problems on leetcode. and contrary to popular belief they do come up outside of interviews - i had to solve a circuit synthesis problem last week and it turned out to be basically DP substring counting problem. took me all of 5 minutes.

dominotw 5y ago
brute force recursive solution + @lru_cache annotation. works for me every time on leetcode.
- mettamage 5y ago
  I've solved some DP problems (I call it: "find the right table/array and then decide whether you feel recursive or iterative"). I know about lru caches. Yet, I'm not fully understanding what your implying with your comment.
  - jvanderbot 5y ago
    Op's saying (I think) that the difference between brute force recursion and DP ( top down ) is memoization, and that the language / library construct lru_cache will perform that memoization for you. Its not a fascinating insight. If you can express a problem as a recursive brute force solution and there's a least a little recalculation involved, you're one step from DP. The lru_cache just does that step. Just google lru_cache edit: "op"

andreygrehov 5y ago

If anyone is interested, I recorded a beginners course about Dynamic Programming during the lock-downs: https://www.youtube.com/playlist?list=PLVrpF4r7WIhTT1hJqZmjP10nxsmrbRvlf https://www.youtube.com/playlist?list=PLVrpF4r7WIhTT1hJqZmjP...