6 ms·

I have definitely seen senior developers put quadratic and even exponential algorithms into production and cause global outages. Code review didn't help, becaus

by fl0ki 2y ago

I have definitely seen senior developers put quadratic and even exponential algorithms into production and cause global outages. Code review didn't help, because their code was reviewed by other people who were interviewed to the same low standard. I continue to insist on algorithm & data structure interviews for software engineer candidates. Not every project needs them as much, but in large enough institutions, every engineer is likely to run into at least one project that does. If they can't figure it out and can't get help from someone who can, their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. There's also a difference between "don't interview for algorithms at all" and "interview for algorithms but occasionally accept an engineer weak in them". In the former case you're unlikely to get lucky enough to end up with any engineers good in algorithms, in the latter case some engineers can help others with algorithms, hopefully in exchange for other relevant skills. In other words, it's worth interviewing to make sure certain mixes of skills end up on your teams, not necessarily as a hard cutoff on one skill that means you pass on other candidates with other skills.

taeric 2y ago
Annoying counterpoint, of course, I've seen senior developers also stall out on avoiding quadratic code to the point that they didn't get the code delivered and then the project got scrapped. The most widespread failure I have seen, by far, is to write code that is so abstracted out that it isn't really clear on how to get the necessary parts inline to get at efficient code. You'll have an obvious path on how to load all data for something, but no obvious path on how to get just the relevant points for the algorithm in. This is super hilarious for code paths that try and hide network hits, such that you don't see that that loop is doing a network call per iteration.
- esafak 2y ago
  But computational complexity is a separate concern from level of abstraction.
  - sopooneo 2y ago
    I would argue that, in practice, they are not completely separate. Too much abstraction makes it hard to tell what the underlying algorithm is. And even if you can figure that out, makes it hard to optimize.
  - chipdart 2y ago
    > But computational complexity is a separate concern from level of abstraction. It really depends. Abstractions can and often do to otherwise avoidable copies. It's pointless to argue about computational complexity of an algorithm if you don't even notice that an abstraction you're using is adding a performance penalty in the form of running a linear-time algorithm in the hot path, and is even preventing the runtime from optimizing away problematic code.
- saagarjha 2y ago
  I’m curious for more details. I imagine this has to be more interesting than “Just write the double for loop.” “No.” “Ok I guess we’re canceling this project.”
  - taeric 2y ago
    Yeah, it was rarely that silly. Usual thing I'm thinking of are the designs where folks think they can make something like facebook messenger with 200ms latency in the system. You know, "by design." (Of course... I swear that was an interview question I was given once.) What you wind up with is someone designing a giant system from parts they have read about that, honestly would probably work. However, it would also require retouching pretty much every system in the company to get things aligned. Which, isn't likely to ever happen. Put a different way, the longer a project spends in design phase, the higher the chance it will not get completed. And all too often people will spend a ton of time in design looking for ways to make sure what they are building will scale to success level loads.
- zarathustreal 2y ago
  Echoing my sibling commenters here, I’d love to hear more about this situation. Usually “inlining” and algorithmic complexity are juxtaposed as orthogonal types of optimization, with algorithmic optimization typically even being the reasoning for why the level of abstraction doesn’t matter very much. You typically get a much better speed up from going from O(n^2) to O(n) algorithms than from implementing e.g. Duff’s device, etc
  - taeric 2y ago
    My apologies, forgot I posted. On phone, so will be brief. Will try a longer response later. (Apologies to other responses, not hitting them all.) Basic point is that many will abstract out data to different locations and ownership lifetimes. So, congrats, you kept us at a lower complexity, but failed to realize you need to first essentially reindex all of the data for that to happen. Now, I grant that often the problem is more in the data scattered over any number of databases. If there are reasons to keep that spread, though, hard to just sweep it aside. And then there are those that don't accept, "send it to a solver." Really annoying how often people assume they can easily beat cplex and friends.
- chipdart 2y ago
  > This is super hilarious for code paths that try and hide network hits, such that you don't see that that loop is doing a network call per iteration. True story: once I worked on a C++ project with a very ambitious junior developer who liked to portray himself as the local authority in performance and code quality. That junior developer once blocked a PR alleging performance problems because a function call was passing strings by value instead of moving them. Except that function call was only called once per run, and its responsibility was to make 1+n calls to a REST service. Whenever anyone complains about performance, ask for benchmarks. Odds are, even if they are technically right they are still wrong.
zerr 2y ago
CP (Competitive Programming, Olympiad/leetcode puzzles) uses Computer Science (Algos & DS) the same way as e.g. Physics and Biology use Mathematics, i.e. it is a completely different discipline with its own trivia knowledge and tricks that has nothing to do with a software engineering.
- saagarjha 2y ago
  Knowing how to do math is generally useful as a biologist or physicist. Sure there’s trivia or whatever but knowing how an array works is definitely important for software engineering, just like knowing how to do algebra is also important for biologists.
  - mathgradthrow 2y ago
    "knowing how to do math", is roughly how one who does not know any math beyond the tricks describe above describes mathematics.
    - saagarjha 2y ago
      Sorry which tricks are you talking about?
  - zerr 2y ago
    My point is that just because Math is useful for both physicists and biologists, that does not mean that for a physicist position you should interview the candidate (a physicist) in Biology. Biology "is a completely different discipline with its own trivia knowledge and tricks".
    - saagarjha 2y ago
      I mean making your entire interview just algorithms is stupid, yes
      - zerr 2y ago
tester756 2y ago
>their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. Anything that supports this? Because I don't believe it at all The only thing that I can think of that could cost company a lot is data corruption / database being destroyed Unless it is very specific case like HFT
- TrackerFF 2y ago
  That is pretty much my stance, too. I've seen a lot of critical fuckups in my years, but I have yet to see someone fuck up everything by using a woefully unsuitable algorithm. That, of course, does not mean that it doesn't happen - could very well be the case at the type of companies I haven't worked for, where performance is absolutely critical...but I have a hard time believing those places don't have safety guards at place. In fact, the vast majority of fabled horror stories I've heard in the industry are seemingly just that...fabled.
  - fl0ki 2y ago
    I agree at the level of source code, but sometimes algorithm skills affect the whole architecture. Even simple things like where certain data and logic would optimally exist can have cascading and compounding consequences for the whole rest of the system. To see better solutions, you have to know what's possible with algorithms and data structures. To give a contrived example that I think should make sense no matter what kind of software people actually work on: Imagine nobody had ever figured out video compression and the only way we could see electronic video required orders of magnitude higher costs. VOD and video conferencing may not even be viable for most people. We'd still be live streaming television over analog radio. It would have vast consequences throughout other technologies and what we can do with them. If a service gets the kind of viral growth that their creators dearly hope they will, that will almost certainly involve a level of scale that would benefit from knowledge of algorithms (including practical mathematics as a subset) and the difference between a good or bad solution could have far-reaching consequences. Of course many problems do have well-known solutions that anybody can look up, but many projects do end up needing solutions novel to them.
jupp0r 2y ago
In the set of errors that cause production outages, inefficient algorithms is a pretty small subset though. I still think it can be a useful proxy for other things you might value in engineers though.
kemiller 2y ago
The problem is that small organizations cargo cult large organization interviews, when for most small organizations (ones that are not building infrastructure for others) speed of delivery is vastly more important than algorithmic purity. Lots of n^2 and even 2^n algorithms are in fact perfectly reasonable in many contexts. Even in large organizations, lots of people aren't touching the massively-scaled systems and are in fact just application developers.
deleted 2y ago
[deleted]
darth_avocado 2y ago
I think the answer to the age old problem of "do algorithmic interviews help" is: it depends. Testing hard algorithms is probably not going to help in figuring out if someone is a good front end engineer who is only expected to build web or mobile apps. It will however be useful to identify if someone who's expected to regularly design system components, like say a cache layer for something, to actually understand the drawbacks of different caching strategies. Algorithms should be tested, but conditionally. On the other hand, every engineer needs to know data structures and should be tested on them.
BeetleB 2y ago
> Not every project needs them as much, but in large enough institutions, every engineer is likely to run into at least one project that does. If they can't figure it out and can't get help from someone who can, their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. I worked at a very large company (over 100K employees) for over a decade. I definitely encountered problems where knowing complexity and graph algorithms helped. Not knowing them would have cost the company almost nothing. I think you're subject to survivorship bias. Many (most?) large Fortune 500 companies make the bulk of their money by simple business logic and grunt work. Not by scaling. Internet based companies (which are the minority) are outliers.
- chipdart 2y ago
  > I think you're subject to survivorship bias. I think some people have a trick, and they desperately try to upsell their trick in order to subtly overstate the importance of their skillset. It's very hard to get anyone to be honest about the importance of their contributions when their livelihood depends on it.
chipdart 2y ago
> I continue to insist on algorithm & data structure interviews for software engineer candidates. Not every project needs them as much, but in large enough institutions, every engineer is likely to run into at least one project that does. If they can't figure it out and can't get help from someone who can, their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. I strongly disagree with this assertion. Using algorithms&data structures trivia in interviews means you will reject candidates who might happen to fail a inane trivia question in spite of otherwise being perfect fits and the ideal candidate. Imposing trivia on algorithms&data structures as a requirement communicates the idea that you don't believe anyone, including the ideal candidate, is unable to learn and develop this skill while working at your organization. This is telling regarding the bar which is set at your organization to nurture development in this skill set. If your org is indeed large, odds are you can find a subject matter expert within your ranks, which means it's utterly pointless to expect your engineers must be able to come up with perfectly optimal solutions on their own. They are a Slack message away from that. The biggest blindspot you convey in your post is the ladder-pulling aspect of data structures & algorithms in today's recruitment process. Subpar engineers use it to depict themselves as competent or raising the bar within their organization by coming up with obscure puzzles that bear absolutely no relationship to the job they are hiring for. They spend days hand-picking a problem for which they have a solution to, they go over the problem as many times as they need, they proceed to grill candidates on this trivia, they make their hire/no hire decision on the 20minute performance a candidate has after being presented with a riddle, and then go back to not have to face that issue ever again in any production setting. The worst offenders are interviewers who pull this stunt in spite of they themselves completely failing to understand their own problem.
- fl0ki 2y ago
  > Using algorithms&data structures trivia in interviews I never said trivia. By definition, trivia is stuff that is not important. I interview for things that are important to the work we do, and I give plenty of hints along the way to give a candidate a chance to show their reasoning abilities even if they have gaps in their knowledge or memory. This is what critics of algorithm interviewing conveniently ignore to make the interview seem unfair and unreasonable. If you had a bad interviewer I'm sorry to hear that. Formal interview training in places like Google explicitly includes how to hint and unblock people effectively, and hiring committee feedback-on-feedback can instruct interviewers how to be more fair and better reflect the candidate's potential. Maybe your interviewer did a bad job and later received that feedback, but it had already soured your perception of algorithm interviews. It's also a total myth that candidates are completely rejected because of one bad algorithm interview. If you have a day of five interviews, your success is not an AND() of the five, but it's also not an OR(). Whoever looks at the feedback weighs the various positive and negative signals to determine if it's a fit for the role. Skills in some areas can outweigh skills in other areas. Algorithms are one of those areas for a reason but it's still only one area and probably only one or two of the five or more interviews people will do. If you know a better way to interview for these skills before hiring someone, please tell the rest of us and change the industry for the better. Until then, this seems to be the best we can do. If your organization has very different needs, you can interview your candidates however best meets those needs. I know my organization needs algorithms and data structure skills, not just trivia but actual fundamental skills that generalize to novel problems we encounter along the way, and I'm going to keep interviewing for those skills. It's still just one interview out of several, and we've hired people that totally flunked my interview, so I'm not even gatekeeping, I'm giving the hiring team the signal they asked me to give them and the rest is up to them.

by fl0ki 2y ago

taeric 2y ago

Annoying counterpoint, of course, I've seen senior developers also stall out on avoiding quadratic code to the point that they didn't get the code delivered and then the project got scrapped. The most widespread failure I have seen, by far, is to write code that is so abstracted out that it isn't really clear on how to get the necessary parts inline to get at efficient code. You'll have an obvious path on how to load all data for something, but no obvious path on how to get just the relevant points for the algorithm in. This is super hilarious for code paths that try and hide network hits, such that you don't see that that loop is doing a network call per iteration.

esafak 2y ago
But computational complexity is a separate concern from level of abstraction.
- sopooneo 2y ago
  I would argue that, in practice, they are not completely separate. Too much abstraction makes it hard to tell what the underlying algorithm is. And even if you can figure that out, makes it hard to optimize.
- chipdart 2y ago
  > But computational complexity is a separate concern from level of abstraction. It really depends. Abstractions can and often do to otherwise avoidable copies. It's pointless to argue about computational complexity of an algorithm if you don't even notice that an abstraction you're using is adding a performance penalty in the form of running a linear-time algorithm in the hot path, and is even preventing the runtime from optimizing away problematic code.
saagarjha 2y ago
I’m curious for more details. I imagine this has to be more interesting than “Just write the double for loop.” “No.” “Ok I guess we’re canceling this project.”
- taeric 2y ago
  Yeah, it was rarely that silly. Usual thing I'm thinking of are the designs where folks think they can make something like facebook messenger with 200ms latency in the system. You know, "by design." (Of course... I swear that was an interview question I was given once.) What you wind up with is someone designing a giant system from parts they have read about that, honestly would probably work. However, it would also require retouching pretty much every system in the company to get things aligned. Which, isn't likely to ever happen. Put a different way, the longer a project spends in design phase, the higher the chance it will not get completed. And all too often people will spend a ton of time in design looking for ways to make sure what they are building will scale to success level loads.
zarathustreal 2y ago
Echoing my sibling commenters here, I’d love to hear more about this situation. Usually “inlining” and algorithmic complexity are juxtaposed as orthogonal types of optimization, with algorithmic optimization typically even being the reasoning for why the level of abstraction doesn’t matter very much. You typically get a much better speed up from going from O(n^2) to O(n) algorithms than from implementing e.g. Duff’s device, etc
- taeric 2y ago
  My apologies, forgot I posted. On phone, so will be brief. Will try a longer response later. (Apologies to other responses, not hitting them all.) Basic point is that many will abstract out data to different locations and ownership lifetimes. So, congrats, you kept us at a lower complexity, but failed to realize you need to first essentially reindex all of the data for that to happen. Now, I grant that often the problem is more in the data scattered over any number of databases. If there are reasons to keep that spread, though, hard to just sweep it aside. And then there are those that don't accept, "send it to a solver." Really annoying how often people assume they can easily beat cplex and friends.
chipdart 2y ago
> This is super hilarious for code paths that try and hide network hits, such that you don't see that that loop is doing a network call per iteration. True story: once I worked on a C++ project with a very ambitious junior developer who liked to portray himself as the local authority in performance and code quality. That junior developer once blocked a PR alleging performance problems because a function call was passing strings by value instead of moving them. Except that function call was only called once per run, and its responsibility was to make 1+n calls to a REST service. Whenever anyone complains about performance, ask for benchmarks. Odds are, even if they are technically right they are still wrong.

zerr 2y ago

CP (Competitive Programming, Olympiad/leetcode puzzles) uses Computer Science (Algos & DS) the same way as e.g. Physics and Biology use Mathematics, i.e. it is a completely different discipline with its own trivia knowledge and tricks that has nothing to do with a software engineering.

saagarjha 2y ago
Knowing how to do math is generally useful as a biologist or physicist. Sure there’s trivia or whatever but knowing how an array works is definitely important for software engineering, just like knowing how to do algebra is also important for biologists.
- mathgradthrow 2y ago
  "knowing how to do math", is roughly how one who does not know any math beyond the tricks describe above describes mathematics.
  - saagarjha 2y ago
    Sorry which tricks are you talking about?
- zerr 2y ago
  My point is that just because Math is useful for both physicists and biologists, that does not mean that for a physicist position you should interview the candidate (a physicist) in Biology. Biology "is a completely different discipline with its own trivia knowledge and tricks".
  - saagarjha 2y ago
    I mean making your entire interview just algorithms is stupid, yes
    - zerr 2y ago

tester756 2y ago

>their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. Anything that supports this? Because I don't believe it at all The only thing that I can think of that could cost company a lot is data corruption / database being destroyed Unless it is very specific case like HFT

TrackerFF 2y ago
That is pretty much my stance, too. I've seen a lot of critical fuckups in my years, but I have yet to see someone fuck up everything by using a woefully unsuitable algorithm. That, of course, does not mean that it doesn't happen - could very well be the case at the type of companies I haven't worked for, where performance is absolutely critical...but I have a hard time believing those places don't have safety guards at place. In fact, the vast majority of fabled horror stories I've heard in the industry are seemingly just that...fabled.
- fl0ki 2y ago
  I agree at the level of source code, but sometimes algorithm skills affect the whole architecture. Even simple things like where certain data and logic would optimally exist can have cascading and compounding consequences for the whole rest of the system. To see better solutions, you have to know what's possible with algorithms and data structures. To give a contrived example that I think should make sense no matter what kind of software people actually work on: Imagine nobody had ever figured out video compression and the only way we could see electronic video required orders of magnitude higher costs. VOD and video conferencing may not even be viable for most people. We'd still be live streaming television over analog radio. It would have vast consequences throughout other technologies and what we can do with them. If a service gets the kind of viral growth that their creators dearly hope they will, that will almost certainly involve a level of scale that would benefit from knowledge of algorithms (including practical mathematics as a subset) and the difference between a good or bad solution could have far-reaching consequences. Of course many problems do have well-known solutions that anybody can look up, but many projects do end up needing solutions novel to them.

jupp0r 2y ago

In the set of errors that cause production outages, inefficient algorithms is a pretty small subset though. I still think it can be a useful proxy for other things you might value in engineers though.

kemiller 2y ago

The problem is that small organizations cargo cult large organization interviews, when for most small organizations (ones that are not building infrastructure for others) speed of delivery is vastly more important than algorithmic purity. Lots of n^2 and even 2^n algorithms are in fact perfectly reasonable in many contexts. Even in large organizations, lots of people aren't touching the massively-scaled systems and are in fact just application developers.

deleted 2y ago

[deleted]

darth_avocado 2y ago

I think the answer to the age old problem of "do algorithmic interviews help" is: it depends. Testing hard algorithms is probably not going to help in figuring out if someone is a good front end engineer who is only expected to build web or mobile apps. It will however be useful to identify if someone who's expected to regularly design system components, like say a cache layer for something, to actually understand the drawbacks of different caching strategies. Algorithms should be tested, but conditionally. On the other hand, every engineer needs to know data structures and should be tested on them.

BeetleB 2y ago

> Not every project needs them as much, but in large enough institutions, every engineer is likely to run into at least one project that does. If they can't figure it out and can't get help from someone who can, their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. I worked at a very large company (over 100K employees) for over a decade. I definitely encountered problems where knowing complexity and graph algorithms helped. Not knowing them would have cost the company almost nothing. I think you're subject to survivorship bias. Many (most?) large Fortune 500 companies make the bulk of their money by simple business logic and grunt work. Not by scaling. Internet based companies (which are the minority) are outliers.

chipdart 2y ago
> I think you're subject to survivorship bias. I think some people have a trick, and they desperately try to upsell their trick in order to subtly overstate the importance of their skillset. It's very hard to get anyone to be honest about the importance of their contributions when their livelihood depends on it.

chipdart 2y ago

> I continue to insist on algorithm & data structure interviews for software engineer candidates. Not every project needs them as much, but in large enough institutions, every engineer is likely to run into at least one project that does. If they can't figure it out and can't get help from someone who can, their weak solution can easily cost the company and its other engineers far more than it would have cost to interview engineers properly and compensate the fewer qualified engineers commensurately. I strongly disagree with this assertion. Using algorithms&data structures trivia in interviews means you will reject candidates who might happen to fail a inane trivia question in spite of otherwise being perfect fits and the ideal candidate. Imposing trivia on algorithms&data structures as a requirement communicates the idea that you don't believe anyone, including the ideal candidate, is unable to learn and develop this skill while working at your organization. This is telling regarding the bar which is set at your organization to nurture development in this skill set. If your org is indeed large, odds are you can find a subject matter expert within your ranks, which means it's utterly pointless to expect your engineers must be able to come up with perfectly optimal solutions on their own. They are a Slack message away from that. The biggest blindspot you convey in your post is the ladder-pulling aspect of data structures & algorithms in today's recruitment process. Subpar engineers use it to depict themselves as competent or raising the bar within their organization by coming up with obscure puzzles that bear absolutely no relationship to the job they are hiring for. They spend days hand-picking a problem for which they have a solution to, they go over the problem as many times as they need, they proceed to grill candidates on this trivia, they make their hire/no hire decision on the 20minute performance a candidate has after being presented with a riddle, and then go back to not have to face that issue ever again in any production setting. The worst offenders are interviewers who pull this stunt in spite of they themselves completely failing to understand their own problem.

fl0ki 2y ago
> Using algorithms&data structures trivia in interviews I never said trivia. By definition, trivia is stuff that is not important. I interview for things that are important to the work we do, and I give plenty of hints along the way to give a candidate a chance to show their reasoning abilities even if they have gaps in their knowledge or memory. This is what critics of algorithm interviewing conveniently ignore to make the interview seem unfair and unreasonable. If you had a bad interviewer I'm sorry to hear that. Formal interview training in places like Google explicitly includes how to hint and unblock people effectively, and hiring committee feedback-on-feedback can instruct interviewers how to be more fair and better reflect the candidate's potential. Maybe your interviewer did a bad job and later received that feedback, but it had already soured your perception of algorithm interviews. It's also a total myth that candidates are completely rejected because of one bad algorithm interview. If you have a day of five interviews, your success is not an AND() of the five, but it's also not an OR(). Whoever looks at the feedback weighs the various positive and negative signals to determine if it's a fit for the role. Skills in some areas can outweigh skills in other areas. Algorithms are one of those areas for a reason but it's still only one area and probably only one or two of the five or more interviews people will do. If you know a better way to interview for these skills before hiring someone, please tell the rest of us and change the industry for the better. Until then, this seems to be the best we can do. If your organization has very different needs, you can interview your candidates however best meets those needs. I know my organization needs algorithms and data structure skills, not just trivia but actual fundamental skills that generalize to novel problems we encounter along the way, and I'm going to keep interviewing for those skills. It's still just one interview out of several, and we've hired people that totally flunked my interview, so I'm not even gatekeeping, I'm giving the hiring team the signal they asked me to give them and the rest is up to them.