8 ms·

One thing that my grad advisor used to emphasize is “failing fast”. That is, for every problem of the form “Is it possible to do X?”, there is a dual problem of

One thing that my grad advisor used to emphasize is “failing fast”. That is, for every problem of the form “Is it possible to do X?”, there is a dual problem of “Can you prove it’s not possible to do X?” Before spending far too much time on the first question, it’s worth it to spend a little bit of time on the second: what’s the quickest way to show that this can’t possibly work? Often this takes the form of looking for statements such as “If X worked, then Y would work too”, and then you go test Y. Just because Y holds, doesn’t mean X does… but if it doesn’t, you know X doesn’t. It can feel like a bit of a diversion in the moment (“why am I wasting my time with Y, when I really care about X?”) but it has saved me months, possibly years of going down rabbit holes in my career. Likewise, it definitely helps with that anxiety you mention, because it means I at least have some motivation that my idea isn’t completely crazy.

srcreigh 3y ago
Can you share a story about this?
- H8crilA 3y ago
  Probably not quite what you wanted, but P=NP. When you look at what does that imply then it's hard to think that it holds. Scott Aaronson has a checklist on how to quicky reject a P?=NP paper, and one of the best methods is to check whether the paper proves something weaker first.
- nemo1618 3y ago
  Here's an example from my own work: I had a function that performed a 256-bit division. I thought there was a good chance it could be optimized, because the dividend was fixed: the function computed 2^256 / d. Surely, if the dividend was always the same (and such a nice, power-of-2 number!), there should be a way to exploit that, right? I poked at it for a few hours before it occurred to me that I could ask someone else who knew a lot about division. So I cold-emailed Daniel Lemire. He replied with the following argument: Suppose you had an algorithm that could compute 2^k / d much more efficiently than the typical algorithm. You could then use this algorithm to speed up any division n / d, by rewriting the expression as (n - 2^k)/d + 2^k/d. This works because both dividends will now be less than 2^k, and division algorithms get faster as the dividend gets smaller. In other words: it's not impossible that there's an efficient algorithm for 2^k / d, but if it does exist, then all the people who have dedicated their time to optimizing integer division over the years have missed some enormous low-hanging fruit. That was compelling enough for me, and I gave up trying to optimize my function. :)
  - deleted 3y ago
    [deleted]
  - peteradio 3y ago
    Feels very much like an appeal to authority though. Fast inverse square root was discovered/implemented as a side task for a first person shooter: https://en.wikipedia.org/wiki/Fast_inverse_square_root https://en.wikipedia.org/wiki/Fast_inverse_square_root.
    - munificent 3y ago
      Appeal to authority is only a fallacy when the person is an authority in an unrelated area. Appealing to an actual authority of the subject in question isn't a fallacy—it's the whole point of having authorities. "90% of dentists agree you should own a motorbike" is an appeal to authority fallacy. "90% of dentists agree you should floss" is not.
      - peteradio 3y ago
        I don't think that's the definition of the fallacy most people have.
        cmcconomy 3y ago
        argumentum ad populum much?
        peteradio 3y ago
        Lol. But seriously appeal to authority fallacy need not be only when the authority is outside their domain. For instance when authority makes a bad argument.
        esquivalience 3y ago
        The dividing line here is that it's fallacious to argue that something is true simply based on authority. That doesn't hold where there are external cogent reasons for following that authority (eg. in this example: it's a difficult problem that's well studied by many, and the body of work has not yet yeileded an answer such that it's unlikely that a more casual hunch will do so).
- gms7777 3y ago
  I often approach this from a “upper bound” and “lower bound” scenario perspective. So suppose I am trying to extract a signal from data that is noisy and sparse. Often the most time consuming part of solving the problem is figuring out how to deal with all of the noise, and I don’t want to waste time doing that just to come up with nothing in the end. A “upper bound” approach would be to simulate some data that is perfectly clean and meets all my model assumptions, and test my basic method on it. If it doesn’t work on the clean data, it won’t work on the noisy data. The “lower bound” approach would be to try the simplest and dumbest thing I can think of to deal with the noise in the real data. If I can pick up some amount of signal even with the dumb approach, it makes me much more confident that if I spend time making a sophisticated noise model, it will be worth it.
- Too 3y ago
  Back off the envelope calculations can usually quickly tell you if something is possible or not. For physical problems. Newtons law of conservation of energy is an obvious first sanity check. Given some objects mass and the desired travel, you can get the required energy and from there the required power. Before you even start designing a prototype.
bell-cot 3y ago
+1...but this may be better advice when you don't have any anxieties/paralysis about the problem than when you do. Turned-out-that-it-was-overconfident approaches to problems have probably wasted far more time than anxieties/paralysis has.
QuarterReptile 3y ago
A risk to this approach: if you (or someone working for you, who is trying to tackle a problem) starts to externalize the motivation to solve the problem, optimization may start to favor 'failing fast' rather than pursuing best approaches (e.g. there's only one chance to try something per day, and you/they develop a pattern of pursuing the simplest possibility because it takes the least research and thus makes for the easiest increment of work day.)
- gms7777 3y ago
  This is possibly a risk, but it's not one I've ever seen play out. I don't think most people favor failing fast. It's something you do because it's good practice, while crossing your fingers that it doesn't fail. I think this is because in order to prove that something doesn't work, you first have to come up with that something -- deciding on a problem and thinking of a potential solution. While some people might find it fun to disprove other people's ideas, I don't think most really find it satisfying to disprove their own. I think the bigger risk to this approach is that it can be disheartening and demotivating to see your ideas that you already got attached to and were looking forward to pursuing shut down. But it's still less demotivating than spending a year on it before watching it fail.
  - QuarterReptile 3y ago
    I had a group working under me who were troubleshooting an intermittent fault in some control circruitry for a nuclear reactor, and they started phoning it in just like this when they realized that there was only enough appetite to describe and propose weird problems to the boss once each day. After about a week, the goal became achievement of the daily failure with a minimum of preparation. It was a highly solvable problem, but they were all avoiding the necessary step of serious research.
    - gms7777 3y ago
      That's reasonable. I think being in the academic world, I don't see it as often because daily achievement isn't really rewarded -- it's all about individual output in the longer term (manuscripts, grants, etc. ), and every "failure" feels like a step away from having a completed manuscript, even when it's necessary. If anything, it's why a lot of academics tend to avoid the "fail fast" techniques, because at a daily level, it feels more productive to be working towards a solution (even if it is a waste of time) than shooting down potential solution after solution.
cliff_badger 3y ago
I sorry, but I am truly confused at the logic here. If X, then Y. To test X works you first you start on Y? If Y succeeds then you don't know if X failed or not && if Y fails then you know that X failed too?
- kqr 3y ago
  Modus tollens. If X -> Y, then the only way for Y to be false is if X is false.
- raincom 3y ago
  Contrapositive (if ~Y, then ~X) is logically equivalent to the original implication (if X, then Y). Instead of proving the latter, you can prove the former contrapositive.
- cortesoft 3y ago
  In formal logic, it is known as "modus tollens"... if 'if x then y' is true, then 'if not y then not x' is also true. The inverse is not necessarily true, however: 'if x then y' does not mean 'if y then x'. In the case of a an X that is hard to figure out on its own, and it is easy to figure out Y, then it might be worth testing Y first, even if you will only get useful information if Y is false.
- ndepoel 3y ago
  An example of how you could fill this in: identify a small subset of the problem that is relatively quick and easy to test. If the entirety of the problem can be solved, then for sure this small subset has to be solvable too. If you can't solve this small sub-problem, then you know there's not much point diving into the larger rabbit hole yet. However if you do solve the sub-problem, then that might show you the potential that exists, it may allow you to already look at adjacent problems using the results of this early test, and also important: it will give you additional motivation to keep going.
- tetha 3y ago
  Well, assume you have a very, very efficient algorithm to check if normal boolean expressions have a solution. It checks some constant number of things for each variable, and then outputs a solution and it works for a large number of things. Using the same logic as the parent comment, I would be very suspicious of the general applicability of this algorithm. Because, if this algorithm was correct, P would be equal to NP based off of this algorithm, because you'd have a polynomial solution to SAT. This, in turn, would invalidate pretty much all practical cryptography, most likely turn bitcoin on its head, and cause a significant number of other disruptions. That is this line of thinking. The formal name is Modus Tollens, but it basically says: If your assumption is right, I can propose a much more preposterous assumption that would also be right. Or I could propose something enabled or validated by your assumption which is much easier to invalidate. I constantly use this in stupid security discussions as well. There are so many people asking about silly threat scenarios, but the specific threat scenarios generally imply that an attacker already has control of critical infrastructure anyway, and all of these nitpicky things they wonder about are just not relevant. Like, if you assume this action to be possible, they have control of the secret management solution, and then we are doomed already.
kqr 3y ago
This is also a useful technique to validate a new product before committing too many resources building it. Imagine that you have built it technically flawlessly but it's not selling copies. Why might that be? Draw up a list of reasons and test each of those as hypotheses before you build the product itself.
bbertelsen 3y ago
Love this comment because it translates so well to any mentally taxing endeavor. Writing a long-running program? Set a few assertions up front so that it fails immediately before wasting your time. Think about those "failure" condition assertions up front and save your time on practically any experiment. Even chess players do this by surrendering early when they know a game has reached a conclusion.
- danuker 3y ago
  > Writing a long-running program? If it applies, have a "testing" version that runs quickly, on limited (but error-prone) data cases. Ideally, run it with the rest of your test suite.
alberth 3y ago
Isn’t this the Scientific Method?
- burnished 3y ago
  This is a method of disproving a hypothesis, so in that sense yes.
hinkley 3y ago
Peripheral to this, some of my best refactors have come from trying to document why we can’t do X, explaining that we can’t do X because of Y, and realizing that Y does not need to exist. There comes a point where explaining why we can’t have something is more painful than just fixing it.
muzani 3y ago
Something similar we do at work is write a quick document on why something isn't possible. A bug can't be reproduced? Take a video of it working fine, and show the logical path for how the bug couldn't happen. Can't build X in 3 months? What's the minimum for X? What's the missing component that takes 2 months? We can't have a partnership with A, but why wouldn't it be possible to have a partnership with A's competitor?

srcreigh 3y ago

Can you share a story about this?

H8crilA 3y ago
Probably not quite what you wanted, but P=NP. When you look at what does that imply then it's hard to think that it holds. Scott Aaronson has a checklist on how to quicky reject a P?=NP paper, and one of the best methods is to check whether the paper proves something weaker first.
nemo1618 3y ago
Here's an example from my own work: I had a function that performed a 256-bit division. I thought there was a good chance it could be optimized, because the dividend was fixed: the function computed 2^256 / d. Surely, if the dividend was always the same (and such a nice, power-of-2 number!), there should be a way to exploit that, right? I poked at it for a few hours before it occurred to me that I could ask someone else who knew a lot about division. So I cold-emailed Daniel Lemire. He replied with the following argument: Suppose you had an algorithm that could compute 2^k / d much more efficiently than the typical algorithm. You could then use this algorithm to speed up any division n / d, by rewriting the expression as (n - 2^k)/d + 2^k/d. This works because both dividends will now be less than 2^k, and division algorithms get faster as the dividend gets smaller. In other words: it's not impossible that there's an efficient algorithm for 2^k / d, but if it does exist, then all the people who have dedicated their time to optimizing integer division over the years have missed some enormous low-hanging fruit. That was compelling enough for me, and I gave up trying to optimize my function. :)
- deleted 3y ago
  [deleted]
- peteradio 3y ago
  Feels very much like an appeal to authority though. Fast inverse square root was discovered/implemented as a side task for a first person shooter: https://en.wikipedia.org/wiki/Fast_inverse_square_root https://en.wikipedia.org/wiki/Fast_inverse_square_root.
  - munificent 3y ago
    Appeal to authority is only a fallacy when the person is an authority in an unrelated area. Appealing to an actual authority of the subject in question isn't a fallacy—it's the whole point of having authorities. "90% of dentists agree you should own a motorbike" is an appeal to authority fallacy. "90% of dentists agree you should floss" is not.
    - peteradio 3y ago
      I don't think that's the definition of the fallacy most people have.
      - cmcconomy 3y ago
        argumentum ad populum much?
        peteradio 3y ago
        Lol. But seriously appeal to authority fallacy need not be only when the authority is outside their domain. For instance when authority makes a bad argument.
        esquivalience 3y ago
        The dividing line here is that it's fallacious to argue that something is true simply based on authority. That doesn't hold where there are external cogent reasons for following that authority (eg. in this example: it's a difficult problem that's well studied by many, and the body of work has not yet yeileded an answer such that it's unlikely that a more casual hunch will do so).
gms7777 3y ago
I often approach this from a “upper bound” and “lower bound” scenario perspective. So suppose I am trying to extract a signal from data that is noisy and sparse. Often the most time consuming part of solving the problem is figuring out how to deal with all of the noise, and I don’t want to waste time doing that just to come up with nothing in the end. A “upper bound” approach would be to simulate some data that is perfectly clean and meets all my model assumptions, and test my basic method on it. If it doesn’t work on the clean data, it won’t work on the noisy data. The “lower bound” approach would be to try the simplest and dumbest thing I can think of to deal with the noise in the real data. If I can pick up some amount of signal even with the dumb approach, it makes me much more confident that if I spend time making a sophisticated noise model, it will be worth it.
Too 3y ago
Back off the envelope calculations can usually quickly tell you if something is possible or not. For physical problems. Newtons law of conservation of energy is an obvious first sanity check. Given some objects mass and the desired travel, you can get the required energy and from there the required power. Before you even start designing a prototype.

bell-cot 3y ago

+1...but this may be better advice when you don't have any anxieties/paralysis about the problem than when you do. Turned-out-that-it-was-overconfident approaches to problems have probably wasted far more time than anxieties/paralysis has.

QuarterReptile 3y ago

A risk to this approach: if you (or someone working for you, who is trying to tackle a problem) starts to externalize the motivation to solve the problem, optimization may start to favor 'failing fast' rather than pursuing best approaches (e.g. there's only one chance to try something per day, and you/they develop a pattern of pursuing the simplest possibility because it takes the least research and thus makes for the easiest increment of work day.)

gms7777 3y ago
This is possibly a risk, but it's not one I've ever seen play out. I don't think most people favor failing fast. It's something you do because it's good practice, while crossing your fingers that it doesn't fail. I think this is because in order to prove that something doesn't work, you first have to come up with that something -- deciding on a problem and thinking of a potential solution. While some people might find it fun to disprove other people's ideas, I don't think most really find it satisfying to disprove their own. I think the bigger risk to this approach is that it can be disheartening and demotivating to see your ideas that you already got attached to and were looking forward to pursuing shut down. But it's still less demotivating than spending a year on it before watching it fail.
- QuarterReptile 3y ago
  I had a group working under me who were troubleshooting an intermittent fault in some control circruitry for a nuclear reactor, and they started phoning it in just like this when they realized that there was only enough appetite to describe and propose weird problems to the boss once each day. After about a week, the goal became achievement of the daily failure with a minimum of preparation. It was a highly solvable problem, but they were all avoiding the necessary step of serious research.
  - gms7777 3y ago
    That's reasonable. I think being in the academic world, I don't see it as often because daily achievement isn't really rewarded -- it's all about individual output in the longer term (manuscripts, grants, etc. ), and every "failure" feels like a step away from having a completed manuscript, even when it's necessary. If anything, it's why a lot of academics tend to avoid the "fail fast" techniques, because at a daily level, it feels more productive to be working towards a solution (even if it is a waste of time) than shooting down potential solution after solution.

cliff_badger 3y ago

I sorry, but I am truly confused at the logic here. If X, then Y. To test X works you first you start on Y? If Y succeeds then you don't know if X failed or not && if Y fails then you know that X failed too?

kqr 3y ago
Modus tollens. If X -> Y, then the only way for Y to be false is if X is false.
raincom 3y ago
Contrapositive (if ~Y, then ~X) is logically equivalent to the original implication (if X, then Y). Instead of proving the latter, you can prove the former contrapositive.
cortesoft 3y ago
In formal logic, it is known as "modus tollens"... if 'if x then y' is true, then 'if not y then not x' is also true. The inverse is not necessarily true, however: 'if x then y' does not mean 'if y then x'. In the case of a an X that is hard to figure out on its own, and it is easy to figure out Y, then it might be worth testing Y first, even if you will only get useful information if Y is false.
ndepoel 3y ago
An example of how you could fill this in: identify a small subset of the problem that is relatively quick and easy to test. If the entirety of the problem can be solved, then for sure this small subset has to be solvable too. If you can't solve this small sub-problem, then you know there's not much point diving into the larger rabbit hole yet. However if you do solve the sub-problem, then that might show you the potential that exists, it may allow you to already look at adjacent problems using the results of this early test, and also important: it will give you additional motivation to keep going.
tetha 3y ago
Well, assume you have a very, very efficient algorithm to check if normal boolean expressions have a solution. It checks some constant number of things for each variable, and then outputs a solution and it works for a large number of things. Using the same logic as the parent comment, I would be very suspicious of the general applicability of this algorithm. Because, if this algorithm was correct, P would be equal to NP based off of this algorithm, because you'd have a polynomial solution to SAT. This, in turn, would invalidate pretty much all practical cryptography, most likely turn bitcoin on its head, and cause a significant number of other disruptions. That is this line of thinking. The formal name is Modus Tollens, but it basically says: If your assumption is right, I can propose a much more preposterous assumption that would also be right. Or I could propose something enabled or validated by your assumption which is much easier to invalidate. I constantly use this in stupid security discussions as well. There are so many people asking about silly threat scenarios, but the specific threat scenarios generally imply that an attacker already has control of critical infrastructure anyway, and all of these nitpicky things they wonder about are just not relevant. Like, if you assume this action to be possible, they have control of the secret management solution, and then we are doomed already.

kqr 3y ago

This is also a useful technique to validate a new product before committing too many resources building it. Imagine that you have built it technically flawlessly but it's not selling copies. Why might that be? Draw up a list of reasons and test each of those as hypotheses before you build the product itself.

bbertelsen 3y ago

Love this comment because it translates so well to any mentally taxing endeavor. Writing a long-running program? Set a few assertions up front so that it fails immediately before wasting your time. Think about those "failure" condition assertions up front and save your time on practically any experiment. Even chess players do this by surrendering early when they know a game has reached a conclusion.

danuker 3y ago
> Writing a long-running program? If it applies, have a "testing" version that runs quickly, on limited (but error-prone) data cases. Ideally, run it with the rest of your test suite.

alberth 3y ago

Isn’t this the Scientific Method?

burnished 3y ago
This is a method of disproving a hypothesis, so in that sense yes.

hinkley 3y ago

Peripheral to this, some of my best refactors have come from trying to document why we can’t do X, explaining that we can’t do X because of Y, and realizing that Y does not need to exist. There comes a point where explaining why we can’t have something is more painful than just fixing it.

muzani 3y ago

Something similar we do at work is write a quick document on why something isn't possible. A bug can't be reproduced? Take a video of it working fine, and show the logical path for how the bug couldn't happen. Can't build X in 3 months? What's the minimum for X? What's the missing component that takes 2 months? We can't have a partnership with A, but why wouldn't it be possible to have a partnership with A's competitor?