3 ms·

Hmm. I see what you’re saying, but I’m not sure that there is a hard cap. Pernosco has allowed me to solve harder problems than I would be able to solve without

by db48x 6y ago

Hmm. I see what you’re saying, but I’m not sure that there is a hard cap. Pernosco has allowed me to solve harder problems than I would be able to solve without it. Or perhaps there is a hard cap, but that cap is much higher than you would think. High–quality tools might get you closer to that cap than simpler tools would. For example, I participated in a project recently to extend Flex so that it could generate both Rust and Go code, in addition to C. Pernosco was an invaluable aid here. I was able to get Flex to generate a C program and also a Rust program, then compare the execution of those two programs. I recorded them both, and opened them each in their own window. Keeping them side–by–side, I could probe the innards of both programs simultaneously. Since both programs were supposed to implement the same state machine (and thus recognize the same list of tokens), it was easy to find places where the Rust version went off the rails. I was able to do in hours what would have taken days with any other method, if I had succeeded at all. The behavior of that state machine is not at all obvious, and I might never have unraveled it without Pernosco.

ajross 6y ago
> I was able to do in hours what would have taken days with any other method See... I think that's just wrong. Here's how I'd do that (knowing nothing about flex but having written similar state machine generators for non-lexing problems): + augment the generated code to dump the sequence of state transitions and tokens to stderr or whatever + run it in both environments + diff the output + Go back and add some metadata like __FILE__/__LINE__ or whatever as needed to find the bad data or state + Repeat as needed Now, is that as button-pushing-trivial as using a replay debugger? No, probably not. But it's not "days" of work either, it's like an hour to get a rig like that put together.
- db48x 6y ago
  Sure, I’ve done the same thing in the past; I used to use print debugging all the time. I still do, if I can’t use rr or Pernosco. But I think that you’re underestimating the time that it would take you. Sure, if you knew ahead of time what you would need to log, it might take an hour or even less. In practice though you’re going to end up going through that loop dozens of times, adding some logging to both sets of source code (or removing something that turned out to be useless or confusing), rebuilding, rerunning the test case, and diffing the outputs. Much better to record everything, and I do mean everything, using rr. Then you can add and remove data from your queries until you understand the bug (or rather, one of the handful of bugs that all existed at the same time) and how to fix it. No changes to the source code needed, no need to recompile them or rerun the test case. However, I am perfectly willing to suppose that you could do in hours what would have taken me days. There are a lot of people in the world, and some of them are bound to be better engineers than I am. In that case, I suspect that using Pernosco, you could have done it in minutes instead of hours. I think it’s safe enough to say that gives me a straight 5× improvement to my abilities, and it would do the same for you. Incidentally, roc has talked about adding automatic diffing of recordings to Pernosco. This would be diffing between multiple runs of the same program, rather than between two related programs that happen to be trying to accomplish the same task, but I can well imagine how much time it will save. Imagine those intermittent tests that fail one time out of a hundred. Currently you can rerun the test until you manage to capture the failure in a recording, and that helps to debug the problem quite a lot. But then imagine comparing a successful run to the failed run, so that you know the bug is in one of the differences between them. It’ll give anyone superpowers.

by db48x 6y ago

ajross 6y ago

> I was able to do in hours what would have taken days with any other method See... I think that's just wrong. Here's how I'd do that (knowing nothing about flex but having written similar state machine generators for non-lexing problems): + augment the generated code to dump the sequence of state transitions and tokens to stderr or whatever + run it in both environments + diff the output + Go back and add some metadata like __FILE__/__LINE__ or whatever as needed to find the bad data or state + Repeat as needed Now, is that as button-pushing-trivial as using a replay debugger? No, probably not. But it's not "days" of work either, it's like an hour to get a rig like that put together.

db48x 6y ago
Sure, I’ve done the same thing in the past; I used to use print debugging all the time. I still do, if I can’t use rr or Pernosco. But I think that you’re underestimating the time that it would take you. Sure, if you knew ahead of time what you would need to log, it might take an hour or even less. In practice though you’re going to end up going through that loop dozens of times, adding some logging to both sets of source code (or removing something that turned out to be useless or confusing), rebuilding, rerunning the test case, and diffing the outputs. Much better to record everything, and I do mean everything, using rr. Then you can add and remove data from your queries until you understand the bug (or rather, one of the handful of bugs that all existed at the same time) and how to fix it. No changes to the source code needed, no need to recompile them or rerun the test case. However, I am perfectly willing to suppose that you could do in hours what would have taken me days. There are a lot of people in the world, and some of them are bound to be better engineers than I am. In that case, I suspect that using Pernosco, you could have done it in minutes instead of hours. I think it’s safe enough to say that gives me a straight 5× improvement to my abilities, and it would do the same for you. Incidentally, roc has talked about adding automatic diffing of recordings to Pernosco. This would be diffing between multiple runs of the same program, rather than between two related programs that happen to be trying to accomplish the same task, but I can well imagine how much time it will save. Imagine those intermittent tests that fail one time out of a hundred. Currently you can rerun the test until you manage to capture the failure in a recording, and that helps to debug the problem quite a lot. But then imagine comparing a successful run to the failed run, so that you know the bug is in one of the differences between them. It’ll give anyone superpowers.