Yes, there are helpful experiments along these lines, but you need to be careful drawing conclusions because the output is random and it's easy to fall for gamb

Yes, there are helpful experiments along these lines, but you need to be careful drawing conclusions because the output is random and it's easy to fall for gambling fallacies. This isn't like debugging a deterministic program; it can take more data than you might expect. For example, let's take "can it solve it at all." How many attempts will you give it before you give up? How many different prompts will you try? If it hasn't solved it yet, there's always an argument that it could, given a better prompt. Also, you might see a problem get solved the first time, take a screenshot, and then assume it can solve the problem reliably when it can't, it was just lucky that one time. Similarly for your other questions. If it does work, and you change one thing and it fails, is your change the cause or was it random? You need to try it both ways multiple times. For everyday purposes this often doesn't matter. It's like asking a random person for directions. If it works, maybe you don't care if it's repeatable, because you're never going to ask for the same directions again.