3 ms·

Everyone says this in the abstract but to concrete examples they shug and say, of course that approach doesn't work, they did X, Y, Z wrong, they should have gi

Everyone says this in the abstract but to concrete examples they shug and say, of course that approach doesn't work, they did X, Y, Z wrong, they should have given it more effort, it could have worked if done properly / this can obviously never work, everyone knew already, it's nothing new etc.

DrScientist 3mo ago
It would need to be beautifully done science - indeed the level of rigour required to confidently show no effect is probably quite a bit larger than the level of rigour required to confidently show a large effect. And that's probably one of the main reasons it's not done more often.
baron3dl 3mo ago
Isn't that exactly the point? If in my conception that concrete example eliminates that single branch, then I can try the others, and only the others. Or confirm the negative. Which is also valid work. In general, your point stands for armchair researcher.
- bonoboTP 3mo ago
  No, good researchers also dismiss such things, because most research out there is crap. If you reward negative results, you'll have a flood of them, and the reason that they are negative will mostly look like it's incompetence (and mostly will be, but even the small not-incompetent will be hard to distinguish). It's asymmetric. If you make something work, beat a benchmark, invent a drug that works etc, the exact way you arrived there has some leeway. In the end, the thing worked out so it's worth knowing about. If your project failed, there can be a million reasons for that, and it's not necessarily that the initial hypothesis or initial idea has been refuted. E.g. in ML, your model didn't learn the task. Okay, there could be a million knobs (hyperparameters) that you set up wrong, or you implemented it wrong, or you should have just added learning rate warmup, or this or that, a million things possible. People fail all the time at projects that others then manage to do later on. Science is not like simply asking the universe some clear question and getting a clear answer. It's a very messy process and even professionals are not super great at it, or at least they simply cannot afford to put so much effort in each single project to make it absolutely airtight such that the failure to make it work can be a legit refutation of the main idea.

DrScientist 3mo ago

It would need to be beautifully done science - indeed the level of rigour required to confidently show no effect is probably quite a bit larger than the level of rigour required to confidently show a large effect. And that's probably one of the main reasons it's not done more often.

baron3dl 3mo ago

Isn't that exactly the point? If in my conception that concrete example eliminates that single branch, then I can try the others, and only the others. Or confirm the negative. Which is also valid work. In general, your point stands for armchair researcher.

bonoboTP 3mo ago
No, good researchers also dismiss such things, because most research out there is crap. If you reward negative results, you'll have a flood of them, and the reason that they are negative will mostly look like it's incompetence (and mostly will be, but even the small not-incompetent will be hard to distinguish). It's asymmetric. If you make something work, beat a benchmark, invent a drug that works etc, the exact way you arrived there has some leeway. In the end, the thing worked out so it's worth knowing about. If your project failed, there can be a million reasons for that, and it's not necessarily that the initial hypothesis or initial idea has been refuted. E.g. in ML, your model didn't learn the task. Okay, there could be a million knobs (hyperparameters) that you set up wrong, or you implemented it wrong, or you should have just added learning rate warmup, or this or that, a million things possible. People fail all the time at projects that others then manage to do later on. Science is not like simply asking the universe some clear question and getting a clear answer. It's a very messy process and even professionals are not super great at it, or at least they simply cannot afford to put so much effort in each single project to make it absolutely airtight such that the failure to make it work can be a legit refutation of the main idea.