4 ms·

i wish more people would publish the things that didn't work. i'd like everything, but the exploration of searched negative space is so wasteful.

locknitpicker 2mo ago
> I wish more people would publish the things that didn't work. This would be a pointless endeavour. One of the most basic mantras of science is "absence of evidence is not evidence of absence". So just because something didn't worked out for you that doesn't mean it doesn't work out for others, or even yourself in the future.
- don_esteban 2mo ago
  You are talking about a different thing, i.e. you have slipped in a level of abstraction that was not there before: In 'searching a path from A to B in a maze' language: The original statement was: (1) The branch to the left from A is a dead end. Your interpretation: (2) There is no path from A to B. (1) is still very useful (reducing the wasted effort) for those trying to find a path from A to B. The OP's point is that in the current environment only positive results are rewarded (I found the path from A to B!), not the negative ones like (1).
  - locknitpicker 2mo ago
    > The original statement was: (1) The branch to the left from A is a dead end. This is where you get things wrong at a very basic and fundamental level. Just because you failed to explore branch A, that does not mean it is a dead end. It just means you came up empty. That is why science is based on observations and theories: it is based on building up on ideas and what works and can be proven. Otherwise you will left with useless papers such as "Bicycles are a dead end because I tried to ride one and I fell".
    - don_esteban 2mo ago
      You are assuming incompetence on the scientist saying that the branch from the left of A is a dead end. While nobody is perfect, there are numerous perfectly valid scientific negative results. You know, there exist things like impossibility proofs in mathematics and computer science. There are equivalents in other sciences (e.g. if X was true, that would lead to Y that is easily observable and clearly not observed). Sometimes that implication has assumptions that might change once the technology/society changes, other times it holds true regardless. Unicycles are a dead end as a practical transportation, because the bicycles have them beat in every way (except portability). A scientific result would be much more along the lines of 'Bicycles without gears have limited applicability, especially in hilly terrain.'
      - bonoboTP 2mo ago
        There are different types of research. Negative AI results are not like Michelson-Morley experiment in physics that can positively establish the lack of something. It's more of a craft, the "make it work" style of thing. Give the same task or goal to two different teams, you will see massively different outcomes. Simply failing to make it work is not conclusive. To make such a negative result acceptable in AI, you'd have to have some clear reason why you think that your particular setup should produce the result you want, that exact configuration and architecture, dataset etc. There are countless projects in AI that fail. And it's not clear at all that it refutes any abstract hypothesis. It's a get-your-hands-dirty field. It can make or break a project whether someone has that tacit knowledge, that black magic experience to know how to properly do the project. People can generate extremely many ideas. You'd need to convince me that your idea (among a million others that people are trying each day) is so significant that its failure is in itself interesting. If you were to review for AI conferences, you'd see the flood of papers that claim to achieve 0.5% or 1% improvement on some benchmark. Now imagine that they didn't even have that to show for it. It got worse by 2% after trying their random idea. Who cares then? Even the +1% with a random idea is quite annoying to accept. But if their random idea really made something work much better, I will at least have some reason to want to see what may be going on there, there can be some signal. With negative results, it's very uninteresting.
znpy 2mo ago
I think you would be interested in the Journal of Trial and Error: https://journal.trialanderror.org/ https://journal.trialanderror.org/
- ciphercipher 2mo ago
  a goldmine!
- baron3dl 2mo ago
  WOW! Thank you!
bonoboTP 2mo ago
Everyone says this in the abstract but to concrete examples they shug and say, of course that approach doesn't work, they did X, Y, Z wrong, they should have given it more effort, it could have worked if done properly / this can obviously never work, everyone knew already, it's nothing new etc.
- DrScientist 2mo ago
  It would need to be beautifully done science - indeed the level of rigour required to confidently show no effect is probably quite a bit larger than the level of rigour required to confidently show a large effect. And that's probably one of the main reasons it's not done more often.
- baron3dl 2mo ago
  Isn't that exactly the point? If in my conception that concrete example eliminates that single branch, then I can try the others, and only the others. Or confirm the negative. Which is also valid work. In general, your point stands for armchair researcher.
  - bonoboTP 2mo ago
    No, good researchers also dismiss such things, because most research out there is crap. If you reward negative results, you'll have a flood of them, and the reason that they are negative will mostly look like it's incompetence (and mostly will be, but even the small not-incompetent will be hard to distinguish). It's asymmetric. If you make something work, beat a benchmark, invent a drug that works etc, the exact way you arrived there has some leeway. In the end, the thing worked out so it's worth knowing about. If your project failed, there can be a million reasons for that, and it's not necessarily that the initial hypothesis or initial idea has been refuted. E.g. in ML, your model didn't learn the task. Okay, there could be a million knobs (hyperparameters) that you set up wrong, or you implemented it wrong, or you should have just added learning rate warmup, or this or that, a million things possible. People fail all the time at projects that others then manage to do later on. Science is not like simply asking the universe some clear question and getting a clear answer. It's a very messy process and even professionals are not super great at it, or at least they simply cannot afford to put so much effort in each single project to make it absolutely airtight such that the failure to make it work can be a legit refutation of the main idea.

locknitpicker 2mo ago

> I wish more people would publish the things that didn't work. This would be a pointless endeavour. One of the most basic mantras of science is "absence of evidence is not evidence of absence". So just because something didn't worked out for you that doesn't mean it doesn't work out for others, or even yourself in the future.

don_esteban 2mo ago
You are talking about a different thing, i.e. you have slipped in a level of abstraction that was not there before: In 'searching a path from A to B in a maze' language: The original statement was: (1) The branch to the left from A is a dead end. Your interpretation: (2) There is no path from A to B. (1) is still very useful (reducing the wasted effort) for those trying to find a path from A to B. The OP's point is that in the current environment only positive results are rewarded (I found the path from A to B!), not the negative ones like (1).
- locknitpicker 2mo ago
  > The original statement was: (1) The branch to the left from A is a dead end. This is where you get things wrong at a very basic and fundamental level. Just because you failed to explore branch A, that does not mean it is a dead end. It just means you came up empty. That is why science is based on observations and theories: it is based on building up on ideas and what works and can be proven. Otherwise you will left with useless papers such as "Bicycles are a dead end because I tried to ride one and I fell".
  - don_esteban 2mo ago
    You are assuming incompetence on the scientist saying that the branch from the left of A is a dead end. While nobody is perfect, there are numerous perfectly valid scientific negative results. You know, there exist things like impossibility proofs in mathematics and computer science. There are equivalents in other sciences (e.g. if X was true, that would lead to Y that is easily observable and clearly not observed). Sometimes that implication has assumptions that might change once the technology/society changes, other times it holds true regardless. Unicycles are a dead end as a practical transportation, because the bicycles have them beat in every way (except portability). A scientific result would be much more along the lines of 'Bicycles without gears have limited applicability, especially in hilly terrain.'
    - bonoboTP 2mo ago
      There are different types of research. Negative AI results are not like Michelson-Morley experiment in physics that can positively establish the lack of something. It's more of a craft, the "make it work" style of thing. Give the same task or goal to two different teams, you will see massively different outcomes. Simply failing to make it work is not conclusive. To make such a negative result acceptable in AI, you'd have to have some clear reason why you think that your particular setup should produce the result you want, that exact configuration and architecture, dataset etc. There are countless projects in AI that fail. And it's not clear at all that it refutes any abstract hypothesis. It's a get-your-hands-dirty field. It can make or break a project whether someone has that tacit knowledge, that black magic experience to know how to properly do the project. People can generate extremely many ideas. You'd need to convince me that your idea (among a million others that people are trying each day) is so significant that its failure is in itself interesting. If you were to review for AI conferences, you'd see the flood of papers that claim to achieve 0.5% or 1% improvement on some benchmark. Now imagine that they didn't even have that to show for it. It got worse by 2% after trying their random idea. Who cares then? Even the +1% with a random idea is quite annoying to accept. But if their random idea really made something work much better, I will at least have some reason to want to see what may be going on there, there can be some signal. With negative results, it's very uninteresting.

znpy 2mo ago

I think you would be interested in the Journal of Trial and Error: https://journal.trialanderror.org/ https://journal.trialanderror.org/

ciphercipher 2mo ago
a goldmine!
baron3dl 2mo ago
WOW! Thank you!

bonoboTP 2mo ago

Everyone says this in the abstract but to concrete examples they shug and say, of course that approach doesn't work, they did X, Y, Z wrong, they should have given it more effort, it could have worked if done properly / this can obviously never work, everyone knew already, it's nothing new etc.

DrScientist 2mo ago
It would need to be beautifully done science - indeed the level of rigour required to confidently show no effect is probably quite a bit larger than the level of rigour required to confidently show a large effect. And that's probably one of the main reasons it's not done more often.
baron3dl 2mo ago
Isn't that exactly the point? If in my conception that concrete example eliminates that single branch, then I can try the others, and only the others. Or confirm the negative. Which is also valid work. In general, your point stands for armchair researcher.
- bonoboTP 2mo ago
  No, good researchers also dismiss such things, because most research out there is crap. If you reward negative results, you'll have a flood of them, and the reason that they are negative will mostly look like it's incompetence (and mostly will be, but even the small not-incompetent will be hard to distinguish). It's asymmetric. If you make something work, beat a benchmark, invent a drug that works etc, the exact way you arrived there has some leeway. In the end, the thing worked out so it's worth knowing about. If your project failed, there can be a million reasons for that, and it's not necessarily that the initial hypothesis or initial idea has been refuted. E.g. in ML, your model didn't learn the task. Okay, there could be a million knobs (hyperparameters) that you set up wrong, or you implemented it wrong, or you should have just added learning rate warmup, or this or that, a million things possible. People fail all the time at projects that others then manage to do later on. Science is not like simply asking the universe some clear question and getting a clear answer. It's a very messy process and even professionals are not super great at it, or at least they simply cannot afford to put so much effort in each single project to make it absolutely airtight such that the failure to make it work can be a legit refutation of the main idea.