4 ms·
I'll put it quite simply. If you promote an ML system that is ostensibly a scientific content generator, the expectation is that it will regurgitate facts, not
by dekhn 4y ago
I'll put it quite simply. If you promote an ML system that is ostensibly a scientific content generator, the expectation is that it will regurgitate facts, not lies. If you can't get that right, don't release it, except to publish a "negative results" paper.
- seydor 4y agoThat is just false. The model had a disclaimer on every page that it is not to be trusted and that it hallucinates. Strawmanning is not a nice academic retort
- paganel 4y agoTo be honest projects that constantly and consistently generate smart falsehoods (the "bears in the sky" that the author of the article mentions) don't have any scientific need, so to speak. Or, if they do, I fail to see it.
- seydor 4y agoThey do. Try asking it about a field that is very novel or very very niche, and it will point out various links. Biology connections (which are often very vague and open) are good for that. I found that it generated some links between mechanisms that i found interesting. Unfortunately the model is down so i cant review it.
- mynameisvlad 4y agoBut how can you possibly trust anything the model spits out as accurste and true if it's happy to spit out a whole thing on bears in space, something that plainly never happened. Sure, it be spitting out these links between mechanisms, but it could have pulled them out of the same place it pulled an article about bears in space: the server's ass. And verification is going to take you far longer on something that actually seems realistic, which is a massive problem.
- seydor 4y agoIt's not supposed to spit out truth. Meta may have overhyped it but they were careful not to claim that it makes true claims, but that it compiles knowledge, and they had a disclaimer that it hallucinates fact. Anyone who has seen language models knows that if you ask it about space bears, it won't reply that there re no space bears, instead it will try to create a link between space and bears. It seems to me that people were deliberately using impossible inputs so that they can claim the model is dangerous for doign what a language model does. And their ultimate purpose was to defame the model and take it down. (And BTW we 've heard those "dangerousness" BS claims before) The usefulness of the model was in under-researched fields and questions, where it can provide actually useful directions.
- mynameisvlad 4y agoMy whole point is said directions are completely useless if you can't trust the system to be based on factual principles. It's easy to dismiss "bears in space" as not factual. It's harder to dismiss "the links between two underresearched fields in biology" without putting in an exorbitant amount of work. Work which likely is going to be useless if the model is just as happy to spit out "bears in space". And that was what I had already said. I asked you how you could possibly trust those links it provided. Because you can't. They may very well be novel links that had never been researched. Or they could have been bears in space. A scientific model which doesn't guarantee some degree of factuality in its responses is completely useless as a scientific model.
- obmelvin 4y agoThey may have had a disclaimer, but they also presented it as a tool to help one summarize research. It was clearly unreliable there. And who cares if it has some good results? How do you know whether it's spitting nonsense* or accurate information? Yes, I do think there is value in engaging with the community for models that aren't perfect. But it needs more work before it can be framed as useful tool for scientific research. * I mean subtle nonsense. Space bears are easy to distinguish..but what if the errors aren't as 'crazy'?
- mrguyorama 4y ago
- dekhn 4y agoI'm saying that if your model needs something like that (I certainly didn't see the disclaimer) then it's not appropriate for science. THe expectation of accuracy is much higher.
- seydor 4y agoYou can see it in this screenshot: https://pbs.twimg.com/media/FhpD9kIWIAAqKCZ.jpg https://pbs.twimg.com/media/FhpD9kIWIAAqKCZ.jpg > WARNING: Outputs may be unreliable! Language Models are prone to hallucinate text. Trained on data up to July 2022 The model wasn't supposed to make science for you and any scientist who used it like that should probably not be a scientist. It's a language model, i would think people know what that means by now
- dekhn 4y agoYes, it's clearly part of my point that language models are insufficient to produce high quality professional text. Facebook's intent with the model was quite clear; the abstract of the paper says this: "In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge... these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community." I didn't expect it to make science for me- that would be quite impressive (it's been the guiding principle for my 30+ years in science)- but I do expect the demo to be less bad.
- msabalau 4y agoSticking a bunch of disclaimers onto cyanide doesn't mean that it is useful to describe it as almond milk. If they had done an adequate and accurate job explaining what, if any, plausible value or potential Galactica had, warnings would not have to have to carry so much weight. Or their PR could have focused on their tricks and optimizations, and not characterised Galactica at all. Instead they'll be known as the people who described a science flavored word salad generator as if it was a tool useful for "summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more" If they had, for example, positioned this as a tool for creating unscientific, but academic sounding anti-vax propaganda, people could have questioned their morals, but not the "fitness to purpose" of their tool.
- bitL 4y agoThey proved that Galactica would be an excellent front-end for a differentiable information retrieval system. This definitely moved the research forward; work on the back-end (i.e. fact retrieval usable for LLMs) is still needed.