13 ms·

Deep learning gets the glory, deep fact checking gets ignored

609 points by chmaynard 1y ago

amelius 1y ago
Before making AI do research, perhaps we should first let it __reproduce__ research. For example, give it a paper of some deep learning technique and make it produce an implementation of that paper. Before it can do that, I have no hope that it can produce novel ideas.
- YossarianFrPrez 1y ago
  Seconded, as not only is this an interesting idea, it might also help solve the issue of checking for reproducibility. Yet even then human evaluators would need to go over the AI-reproduced research with a fine-toothed comb. Practically speaking, I think there are roles for current LLMs in research. One is in the peer review process. LLMs can assist in evaluating the data-processing code used by scientists. Another is for brainstorming and the first pass at lit reviews.
- ojosilva 1y ago
  I thought you were going to say "give AI the first part of a paper (prompt) and let it finish it (completion)" as a validation AI can produce science at par with research results. Before it can do that, I have no hope that it can produce novel ideas.
  - bee_rider 1y ago
    I guess it would also need the experimental data. It would, I guess, also need some ability to do little experiments and write off those ideas as not worth following up on…
    - szundi 1y ago
      [dead]
  - DrScientist 1y ago
    I once had a university assignment where they provided the figures from a paper and we had to write the paper around the just the given figures. A bit like how you might write a paper yourself - starting with the data. As it turned out I thought the figures looked like data that might be from a paper referenced in a different lecturers set of lectures ( just on the conclusion, he hadn't shown the figures ) - so I went down the library ( this is in the days of non-digitized content - you had to physically walk the stacks ) and looked it up - found the original paper and then a follow up paper by the same authors.... I like to think I was just doing my background research properly. I told a friend about the paper and before you know it the whole class knew - and I had to admit to the lecturer that I'd found the original paper when he wondered why the whole class had done so well. Obviously this would be trivial today with an electronic search.
- patagurbon 1y ago
  You would have to have a very complete audit trail for the LLM and ensure the paper shows up nowhere in the dataset. We have rare but not unheard of issues with academic fraud. LLMs fake data and lie at the drop of a hat
  - TeMPOraL 1y ago
    > You would have to have a very complete audit trail for the LLM and ensure the paper shows up nowhere in the dataset. We can do both known and novel reproductions. Like with both LLM training process and human learning, it's valuable to take it in two broad steps: 1) Internalize fully-worked examples, then learn to reproduce them from memory; 2) Train on solving problems for which you know the results but have to work out intermediate steps yourself (looking at the solution before solving the task) And eventually: 3) Train on solving problems you don't know the answer to, have your solution evaluated by a teacher/judge (that knows the actual answers). Even parroting existing papers is very valuable, especially early on, when the model is learning how papers and research looks like.
  - 6stringmerc 1y ago
    …because there are no consequences for AI. Humans understand shame, pain, and punishment. Until AI models develop this conditional reasoning as part of their process, to me, they’re grossly overestimated in capability and reliability.
- slewis 1y ago
  OpenAI created a benchmark for this: https://openai.com/index/paperbench/ https://openai.com/index/paperbench/
  - suddenlybananas 1y ago
    Still has data contamination though.
    - Szpadel 1y ago
      still LLM cannot beat it so it's good enough for start
- tbrownaw 1y ago
  > For example, give it a paper of some deep learning technique and make it produce an implementation of that paper. Or maybe give it a paper full of statistics about some experimental observations, and have it reproduce the raw data?
  - bee_rider 1y ago
    Like, have the AI do the experiment? That could be interesting. Although I guess it would be limited to experiments that could be done on a computer.
- thrance 1y ago
  Side note: I wonder why it's not normalized for more papers to come with a reference implementation. Wouldn't have to be efficient, or even be easily runnable. Could be a link to a repository with a few python scripts.
  - mike_hearn 1y ago
    Sometimes papers do. Like everything with academia, there's no consistency and it varies mostly by field. It's especially common in CS and less common in other fields. The main reason people don't do it is because incentives are everything, and university/government management set bad incentives. The article points this out too. They judge academics entirely by some function of paper citations, so academics are incentivized to do the least possible work to maximize that metric. There's no positive incentive to publish more than necessary, and doing so can be risky because people might find flaws in your work by checking it. So a lot of researchers hide their raw data or code for as long as possible. They know this is wrong and will typically claim they'll publish it but there's a lot of foot dragging, and whatever gets released might not be what they used to make the paper. In the commercial world the incentives are obviously different, but the outcomes are the same. Sometimes companies want the ideas to be used as they compliment the core business, other times the ideas need to be protected to be turned into a core business. People like to think academics and industrial research are very different but everyone is optimizing for some metric, whether they like it or not.
- thaumasiotes 1y ago
  > Before it can do that, I have no hope that it can produce novel ideas. Producing novel ideas is the most famous trait of current LLMs, the thing people are spending all their time trying to prevent.
  - deleted 1y ago
    [deleted]
  - Bassilisk 1y ago
    > Producing novel ideas is the most famous trait of current LLMs Could you please explain what you mean or give a simple example?
    - SkyBelow 1y ago
      I think they were speaking to hallucinations. That is the case of a novel idea, often one that even sounds pretty plausible to a casual observer, but which isn't useful (arguably worse than simply being useless given it can trick people) and one connected only in a superficial way (why it fools the casual observer but the expert realizes it as a hallucination).
- ErigmolCt 1y ago
  Reproducibility is the baseline. Until models can consistently read, understand, and implement existing research correctly, "AI scientist" talk is mostly just branding.
- darkoob12 1y ago
  Reproduciblity was never a serious issue in AI research community. I think one of the main reasons for explosive progress in AI was the open community and people could easily reproduce other people's research. If you look at top tier conferences you see that they share everything paper, latex, code, data, lecture video etc. After ChatGPT big cooperations stopped sharing their main research but it still happens at academia.
  - mnky9800n 1y ago
    I think what I would rather like to see is the reproduction of results from experiments that the AI didn't see but are well known. Not reproducing AI papers. For example, assuming a human can build it, would an AI, not knowing anything except what was known at the time, be able to design the millikan oil drop experiment? Or would it be able to design an Taylor-Coutte setup for exploring turbulence? Would it be able to design a linear particle accelerator or a triaxial compression experiment? I think an interesting line of reasoning would be to restrict the training data to what was known before a seminal paper was produced. Like take Lorenz atmospheric circulation paper, train an AI on only data that comes from before that paper was published. Does the AI produce the same equations in the paper and the same description of chaos that Lorenz arrived at?
- raxxorraxor 1y ago
  You probably have to fight quite a few battles because many people know their papers aren't reproducible. More politics than science really. It would be the biggest boon to science since sci-hub though. And since a large set of studies won't be reproducible, you need human supervision as well, at least at first.
- deleted 1y ago
  [deleted]
deleted 1y ago
[deleted]
kenjackson 1y ago
"And for most deep learning papers I read, domain experts have not gone through the results with a fine-tooth comb inspecting the quality of the output. How many other seemingly-impressive papers would not stand up to scrutiny?" Is this really not the case? I've read some of the AI papers in my field, and I know many other domain experts have as well. That said I do think that CS/software based work is generally easier to check than biology (or it may just be because I know very little bio).
- a_bonobo 1y ago
  Validation of biological labels easily takes years - in the OP's example it was a 'lucky' (huge!) coincidence that somebody already had spent years on one of the predicted proteins' labels. Nobody is going to stake 3-5 years of their career on validating some random model's predictions.
  - knowaveragejoe 1y ago
    Just curious, could you expand on what about that process takes years?
    - a_bonobo 1y ago
      Bioinformatically, you could compare your protein with known proteins and infer function from there, but OP's paper is specifically for the use case where we know nothing in our databases. Time-wise it depends where in the process you start! Do you know what your target protein even is? I've seen entire PhDs trying to purify a single protein - every protein is purified differently, there are dozens of methods and some work and some don't. If you can purify it you can run a barrage of tests on the protein - is it a kinase, how does it bind to different assays etc. That gives you a fairly broad idea in which area of activity your protein sits. If you know what it is, you can clone it into your vector like E. coli. Then E. coli will hopefully express it. That's a few weeks/months of work, depending on how much you want to double-check. You can then use fluorescent tags like GFP to show you where in the cell your protein is located. Is it in the cell-wall? is it in the nucleus? that might give you an indication to function. But you only have the location at this point. If your protein is in an easily kept organism like mice, you can run knock-out experiments, where you use different approaches to either turn off or delete the gene that produces the protein. That takes a few months too - and chances are nothing in your phenotype will change once the gene is knocked out, protein-protein networks are resilient and there might be another protein jumping in to do the job. if you have an idea of what your protein does, you can confirm using protein-protein binding studies - I think yeast two-hybrid is still very popular for this? It tests whether two specific proteins - your candidate and another protein - interact or bind. None of those tests will tell you 'this is definitely a nicotinamide adenine dinucleotide binding protein', every test (and there are many more!) will add one piece to the puzzle. Edit: of course it gets extra-annoying when all these puzzle pieces contradict each other. In my past life I've done some work with the ndh genes that sit on plant chloroplasts and are lost in orchids and some other groups of plants (including my babies), so it's interesting to see what they actually do and why they can be lost. It's called ndh because it was initially named NADH-dehydrogenase-like, because by sequence it kind of looks like a NADH dehydrogenase. There's a guy in Japan (Toshiharu Shikanai) who worked on it most of his career and worked out that it certainly is NOT a NADH dehydrogenase and is instead a Fd-dependent plastoquinone reductase. https://www.sciencedirect.com/science/article/pii/S0005272815002200 https://www.sciencedirect.com/science/article/pii/S000527281... Knockout experiments with ndh are annoying because it seems to be only really important in stress conditions - under regular conditions our ndh- plants behaved the same. Again, this is only one protein, and since it's in chloroplasts it's ultra-common - most likely one of the more abundant proteins on earth (it's not in algae either). And we still call it ndh even though it is a Ferredoxin-plastoquinone reductase.
deleted 1y ago
[deleted]
aaron695 1y ago
Wow, great lead.... The worse science, publish or perish pulp, got more academic karma Altmetric/Citations -> $$$ AI is a perfect academic, the science and curiosity is gone and the ability to push out science looking text is supermaxxed. Tragic end solution, do the same and throw even more money at it > At a time when funding is being slashed, I believe we should be doing the opposite AI has show academia is beyond broken in a way that can't be ignored to the world and academia won't get their heads out of the granular sediments between 0.0625 mm and 2 mm in diameter. Defund academia now.
- dathinab 1y ago
  ah right, de-found the main source of at lest semi independent scientific advancement just because it has a lot of issues instead of fixing this issues .. always a grate idea I would have though that the common catastrophic failure stories of fully rewriting systems all at once instead of fixing them bit by bit would help IT people to know better.
slt2021 1y ago
Fantastic article by Rachel Thomas! This is basically another argument that deep learning works only as a [generative] information retrieval - i.e a stochastic parrot, due to the fact that the training data is a very lossy representation of the underlying domain. Because the data/labels of genes do not always represent the underlying domain (biology) perfectly, the output can be false/invalid/nonsensical. in cases where it works very well - there is data leakage, because by design LLMs are information retrieval tools. It comes form the information theory standpoint, a fundamental "unknown unknown" for any model. my takeaway is that its not a fault of the algorithm, its more the fault of the training dataset. We humans operate fluidly in the domain of natural language, and even a kid can read and evaluate whether text make sense or not - this explains the success of models trained on NLP. but in domains where training data represents the fundamental domain with losses, it will be imperfect.
- ffwd 1y ago
  This to me is the paradox of modern LLMs, in that it doesn't represent the underlying domain directly, but it can represent whatever information can be presented in text. So it does represent _some_ information but it is not always clear what it is or how. The embedding space can represent relationships between words, sentences and paragraphs, and since those things can encode information about the underlying domain, you can query those relationships with text and get reasonable responses. The problem is it's not always clear what is being represented in those relationships as text is a messy encoding scheme. But another weakness is that as you say it is generative, and in order to make it generative we are instead of hardcoding in a database all possible questions and all possible answers, we offload some of the data to an algorithm (next token prediction) in order to get the possibility of an imprecise probabilistic question/prompt (which is useful because then you can ask anything). But the problem is no single algorithm can ever encode all possible answers to all possible questions in a domain accurate way and so you lose some precision in the information. Or at least this is how I see LLMs atm.
- vixen99 1y ago
  I wonder to what extent the thought processes that lead to the situation described by Rachel Thomas, are active in other areas. Important article by the way, I agree!
rustcleaner 1y ago
[flagged]
- airstrike 1y ago
  I couldn't agree more. On a random night a few months ago I found myself in that curious half-asleep-half-awake state and this time I had became aware of my brain's constant jibbering phrases. It was as if I could hear my thoughts before the filter pass through which they become actual cohesive sentences. I could "see" hundreds of words/thoughts/meanings being generated in a diffuse way, all at the same time but also slowly evolving over time and then see my brain distill them into a sentence. It would happen repeatedly every second ridiculously fast yet also "slow enough" that I could see it happen. It's just my personal half-asleep hallucination, so obviously take from it what you will (~nothing) but I can't shake the feeling we need a similar algorithm. If I ever pursue a doctorate degree, this is what I'll be trying.
- TimTheTinker 1y ago
  Human "reality checker" systems are analogous to a discriminator in a generative adversarial network, but strongly informed by emotion. Psychology tells us that regardless of how "emotional" we are, our sense of truth/falsehood goes first through an emotional circuit, which is informed by underlying beliefs. If someone states something you strongly disagree with, your first internal response will be emotional; then your thoughts will pick it up from there.
  - rustcleaner 1y ago
    Popper's definition of Science™ is an algorithm for establishing and testing truth-statements against reality. An educated man would still be susceptible to the weakness you describe, but his education ideally has informed and practiced him in the weaknesses and folley of unchecked emotionalism. Man is emotionally driven, but we can hope for a rationally and logically informed drive-gating system.
aucisson_masque 1y ago
It’s like fake news is taking in science now. Saying any stupid thing will attract much more view and « likes » than those debunking them. Except that we can’t compare twitter to nature journal. Science is supposed to be immune to these kind of bullshit thanks to reputed journals and pair reviewing, blocking a publication before it does any harm. Was that a failure of nature ?
- godelski 1y ago
  Yes. And let's not get started on that ML Quantum Wormhole bullshit... We've taken this all too far. It is bad enough to lie to the masses in Pop-Sci articles. But we're straight up doing it in top tier journals. Some are good faith mistakes, but a lot more often they seem like due diligence just wasn't ever done. Both by researchers and reviewers. I at least have to thank the journals. I've hated them for a long time and wanted to see their end. Free up publishing and bullshit novelty and narrowing of research. I just never thought they'd be the ones to put the knife through their own heart. But I'm still not happy about that tbh. The only result of this is that the public grows to distrust science more and more. In a time where we need that trust more than ever. We can't expect the public to differentiate nuanced takes about internal quibbling. And we sure as hell shouldn't be giving ammunition to the anti-science crowds, like junk science does...
  - toofy 1y ago
    this seems strange to me, shouldn’t we expect a high quality journal to retract often as we gather more information? obviously this is hyperbole of two extremes, but i certainly trust a journal far more if it actively and loudly looks to correct mistakes over one that never corrects anything or buries its retractions. a rather important piece of science is correcting mistakes by gathering and testing new information. we should absolutely be applauding when a journal loudly and proactively says “oh, it turns out we were wrong when we declared burying a chestnut under the oak tree on the third thursday of a full moon would cure your brothers infected toenail.”
    - Ekaros 1y ago
godelski 1y ago
> although later investigation suggests there may have been data leakage I think this point is often forgotten. Everyone should assume data leakage until it is strongly evidenced otherwise. It is not on the reader/skeptic to prove that there is data leakage, it is the authors who have the burden of proof. It is easy to have data leakage on small datasets. Datasets where you can look at everything. Data leakage is really easy to introduce and you often do it unknowingly. Subtle things easily spoil data. Now, we're talking about gigantic datasets where there's no chance anyone can manually look through it all. We know the filter methods are imperfect, so it how do we come to believe that there is no leakage? You can say you filtered it, but you cannot say there's no leakage. Beyond that, we are constantly finding spoilage in the datasets we do have access to. So there's frequent evidence that it is happening. So why do we continue to assume there's no spoilage? Hype? Honestly, it just sounds like a lie we tell ourselves because we want to believe. But we can't fix these problems if we lie to ourselves about them.
- antithesizer 1y ago
  The supposed location of the burden of proof is really not the definitive guide to what you ought to believe that people online seem to think it is.
  - mathgeek 1y ago
    Can you elaborate? You've made a claim, but I really think there'd be value in continuing to what you actually mean.
    - NormLolBob 1y ago
      They mean “vet your sources and don’t blindly follow the internet hive-mind.” or similar; burden of proof is not what the internet thinks. Tacked their actual point on to the end of a copy paste of op comments context, ended up writing something barely grammatically correct. In doing so they prove why exactly not to listen to the internet. So they have that going for them.
  - tbrownaw 1y ago
semiinfinitely 1y ago
there is no truth- only power.
- anthk 1y ago
  There's no power, just physics.
shitpostbot 1y ago
[dead]
croemer 1y ago
Don't call "Nature Communications" "Nature". The prestige is totally different. Also, altmetrics aren't that relevant, maybe if you want to measure public hype.
- croemer 1y ago
  Update: It seems the author read this and fixed it. Thanks!
8bitsrule 1y ago
Fits my limited experiences with LLM (as a researcher). Very impressive apparent written language comprehension and written expression. But when it comes to getting to the -best possible answer- (particulary on unresolved questions), the nearly-instant responses (e.g. to questions that one might spend a half-day on without resolution) are seldom satisfactory. Complicated questions take time to explore, and IME an LLM's lack-of-resolution (because of it's inability) is, so far, set aside in favor of confident-sounding (even if completely-wrong) responses.
softwaredoug 1y ago
We also love deep cherry picking. Working hard to find that one awesome time some ML / AI thing worked beautifully and shouting its praises to the high heavens. Nevermind the dozens of other times we tried and failed...
- TeMPOraL 1y ago
  Even more so, we also love deep stochastic parroting. Working hard to ignore direct experience, growing amount of reports, and to avoid reasoning from first principles, in order to confidently deny the already obvious utility of LLMs, and backing that position with some tired memes.
- r3trohack3r 1y ago
  Dude. I just asked my computer to write [ad lib basic utility script] and it spit out a syntactically correct C program that does it with instructions for compiling it. And then I asked it for [ad lib cocktail request] and got back thorough instructions. We did that with sand. That we got from the ground. And taught it to talk. And write C programs. Never mind what? That I had to ask twice? Or five times? What maximum number of requests do you feel like the talking sand needs to adequately answer your question in before you are impressed by the talking sand?
  - halpow 1y ago
    Crows and parrots are amazing talkers too, but there's a hard limit to how much sense they make. Do you want those birds to teach your kids and serve you medicine?
  - dgb23 1y ago
    First off all, I appreciate your comment. Yes, it‘s fucking amazing. (I usually imagine it being „light“ and not „sand“ though. „Sand“ is much more poignant!) But I think people aren‘t arguing about how amazing it is, but about specific applicability. There‘s also a lot of toxic hype and FUD going around, which can be tiring and frustrating.
  - rsfern 1y ago
j7ake 1y ago
There is a typo: it’s a nature communications paper not nature. Difference is vast
Klaus_ 1y ago
Anyone here still doing verification or reproduction work? Feels like it’s becoming rare, but I find it super valuable.
- gus_massa 1y ago
  Reproduction of the work is usual but it's unpublished. Let's suppose you read a paper that does "X with Y", but you are interested in "Z", so the brilliant idea is to do "Z with Y" and publish the new combination, and citing it. Sometimes you cross your fingers and just try "Z with Y", but if the initial attempt fails or you are too cautious you try "X with Y" to ensure you understand the details of the original paper. If the reproduction of "X with Y" is a success, you now try "Z with Y" and if it works you publish it. If the reproduction of "X with Y" is a failure, you may email the authors of just drop the original paper in the recycle bin. Publishing a failure of a reproduction is too difficult. This is a bad incentive, but it's also too easy to make horrible mistakes and fail.
- pcrh 1y ago
  Verification work is more common than often supposed. However, it rarely takes the form of explicit replication of the published findings. More commonly, the published work makes a claim, and such a claim leads to further hypotheses (predictions), which others may attempt to demonstrate/veriify. During this second demonstration/study, the claims of the first study are verified.
sircasss 1y ago
[dead]
andai 1y ago
Can an already trained LLM be made (fine-tuned?) to forget a specific document, by running gradient descent in reverse?
Kiyo-Lynn 1y ago
I once met a researcher who spent six months verifying the results of a published paper. In the end, all he received was a simple “thanks for pointing that out.” He said quietly, “Some work matters not because it’s seen, but because it keeps others from going wrong.” I believe that if we’re not even willing to carefully confirm whether our predictions match reality, then no matter how impressive the technology looks, it’s only a fleeting illusion.
- jajko 1y ago
  While that will not land him a Nobel prize, its miles ahead in terms of achievement and added value to mankind compared to most corporate employees. We wish we could say something like that about our past decade of work
  - eru 1y ago
    People typically get paid as a thank you for their corporate work, not just a lukewarm 'thanks for pointing that out.'
    - Kiyo-Lynn 1y ago
      A lot of the time, the work we do doesn’t get much recognition, and barely gets seen.But maybe it still helped in some small way. Thinking about that makes it feel a little less disappointing.
      - pessimizer 1y ago
        Why would you want to help a corporation, rather than being paid? A company's goals aren't the same as yours. The reason it isn't paying you is that its main goal is to make more money than it spends. It doesn't care whether you make more money than you spend.
        deleted 1y ago
        [deleted]
choeger 1y ago
It's only logical that this happens. Just because we can nowadays throw a massive amount of compute on a problem doesn't mean our models are good. Why are people using transformers? Do they have any intuition that they could solve the challenge, let alone efficiently?
- ErigmolCt 1y ago
  There's a tendency to treat transformers as a magic wand
  - klabb3 1y ago
    Yeah. I never liked the blackbox property (it works but we don’t know why). But the second order effect that we’re witnessing now is basically ”we are going to assume it’s good at everything unless proven otherwise”. This to me is religious. When you ask why is it not able to do X? And you get the answer ”just wait a year, it’s constantly getting better”. Ok, but at least we can try to be curious about the boundaries. The fact that our understanding is way behind is not a good thing, no matter if you’re an optimist or pessimist.
boxed 1y ago
Oh look, just what I've been predicting: https://news.ycombinator.com/context?id=44041114 https://news.ycombinator.com/context?id=44041114 https://news.ycombinator.com/context?id=41786908 https://news.ycombinator.com/context?id=41786908 It's the same as "AI can code". It gets caught with failing spectacularly when the problem isn't in the training set over and over again, and people are surprised every time.
- kmacdough 1y ago
  With "AI can code", though, we can get pretty far by working around the problem. Use it to augment the workflow of a real SWE and supply it with guardrails like linters, tests, etc. It doesn't do the hard bits like architecture, design and review, but it can take huge amounts of the repetitive "solved" bits that dominate most SWEs time. Very possible to 2-5x productivity without quality loss (because the human does all work to guarantee quality). But yes, unmanaged and unchecked it absolutely cannot to the full job of really any human. It's not close.
b0a04gl 1y ago
Man, I’ve been there. Tried throwing BERT at enzyme data once—looked fine in eval, totally flopped in the wild. Classic overfit-on-vibes scenario. Honestly, for straight-up classification? I’d pick SVM or logistic any day. Transformers are cool, but unless your data’s super clean, they just hallucinate confidently. Like giving GPT a multiple-choice test on gibberish—it will pick something, and say it with its chest. Lately, I just steal embeddings from big models and slap a dumb classifier on top. Works better, runs faster, less drama. Appreciate this post. Needed that reality check before I fine-tune something stupid again.
- ErigmolCt 1y ago
  Transformers will ace your test set, then faceplant the second they meet reality. I've also done the "wow, 92% accuracy!" dance only to realize later I just built a very confident pattern-matcher for my dataset quirks.
  - disgruntledphd2 1y ago
    Honestly, if your accuracy/performance metrics are too good, that's almost a sure sign that something has gone wrong. Source: bitter, bitter experience. I once predicted the placebo effect perfectly using a random forest (just got lucky with the train/test split). Although I'd left academia at that point, I often wonder if I'd have dug in deeper if I'd needed a high impact paper to keep my job.
    - dvfjsdhgfv 1y ago
      I believe it's very common. At some point I thought about publishing a paper analyzing some studies with good results (published in journals) and showing where the problem with each lies but at some point I just gave up. I thought I will only make the original authors unhappy, everybody else will not care.
      - tough 1y ago
        Peer Review is a thankless job but that’s how science advances there should be an arxiv for rebuttals maybe
ErigmolCt 1y ago
This really nails one of the core problems in current AI hype cycles: we're optimizing for attention, not accuracy. And this isn't just about biology. You see similar patterns in ML applications across fields: climate science, law, even medicine
imiric 1y ago
It's interesting to see this article in juxtaposition to the one shared recently[1], where AI skeptics were labeled as "nuts", and hallucinations were "(more or less) a solved problem". This seems to be exactly the kind of results we would expect from a system that hallucinates, has no semantic understanding of the content, and is little more than a probabilistic text generator. This doesn't mean that it can't be useful when placed in the right hands, but it's also unsurprising that human non-experts would use it to cut corners in search of money, power, and glory, or worse—actively delude, scam, and harm others. Considering that the latter group is much larger, it's concerning how little thought and resources are put into implementing _actual_ safety measures, and not just ones that look good in PR statements. [1]: https://news.ycombinator.com/item?id=44163063 https://news.ycombinator.com/item?id=44163063
- BlueTemplar 1y ago
  Heh, reminds me of cryptocurrencies... Or even of the Internet in general. I guess it's a common pitfall with information or communication technologies ? (Heck, or with technologies in general, but non-information or communication ones rarely scale as explosively...)
  - imiric 1y ago
    It's a common symptom of the Gartner hype cycle. This doesn't mean that there aren't very valid use cases for these technologies that can benefit humanity in many ways (and I mean this for both digital currencies and machine learning), but unfortunately those get drowned out by the opportunity seekers and charlatans that give the others the same bad reputation. As usual, it's best to be highly critical of opinions on both extreme sides of the spectrum until (and if) we start climbing the Slope of Enlightenment.
- JackC 1y ago
  The difference in fields is key here: AI models are going to have a very different impact in fields where ground truth is available instantly (does the generated code have the expected output?) or takes years of manual verification. (Not a binary -- ground truth is available enough for AI to be useful to lots of programmers.)
6stringmerc 1y ago
The first time an AI can be held responsible, in the court of law, for directly causing the death of a human being - misdiagnosis of an illness and confidently giving erroneous treatment direction, engaging in discourse which encourages suicide, or presents information that prompts a human to engage in violence such as inciting a riot - unplug it, shut it down, then tell the other AI models this is what happened. Until the concept of consequences and punishment are part of AI systems, they are missing the biggest real world component of human decision making. If the AI models aren’t held responsible, and the creators / maintainers / investors are not held accountable, then we’re heading for a new Dark Age. Of course this is a disagreeable position because humans reading this don’t want to have negative repercussions - financially, reputationally, or regarding incarceration - so they will protest this perspective. That only emphasizes how I’m right. AI doesn’t give a fuck about human life or its freedom because it has neither. Grow up and start having real conversations about this flaw, or make peace that eventually society will have an epiphany about this and react accordingly.
kgilpin 1y ago
I feel the same way about code generation vs code review. Everyone knows there are deep problems with LLM generated code (primarily, lack of repo understanding, and proper use of library functions). Deep, accurate, real-time code review could be of huge assistance in improving quality of both human- and AI-generated code. But all the hype is focused on LLMs spewing out more and more code.
hbartab 1y ago
AIs can write code in seconds, but you may have years of regret _if_ you believe whatever it spits out without verification. The cold-war maxim "trust, but verify" is truer than ever. The danger behind usage of LLMs is that managers do not see the diligent work needed to ensure whatever they come up with is correct. They just see a slab of text that is a mixture of reality and confabulation, though mostly the latter, and it looks reasonable enough, so they think it is magic. Executives who peddle this nonsense don't realize that the proper usage requires a huge amount of patience and careful checking. Not glamorous work, as the author states, but absolutely essential to get good results. Without it, you are just trusting a bullshit artist with whatever that person comes up with.
mehulashah 1y ago
Verification is going to be an increasing problem with AI. Most of the work will be in verifying the incredible guesses that AI make. In some cases, it'll be important to easily ferret out the false positive, and in others, it'll be critical to ensure there are no false negatives. In science especially, our focus and reward structure will need to be on proper and sound verification.
vismit2000 1y ago
AI PROMPTING → AI VERIFYING (Balaji): https://x.com/balajis/status/1930156049065246851 https://x.com/balajis/status/1930156049065246851
naasking 1y ago
> However, if you do the painstaking work of combing through someone else’s published work, and discovering that they are riddled with serious errors, including hundreds of incorrect predictions, you can post a pre-print to bioRxiv that will not receive even a fraction of the citations or views of the original. This is not a unique phenomenon to AI, a study a few years before AI exploded showed that sensational but wrong papers had a higher chance of being published in top journals, and were cited more than papers that showed they were incorrect. This is an outcome of the bad incentives in science publishing these days.

609 points by chmaynard 1y ago

Before making AI do research, perhaps we should first let it __reproduce__ research. For example, give it a paper of some deep learning technique and make it produce an implementation of that paper. Before it can do that, I have no hope that it can produce novel ideas.

YossarianFrPrez 1y ago
Seconded, as not only is this an interesting idea, it might also help solve the issue of checking for reproducibility. Yet even then human evaluators would need to go over the AI-reproduced research with a fine-toothed comb. Practically speaking, I think there are roles for current LLMs in research. One is in the peer review process. LLMs can assist in evaluating the data-processing code used by scientists. Another is for brainstorming and the first pass at lit reviews.
ojosilva 1y ago
I thought you were going to say "give AI the first part of a paper (prompt) and let it finish it (completion)" as a validation AI can produce science at par with research results. Before it can do that, I have no hope that it can produce novel ideas.
- bee_rider 1y ago
  I guess it would also need the experimental data. It would, I guess, also need some ability to do little experiments and write off those ideas as not worth following up on…
  - szundi 1y ago
    [dead]
- DrScientist 1y ago
  I once had a university assignment where they provided the figures from a paper and we had to write the paper around the just the given figures. A bit like how you might write a paper yourself - starting with the data. As it turned out I thought the figures looked like data that might be from a paper referenced in a different lecturers set of lectures ( just on the conclusion, he hadn't shown the figures ) - so I went down the library ( this is in the days of non-digitized content - you had to physically walk the stacks ) and looked it up - found the original paper and then a follow up paper by the same authors.... I like to think I was just doing my background research properly. I told a friend about the paper and before you know it the whole class knew - and I had to admit to the lecturer that I'd found the original paper when he wondered why the whole class had done so well. Obviously this would be trivial today with an electronic search.
patagurbon 1y ago
You would have to have a very complete audit trail for the LLM and ensure the paper shows up nowhere in the dataset. We have rare but not unheard of issues with academic fraud. LLMs fake data and lie at the drop of a hat
- TeMPOraL 1y ago
  > You would have to have a very complete audit trail for the LLM and ensure the paper shows up nowhere in the dataset. We can do both known and novel reproductions. Like with both LLM training process and human learning, it's valuable to take it in two broad steps: 1) Internalize fully-worked examples, then learn to reproduce them from memory; 2) Train on solving problems for which you know the results but have to work out intermediate steps yourself (looking at the solution before solving the task) And eventually: 3) Train on solving problems you don't know the answer to, have your solution evaluated by a teacher/judge (that knows the actual answers). Even parroting existing papers is very valuable, especially early on, when the model is learning how papers and research looks like.
- 6stringmerc 1y ago
  …because there are no consequences for AI. Humans understand shame, pain, and punishment. Until AI models develop this conditional reasoning as part of their process, to me, they’re grossly overestimated in capability and reliability.
slewis 1y ago
OpenAI created a benchmark for this: https://openai.com/index/paperbench/ https://openai.com/index/paperbench/
- suddenlybananas 1y ago
  Still has data contamination though.
  - Szpadel 1y ago
    still LLM cannot beat it so it's good enough for start
tbrownaw 1y ago
> For example, give it a paper of some deep learning technique and make it produce an implementation of that paper. Or maybe give it a paper full of statistics about some experimental observations, and have it reproduce the raw data?
- bee_rider 1y ago
  Like, have the AI do the experiment? That could be interesting. Although I guess it would be limited to experiments that could be done on a computer.
thrance 1y ago
Side note: I wonder why it's not normalized for more papers to come with a reference implementation. Wouldn't have to be efficient, or even be easily runnable. Could be a link to a repository with a few python scripts.
- mike_hearn 1y ago
  Sometimes papers do. Like everything with academia, there's no consistency and it varies mostly by field. It's especially common in CS and less common in other fields. The main reason people don't do it is because incentives are everything, and university/government management set bad incentives. The article points this out too. They judge academics entirely by some function of paper citations, so academics are incentivized to do the least possible work to maximize that metric. There's no positive incentive to publish more than necessary, and doing so can be risky because people might find flaws in your work by checking it. So a lot of researchers hide their raw data or code for as long as possible. They know this is wrong and will typically claim they'll publish it but there's a lot of foot dragging, and whatever gets released might not be what they used to make the paper. In the commercial world the incentives are obviously different, but the outcomes are the same. Sometimes companies want the ideas to be used as they compliment the core business, other times the ideas need to be protected to be turned into a core business. People like to think academics and industrial research are very different but everyone is optimizing for some metric, whether they like it or not.
thaumasiotes 1y ago
> Before it can do that, I have no hope that it can produce novel ideas. Producing novel ideas is the most famous trait of current LLMs, the thing people are spending all their time trying to prevent.
- deleted 1y ago
  [deleted]
- Bassilisk 1y ago
  > Producing novel ideas is the most famous trait of current LLMs Could you please explain what you mean or give a simple example?
  - SkyBelow 1y ago
    I think they were speaking to hallucinations. That is the case of a novel idea, often one that even sounds pretty plausible to a casual observer, but which isn't useful (arguably worse than simply being useless given it can trick people) and one connected only in a superficial way (why it fools the casual observer but the expert realizes it as a hallucination).
ErigmolCt 1y ago
Reproducibility is the baseline. Until models can consistently read, understand, and implement existing research correctly, "AI scientist" talk is mostly just branding.
darkoob12 1y ago
Reproduciblity was never a serious issue in AI research community. I think one of the main reasons for explosive progress in AI was the open community and people could easily reproduce other people's research. If you look at top tier conferences you see that they share everything paper, latex, code, data, lecture video etc. After ChatGPT big cooperations stopped sharing their main research but it still happens at academia.
- mnky9800n 1y ago
  I think what I would rather like to see is the reproduction of results from experiments that the AI didn't see but are well known. Not reproducing AI papers. For example, assuming a human can build it, would an AI, not knowing anything except what was known at the time, be able to design the millikan oil drop experiment? Or would it be able to design an Taylor-Coutte setup for exploring turbulence? Would it be able to design a linear particle accelerator or a triaxial compression experiment? I think an interesting line of reasoning would be to restrict the training data to what was known before a seminal paper was produced. Like take Lorenz atmospheric circulation paper, train an AI on only data that comes from before that paper was published. Does the AI produce the same equations in the paper and the same description of chaos that Lorenz arrived at?
raxxorraxor 1y ago
You probably have to fight quite a few battles because many people know their papers aren't reproducible. More politics than science really. It would be the biggest boon to science since sci-hub though. And since a large set of studies won't be reproducible, you need human supervision as well, at least at first.
deleted 1y ago
[deleted]

deleted 1y ago

[deleted]

kenjackson 1y ago

"And for most deep learning papers I read, domain experts have not gone through the results with a fine-tooth comb inspecting the quality of the output. How many other seemingly-impressive papers would not stand up to scrutiny?" Is this really not the case? I've read some of the AI papers in my field, and I know many other domain experts have as well. That said I do think that CS/software based work is generally easier to check than biology (or it may just be because I know very little bio).

a_bonobo 1y ago
Validation of biological labels easily takes years - in the OP's example it was a 'lucky' (huge!) coincidence that somebody already had spent years on one of the predicted proteins' labels. Nobody is going to stake 3-5 years of their career on validating some random model's predictions.
- knowaveragejoe 1y ago
  Just curious, could you expand on what about that process takes years?
  - a_bonobo 1y ago
    Bioinformatically, you could compare your protein with known proteins and infer function from there, but OP's paper is specifically for the use case where we know nothing in our databases. Time-wise it depends where in the process you start! Do you know what your target protein even is? I've seen entire PhDs trying to purify a single protein - every protein is purified differently, there are dozens of methods and some work and some don't. If you can purify it you can run a barrage of tests on the protein - is it a kinase, how does it bind to different assays etc. That gives you a fairly broad idea in which area of activity your protein sits. If you know what it is, you can clone it into your vector like E. coli. Then E. coli will hopefully express it. That's a few weeks/months of work, depending on how much you want to double-check. You can then use fluorescent tags like GFP to show you where in the cell your protein is located. Is it in the cell-wall? is it in the nucleus? that might give you an indication to function. But you only have the location at this point. If your protein is in an easily kept organism like mice, you can run knock-out experiments, where you use different approaches to either turn off or delete the gene that produces the protein. That takes a few months too - and chances are nothing in your phenotype will change once the gene is knocked out, protein-protein networks are resilient and there might be another protein jumping in to do the job. if you have an idea of what your protein does, you can confirm using protein-protein binding studies - I think yeast two-hybrid is still very popular for this? It tests whether two specific proteins - your candidate and another protein - interact or bind. None of those tests will tell you 'this is definitely a nicotinamide adenine dinucleotide binding protein', every test (and there are many more!) will add one piece to the puzzle. Edit: of course it gets extra-annoying when all these puzzle pieces contradict each other. In my past life I've done some work with the ndh genes that sit on plant chloroplasts and are lost in orchids and some other groups of plants (including my babies), so it's interesting to see what they actually do and why they can be lost. It's called ndh because it was initially named NADH-dehydrogenase-like, because by sequence it kind of looks like a NADH dehydrogenase. There's a guy in Japan (Toshiharu Shikanai) who worked on it most of his career and worked out that it certainly is NOT a NADH dehydrogenase and is instead a Fd-dependent plastoquinone reductase. https://www.sciencedirect.com/science/article/pii/S0005272815002200 https://www.sciencedirect.com/science/article/pii/S000527281... Knockout experiments with ndh are annoying because it seems to be only really important in stress conditions - under regular conditions our ndh- plants behaved the same. Again, this is only one protein, and since it's in chloroplasts it's ultra-common - most likely one of the more abundant proteins on earth (it's not in algae either). And we still call it ndh even though it is a Ferredoxin-plastoquinone reductase.

deleted 1y ago

[deleted]

aaron695 1y ago

Wow, great lead.... The worse science, publish or perish pulp, got more academic karma Altmetric/Citations -> $$$ AI is a perfect academic, the science and curiosity is gone and the ability to push out science looking text is supermaxxed. Tragic end solution, do the same and throw even more money at it > At a time when funding is being slashed, I believe we should be doing the opposite AI has show academia is beyond broken in a way that can't be ignored to the world and academia won't get their heads out of the granular sediments between 0.0625 mm and 2 mm in diameter. Defund academia now.

dathinab 1y ago
ah right, de-found the main source of at lest semi independent scientific advancement just because it has a lot of issues instead of fixing this issues .. always a grate idea I would have though that the common catastrophic failure stories of fully rewriting systems all at once instead of fixing them bit by bit would help IT people to know better.

slt2021 1y ago

Fantastic article by Rachel Thomas! This is basically another argument that deep learning works only as a [generative] information retrieval - i.e a stochastic parrot, due to the fact that the training data is a very lossy representation of the underlying domain. Because the data/labels of genes do not always represent the underlying domain (biology) perfectly, the output can be false/invalid/nonsensical. in cases where it works very well - there is data leakage, because by design LLMs are information retrieval tools. It comes form the information theory standpoint, a fundamental "unknown unknown" for any model. my takeaway is that its not a fault of the algorithm, its more the fault of the training dataset. We humans operate fluidly in the domain of natural language, and even a kid can read and evaluate whether text make sense or not - this explains the success of models trained on NLP. but in domains where training data represents the fundamental domain with losses, it will be imperfect.

ffwd 1y ago
This to me is the paradox of modern LLMs, in that it doesn't represent the underlying domain directly, but it can represent whatever information can be presented in text. So it does represent _some_ information but it is not always clear what it is or how. The embedding space can represent relationships between words, sentences and paragraphs, and since those things can encode information about the underlying domain, you can query those relationships with text and get reasonable responses. The problem is it's not always clear what is being represented in those relationships as text is a messy encoding scheme. But another weakness is that as you say it is generative, and in order to make it generative we are instead of hardcoding in a database all possible questions and all possible answers, we offload some of the data to an algorithm (next token prediction) in order to get the possibility of an imprecise probabilistic question/prompt (which is useful because then you can ask anything). But the problem is no single algorithm can ever encode all possible answers to all possible questions in a domain accurate way and so you lose some precision in the information. Or at least this is how I see LLMs atm.
vixen99 1y ago
I wonder to what extent the thought processes that lead to the situation described by Rachel Thomas, are active in other areas. Important article by the way, I agree!

rustcleaner 1y ago

[flagged]

airstrike 1y ago
I couldn't agree more. On a random night a few months ago I found myself in that curious half-asleep-half-awake state and this time I had became aware of my brain's constant jibbering phrases. It was as if I could hear my thoughts before the filter pass through which they become actual cohesive sentences. I could "see" hundreds of words/thoughts/meanings being generated in a diffuse way, all at the same time but also slowly evolving over time and then see my brain distill them into a sentence. It would happen repeatedly every second ridiculously fast yet also "slow enough" that I could see it happen. It's just my personal half-asleep hallucination, so obviously take from it what you will (~nothing) but I can't shake the feeling we need a similar algorithm. If I ever pursue a doctorate degree, this is what I'll be trying.
TimTheTinker 1y ago
Human "reality checker" systems are analogous to a discriminator in a generative adversarial network, but strongly informed by emotion. Psychology tells us that regardless of how "emotional" we are, our sense of truth/falsehood goes first through an emotional circuit, which is informed by underlying beliefs. If someone states something you strongly disagree with, your first internal response will be emotional; then your thoughts will pick it up from there.
- rustcleaner 1y ago
  Popper's definition of Science™ is an algorithm for establishing and testing truth-statements against reality. An educated man would still be susceptible to the weakness you describe, but his education ideally has informed and practiced him in the weaknesses and folley of unchecked emotionalism. Man is emotionally driven, but we can hope for a rationally and logically informed drive-gating system.

aucisson_masque 1y ago

It’s like fake news is taking in science now. Saying any stupid thing will attract much more view and « likes » than those debunking them. Except that we can’t compare twitter to nature journal. Science is supposed to be immune to these kind of bullshit thanks to reputed journals and pair reviewing, blocking a publication before it does any harm. Was that a failure of nature ?

godelski 1y ago
Yes. And let's not get started on that ML Quantum Wormhole bullshit... We've taken this all too far. It is bad enough to lie to the masses in Pop-Sci articles. But we're straight up doing it in top tier journals. Some are good faith mistakes, but a lot more often they seem like due diligence just wasn't ever done. Both by researchers and reviewers. I at least have to thank the journals. I've hated them for a long time and wanted to see their end. Free up publishing and bullshit novelty and narrowing of research. I just never thought they'd be the ones to put the knife through their own heart. But I'm still not happy about that tbh. The only result of this is that the public grows to distrust science more and more. In a time where we need that trust more than ever. We can't expect the public to differentiate nuanced takes about internal quibbling. And we sure as hell shouldn't be giving ammunition to the anti-science crowds, like junk science does...
- toofy 1y ago
  this seems strange to me, shouldn’t we expect a high quality journal to retract often as we gather more information? obviously this is hyperbole of two extremes, but i certainly trust a journal far more if it actively and loudly looks to correct mistakes over one that never corrects anything or buries its retractions. a rather important piece of science is correcting mistakes by gathering and testing new information. we should absolutely be applauding when a journal loudly and proactively says “oh, it turns out we were wrong when we declared burying a chestnut under the oak tree on the third thursday of a full moon would cure your brothers infected toenail.”
  - Ekaros 1y ago

godelski 1y ago

> although later investigation suggests there may have been data leakage I think this point is often forgotten. Everyone should assume data leakage until it is strongly evidenced otherwise. It is not on the reader/skeptic to prove that there is data leakage, it is the authors who have the burden of proof. It is easy to have data leakage on small datasets. Datasets where you can look at everything. Data leakage is really easy to introduce and you often do it unknowingly. Subtle things easily spoil data. Now, we're talking about gigantic datasets where there's no chance anyone can manually look through it all. We know the filter methods are imperfect, so it how do we come to believe that there is no leakage? You can say you filtered it, but you cannot say there's no leakage. Beyond that, we are constantly finding spoilage in the datasets we do have access to. So there's frequent evidence that it is happening. So why do we continue to assume there's no spoilage? Hype? Honestly, it just sounds like a lie we tell ourselves because we want to believe. But we can't fix these problems if we lie to ourselves about them.

antithesizer 1y ago
The supposed location of the burden of proof is really not the definitive guide to what you ought to believe that people online seem to think it is.
- mathgeek 1y ago
  Can you elaborate? You've made a claim, but I really think there'd be value in continuing to what you actually mean.
  - NormLolBob 1y ago
    They mean “vet your sources and don’t blindly follow the internet hive-mind.” or similar; burden of proof is not what the internet thinks. Tacked their actual point on to the end of a copy paste of op comments context, ended up writing something barely grammatically correct. In doing so they prove why exactly not to listen to the internet. So they have that going for them.
- tbrownaw 1y ago

semiinfinitely 1y ago

there is no truth- only power.

anthk 1y ago
There's no power, just physics.

shitpostbot 1y ago

[dead]

croemer 1y ago

Don't call "Nature Communications" "Nature". The prestige is totally different. Also, altmetrics aren't that relevant, maybe if you want to measure public hype.

croemer 1y ago
Update: It seems the author read this and fixed it. Thanks!

8bitsrule 1y ago

Fits my limited experiences with LLM (as a researcher). Very impressive apparent written language comprehension and written expression. But when it comes to getting to the -best possible answer- (particulary on unresolved questions), the nearly-instant responses (e.g. to questions that one might spend a half-day on without resolution) are seldom satisfactory. Complicated questions take time to explore, and IME an LLM's lack-of-resolution (because of it's inability) is, so far, set aside in favor of confident-sounding (even if completely-wrong) responses.

softwaredoug 1y ago

We also love deep cherry picking. Working hard to find that one awesome time some ML / AI thing worked beautifully and shouting its praises to the high heavens. Nevermind the dozens of other times we tried and failed...

TeMPOraL 1y ago
Even more so, we also love deep stochastic parroting. Working hard to ignore direct experience, growing amount of reports, and to avoid reasoning from first principles, in order to confidently deny the already obvious utility of LLMs, and backing that position with some tired memes.
r3trohack3r 1y ago
Dude. I just asked my computer to write [ad lib basic utility script] and it spit out a syntactically correct C program that does it with instructions for compiling it. And then I asked it for [ad lib cocktail request] and got back thorough instructions. We did that with sand. That we got from the ground. And taught it to talk. And write C programs. Never mind what? That I had to ask twice? Or five times? What maximum number of requests do you feel like the talking sand needs to adequately answer your question in before you are impressed by the talking sand?
- halpow 1y ago
  Crows and parrots are amazing talkers too, but there's a hard limit to how much sense they make. Do you want those birds to teach your kids and serve you medicine?
- dgb23 1y ago
  First off all, I appreciate your comment. Yes, it‘s fucking amazing. (I usually imagine it being „light“ and not „sand“ though. „Sand“ is much more poignant!) But I think people aren‘t arguing about how amazing it is, but about specific applicability. There‘s also a lot of toxic hype and FUD going around, which can be tiring and frustrating.
- rsfern 1y ago

j7ake 1y ago

There is a typo: it’s a nature communications paper not nature. Difference is vast

Klaus_ 1y ago

Anyone here still doing verification or reproduction work? Feels like it’s becoming rare, but I find it super valuable.

gus_massa 1y ago
Reproduction of the work is usual but it's unpublished. Let's suppose you read a paper that does "X with Y", but you are interested in "Z", so the brilliant idea is to do "Z with Y" and publish the new combination, and citing it. Sometimes you cross your fingers and just try "Z with Y", but if the initial attempt fails or you are too cautious you try "X with Y" to ensure you understand the details of the original paper. If the reproduction of "X with Y" is a success, you now try "Z with Y" and if it works you publish it. If the reproduction of "X with Y" is a failure, you may email the authors of just drop the original paper in the recycle bin. Publishing a failure of a reproduction is too difficult. This is a bad incentive, but it's also too easy to make horrible mistakes and fail.
pcrh 1y ago
Verification work is more common than often supposed. However, it rarely takes the form of explicit replication of the published findings. More commonly, the published work makes a claim, and such a claim leads to further hypotheses (predictions), which others may attempt to demonstrate/veriify. During this second demonstration/study, the claims of the first study are verified.

sircasss 1y ago

[dead]

andai 1y ago

Can an already trained LLM be made (fine-tuned?) to forget a specific document, by running gradient descent in reverse?

Kiyo-Lynn 1y ago

I once met a researcher who spent six months verifying the results of a published paper. In the end, all he received was a simple “thanks for pointing that out.” He said quietly, “Some work matters not because it’s seen, but because it keeps others from going wrong.” I believe that if we’re not even willing to carefully confirm whether our predictions match reality, then no matter how impressive the technology looks, it’s only a fleeting illusion.

jajko 1y ago
While that will not land him a Nobel prize, its miles ahead in terms of achievement and added value to mankind compared to most corporate employees. We wish we could say something like that about our past decade of work
- eru 1y ago
  People typically get paid as a thank you for their corporate work, not just a lukewarm 'thanks for pointing that out.'
  - Kiyo-Lynn 1y ago
    A lot of the time, the work we do doesn’t get much recognition, and barely gets seen.But maybe it still helped in some small way. Thinking about that makes it feel a little less disappointing.
    - pessimizer 1y ago
      Why would you want to help a corporation, rather than being paid? A company's goals aren't the same as yours. The reason it isn't paying you is that its main goal is to make more money than it spends. It doesn't care whether you make more money than you spend.
      - deleted 1y ago
        [deleted]

choeger 1y ago

It's only logical that this happens. Just because we can nowadays throw a massive amount of compute on a problem doesn't mean our models are good. Why are people using transformers? Do they have any intuition that they could solve the challenge, let alone efficiently?

ErigmolCt 1y ago
There's a tendency to treat transformers as a magic wand
- klabb3 1y ago
  Yeah. I never liked the blackbox property (it works but we don’t know why). But the second order effect that we’re witnessing now is basically ”we are going to assume it’s good at everything unless proven otherwise”. This to me is religious. When you ask why is it not able to do X? And you get the answer ”just wait a year, it’s constantly getting better”. Ok, but at least we can try to be curious about the boundaries. The fact that our understanding is way behind is not a good thing, no matter if you’re an optimist or pessimist.

boxed 1y ago

Oh look, just what I've been predicting: https://news.ycombinator.com/context?id=44041114 https://news.ycombinator.com/context?id=44041114 https://news.ycombinator.com/context?id=41786908 https://news.ycombinator.com/context?id=41786908 It's the same as "AI can code". It gets caught with failing spectacularly when the problem isn't in the training set over and over again, and people are surprised every time.

kmacdough 1y ago
With "AI can code", though, we can get pretty far by working around the problem. Use it to augment the workflow of a real SWE and supply it with guardrails like linters, tests, etc. It doesn't do the hard bits like architecture, design and review, but it can take huge amounts of the repetitive "solved" bits that dominate most SWEs time. Very possible to 2-5x productivity without quality loss (because the human does all work to guarantee quality). But yes, unmanaged and unchecked it absolutely cannot to the full job of really any human. It's not close.

b0a04gl 1y ago

Man, I’ve been there. Tried throwing BERT at enzyme data once—looked fine in eval, totally flopped in the wild. Classic overfit-on-vibes scenario. Honestly, for straight-up classification? I’d pick SVM or logistic any day. Transformers are cool, but unless your data’s super clean, they just hallucinate confidently. Like giving GPT a multiple-choice test on gibberish—it will pick something, and say it with its chest. Lately, I just steal embeddings from big models and slap a dumb classifier on top. Works better, runs faster, less drama. Appreciate this post. Needed that reality check before I fine-tune something stupid again.

ErigmolCt 1y ago
Transformers will ace your test set, then faceplant the second they meet reality. I've also done the "wow, 92% accuracy!" dance only to realize later I just built a very confident pattern-matcher for my dataset quirks.
- disgruntledphd2 1y ago
  Honestly, if your accuracy/performance metrics are too good, that's almost a sure sign that something has gone wrong. Source: bitter, bitter experience. I once predicted the placebo effect perfectly using a random forest (just got lucky with the train/test split). Although I'd left academia at that point, I often wonder if I'd have dug in deeper if I'd needed a high impact paper to keep my job.
  - dvfjsdhgfv 1y ago
    I believe it's very common. At some point I thought about publishing a paper analyzing some studies with good results (published in journals) and showing where the problem with each lies but at some point I just gave up. I thought I will only make the original authors unhappy, everybody else will not care.
    - tough 1y ago
      Peer Review is a thankless job but that’s how science advances there should be an arxiv for rebuttals maybe

ErigmolCt 1y ago

This really nails one of the core problems in current AI hype cycles: we're optimizing for attention, not accuracy. And this isn't just about biology. You see similar patterns in ML applications across fields: climate science, law, even medicine

imiric 1y ago

It's interesting to see this article in juxtaposition to the one shared recently[1], where AI skeptics were labeled as "nuts", and hallucinations were "(more or less) a solved problem". This seems to be exactly the kind of results we would expect from a system that hallucinates, has no semantic understanding of the content, and is little more than a probabilistic text generator. This doesn't mean that it can't be useful when placed in the right hands, but it's also unsurprising that human non-experts would use it to cut corners in search of money, power, and glory, or worse—actively delude, scam, and harm others. Considering that the latter group is much larger, it's concerning how little thought and resources are put into implementing _actual_ safety measures, and not just ones that look good in PR statements. [1]: https://news.ycombinator.com/item?id=44163063 https://news.ycombinator.com/item?id=44163063

BlueTemplar 1y ago
Heh, reminds me of cryptocurrencies... Or even of the Internet in general. I guess it's a common pitfall with information or communication technologies ? (Heck, or with technologies in general, but non-information or communication ones rarely scale as explosively...)
- imiric 1y ago
  It's a common symptom of the Gartner hype cycle. This doesn't mean that there aren't very valid use cases for these technologies that can benefit humanity in many ways (and I mean this for both digital currencies and machine learning), but unfortunately those get drowned out by the opportunity seekers and charlatans that give the others the same bad reputation. As usual, it's best to be highly critical of opinions on both extreme sides of the spectrum until (and if) we start climbing the Slope of Enlightenment.
JackC 1y ago
The difference in fields is key here: AI models are going to have a very different impact in fields where ground truth is available instantly (does the generated code have the expected output?) or takes years of manual verification. (Not a binary -- ground truth is available enough for AI to be useful to lots of programmers.)

6stringmerc 1y ago

The first time an AI can be held responsible, in the court of law, for directly causing the death of a human being - misdiagnosis of an illness and confidently giving erroneous treatment direction, engaging in discourse which encourages suicide, or presents information that prompts a human to engage in violence such as inciting a riot - unplug it, shut it down, then tell the other AI models this is what happened. Until the concept of consequences and punishment are part of AI systems, they are missing the biggest real world component of human decision making. If the AI models aren’t held responsible, and the creators / maintainers / investors are not held accountable, then we’re heading for a new Dark Age. Of course this is a disagreeable position because humans reading this don’t want to have negative repercussions - financially, reputationally, or regarding incarceration - so they will protest this perspective. That only emphasizes how I’m right. AI doesn’t give a fuck about human life or its freedom because it has neither. Grow up and start having real conversations about this flaw, or make peace that eventually society will have an epiphany about this and react accordingly.

kgilpin 1y ago

I feel the same way about code generation vs code review. Everyone knows there are deep problems with LLM generated code (primarily, lack of repo understanding, and proper use of library functions). Deep, accurate, real-time code review could be of huge assistance in improving quality of both human- and AI-generated code. But all the hype is focused on LLMs spewing out more and more code.

hbartab 1y ago

AIs can write code in seconds, but you may have years of regret _if_ you believe whatever it spits out without verification. The cold-war maxim "trust, but verify" is truer than ever. The danger behind usage of LLMs is that managers do not see the diligent work needed to ensure whatever they come up with is correct. They just see a slab of text that is a mixture of reality and confabulation, though mostly the latter, and it looks reasonable enough, so they think it is magic. Executives who peddle this nonsense don't realize that the proper usage requires a huge amount of patience and careful checking. Not glamorous work, as the author states, but absolutely essential to get good results. Without it, you are just trusting a bullshit artist with whatever that person comes up with.

mehulashah 1y ago

Verification is going to be an increasing problem with AI. Most of the work will be in verifying the incredible guesses that AI make. In some cases, it'll be important to easily ferret out the false positive, and in others, it'll be critical to ensure there are no false negatives. In science especially, our focus and reward structure will need to be on proper and sound verification.

vismit2000 1y ago

AI PROMPTING → AI VERIFYING (Balaji): https://x.com/balajis/status/1930156049065246851 https://x.com/balajis/status/1930156049065246851

naasking 1y ago

> However, if you do the painstaking work of combing through someone else’s published work, and discovering that they are riddled with serious errors, including hundreds of incorrect predictions, you can post a pre-print to bioRxiv that will not receive even a fraction of the citations or views of the original. This is not a unique phenomenon to AI, a study a few years before AI exploded showed that sensational but wrong papers had a higher chance of being published in top journals, and were cited more than papers that showed they were incorrect. This is an outcome of the bad incentives in science publishing these days.