12 ms·
AI in my plasma physics research didn’t go the way I expected
- nicoco 1y agoI am not a AI booster at all, but the fact that negative results are not published and that everyone is overselling their stuff in research papers is unfortunately not limited to AI. This is just a consequence of the way scientists are evaluated and of the scientific publishing industry, which basically suffers from the same shit than traditional media does (craving for audience). Anyway, winter is coming, innit?
- croes 1y agoBut AI makes it easier to write convincing looking papers
- moravak1984 1y agoSure, it's not. But often on AI papers one sees remarks that actually mean: "...and if you throw in one zillion GPUs and make them run until the end of time you get {magic_benchmark}". Or "if you evaluate this very smart algo in our super-secret, real-life dataset that we claim is available on request, but we'd ghost you if you dare to ask, then you will see this chart that shows how smart we are". Sure, it is often flag-planting, but when these papers come from big corps, you cannot "just ignore them and keep on" even when there are obvious flaws/issues. It's a race over resources, as a (former) researcher on a low-budget university, we just cannot compete. We are coerced to believe whatever figure is passed on in the literature as "benchmark", without possibility of replication.
- nicoco 1y agoI agree with that. Classically used "AI benchmarks" need to be questioned. In my field, these guys have dropped a bomb, and no one seem to care: https://hal.science/hal-04715638/document https://hal.science/hal-04715638/document
- baxtr 1y agoCan you give brief summary why this paper is a breakthrough for an outsider of the field?
- mzl 1y agoChecking it shortly (I haven't seen the paper before) this seems to be a very good analysis of how results are reported specifically for medical imaging benchmarks. As is often the case with statistics, selecting just a single number to report (whatever that number is) will hide a lot of different behaviours. Here, they show that just using the mean is a bad way to report data as the confidence intervals (reconstructed by the methods in the paper in most cases) show that the models can't really be distinguished based on their mean.
- amarcheschi 1y agoHell, I was asked to use confidence interval as well as average values for by bs thesis when doing ml benchmarks and scientist publishing results in medical fields aren't doing it... How can something like that happen? I mean, i had a supervisor tell me "add the confidence interval to the results as well", and explained me why. I guess that at nobody ever told them? Or they didn't care? Or it's just a honest mistake
- stogot 1y agoIs it because it’s word-of-mouth and not written down in some NSF (or other organization) guidance? Thiss seems to be the issue
- amarcheschi 1y agoThat might be, but couldn't a paper be asked to include that to be published? It looks like an important information
- nicoco 1y agoI don't think it qualifies as a breakthrough. In short: 1. Segmentation is a very classical in medical image processing. 2. Everyday there are papers claiming that they beat the state of the art 3. This paper says that most of the time, the state of the art has not been beat because they actually are in the margin of error.
- aleph_minus_one 1y ago> It's a race over resources, as a (former) researcher on a low-budget university, we just cannot compete. We are coerced to believe whatever figure is passed on in the literature as "benchmark", without possibility of replication. The central purpose of university research has basically always been that researchers work on hard, foundational topics that are more long-term so that industry is hardly willing to do them. On the other hand, these topics are very important, that is why the respective country is willing to finance this foundational research. Thus, if you are at a university, once your research topic becomes an arms race with industry, you simply work either at the wrong place (university instead of industry) or on a "wrong" topic in the respective research area (look for some much more long-term, experimental topics that, if you are right, might change the whole research area in, say, 15 years, instead of some high resource-intensive, minor improvements to existing models).
- KurSix 1y agoAI just happens to be the current hype magnet, so the cracks show more clearly
- asoneth 1y agoI published my first papers a little over fifteen years ago on practical applications for AI before switching domains. Recently I've been sucked back in. I agree it's a problem across all of science, but AI seems to attract more than it's fair share of researchers seeking fame and fortune. Exaggerated claims and cherry-picking data seem even more extreme in my limited experience, and even responsible researchers end up exaggerating a bit to try and compete.
- BenFranklin100 1y agoThe author is a Princeton PhD grad working in physics. Funding for this type of work usually comes from the NSF. NSF is under attack by DOGE, and Trump has proposed slashing the NSF budget by 55%. A reason used to justify these massive cuts is that AI will soon replace traditional research. This post demonstrates this assumption is likely false.
- surfingdino 1y agoWhen politicians get involved in research scientific proof and reason aren't always winning.
- adastra22 1y agoPoliticians have been involved in research for over a century.
- toolslive 1y agoDidn't a politician invent the internet ?
- TypingOutBugs 1y agoWho?
- sundarurfriend 1y agoAl Gore played a big role in getting political (and hence economic) support for the expansion of the Internet. https://en.wikipedia.org/wiki/Al_Gore_and_information_technology https://en.wikipedia.org/wiki/Al_Gore_and_information_techno... : > Al Gore, a strong and knowledgeable proponent of the Internet, promoted legislation that resulted in President George H.W Bush signing the High Performance Computing and Communication Act of 1991. This Act allocated $600 million > In the early 1990s the Internet was big news ... In the fall of 1990, there were just 313,000 computers on the Internet; by 1996, there were close to 10 million. The networking idea became politicized during the 1992 Clinton–Gore election campaign, where the rhetoric of the information highway captured the public imagination. Your parent comment is either joining in on the ridicule side or at least in misquoting: > Gore became the subject of controversy and ridicule when his statement, "I took the initiative in creating the Internet", was widely quoted out of context. It was often misquoted by comedians and figures in American popular media who framed this statement as a claim that Gore believed he had personally invented the Internet.[54] Gore's actual words were widely reaffirmed by notable Internet pioneers, such as Vint Cerf and Bob Kahn, who stated, "No one in public life has been more intellectually engaged in helping to create the climate for a thriving Internet than the Vice President."
- rhubarbtree 1y agoI think this is mostly just a repeat of the problems of academia - no longer truth-seeking, instead focused on citations and careerism. AI is just a.n.other topic where that is happening.
- geremiiah 1y agoI don't want to generalize because I do not know how widespread this pattern is, but my job has me hopping between a few HPC centers around Germany, and a pattern I notice is that, a lot of these places are chuck full of reject physicists, and a lot of the AI funding that gets distributed gets gobbled up by these people and the consequence of which is a lot of these ML4Science projects. I personally think it is a bit of a shame, because HPC centers are not there to only serve physicists, and especially with AI funding we in Germany should be doing more AI-core research.
- ktallett 1y agoHPCs are usually in Collab with universities for specific science research. Using up their resources is hopping on the bandwagon to damage another industry.an industry (AI) which is neither new nor anywhere close to being anything more than an personal assistant at the moment. Not even a great one at that.
- shusaku 1y ago> a pattern I notice is that, a lot of these places are chuck full of reject physicists Utter nonsense, these are some of the smartest people in the world who do incredibly valuable science.
- loa_in_ 1y agoExactly. Passing academia is the opposite of being a reject.
- barrenko 1y agoSeriously don't understand what "no longer" does here.
- deleted 1y ago[deleted]
- raesene9 1y agoInteresting article. There is always risk that a new hot technique will get more attention that it ultimately warrants. For me the key quote in the article is "Most scientists aren’t trying to mislead anyone, but because they face strong incentives to present favorable results, there’s still a risk that you’ll be misled." Understanding people's incentives is often very useful when you're looking at what they're saying.
- ktallett 1y agoThere are those who have realised they can make a lot of cash from it and also get funding by using the term AI. But at the end of the day what software doesn't have some machine learning built in. It's nothing new, nor is the current implementations particularly extraordinary or accurate.
- asdff 1y agoPlenty of software has zero ML. But either way not all ML is the same. There are many different algorithms each with their own tradeoffs. AI as it is presently marketed however usually means one type of AI, large language model, which also has tradeoffs, and is a bit new to the scene compared to say markov chains whose history starts in the early 1900s.
- overfeed 1y agoAI is a fuzzy term, and a moving target. Expert Systems have zero ML and were considered cutting-edge AI once upon a time.
- mooreds 1y ago> also get funding by using the term AI. Don't underestimate this. I'm peripherally in the startup world and funding has dried up for everyone unless you have some kind of AI story. So people shoehorn in AI to their company stories.
- pawanjswal 1y agoAppreciate the honesty. AI isn’t magic, and it’s refreshing to see someone actually say it out loud.
- omneity 1y agoThe article initially appears to suggest that all AI in science (or at least the author’s field) is hype. But their gripe seems to be specific to an architecture named PINN that seems to be overhyped, as they mention in the end how they end up using other DL models to successfully compute PDEs faster than traditional numerical methods.
- hyttioaoa 1y agoHe published a whole paper providing a systematic analysis of a wide range of models. There's a whole section on that. So it's not specific to PINN.
- BlueTemplar 1y agoThe use of the term «AI» is, yet again, annoying by its vagueness. I'm assuming that they do not refer to the general use of machines to solve differential equations (whether exactly or approximately), which is centuries old (Babbage's engine). But then how restricted these «Physics-Informed Neural Networks» are ? Are there other methods using Neural Networks to solve differential equations ?
- nottorp 1y agoReplace PINN with any "AI" solution for anything and you'll still find it overhyped. The only realistic evaluations of "AI" so far are those that admit it's only useful for experts to skip some boring work. And triple check the output after.
- geremiiah 1y agoIt's more widespread than PINNs. PINNs have been widely known to be rubbish a long time ago. But the general failure of using ML for physics problems is much more widespread. Where ML generally shines is either when you have relatively lots of experimental data with respect to a fairly narrow domain. This is the case for machine learned interatomic potentials MLIPs which have been a thing since the '90s. Also potentially the case for weather modelling (but I do not want to comment about that). Or when you have absolute insane amounts of data, and you train a really huge model. This is what we refer to as AI. This is basically why Alphafold is successful, and Alphafold still fails to produce good results when you query it on inputs that are far from any data points in its training data. But most ML for physics problems tend to be somewhere in between. Lacking experimental data and working with not enough simulation data because it is so expensive to produce. And also training models that are not large enough, because inference would be too slow, anyway, if they were too big. And then expecting these models to learn a very wide range of physics. And then everyone jumps in on the hype train, because it is so easy to give it a shot. And everyone gets the same dud results. But then they publish anyway. And if the lab/PI is famous enough or if they formulate the problem in a way that is unique and looks sciency or mathy, they might even get their paper in a good journal/conference and get lots of citations. But in the end, they still only end up with the same results as everyone else: replicates the training data to some extent, somebody else should work on the generalizability problem.
- kumarvvr 1y agoAre complex math problems just solvable by LLMs, as a stream of language tokens? I mean, there ought to be an element of abstract thought, abstract reasoning, abstract inter-linking of concepts, etc, to enable mathematicians to solve complex math theorems and problems. What am I missing?
- geremiiah 1y agoLLMs are not involved anywhere. You start with some data. Either simulation data or experimental data. Then you train a model to either learn a time evolution operator or a force field. Then you apply it to more input data, and you visualize the results. One typical use case is that the simulation data takes months to generate. So for experimental use cases, it is very slow. So the idea was, to train a model that can learn the underlying physics. The model will be small enough so that inference won't be prohibitively expensive. So you can then use the ML model in lieu of the classical physics based model. Where this usually fails is that while ML models can be trained well enough to replicate the training data, they typically fail to generalize well outside of the domain and regime of the training data. So unless your experimental problems are entirely within the same domains and regimes as the training data, your model is of not much use. So claims of generalizability and applicability are always dubious. Lots of publications on this topic follow the same pattern: conceive of a new architecture or formalism, train an ML model on widely available data, results show that it can reproduce the training data to some extent, mention generalizability in the discussion but never test it.
- thrdbndndn 1y agoI understand lots of people would (rightfully) say "no shit", but I think it's good to actually describe how it is in details. So kudos to the author.
- intended 1y agoVerification is at the heart of economic and intellectual activity.
- yapyap 1y agoGlad to see some people who bought into the nonsense are waking up
- plasticeagle 1y agoDoes anybody else find it peculiar that the majority of these articles about AI say things like "of course I don't doubt that AI will lead to major discoveries", and then go on to explain how they aren't useful in any field whatsoever? Where are the AI-driven breakthroughs? Or even the AI-driven incremental improvements? Do they exist anywhere? Or are we just using AI to remix existing general knowledge, while making no progress of any sort in any field using it?
- moffkalast 1y ago> AlphaEvolve’s procedure found an algorithm to multiply 4x4 complex-valued matrices using 48 scalar multiplications, improving upon Strassen’s 1969 algorithm that was previously known as the best in this setting. This finding demonstrates a significant advance over our previous work, AlphaTensor, which specialized in matrix multiplication algorithms, and for 4x4 matrices, only found improvements for binary arithmetic. > To investigate AlphaEvolve’s breadth, we applied the system to over 50 open problems in mathematical analysis, geometry, combinatorics and number theory. The system’s flexibility enabled us to set up most experiments in a matter of hours. In roughly 75% of cases, it rediscovered state-of-the-art solutions, to the best of our knowledge. > And in 20% of cases, AlphaEvolve improved the previously best known solutions, making progress on the corresponding open problems. For example, it advanced the kissing number problem. This geometric challenge has fascinated mathematicians for over 300 years and concerns the maximum number of non-overlapping spheres that touch a common unit sphere. AlphaEvolve discovered a configuration of 593 outer spheres and established a new lower bound in 11 dimensions. https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf https://storage.googleapis.com/deepmind-media/DeepMind.com/B... (this is an LLM driven pipeline)
- Ygg2 1y agoThat's less LLM and more three projects by DeepMind team. And it's far from commercial availability.
- 1y ago
- reify 1y agoExtremely diplomatic Most scientists aren’t trying to mislead anyone, but because they face strong incentives to present favorable results, there’s still a risk that you’ll be misled. In other words scientists are trying to mislead everyone because there are a lot of incentives; money and professional status to name just two. A common problem across all disciplines of science.
- nathias 1y agopeople will continue to publish cope articles about how AI is useless, far after superintelligence will be reached
- vaylian 1y agocitation needed
- tonii141 1y agoThis article addresses the misconception that arises when someone lacks a clear understanding of the underlying mathematics of neural networks and mistakenly believes they are a magical solution capable of solving every problem. While neural networks are powerful tools, using them effectively requires knowledge and experience to determine when they are appropriate and when alternative approaches are better suited.
- constantcrying 1y agoThis does not apply to PINNs though. They were used and investigated by people deeply knowledgeable about numerics and neural networks, they just totally failed to live up to expectation.
- tonii141 1y ago"they just totally failed to live up to expectation" Because the expectation was too high. If you are aiming for precision, neural networks might not be the best solution for you. That is why generative AI works so well, it doesn’t need to be extremely precise. On the other hand you don't see people use neural networks in system control for cricital processes.
- sgt101 1y agoI think that while the mathematics of neural networks are clearly completely understood we do not really understand why neural networks behave the way that they do when combined with large amounts of real world data. In particular the ability of auto regressive transformer based networks to produce sequences speech while being immutable still shocks me whenever I think about it. Of course, this says as much about what we think of ourselves and other humans as it does about the matrices. I also think that the weather forcasting networks are quite shocking, the compression that they have achieved in modeling the physical system that produces weather is frankly.... wrong... but it obviously does actually work.
- shalmanese 1y agoThis is less an article about AI and more about, one of the less talked about functions of a PhD program is becoming literate at “reading” academic claims beyond their face value. None of the claims made in the article are surprising because they’re the natural outgrowth of the hodgepodge of incentives we’ve accreted as what we call “science” over time and you just need to practice over time to be able to place the output of science in the proper context and understand that a “paper” is an artifact of a sociotechnical system with all the entailing complexity that demands.
- Flamentono2 1y agoI'm not sure why people on HN (of all places) are so divided regarding the perception of AI/ML. I have not seen anything like it before. We literaly had not system or way of even doing things like code generation based on text input. Just last week i asked for a script to do image segmentation with a basic UI and claude just generated that for me in under 1 Minute. I could list tons of examples which are groundbreaking. The whole Image generation stack is completly new. That blog article is fair enough, there is hype around this topic for sure, but alone for every researcher who needs to write code for their research, AI can make them already a lot more efficient. But i do believe, that we have entered a new ara: An ara were we take data again very serious. A few years back, you said 'the internet doesn't forget' then we realized that yes the internet starts to forget. Google deleted pages, removed the cache feature and it felt like we stoped caring for data because we didn't knew what to do with it. Then ai came along. And not only is now data king again but we are now in the mids of reinforcment ara: We now give feedback and the systems incorporate that feedback into their training/learning. And the ai/ml topic is getting worked on on every single aspect of it: Hardware, Algorithm, use cases, data, tools, protocols, etc. We are in the middle of incorporating and building for and on it. This takes a little bit of time. Still the progress is crazy exhausting. We will only see in a few years if there is a real ceiling. We do need more GPUs, bigger Datacenters to do a lot more experiments on AI architecture and algorithm. We have a clear bottleneck. Big companies train one big model for weeks and month.
- KurSix 1y agoBut on the flip side, the "AI will revolutionize science" narrative feels way ahead of what the evidence supports
- callc 1y ago> “I'm not sure why people on HN (of all places) are so divided regarding the perception of AI/ML.” Everyone is a rational actor from their individual perspective. The people hyping AI, and the people dismissing the hype both have good reasons. The is justification to see this new tech as ground breaking. There is justification to be weary about massive theft of data and dismissiveness of privacy. First, acknowledge and respect that there are so many opinions on any issue. Take yourself out of the equation for a minute. Understand the other side. Really understand it. Take a long walk in other people’s shoes.
- spwa4 1y agoTLDR: AI is like any new method in software engineering. It is not a general solution, and by itself not that useful, only as an addition. Unless an expert human takes a LOT of time to fine-tune the method (ie. automatically selecting what works well in what case, using the best method in almost all cases) it only performs well in a very small subset of cases.
- KurSix 1y agoThe comparison to the replication crisis is spot on
- therebase 1y agoIt is not only about the results we create with these tools, but as well about the effect they have on us as a result. Just about tech engineering here but I do think it transfers to science as well. https://dev.to/sebs/the-quiet-crisis-how-is-ai-eroding-our-technical-competence-2in2 https://dev.to/sebs/the-quiet-crisis-how-is-ai-eroding-our-t...
- sublimefire 1y agoGreat analysis and spot on examples. Another issue with AI related research is that a lot of papers are new and not that many get published in “proper” places, yet being quoted right/left/center, just look at google scholar. It is hard to repro the results and check the validity of some statements, not to mention that research which was done 4 years ago used one set of models and now another set of models with different training data is used in tests. It is hard to establish what really affects the results and if the conclusions are applicable to some specific property of the outdated model or if it is even generalisable.
- eviks 1y ago> I found that AI methods performed much worse than advertised. Lesson learned: don't trust ads > Most scientists aren’t trying to mislead anyone More learning ahead, the exciting part of being a scientist!
- wrren 1y agoAI companies are hugely motivated to show beyond-human levels of intelligence in their models, even if it means flubbing the numbers. If they manage to capture the news cycle for a bit, it's a boost to confidence in their products and maybe their share price if they're public. The articles showing that these advances are largely junk aren't backed by corporate marketing budgets or the desires of the investor class like the original announcements were.
- rajnathani 1y agoJust 2 days ago, there was an HN post about an AI-aided discovery of a fast matrix multiplication algorithm ("X X^t can be faster" | 198 points, 61 comments): https://news.ycombinator.com/item?id=44006824 https://news.ycombinator.com/item?id=44006824
- i_c_b 1y agoI'm probably saying something obvious here, but it seems like there's this pre-existing binary going on ("AI will drive amazing advances and change everything!" "You are wrong and a utopian / grifter!") that takes up a lot of oxygen, and it really distracts from the broader question of "given the current state of AI and its current trajectory, how can it be fruitfully used to advance research, and to what's the best way to harness it?" This is the sort of thing I mean, I guess, by way of close parallel in a pre-AI context. For a while now, I've been doing a lot of private math research. Whether or not I've wasted my time, one thing I've found utterly invaluable has been the OEIS.org website, where you can just enter sequence of numbers and then search for it to see what contexts it shows up in. It's basically a search engine for numerical sequences. And the reason it has been invaluable is that I will often encounter some sequence of integers, I'll be exploring it, and then when I search for it on OEIS, I'll discover that that sequence shows up in much different mathematical contexts. And that will give me an opening to 1) learn some new things and recontextualize what I'm already exploring and 2) give me raw material to ask new questions. Likewise, Wolfram Mathematica has been a godsend. And it's for similar reasons - if I encounter some strange or tricky or complicated integral or infinite sum, it is frequently handy to just toss it into Mathematica, apply some combination of parameter constraints and Expands and FullSimplify's, and see if whatever it is I'm exploring connects, surprisingly, to some unexpected closed form or special function. And, once again, 1) I've learned a ton this way and gotten survey exposure to other fields of math I know much less well, and 2) it's been really helpful in iteratively helping me ask new, pointed questions. Neither OEIS nor Mathematica can just take my hard problems and solve them for me. A lot of this process has been about me identifying and evolving what sorts of problems I even find compelling in the first place. But these resources have been invaluable in helping me broaden what questions I can productively ask, and it's through something more like a high powered, extremely broad, extremely fast search. There's a way that my engagement with these tools has made me a lot smarter and a lot broader-minded, and it's changed the kinds of questions I can productively ask. To make a shaky analogy, books represent a deeply important frozen search of different fields of knowledge, and these tools represent a different style of search, reorganizing knowledge around whatever my current questions are - and acting in a very complementary fashion to books, too, as a way to direct me to books and articles once I have enough context. Although I haven't spent nearly as much time with it, what I've just described about these other tools certainly is similar to what I've found with AI so far, only AI promises to deliver even more so. As a tool for focused search and reorganization of survey knowledge about an astonishingly broad range of knowledge, it's incredible. I guess I'm trying to name a "broad" rather than "deep" stance here, concerning the obvious benefits I'm finding with AI in the context of certain kinds of research. Or maybe I'm pushing on what I've seen called, over in the land of chess and chess AI, a centaur model - a human still driving, but deeply integrating the AI at all steps of that process. I've spent a lot of my career as a programmer and game designer working closely with research professors in R1 university settings (in both education and computer science), and I've particularly worked in contexts that required researchers to engage in interdisciplinary work. And they're all smart people (of course), but the silofication of various academic disciplines and specialties is obviously real and pragmatically unavoidable, and it clearly casts a long shadow on what kind of research gets done. No one can know everything, and no one can really even know too much of anything out of their own specialties within their own disciplines - there's simply too much to know. There are a lot of contexts where "deep" is emphasized over "broad" for good reasons. But I think the potential for researchers to cheaply and quickly and silently ask questions outside of their own specializations, to get fast survey level understandings of domains outside of their own expertise, is potentially a huge deal for the kinds of questions they can productively ask. But, insofar as any of this is true, it's a very different way of harnessing of AI than just taking AI and trying to see if it will produce new solutions to existing, hard, well-defined problems. But who knows, maybe I'm wrong in all of this.
- Workaccount2 1y agoThis is the second article in a week where someone is writing about how "AI" has failed them in their field (here it's physics, the other article was radiology), and in both articles they are using now ancient mid 2010's deep learning NNs. I don't know if it's intentional, but the word "AI" means different things almost every year now. Its worse than papers getting released with "LLMs unable to do basic math" and then you see they used GPT-3 for the study.
- scuff3d 1y ago"I suspect that scientists are switching to AI less because it benefits science, and more because it benefits them." This is a huge problem in software, and it's not restricted to AI. So much of what has been adopted over the years has everything to do with making the programmers life easier, but nothing to do with producing better software. AI is a continuation of that.
- indoordin0saur 1y agoI saw the name of the blog owner (A "Timothy B. Lee") and was surprised to see that the ~70 year old inventor of HTTP and the web had such an active and cutting-edge blog.
- stonemetal12 1y agoAI for science is industrial scale P hacking. Dump in all the world's knowledge in to an AI and see what falls out.
- gowld 1y agoThere's an easy fix for that: Choose a smaller (more appropriate) P.
- deleted 1y ago[deleted]
- blitzar 1y agoI got fooled by a Ponzi Scheme–here's what it taught me about how to make money.
- toss1 1y ago>>Most scientists aren’t trying to mislead anyone, but because they face strong incentives to present favorable results, there’s still a risk that you’ll be misled. >>We also found evidence, once again, that researchers tend not to report negative results, an effect known as reporting bias. >>But unfortunately, the scientific literature is not a reliable source for evaluating the success of AI in science. >> One issue is survivorship bias. Because AI research, in the words of one researcher, has “nearly complete non-publication of negative results,” we usually only see the successes of AI in science and not the failures. But without negative results, our attempts to evaluate the impacts of AI in science typically get distorted. While these biases will absolutely create overconfidence and wasted effort, the fact that there are rapid advances with some clear successes such as protein folding, drug discovery, &weather forecasting, leads me to expect there will be more very significant advances, in no small part because of the massive investment in funds and time to the problem of making AI-based advances. For exactly the reasons this researcher spent his time and funds to research this, despite his negative results, there was learning, and the effect of millions of people effectively searching & developing will result in more great good advances being found and/or built. Whether they are worth the total financial & human capital being spent is another question, but I'm expecting that to be also positive
- bwfan123 1y agonice expose of human biases involved, need more of these to balance the hype. 1) Instead of identifying a problem and then trying to find a solution, we start by assuming that AI will be the solution and then looking for problems to solve. hammer in search of a nail 2) nearly complete non-publication of negative results survivorship (and confirmation bias) 3) same people who evaluate AI models also benefit from those evaluations power of incentives (and conflicts therein) 4) ai bandwagon effect, and fear of missing out social-proof
- abhinavsns 1y agoThere is a reason why AI fails: https://open.substack.com/pub/asimai/p/the-allure-of-ai-for-numerical-simulations https://open.substack.com/pub/asimai/p/the-allure-of-ai-for-...
- ausbah 1y ago> After a few weeks of failure, I messaged a friend at a different university, who told me that he too had tried using PINNs, but hadn’t been able to get good results. not really related to AI but this reflects a lesson I learned too late during some research in college: constant collaboration is important because it helps you avoid retreading over areas where others have already failed
- mmarian 1y agoOr the need for researchers to publish their failured experiments?
- thearn4 1y agoAnother reason why the idea of AI agents for science hasn't made much sense to me. Research is an extremely collaborative set of activities. How good would a researcher be who is very good at literature review, but never actually talks to anyone, goes to any conferences, etc?
- angry_moose 1y agoI've been "lucky" enough to get to trial some AI FEM-like structural solvers. At best, they're sortof ok for linear, small deformation problems. The kind of models where we could get an exact solution in ~5 minutes vs a fairly sloppy solution in ~30 seconds. Start throwing anything non-linear in and they just fall apart. Maybe enough to do some very high-level concept selection but even that isn't great. I'm reasonably convinced some of them are just "curvature detectors" - make anything straight blue, anything with high curvature red, and interpolate everything else.
- amelius 1y agoCould you use these models as a preconditioner in an iterative solver?
- angry_moose 1y agoI don't see any reason its not theoretically possible but I doubt it would be that beneficial. You'd have to map the results back onto the traditional model which has overhead; and using shaky results as a precondition is going to negate a lot of the benefits, especially if its (incorrectly) predicting the part is already in the non-linear stress range which I've seen before. Force balances are all over the place as well (if they even bother to predict them at all, which its not always clear) so it could even be starting from a very unstable point. Its relatively trivial to just use the native solution from a linear solution as the starting point instead, which is basically what is done anyway with auto time stepping.
- xeonmc 1y agoSo it’s more like a “second principles” solver, it cannot synthesize anything that it hadn’t already seen before.
- cadamsdotcom 1y agoHard to tell between “doesn’t work” and “too early”.
- mmarian 1y agoLine needs to be drawn somewhere though, otherwise you could make the same case for crypto and AR/VR.
- -__---____-ZXyw 1y agoDid the title get changed, or have I started hallucinating? Title is: "I got fooled by AI-for-science hype—here's what it taught me"
- kjhughes 1y agoIt got changed (for the worse, in my opinion) away from the original title. The original title is supposed to be favored here unless it has a serious problem. This original title had no serious problem, unless accurately summarizing a PhD candidate's thoughtful critique of some questionable AI contributions to scientific research is a serious problem.
- tanderson92 1y agoThe present title is more friendly to VCs and the tech industry, shocking no one.
- tanderson92 1y agoNo, you are not hallucinating: https://web.archive.org/web/20250520152757/https://news.ycombinator.com/item?id=44037941#44046657 https://web.archive.org/web/20250520152757/https://news.ycom...
- shantnutiwari 1y agoCould it be we are all scared, because if we call the Emperor naked, and 15 years from now someone finds a useful case for AI(even if its completely different to what exists today), everyone will point to our post and say "Hahaha look at those Luddites, didnt even believe AI was real LOL"
- Ukv 1y agoThe recent high level of funding (Stargate, HUMAIN, ...), seemingly prompted mostly by LLMs, could plausibly be an emperor's new clothes fear of missing out among investors - will have to wait and see how it pans out. But for 2010s-era machine learning this article is talking about, I feel it largely already has been validated - from shunned and unfunded at the start of the decade to being the almost universal go-to for any NLP or computer vision task by the end. The article itself lists a few use-cases (protein folding, weather forecasting, drug discovery), and I think it's unlikely you've gone through the day without encountering at least a few more (maybe search engines query-understanding, language translation, generated video captions, OCR, or using a product that was scanned for defects). Not that every ML method will work out first try when applied to a new problem, but it's far from the case that we're waiting 15 years hoping for someone to maybe find a use-case for the field.
- jxjnskkzxxhx 1y ago> AI adoption is exploding among scientists less because it benefits science and more because it benefits the scientists themselves. This is true in so many aspects of human life - anyone trying to run an organisation should be aware of it.