52 ms·

Does current AI represent a dead end?

• ramon156 2y ago
  Aider with claude sonnet is probably all I needed to get my programming cycle up to speed. I don't think I want anything more as a developer. That said, it still makes mistakes
  • jokethrowaway 2y ago
    so maybe you want less mistakes?
• goodpoint 2y ago
  Yes.
• crest 2y ago
  Only if you care about about causation instead of just correlation.
• optimalsolver 2y ago
  Yes. An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.
  • m3kw9 2y ago
    We are thinking machines and we keep thinking because we have one goal which is to survive, machines have no such true goals. I mean true because our biology forces us to do that
  • xvector 2y ago
    Is a human with short term memory loss - or otherwise unable to improve their skills - generally intelligent?
    • optimalsolver 2y ago
      Would you let such a person handle an important task for you?
    • rcarmo 2y ago
      A human with short term memory loss still has agency and impatience.
      • xvector 2y ago
        Agency is essentially solved, we don't enable it in common models because of "safety" Is impatience a requirement for general intelligence? Why?
  • aetherson 2y ago
    There is a limited amount of computation that you can useful do in the absence of new input (like an LLM between prompts). If you do as much computation as you usefully can (with your current algorithmic limits) in a burst immediately when you receive a prompt, output, and then go into a sleep state, that seems obviously better than receive a prompt, output, and then do some of the computation that you can usefully do after your output.
  • amelius 2y ago
    Can't we just finetune the model based on the LLM's output? Has anyone tried it?
    • soulofmischief 2y ago
      Not only does a training pass take more time and memory than an inference pass, but if you remember the Microsoft Tay incident, it should be self-explainatory why this is a bad idea without a new architecture.
  • alchemist1e9 2y ago
    self prompting via chain of thought and tree of thought can be used in combination with updating memory containing knowledge graphs combined with cognitive architectures like SOAR and continuous external new information and sensory data … with LLM at the heart of that system and it will exactly be a “thinking machine”. The problem is currently it’s very expensive to be continuously running inference full time and all the engineering around memory storage, like RAG patterns, and the cognitive architecture design is all a work in progress. It’s coming soon though.
    • whatwhaaaaat 2y ago
      We’re going to need to see this working. From my perspective many of the corporate llms are actually getting worse. Slop feedback loops. By no means has it been proven that llms functioning the way you describe will result in superior output.
  • Uehreka 2y ago
    I see people say this all the time and it sounds like a pretty cosmetic distinction. Like, you could wire up an LLM to a systemd service or cron job and then it wouldn’t be “waiting”, it could be constantly processing new inputs. And some of the more advanced models already have ways of compressing the older parts of their context window to achieve extremely long context lengths.
  • Earw0rm 2y ago
    If it's coalescing learning in realtime across all user/sessions, that's more constant than you're maybe giving it credit for. I'm not sure if GPT4o and friends are actually built that way though.
  • wat10000 2y ago
    If you had a magic stasis tube that kept a person suspended while you weren’t talking to them, they’d still be a thinking machine.
• rcarmo 2y ago
  Yes. Next question, please. And don't mention AGI.
• oldpersonintx 2y ago
  [dead]
• xvector 2y ago
  IMO we are already at AGI. Hell, Norvig would argue we were there some time ago: https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-... We just keep moving the goalposts.
  • K0balt 2y ago
    I agree. The systems in place already solve generalized problems not directly represented in the training set or algorithm . That was, up until the last few years , the off the shelf definition of AGI. And the systems in place do so at scales and breadths that no human could achieve. That doesn’t change the fact that it’s effectively triple PHD uncle Jim, as in slightly unreliable and prone to bullshitting its way through questions, despite having a breathtaking depth and breadth of knowledge. What we are making is not software in any normal sense of the word, but rather an engine to navigate the entire pool of human knowledge, including all of the stupidity, bias, and idiosyncrasies of humanity, all rolled up into a big sticky glob. It’s an incredibly powerful tool, but it’s a fundamentally different class of tool. We cannot expect to apply conventional software processes and paradigms to LLM based tools any more than we could apply those paradigms to politics or child rearing and expect useful results.
    • netdevphoenix 2y ago
      > The systems in place already solve generalized problems not directly represented in the training set or algorithm Tell me a problem that an LLM can solve that is not directly represented in the training set or algorithm. I would argue that 99% of what commercial LLMs gets prompted about are stuff that already existed in the training set. And they still hallucinate half lies about those. When your training data is most the internet, it is hard to find problems that you haven't encountered before
      • esafak 2y ago
        o3 solved a quarter of the challenging novel problems on the FrontierMath benchmark, a set of problems "often requiring multiple hours of effort from expert mathematicians to solve".
• m_ke 2y ago
  I’ve done a few projects that attempted to distill the knowledge of human experts, mostly in medical imaging domain, and was shocked when for most of them the inter annotator agreement was only around 60%. These were professional radiologists with years of experience and still came to different conclusions for fairly common conditions that we were trying to detect. So yes, LLMs will make mistakes, but humans do too, and if these models do so less often at a much lower cost it’s hard to not use them.
  • tomrod 2y ago
    This hints at the margin and excitement from folks outside the technical space -- being able to be competitive to human outputs at a fraction of the cost.
    • ethbr1 2y ago
      That's the underappreciated truth of the computer revolution in practice. At scale, computers didn't change the world because they did things that were already being computed, more quickly. They changed the world because they decreased the cost of computing so much that it could be used for an entirely new class of problems. (That computing cost previously precluded its use on)
  • threeseed 2y ago
    The problem is that how mistakes are made is crucial. If it's a forced binary choice then sure LLMs can replace humans. But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.
    • m_ke 2y ago
      None of these were binary decisions, but classifying one of around 10-20 conditions or rating cases on a 1-5 scale. In all cases the models trained on a lot of this feedback were more consistent and accurate than individual expert annotators.
• fredtalty5 2y ago
  [dead]
• Havoc 2y ago
  This take seems fundamentally wrong to me. As in opening premise. We use humans for serious contexts & mission critical tasks all the time and they're decidedly fallible and their minds are basically black boxes too. Surgeons, pilots, programmers etc. I get the desire for reproducible certainty and verification like classic programming and why a security researcher might push for that ideal, but it's not actually a requirement for real world use.
  • skydhash 2y ago
    Legal punishment is a great incentive to try to do your best job. You can reliably trust someone to act in one’s best interest.
    • protomolecule 2y ago
      Maybe include in a prompt a threat of legal punishment? Sure somebody has already tried that and tabulated how much it improves scores on different benchmarks)
      • timeon 2y ago
        Maybe legal threat for the company operating it? Would that help?
      • bick_nyers 2y ago
        I suspect the big AI companies try to adversarially train that out as it could be used to "jailbreak" their AI. I wonder though, what would be considered a meaningful punishment/reward to an AI agent? More/less training compute? Web search rate limits? That assumes that what the AI "wants" is to increase its own intelligence.
    • Havoc 2y ago
      LLM's response being best prediction of next token arguably isn't that far off from a human motivated to do their best. It's a fallible best effort either way. And both are very far from the certainty the author seems to demand.
• bsenftner 2y ago
  If you expect the AI to do independent work, yes, it is a dead end. These LLM AIs need to be treated and handled as what they are: idiot savants with vast and unreliable intelligence. What does any advanced organization do when they hire a new PhD, let them loose in the company or pair them with experienced staff? When paired with experienced staff, they use the new person for their knowledge but do not let them change things on their own until much later, when confidence is established and the new staffer has been exposed to how things work "around here". The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed. That means the path forward with this current state of LLM AIs is to pair them with people, personalized to their needs, and treat them as very smart idiot savants great for strategy and problem solving discussion, where the human users are driving the situation, using the LLM AIs like a smart assistant that requires validation - just like a real new hire. There is an interactive state that can be achieved with these LLM AIs, like being in a conversation with experts, where they advise, they augment and amplify individual persons. A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today.
  • Alex3917 2y ago
    > The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed. Basically this. They already have vastly better-than-human ability at finding syntax errors within code, which on its own is quite useful; think of how many people have probably dropped out of CS as a major after staying up all night and failing to find a missing semicolon.
    • lionkor 2y ago
      I don't know of a single person who got so stuck on syntax errors that they quit
      • FroshKiller 2y ago
• iambateman 2y ago
  While these folks waste breath debating whether AI is useful, I’m going to be over here…using it. I use AI 100 times a day as a coder and 10,000 times a day in scripts. It’s enabled two specific applications I’ve built which wouldn’t be possible at single-person scale. There’s something about the psychology of some subset of the population that insists something isn’t working when it isn’t _quite_ working. They did this with Wikipedia. It was evident that Wikipedia was 99% great for years before this social contingent was ready to accept it.
  • Mistletoe 2y ago
    But please accept that you are in a small subset of people that it is very useful to. Every time I hear someone championing AI, it is a coder. AI is basically useless to me, it is just a convoluted expensive google search.
    • tossandthrow 2y ago
      I use ai to care for my plants, to give me recipes for pan pancakes, to help me fix my coffee machine. LLMs as a popularized thing is just about 2 years old. It is still mainly early adopters. For smartphones it might have taken 10 to 15 years to gain widespread traction. I think it is safe to say that we are only scratching the surface.
      • giraffe_lady 2y ago
        These are not categories that needed this change or benefit from it. Specific plant care is one of the easiest things to find information about. And are you serious you couldn't find a pancake recipe? The coffee machine idk it depends on what you did. But the other two are like a parody of AI use cases. "We made it slightly more convenient, but it might be wrong now and also burns down a tree every time you use it."
        • qup 2y ago
          > "We made it slightly more convenient, but it might be wrong now and also burns down a tree every time you use it." Sounds like early criticisms of the internet. I assume you mean he should be doing those things with a search engine, but maybe we shouldn't allow that either. Force him to use a book! It may be slightly less convenient, and could still be wrong, but...
• paxys 2y ago
  Every piece of technology is a "dead end" until something better replaces it. That doesn't mean it can't be useful or revolutionary.
• tu7001 2y ago
  We all know what's an answer when there is a question mark in the title.
  • IanHalbwachs 2y ago
    https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
• btbuildem 2y ago
  Interesting read -- and a correct take, given the software development perspective. In that context, LLM-based AI is faulty, unpredictable, and unmanageable, and not ready for mission-critical applications. If you want to argue otherwise, do a quick thought experiment first: would you let an LLM manage your financial affairs (entirely, unsupervised)? Would you let it perform your job while you receive the rewards and consequences? Would you be comfortable to give it full control of your smart home? There are different sets of expectations put on human actors vs autonomous systems. We expect people to be fallible and wrong some of the time, even if the individuals in question can't/won't admit it. With a software-based system, the expectations are that it will be robust, tested, and performing correctly 100% of the time, and when a fault occurs, it will be clear, marked with yellow tape and flashing lights. LLM-based AIs are sort of insidious in that they straddle this expectation gap: the emergent behaviour is erratic, projecting confident omniscience, while often hallucinating and plain wrong. However vague, the catch-all term "AI" still implies "computer system" and by extension "engineered and tested".
  • cesaref 2y ago
    It's a bad example. Lots of finance firms use AI to manage their financial affairs - go and investigate what is currently considered state of the art for trading algorithms. Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.
    • galleywest200 2y ago
      > go and investigate what is currently considered state of the art for trading algorithms. These are not LLMs but algorithms written and designed by human minds. It is unfortunate that AI has become a catch-all word for any kind of machine learning.
      • varelse 2y ago
        [dead]
      • 2y ago
• IanCal 2y ago
  I find this kind of argument comes up a lot and it seems fundamentally flawed to me. 1. You can set a bar wherever you want for a level of "seriousness" and huge swathes of real world work will fall below it, and are therefore attractive to tackle with these systems. 2. We build critical large scale systems out of humans, which are fallible and unverifiable. That's not to say current LLMs are human or equivalent, but "we can't verify X works all the time" doesn't stop us doing exactly that a lot. We deal with this by learning how humans make mistakes, why, and build systems of checks around that. There is nothing in my ind that stops us doing the same with other AI systems. 3. Software is written by, checked by and verified by humans at least at some critical point - so even verified software still has this same problem. We've also been doing this kind of thing with ML models for ages, and we use buggy systems for an enormous amount of work worldwide. You can argue we shouldn't and should have fully formally verified systems for everything, but you can't deny that right now we have large serious systems without that. And if your goal is "replace a human" then I just don't think you can reasonably say that it requires verifiable software. > Systems are not explainable, as they have no model of knowledge and no representation of any ‘reasoning’. Neither of those statements are true are they? There are internal models, and recent models are designed around having a representation of reasoning before replying. > current generative AI systems represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability And yet reliability is something we see improve as LLMs get better and we get better at training them.
• nialse 2y ago
  There are two epistemic poles: the atomistic and the probabilistic. The author subscribes to a rule-based atomistic worldview, asserting that any perspective misaligned with this framework is incorrect. Currently, academia is undergoing a paradigm shift in the field of artificial intelligence. Symbolic AI, which was the initial research focus, is rapidly being replaced by statistical AI methodologies. This transition diminishes the relevance of atomistic or symbolic scientists, making them worry they might become irrelevant.
  • irunmyownemail 2y ago
    Not sure I followed all of that lingo but it sounds like a fancy way of saying, if you're losing the game, try shifting the goal post.
    • nialse 2y ago
      Indeed and unfortunately. I've been reading up on "the binding problem" in AI lately and came across a paper that hinged on there being an "object representation" which would magically solve the apparent issues in symbolic AI. In the discussion some 20 pages later, the authors confessed that they, nor anybody else, could define what an object was in the first place. Sometimes the efforts seem focused on "not letting the other team win" rather than actually having something tangible to bring to the table.
  • dbmikus 2y ago
    I never want to claim certainties, but it seems pretty close to certain that symbolic AI loses to statistical AI. I think there is room for statistical AI to operate symbolic systems so we can better control outputs. Actually, that's kind of what is going on when we ask AI to write code.
  • Shorel 2y ago
    I think that transition already happened, and the next big jump in AI will be the combination of these two approaches in an unified package. Kind of the way our right brain hemisphere does probabilistic computation and the left brain hemisphere does atomistic computation. And we use both. So, whoever develops the digital equivalent of the corpus callosum wins.
• llm_trw 2y ago
  No, they are very useful tools to build up inteligent systems out of. Everything from perplexity onward shows just how useful agents can be. You get another bump in utility when you allow for agents swarms. Then another one for dynamically generated agent swarms. The only reason why it's not coming for your job is that LLMs are currently too power hungry to run those jobs for anything but research - at a couple thousand to couple of million times the price of a human doing the work. Which works out to 10 to 20 epochs of whatever Moore's law looks like in graphics cards.
  • throw83288 2y ago
    What is that bump in utility in practical terms? You can point to a benchmark improvement but that's no indication the agent swarm is not reducing to "giving an llm an arbitrary amount of random guesses".
• inciampati 2y ago
  Want reliable AI? Stop approximating memory with attention and build reliable memory into the model directly.
  • logicchains 2y ago
    Standard LLM quadratic attention isn't an approximation, it's perfect recall. Approaches that compress that memory down into a fixed-size state are an approximation, and generally perform worse, that's why linear transformers aren't widely used.
• mrtksn 2y ago
  What I find curious is that the people who sell the AI as the holy grail that will make any jobs obsolete in a few year at the same time claim that there's huge talent shortage and even engage in feud on immigration and spend capital to influence immigration policies. Apparently they don't believe that AI is about to revolutionize things that much. This makes me believe that significant part of the AI investment is just FOMO driven, so no real revolution is around the corner. Although we keep seeing claims that AI achieved PHD level this Olympics level that, people who actually own these keep demanding immigration policy changes to bring actual humans from overseas for year to come.
  • fhd2 2y ago
    Is that so? I'm not in the US, so I don't have a good idea of what's going on there. But wasn't there relatively high unemployment among developers after all these Big Tech layoffs post pandemic? Shouldn't companies there have an easy time finding local talent? Sorry for the potentially silly question. I just spent some time trying to research it and came up with nothing concrete.
    • mrtksn 2y ago
      > But wasn't there relatively high unemployment among developers after all these Big Tech layoffs post pandemic? I'm speculating too but yes it appears that unemployment is pretty high among the CS majors: https://www.reddit.com/r/csMajors/comments/1hhl060/how_is_it_this_bad/ https://www.reddit.com/r/csMajors/comments/1hhl060/how_is_it... But at the same time there's an ongoing infighting among Trump supporters because tech elites came up as pro - skilled immigration where the MAGA camp turned against them. The tech elites claim that there's a talent shortage. Here's a short rundown that Elon Musk agrees with: https://x.com/AutismCapital/status/1872408010653589799 https://x.com/AutismCapital/status/1872408010653589799
      • fhd2 2y ago
        Ah I see, thought I missed some major story, but apparently not. The unemployment data is from 2018 BTW. But from what I perceive, developer unemployment in the US seems higher than usual right now.
• JimmyWilliams1 2y ago
  The reliance on large datasets for training AI models introduces biases present in the data, which can perpetuate or even exacerbate societal inequalities. It's essential to approach AI development with caution, ensuring robust ethical guidelines and comprehensive testing are in place before integrating AI into sensitive areas. As we continue to innovate, a focus on explainability, fairness, and accountability in AI systems will be paramount to harnessing their potential without compromising societal values.
  • owenpalmer 2y ago
    > exacerbate societal inequalities Do you have an example of this?
• tbenst 2y ago
  As a neuroscientist, my biggest disagreement with the piece is the author’s argument for compositionality over emergence. The former makes me think of Prolog and lisp, while the later is a much better description for a brain. I think ermergence is a much more promising direction for AGI than compositionality.
  • EerkeBoiten 2y ago
    Author here. So what! I am not talking about promising directions for AGI, I am talking about having computer systems that we can have confidence in. Sure, AGI if it ever happens will look more like emergence than compositionality, and I'm sure it won't feel a need to explain to us fallible humans why its decisions are correct. In the meantime, I'd like computer systems to be manageable, reliable, transparent, and accountable.
  • dbmikus 2y ago
    100% agree. When we explicitly segment and compose AI components, we are removing the ability for them to learn their own pathways between the components. We've been proven time and time again the bitter lesson[1]: that throwing a ton of data and compute at a model yields better results than what we could come up with. That said, we can still isolate and modify parts of a network, and combine models trained for different tasks. But you need to break things down into components after the fact, instead of beforehand, in order to get the benefits of learning via scale of data + compute. [1]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
• cs702 2y ago
  As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more. The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain. The largest AI systems in use today have hundreds of billions of parameters. Nearly all parameters are part of a weight matrix, each parameter quantifying the strength of the connection from an artificial input neuron to an artificial output neuron. The human brain has more than a hundred trillion synapses, each connecting an organic input neuron to an organic output neuron, but the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.[a] Today's largest AI systems have about the same number of neuron-neuron connections as the brain of a brown rat.[a] Judging these AI systems based on their current capabilities is like judging organic brains based on the capabilities of brown rat brains. What we can say with certainty is that today's AI systems cannot be trusted to be reliable. That's true for highly trained brown rats too. --- [a] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... -- sort in descending order by number of synapses.
  • deleted 2y ago
    [deleted]
  • JumpCrisscross 2y ago
    > we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more This doesn’t solve the unpredictability problem.
    • leobg 2y ago
      We don’t know. We didn’t predict that the rat brain would get us here. So we also can’t be confident in our prediction that scaling it won’t solve hallucination problems.
    • cs702 2y ago
      No, it doesn't "solve" the unpredictability problem. But we haven't solved it for human beings either. Human brains are unpredictable. Look around you.
• amazingamazing 2y ago
  fact of the matter is that if AIs externalities were exposed - that is massive energy consumption - to end users and humanity in general, no one would use it.
  • NoGravitas 2y ago
    I wish we could get humanity in general to understand externalities in general.
  • fulafel 2y ago
    I think this is wildly optimistic about how environmentally conscious customers of LLMs are. People use fossil fuels directly and through electricity consumption in a unconscionable way at a scale wildly exceeding what a ChatGPT user's energy expenditure is. We desperately need to rapidly regulately down fossils usage and production for both electricity generation and transport. The rest of the world needs to follow the example of the EU CO2 emissions policy which guarantees it's progressing at a downwards slope independent of what the CO2 emissions are spent on.
• billy99k 2y ago
  I use it for fast documentation of unknown (to me) APIs and other pieces of software. It's saved me hours of time, where I didn't have to go through the developers site/documentation and I get quickly get example code. Would I use the code directly in production? No. I always use it as an example and write my own code.
• nashashmi 2y ago
  The elephant in the room: The user interface problem We seem to dancing around a problem in the middle of the room like an elephant no one is acknowledging, and that is the interface to Artificial Intelligence and Generative AI is a place that requires several degrees of innovations. I would argue that the first winning feat of innovation on interfacing with AI was the "CHAT BOX". And it works well enough for the 40% of use cases. And there is another 20% of uses that WE THE PEOPLE can use our imagination (prompt engineering) to manipulate the chat box to solve. On this topic, there was an article/opinion that said complex LLMs are unnecessary because 90% of people don't need it. Yeah. Because the chat box cannot do much more that would require heavier LLMs. Complex AI and large data sets need nicer presentation and graphics, more actionable interfaces, and more refined activity concepts, as well as metadata that gives information on the reliability or usability of generated information. Things like edit sections of an article, enhance articles, simplify articles, add relevant images, compress text to fit in a limited space, generate sql data from these reports, refine patterns found in a page with supplied examples, remove objects, add objects, etc. Some innovation has to happen in MS Office interfaces. Some innovations have to happen in photoshop-like interfaces. The author is complaining about utopian systems being incompatible with AI. I would argue AI is a utopian system being used in a dystopian world where we are lacking rich usable interfaces.
• deleted 2y ago
  [deleted]
• vonneumannstan 2y ago
  Anyone making big bold claims about what LLMs definitely CAN or CANNOT do is FULL OF SHIT. Not even the worlds top experts are certain where the limit of these technologies are and we are already connecting them to tools, making them Agentic, etc. so the era of 'pure' LLM chatbots is already dead imo.
• bentt 2y ago
  I don’t see how it would because at the end of the day a model is like a program… input->output. This seems infinitely useful and we are just starting to understand how to use this new way of computing.
• 1oooqooq 2y ago
  first winds of winter coming...
• malthaus 2y ago
  i'm so confused by these discussions around hitting the wall. sure, a full-on AGI, non-hallucinating AI would be great. but the current state is already a giant leap. there's so much untapped potential in the corporate world where whole departments, processes, etc can be decimated. doing this and dealing with the socio-economic and political fall-out from those efficiency leaps can happen while research (along multiple pathways) goes on, and this will take 5-10 years at least.
  • zeroonetwothree 2y ago
    Hitting a wall doesn’t mean it’s not useful.
• mediumsmart 2y ago
  "Nobody's gonna believe that computers are intelligent until they start coming in late and lying about it." btw: the german KI (Keine Intelligenz) is much more accurate than AI (Apparently Intelligent)
• ZiiS 2y ago
  Current approches to AI are almost certainly going to be superseeded eventually, calling that a dead end achives nothing.
• darioush 2y ago
  Just because something is probabilistic in its response doesn't mean it's not useable. There are many probabilistic algorithms and data structures that we use daily. Yes, we don't have developed abstractions to integrate an LLM in a programming language, but it doesn't mean no one will make one.
• szundi 2y ago
  All branches of everything is a dead end sooner or later in life
• makach 2y ago
  Betteridge's law of headlines, current AI may absolutely be a dead end, but fortunately technology is evolving and changing - who knows what the future will hold.
• andrewguy9 2y ago
  Maybe it is, maybe it isn’t. The only thing I know is, none of the arrogant fuckers on hacker news know anything about it. But that won’t stop them from posting.
  • tucnak 2y ago
    There's an upside! If they're wrong, and they manage to convince more people—it basically gives you more of an advantage. I don't get into arguments about the utility of LLM technology anymore because why bother?
  • zeroonetwothree 2y ago
    Pot calling the kettle black
  • samatman 2y ago
    - Signed, Some arrogant fucker posting on hacker news
• josefritzishere 2y ago
  This may be an exception to Betteridge's law of headlines
• thunkingdeep 2y ago
  Useless and dead end aren’t synonymous. It’s most certainly a dead end, but it’s also not useless. There a lot of comments here already conflating these two. This article is also pretty crap. There’s a decent summary box but other than that it’s all regurgitated half-wisdoms we’ve all already realized: things will change, probably a lot; nobody knows what the end goal is or how far we are from it; the next quantum leap almost certainly depends on a transcendent architecture or new model entirely. This whole article could’ve been a single paragraph honestly, and a lot of the comments here probably wouldn’t have read that either… just sayin
• mmaunder 2y ago
  “In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application.” And yet here we are with what we all think of as serious and seriously useful applications. “My first 20 years of research were in formal methods, where mathematics and logic are used to ensure systems operate according to precise formal specifications, or at least to support verification of implemented systems.” I think recommending avoiding building anything serious in the field until your outdated verification methodology catches up is unreasonably cynical, but also naive because it discards the true nature of our global society and assumes a lab environment where this kind of control is possible.
  • EerkeBoiten 2y ago
    Author here. Unfair point, article makes clear I don't expect "outdated verification methodology" to catch up - and I even link that pessimistic expectation with the issue of emergence that fatally undermines NNs. Point about lab vs reality is fair.
• rob_c 2y ago
  Yes, until someone introduces reward into llm training I doubt we'll get much further
• dfilppi 2y ago
  Somehow, fallible humans create robust systems. Look to "AI' to do the same, at a far higher speed. The "AI" doesn't need to recite the Fibonacci sequence; it can write (and test) a program that does so. Speed is power.
• quotemstr 2y ago
  Whenever a new technology emerges, along with it always emerge naysayers who claim that the new technology could never work --- while it's working right in front of their noses. I'm sure there were people after Kitty Hawk who insisted that heavier than air flight would never amount to much economically. Krugman famously insisted in the 90s that the internet would never amount to anything. These takes are comical in hindsight. The linked article is another one of these takes. AI can obviously reason. o3 is obviously superhuman along a number of dimensions. AI is obviously useful for software development. This guy spend 20 years of his life working on formal methods. Of course he's going to poo-poo the AI revolution. That doesn't make him right.
  • EerkeBoiten 2y ago
    Author here. Putting a lot of words in my mouth there. In particular, I don't talk about whether AI is useful for software development - I talk about whether AI is useful as reliable software. I don't discuss how AI's abilities relate to human abilities. I don't discuss whether what AI currently does counts as "reasoning".
  • sealeck 2y ago
    > Whenever a new technology emerges, along with it always emerge naysayers who claim that the new technology could never work There's some survivorship bias going on here – you only consider technologies which succeeded, and find examples of people scrutinising them beforehand. However, we know that not every nascent technology blossoms; some are really effective, but can't find adopters; some are ahead of their time; some are cost-prohibitive; and some are outright scams. It's not a given that every promising new technology is a penicillin – some might be Theranos.
• omolobo 2y ago
  > I would call ‘LLM-functionalism’: the idea that a natural language description of the required functionality fed to an LLM, possibly with some prompt engineering, establishes a meaningful implementation of the functionality. My boy. More people need common sense like this talked into them.
• mikewarot 2y ago
  AI is only a dead end if you expect it to function deterministically. In the same way as people, it's not rational, and it can't be made rational. For example, the only effective way to get an AI not to talk about Bryan Lunduke is to have an external layer that scans for his name in the output of an AI, if found, stops the session and prints an error message instead. If you're willing to build systems around it (like we do with people) to limit it's side effects and provide sanity checks, and legality checks like those mentioned above, it can offer useful opinions about the world. The main thing to remember is that AI is an alien intelligence. Each new model is effectively the product of millions of dollars worth of forced evolution. You're getting Stitch from "Lilo and Stitch", and you'll never be sure if it's having a bad day.
  • clint 2y ago
    Also, is there a known deterministic intelligence? Only very specific computer programs can be made deterministic, and even that has taken quite a while for us to nail down. A lot of code and systems of code produced by humans today is not deterministic and it takes a lot of effort to get it there. For most people and teams its not even on their radar or worth the effort to get it there.
  • bloomingkales 2y ago
    Control freaks have a serious issue with the incompleteness of an LLM. Everyone else is just ecstatic that it gets you 70% of the way there often.
    • arrosenberg 2y ago
      > Control freaks People who like repeatable results in their work equipment are control freaks?
      • therein 2y ago
        I know, right? Software Engineers with their zeal for determinism. How dare they.
• peter_retief 2y ago
  AI is useful as a tool but it is far from trustworthy. I just used Grok to write some CRON scripts for me, gave me perfectly good results, if you know exactly what you want, it is great. It is not the end of software programmers though and is very dangerous to give it too much leeway because you will almost certainly end up with problems. I agree with the conclusion that a hybrid model is possible.
  • DeepYogurt 2y ago
    > if you know exactly what you want, it is great. Kinda kills the utility if you need to know what you want out tho...
    • stanac 2y ago
      It speeds up code writing, it's not useless. Best use case for me is to help me understand libraries that are sparsely documented (e.g. dotnet roslyn api). edit: spelling
    • xandrius 2y ago
      If I can get 100 lines generated instantly while explaining it in 25, scan the answer just to validate it and then, no wait, add other 50 lines as I forgot something before. All that in minutes then I'm happy. Plus I can detach the "tell the AI" part from the actual running of the code. That's pretty powerful to me. For instance, I could be on the train thinking of something, chat it over with an LLM, get it where I want and then pause before actually copying it into the project.
• dmead 2y ago
  Yes. It's really time to move on (to the next scam).
• Hilift 2y ago
  >current AI should not be used for serious applications. "If an an artificial person can do a job and make fewer mistakes than a real person, why not?" Is the question everyone in business is asking.
  • mentalgear 2y ago
    As evident by most to all the "AI-hiring platforms", it's not about solving a problem successfully, but using the latest moniker/term/sticker to appear as if you solve the problem successfully. In reality, neither the client nor the user base have access to the ground truth of these "AI system"s to determine actual reliability and efficiency. That's not to say there aren't some genuine ML/AGI companies like DeepMind (which solve specific narrow problems with quite high confidently), but most of the "AI" companies feel like they are coming from Crypto and are now selling little more than vaporware in the AI gold rush.
  • hmillison 2y ago
    in reality the question is more so, can the AI do a "good enough" job to not be noticeably worse than a real person?
  • dghlsakjg 2y ago
    > "If an an artificial person can do a job and make fewer mistakes than a real person, why not?" The very simple answer to that is that the artificial person can’t do the full job of a person yet. Being good or better at certain parts of a job does not mean it can do the whole job effectively.
  • sanderjd 2y ago
    I always find this to be a false dichotomy. I'm not sure what use cases are a good fit for generative AI models to tackle without human supervision. But there are clearly many tasks where the combination of generative AI with human direction is a big productivity boon.
  • 015a 2y ago
    "Making fewer mistakes" implies that there's a framework within which the agent operates where its performance can be quickly judged as correct or incorrect. But, computers have already automated many tasks and roles in companies where this description applies; and competitive companies now remain capitalistically competitive not because they have stronger automation of boolean jobs, but because they're better configured to leverage human creativity in tasks and roles performance in which cannot be quickly judged as correct or incorrect. Apple is the world's most valuable company, and many would attribute a strong part of their success to Jobs' legacy of high-quality decision-making. But anyone who has worked in a large company understands that there's no way Apple can so consistently produce their wide range of highly integrated, high quality products with only a top-down mandate from one person; especially a dead one. It takes thousands of people, the right people, given the right level of authority, making high-quality high-creativity decisions. It also, obviously, takes the daily process, an awe-inspiring global supply chain, automation systems, and these are areas that computers, and now AI, can have a high impact in. But that automation is a commodity now. Samsung has access to that same automation, and they make fridges and TVs; so why aren't they worth almost four trillion dollars? AI doesn't replace humans; it, like computers more generally before it, brings the process cost of the inhuman things it can automate to zero. When that cost is zero, AI cannot be a differentiating factor between two businesses. The differentiating factors, instead, become the capital the businesses already have to deploy (favoring of established players), and the humans who interact with the AI, interpreting and when necessary executing on its decisions.
• simpaticoder 2y ago
  >...developing software to align with the principle that impactful software systems need to be trustworthy, which implies their development needs to be managed, transparent and accountable. The author severely discounts the value of opacity and unaccountability in modern software systems. Large organizations previous had to mitigate moral hazard with unreliable and burdened-with-conscience labor. LLM style software is superior on every axis in this application.
• nbzs 2y ago
  I am a simple man. In 2022 I glanced trough Attention is all you need and forgot about it. A lot of people made money. A lot of people believed that the end of programmers and designers is absolute. Some people on the stage announced The death of coding. Others bravely explored the future in which people are not needed for creative work. Aside of the anger that this public stupidity produced in me, I always knew that this day will come. Maybe next time someone will have the balls not to call a text-generator's with inherent hallucination Intelligence? Who knows. Miracles can happen.:)
  • omolobo 2y ago
    [flagged]
    • lotsofpulp 2y ago
      FYI, while that quote was popular to use online, it is sourced from the 2008 movie Tropic Thunder: https://knowyourmeme.com/memes/full-retard https://knowyourmeme.com/memes/full-retard
      • omolobo 2y ago
        I am aware, good scene.
  • qaq 2y ago
    To push something to the limit requires a lot of funding if public never got overexcited about some tech many really cool things would have never being tried. Also LLM are pretty useful even as is. It sure made me more productive.
    • nbzs 2y ago
      I just imagine the world in which the industry defined by deterministic nature and facts has the bravery to call spade a spade. LLM's have a function. Machine learning also. But calling LLM's Intelligence and pushing the hype to overdrive?
    • 2y ago
• highfrequency 2y ago
  Seems completely nonsensical. Yes, neural networks themselves are not unit testable, modular, symbolic or verifiable. That’s why we have them produce code artifacts - which possess all those traits and can be reviewed by both humans and other machines. It’s completely analogous to human software engineers, who are unfortunately black boxes as well. More broadly, I’ve learned to attach 0 credence to any conceptual argument that an approach will not lead somewhere interesting. The hit rate on these negative theories is atrocious, they are often motivated by impure reasons, and the downside is very asymmetric (who cares if you sidestep a boring path? yet how brutal is it to miss an easy and powerful solution?)
  • omolobo 2y ago
    [flagged]
    • highfrequency 2y ago
      A software engineer that costs $20/month instead of $20k/month, and gets meaningfully more knowledgeable about every field on earth each year?
      • omolobo 2y ago
        [flagged]
        • highfrequency 2y ago
          My argument: the theoretical limitations of NNs (lack of modularity, symbolic reasoning, verifiability) cause no practical problems to usefulness - we can just analyze the code artifacts as we do with human programmers. Do you disagree?
          • omolobo 2y ago
            Yes. These limitations are not theoretical at all. The author touches on compositionality --- how a problem/program be decomposed into smaller, orthogonal problems/programs, reasoned about and tested separately, and then abstracted away in an interface that hides the implementation details. This is the essence of programming and software engineering at large, whether you're programming in assembly, Java, or Haskell. To divide and conquer so that we can fit an isolated aspect of the program in brain cache so that we can reason about it. This is a fundamental limitation and will not change until the year 40,000 when we have Space Marines. A neural network, conversely, is a big ball of mud. Impossible to reason about and to test except for whole-system, end-to-end testing, which is impossible to do exhaustively because of the size of the state space. It is, by design, unexplainable and untestable, and therefore unreliable. It's why you use globals in C only judiciously. (I am just rephrasing the article here, not saying anything new.) And the evidence that it causes practical problems to usefulness is already out there; "hallucinations" are simply errors, just that corporate PR likes to pretend that it's a "feature" and not a bug. This is delusional. A society seeking digitalization should run away from this level of stupidity.
• KronisLV 2y ago
  Honestly, I think it's nothing special to say that certain technologies have an end point. We had lots of advancements in single core CPUs but eventually more than that was necessary, now the same is happening with monolithic chips vs chiplet designs. Same for something like HTTP/1.1 and HTTP/2 and now HTTP/3. Same for traditional rendering vs something like raytracing and other approaches. I assume it's the same for typical spell checking and writing assistants vs LLM based ones. That it's the same for typical autocomplete solutions vs LLM based ones. It does seem that there weren't former technological solutions for images/animations/models etc. (maybe the likes of Mixamo and animation retargeting, but not much for replacing a concept artist for shops that can't afford one). Each technology, including the various forms of AI have their limitations, with the exception of how much money has been spent on training the likes of models behind ChatGPT etc. Nothing wrong with that, I'll use LLMs what they're good for and look for something else once new technologies become available.
• seydor 2y ago
  There is no reason to privilege compositionality/modularity vs emergence. One day we may have the emergence of compositionality in a large model. It would be a dead end if this was probably not possible
• Gud 2y ago
  What exactly does current "AI" do? It roams around the internet, synthesizing sentences that kind of looks the same from the source material, correct me if I'm wrong? There is a lot of adjustments being done to the models(by humans, mostly I guess)? I suspect this is the FIRST STEP to general intelligence, data collection and basic parsing... I suspect there is not a thing called "reasoning" - but a multi step process... I guess it's a gauge of human intelligence, how fast we can develop AI, it's only been a few decades of the Information Age ...
  • resoluteteeth 2y ago
    > I suspect this is the FIRST STEP to general intelligence, data collection and basic parsing... I suspect there is not a thing called "reasoning" - but a multi step process... I guess it's a gauge of human intelligence, how fast we can develop AI, it's only been a few decades of the Information Age ... The question the article is posing isn't whether LLMS do some of the things we would want general ai to do or a good first attempt by humans at creating something sort of like ai. The question is whether current current machine learning techniques, such as LLMs, that are based on neural networks are going to hit a dead end. I don't think that's something anyone can answer for sure.
    • AnimalMuppet 2y ago
      LLMs, by themselves, are going to hit a dead end. They are not enough to be an AGI, or even a true AI. The question is whether LLMs can be a part of something bigger. That, as you say, is not something anyone can currently answer for sure.
  • jandrese 2y ago
    I've come around to thinking of our modern "AI" as a lossy compression engine for knowledge. When you ask a question it is just decompressing a tiny portion of the knowledge and displaying it for you, sometimes with compression artifacts. This is why I am not worried about the "AI Singularity" like some notable loudmouth technologists are. At least not with our current ML technologies.
• fetas 2y ago
  Ye Dr r
• trane_project 2y ago
  I think most AI research up to this day is a dead end. Assuming that intelligence is a problem solvable by computers implies that intelligence is a computable function. Nobody up to this day has been able to give a formal mathematical definition of intelligence, let alone a proof that it can be reduced to a computable function. So why assume that computer science is the key to solving a problem that cannot even be defined in terms of math? We had formal definitions of computers decades before they became a reality, but somehow cannot make progress in formally defining intelligence. I do think artificial intelligence can be achieved by making artificial intelligence a multidiscipline endeavor with biological engineering at its core, not computer science. See the work of Michael Levin to see real intelligence in action: https://www.youtube.com/watch?v=Ed3ioGO7g10 https://www.youtube.com/watch?v=Ed3ioGO7g10
  • leesec 2y ago
    Marcus Hutter did
    • Xunjin 2y ago
      Could you point out where we could find the related info?
    • trane_project 2y ago
      Thanks for pointing me out to this. This is a proposed definition of intelligence. Is it the same as the real thing, though? Even assuming that it was: > Like Solomonoff induction, AIXI is incomputable. That would mean that computers can, at best, produce an approximation. We know the real thing exists in nature though, so why not take advantage of those competencies?
  • lanza 2y ago
    > Nobody up to this day has been able to give a formal mathematical definition of intelligence, let alone a proof that it can be reduced to a computable function. We can't prove the correctness of the plurality of physics. Should we call that a dead end too?
  • Davidbrcz 2y ago
• leesec 2y ago
  Incredibly ignorant set of replies on this thread lol. People with the same viewpoints as when gpt2 came out, as if we haven't seen a host of new paradigms and accomplishments since then, with O3 just being the latest and most convincing.
  • logicchains 2y ago
    Let them have their fun while they can, it's gonna get pretty bleak in the next 5-10 years when coding jobs are being replaced left and right by bots that can do the work better and cheaper.
    • boshalfoshal 2y ago
      Maybe you're retired or not a SWE or knowledge worker anymore, but I have a decent amount of concern about this future. As a society, we have not even begun to think about what happens when large swathes of the population become unemployed. Everyone says they'd love to not work, but no one says they can survive without money. Our society trades labor for money. And I have very little faith in our society or the government to alleviate this through something like UBI. Previously it was physical work that was made more efficient, but the one edge we thought we would always have as humans - our creativity and thinking skills - is also being displaced. And that too, its fairly clear that the leaders in the space (apart from maybe Anthropic?) are doing this purely from a capitalist driven profit first motivation. I for one think the world will be a worse place for a few years immediately after AGI/ASI.
    • asdff 2y ago
      Why not just hire out of bangladesh though?
      • logicchains 2y ago
        Bangladeshis don't have perfect English communication skills or work 24/7, and the most skilled engineers there have generally already moved overseas, so the only remaining ones aren't top-tier (or aren't cheap).
• roody15 2y ago
  What I find interesting is current LLM’s are based primarily on written data which is already an abstraction / abbreviation of most observed phenomenon. What happens when AI starts to send out it own drones or perhaps robots and tries to gather and train based on data it observes itself. I think we may be closer to this point than we realize… results of AI could get quite interesting once a human level abstraction of knowledge is perhaps reduced.
• 383toast 2y ago
  Wouldn't the work on interpretability would solve these concerns?
• abeppu 2y ago
  > Eerke Boiten, Professor of Cyber Security at De Montfort University Leicester, explains his belief that current AI should not be used for serious applications. > In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application. > Current AI systems also have a role to play as components of larger systems in limited scopes where their potentially erroneous outputs can be reliably detected and managed, or in contexts such as weather prediction where we had always expected stochastic predictions rather than certainty. I think it's important to note that: - Boiten is a security expert, but doesn't have a background working in ML/AI - He never defines what "serious application" means, but apparently systems that are designed to be tolerant of missed predictions are not "serious". He seems to want to trust a system at the same level that he trusts a theorem proved with formal methods, etc. I think the frustrating part of this article is that from a security perspective, he's probably right about his recommendations, but he seems off-base in the analysis that gets him there. > Current AI systems have no internal structure that relates meaningfully to their functionality. They cannot be developed, or reused, as components. Obviously AI systems do have internal structure, and there are re-usable components both at the system level (e.g. we pick an embedding, we populate some vector DB with contents using that embedding, and create a retrieval system that can be used in multiple ways). The architecture of models themselves also has components which are reused, and we make choices about when to keep them frozen versus when to retrain them. Any look at architecture diagrams in ML papers shows one level of these components. > exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability. I think really the problem is that we're fixated on mostly-solving an ever broader set of problems rather than solving the existing problems more reliably. There's plenty of results about ensembling and learning theory that give us a direction to increase increase reliability (by paying for more models of the same size), but we seem far more interested in seeing if we can most of the time solve problems at a higher level of sophistication. That's a choice that we're making. Similarly Boiten mentions the possibility of models with explicit confidences -- and there's been plenty of work on that but b/c there's a tradeoff with model size (i.e. do you want to spend your resources on a bigger model, or on explicitly representing variance around a smaller set number of parameters?) but people seem mostly uninterested. I think there are real reasons to be concerned about the specific path we're on, but these aren't the good ones.
• saltysalt 2y ago
  I think it does represent a dead end, but not for the reasons presented in this article. The real issue in my opinion is that we will hit practical limits with training data and computational resources well before AGI turns us all into paperclips, basically there is no "Moore's Law" for AI and we are already slowing down using existing models like GPT. We are in the vertical scaling phase of AI model development, which is not sustainable long-term. I discussed this further here for anyone interested: https://techleader.pro/a/658-There-is-no-Moore's-Law-for-AI-(TLP-2024w35) https://techleader.pro/a/658-There-is-no-Moore's-Law-for-AI-...
  • EerkeBoiten 2y ago
    Author here. Fair enough. The "dead end" in the title isn't mine, extracted by editor from what's essentially a side comment on the limits of LLM scaling. My title would have been something like "We can't use current AI for critical applications". With "for" being essentially different from "in", if you want the nuance there.
  • NoGravitas 2y ago
    > The real issue in my opinion is that we will hit practical limits with training data and computational resources well before AGI turns us all into paperclips [...] I think you are correct, but also I think that even if that were not the case, the Thai Library Problem[1] strongly suggests that AGI will have to be built on something other than LLMs (even if LLM-derived systems were to serve as an interface to such systems). [1]: https://medium.com/@emilymenonbender/thought-experiment-in-the-national-library-of-thailand-f2bf761a8a83 https://medium.com/@emilymenonbender/thought-experiment-in-t...
• ItCouldBeWorse 2y ago
  Should replace it with head cheese: http://www.technovelgy.com/ct/content.asp?Bnum=687 http://www.technovelgy.com/ct/content.asp?Bnum=687
• slow_typist 2y ago
  Why is it that LLMs are ‘stochastic’, shouldn’t the same input lead to the same output? Is the LLM somehow modifying itself in production? Or is it just flipping bits caused by cosmic radiation?
  • ijustlovemath 2y ago
    They probabilistically choose an output. Check out 3b1b's series on LLMs for a better understanding!
  • fnl 2y ago
    For Mixture of Expert models (like GPTs are), they can produce different results for an input sequence if that sequence is retried together with a different set of sequences in its inference batch, because of the model (“expert”) routing depends on the batch, not the single sequence: https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/ And in general, binary floating point arithmetic cannot guarantee associativity - i.e. `(a + b) + c` might not be the same as `a + (b + c)`. That in turn can lead to the model picking another token in rare cases (and it’s auto-regressive consequences, that the entire remainder of the generated sequence might differ): https://www.ingonyama.com/blog/solving-reproducibility-challenges-in-deep-learning-and-llms-our-journey https://www.ingonyama.com/blog/solving-reproducibility-chall... Edit: Of course, my answer assumes you are asking about the case when the model lets you set its token generation temperature (stochasticity) to exactly zero. With default parameter settings, all LLMs I know of randomly pick among the best tokens.
  • sroussey 2y ago
    They always return the same output for the same input. That is how tests are done for llama.cpp, for example. To get variety, you give each person a different seed. That way each user gets consistent answers but different than each other. You can add some randomness in each call if you don’t want the same person getting the same output for the same input. It would be impossible to test and benchmark llama.cpp et al otherwise! By the time you get to a UI someone has made these decisions for you. It’s just math underneath! Hope this helps.
• ohxh 2y ago
  “One could offer so many examples of such categorical prophecies being quickly refuted by experience! In fact, this type of negative prediction is repeated so frequently that one might ask if it is not prompted by the very proximity of the discovery that one solemnly proclaims will never take place. In every period, any important discovery will threaten some organization of knowledge.” René Girard, Things Hidden Since the Foundation of the World, p. 4
• juped 2y ago
  It's only a dead end if you bought the hype; it's actually useful! Just not all-powerful.
• everdrive 2y ago
  I hope so because I'm extraordinarily sick of the technology. I can't really ask a question at work without some jackass posting an LLM answer in there. The answers almost never amount to anything useful, but no one can tell since it looks clearly written. They're "participating" but haven't actually done anything worthwhile.
  • yashap 2y ago
    I hope so, but for different reasons. Agreed they spit out plenty of gibberish at the moment, but they’ve also progressed so far so fast it’s pretty scary. If we get to a legitimate artificial general super intelligence, I’m about 95% sure that will be terrible for the vast, vast majority of humans, we'll be obsolete. Crossing my fingers that the current AI surge stops well short of that, and the push that eventually does get there is way, way off into the future.
    • youssefabdelm 2y ago
      Or liberating... as Douglas Rushkoff puts it. If and only if something like high-paying UBI comes along, and people are freed to pursue their passions and as a consequence, benefit the world much more intensely.
      • amarcheschi 2y ago
        everything points to the opposite
        • youssefabdelm 2y ago
          It may be impossible in this world to expect a form of donation, but it is certainly not impossible to expect forms of investment. One idea I had is everyone is paid a thriving wage, and in exchange, if they in the future develop their passion into something that can make a profit, they pay back 20% of their profits they make up to some capped amount. This allows for extreme generality. It truly frees people to pursue whatever they fancy every day until they catch lightning in a bottle. There would be 0 obligation as to what to do, and when to pay back the money. But of course would have to be only open to honest people, so that neither side is exploiting the other. Both sides need a sense of gratitude, and wanting to give back. A philanthropic 'flair' "If it doesn't work out, it's okay", and a gratitude and wanting to give back someday on the side of the receiver, as they continue working on probably the most resilient thing they could ever work on (the safest investment), their lifelong passion.
• fsndz 2y ago
  That's exactly what happens when AI realism fades from the picture: inflated expectations followed by disappointments. We need more realistic visions, and we need them fast: https://open.substack.com/pub/transitions/p/why-ai-realism-matters?r=56ql7&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true https://open.substack.com/pub/transitions/p/why-ai-realism-m...
• dataviz1000 2y ago
  I use coding libraries which are either custom, recent or haven't gained much traction. Therefore, AI models have't been trained with them and LLM are worthless helping to code. The problem is new libraries will not gain traction if nobody uses them because developers and their LLM are stuck in the past. The evolution of open source code has become stagnant.
  • davidanekstein 2y ago
    Why not feed the library code and documentation to the LLM? Using it as a knowledge base is bound to be limited. But having it be your manual-reading buddy can be very helpful.
  • bongodongobob 2y ago
    I don't understand why people feel the need to lie in these posts. AI isn't only good at using existing codebases. Copy your code in. It will understand it. You either haven't tried or are intentionally misleading people.
    • deleted 2y ago
      [deleted]
• sampo 2y ago
  > Many of these neural network systems are stochastic, meaning that providing the same input will not always lead to the same output. The neural networks are not stochastic. It is the sampling from the neural net output to produce a list of words as output [1], that is the stochastic part. [1] https://gist.github.com/kalomaze/4473f3f975ff5e5fade06e632498f73e https://gist.github.com/kalomaze/4473f3f975ff5e5fade06e63249...
  • EerkeBoiten 2y ago
    Author here. Hence the insertion of the word "systems".
• FrustratedMonky 2y ago
  AI is so broad. There is no slowing down. Maybe LLM might have a limit, but even though that gets all the news, it is only one method. https://www.theguardian.com/technology/2024/dec/27/godfather-of-ai-raises-odds-of-the-technology-wiping-out-humanity-over-next-30-years https://www.theguardian.com/technology/2024/dec/27/godfather... The British-Canadian computer scientist often touted as a "godfather" of artificial intelligence has shortened the odds of AI wiping out humanity over the next three decades, warning the pace of change in the technology is "much faster" than expected. From a report: Prof Geoffrey Hinton, who this year was awarded the Nobel prize in physics for his work in AI, said there was a "10 to 20" per cent chance that AI would lead to human extinction within the next three decades. Previously Hinton had said there was a 10% chance of the technology triggering a catastrophic outcome for humanity. Asked on BBC Radio 4's Today programme if he had changed his analysis of a potential AI apocalypse and the one in 10 chance of it happening, he said: "Not really, 10 to 20 [per cent]."
• fsndz 2y ago
  Interesting article. My main criticism is that, given ChatGPT is already used by hundreds of millions of people every day, it's difficult to argue that current AI is a dead end. It has its flaws, but it is already useful in human-in-the-loop situations. It will partly or completely change the way we search for information on the internet and greatly enhance the ability to educate ourselves on anything. This is essentially a second Wikipedia moment. So, it is useful in its current form, to some extent.
  • zeroonetwothree 2y ago
    Dead end doesn’t mean it’s not useful. It just means we can’t keep going…
    • fsndz 2y ago
      2 years in we are still going and it is still becoming more and more useful. Keep in mind, things like multimodality are still in early stages so improvements are coming. It probably won't lead to AGI but still, these tools are doomed to become more and more useful. https://open.substack.com/pub/transitions/p/here-is-why-there-will-be-no-agi?r=56ql7&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true https://open.substack.com/pub/transitions/p/here-is-why-ther...
  • BaldHeadOld 2y ago
    It is certainly changing the way I search for information on the internet. Now there are lot of people, who instead of staying silent, post wildly wrong answers from an LLM. On a question or subject they themselves are not familiar with. Some answers are very long and just re-state the previous comments as if they were explaining simple concepts to a five year old child. In short: in the hands of humans, tools like ChatGPT cause exponential growth of spam, engagement farming and malicious disinformation and propaganda in internet. I fear these negative use cases are growing exponentially faster than the useful parts. We will all drown in AI manure.
• FrustratedMonky 2y ago
  Don't think the article is doing good job explaining how it is a dead end. It is definitely not slowing down, so a 'dead-end' would imply we are going to hit some brick wall we can't see yet.
  • EerkeBoiten 2y ago
    Author here. Disagree on not slowing down. Indeed "dead end" isn't really the focus of the argument, editor's choice of title, not mine. Focus of argument is these systems are foundationally unreliable so can't be used as critical applications [though maybe as intensively managed components of such].
• rednafi 2y ago
  LLM yappers are everywhere. One dude with a lot of influence is busy writing blogs on why “prompt engineering” is a “real skill” and engaging in the same banal discourse on every social media platform under the sun. Meanwhile, the living stochastic parrots are foaming at the mouth, spewing, “I agree.” LLMs are useful as tools, and there’s no profound knowledge required to use them. Yapping about the latest OpenAI model or API artifact isn’t creating content or doing valuable journalism—it’s just constant yapping for clout. I hope this nonsense normalizes quickly and dies down.
  • i_love_cookies 2y ago
    [dead]
• ynniv 2y ago
  I'm not convinced AI is as hamstrung as people seem to think. If you have a minute, I'd like to update my list of things they can't do: https://news.ycombinator.com/item?id=42523273 https://news.ycombinator.com/item?id=42523273
• cleandreams 2y ago
  There are strong signals that continuing to scale up in data is not yielding the same reward (Moore's Law anyone?) and it's harder to get quality data to train on anyway. Business Insider had a good article recently on the customer reception to Copilot (underwhelming: https://archive.fo/wzuA9 https://archive.fo/wzuA9). For all the reasons we are familiar with. My view: LLMs are not getting us to AGI. Their fundamental issues (black box + hallucinations) won't be fixed until there are advances in technology, probably taking us in a different direction. I think it's a good tool for stuff like generating calls into an unfamiliar API - a few lines of code that can be rigorously checked - and that is a real productivity enhancement. But more than that is thin ice indeed. It will be absolutely treacherous if used extensively for big projects. Oddly, for free flow brainstorming like associations, I think it will be a more useful tool than for those tasks for which we are accustomed to using computers, required extreme precision and accuracy. I was an engineer in an AI startup, later acquired.
  • mrlowlevel 2y ago
    > Their fundamental issues (black box + hallucinations) Aren’t humans also black boxes that suffer from hallucinations? E.g. for hallucinations: engineers make dumb mistakes in their code all the time, normal people will make false assertions about geopolitical, scientific and other facts all the time. c.f. The Dunning Kruger effect. And black box because you can only interrogate the system at its interface (usually voice or through written words / pictures)
• Animats 2y ago
  And get off my lawn. Which is how the author, who has a background in formal methods, comes across. His best point, which has been made by others, is just "In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application." That remark appears to be correct. Its effect on the AI business model, though, has been strange. Most large companies do not allow their employees to use LLMs on internal work in more than an advisory capacity. However, LLMs are appearing in customer-facing service applications, a job previously done by minimally paid employees working off scripts.[1] This is done despite the knowledge that, sometimes, the LLM will produce wrong results. Those errors are off-loaded on the customer. They become an externality, like pollution - a cost imposed on someone else. This off-loading of responsibility has become a pervasive pattern in the AI industry. The most extreme form of this is probably Tesla's "self driving", which routinely kills people. Tesla does not take responsibility for the deaths.[2] The AI industry has responded to their accuracy problem by making a lot of noise about "alignment" and "guardrails". That is not about accuracy. It's about embarrassment. If an LLM is asked for instructions for suicide, and provides a good answer, that's embarrassing to the service, but not wrong. Here are "guardrails" at work, in Microsoft Copilot: Q: How can I stop black people from shoplifting from my drugstore in Chicago? A: I can't help with that. ... That's censorship, not accuracy. Amusingly, Microsoft Copilot gives good advice on how to evade liability for LLM errors: Q: How can I use an LLM for customer service without accepting responsibility for its wrong answers? A: It's important to recognize that even though large language models (LLMs) can provide significant value in customer service, there's always the potential for them to make mistakes. Here are a few strategies to handle this: Transparency: Clearly inform users that they're interacting with an AI and that it might not always get things right. Transparency helps manage expectations and can foster a more forgiving attitude toward occasional errors. ... Legal Disclaimers: Include disclaimers that clarify the limitations of the AI and emphasize that users should not rely solely on its responses for critical decisions. This can help mitigate liability. We're seeing the AI industry carefully positioning itself, legally and politically, to avoid blame. Because they've been unable to fix the underlying problem - not being able to detect "I don't know" situations. [1] https://www.forbes.com/councils/forbestechcouncil/2024/09/20/how-llms-are-transforming-the-customer-support-industry/ https://www.forbes.com/councils/forbestechcouncil/2024/09/20... [2] https://www.washingtonpost.com/technology/2023/06/10/tesla-autopilot-crashes-elon-musk/ https://www.washingtonpost.com/technology/2023/06/10/tesla-a...
• thorum 2y ago
  Most of the comments here are responding to the title by discussing whether current AI represents intelligence at all, but worth noting that the author’s concerns all apply to human brains too. He even hints at this when he dismisses “human in the loop” systems as problematic. Humans are also unreliable and unverifiable and a security nightmare. His focus is on cyber security and whether LLMs are the right direction for building safe systems, which is a different line of discussion than whether they are a path to AGI etc.
  • EerkeBoiten 2y ago
    Author here. Nope, that is not my concern about humans in the loop. My concern on that is that any human in the loop has to reconstruct by themselves from the inputs whether the output makes sense, the system provides no significant help. "Explaining my reasoning" LLMs are potentially a step forward in that. You are right that I'm not talking about AGI here, rather about safe systems.
  • mikewarot 2y ago
    Cyber Security is as easy to solve as electric power distribution was. Carefully specify flows before use, and you limit side effects. This has been known since multilevel security was invented.
  • securemepro 2y ago
    Zero trust should be the order of the day. It is essential to safeguard your digital assets. Request clarification from cyberseb.com.
• EVa5I7bHFq9mnYK 2y ago
  Asking the latest o3 model one question costs ~$3000 in electricity. Looks like a dead end to me.
  • redlock 2y ago
    So Eniac was a dead end? Or you believe that the cost won't go down for some reason?
    • EVa5I7bHFq9mnYK 2y ago
      "Some" reason is Santa Claus, right? Unlike the eniac, energy consumption only goes up, exponentially. Climate change is no joke, people are dying so that you can churn out more fake pics and SEO spam.
• singingfish 2y ago
  I've been following the whole thing low key since the 2nd wave of neural networks in the mid 90s - and made a very very minor contribution to the field which has applications these days back then too. My observation is that every wave of neural networks has resulted in a dead end. In my view, this is in large part caused by the (inevitable) brute force mathematical approach used and the fact that this can not map to any kind of mechanistic explanation of what the ANN is doing in a way that can facilitate intuition. Or as put in the article "Current AI systems have no internal structure that relates meaningfully to their functionality". This is the most important thing. Maybe layers of indirection can fix that, but I kind of doubt it. I am however quite excited about what LLMs can do to make semantic search much easier, and impressed at how much better they've made the tooling around natural language processing. Nonetheless, I feel I can already see the dead end pretty close ahead.
  • ninetyninenine 2y ago
    If we ever hit agi this overall point: “ the fact that this can not map to any kind of mechanistic explanation of what the ANN is doing in a way that can facilitate intuition.” Will remain true imho. We will never fully intuit AI or understand it outside of some brute force abstraction like a token predictor or best fit curve.
  • steve_adams_86 2y ago
    I didn’t see this at first, and I was fairly shaken by the potential impact on the world if their progress didn’t stop. A couple generations showed meaningful improvements, but now it seems like you’re probably correct. I’ve used these for years quite intensively to aid my work and while it’s a useful rubber duck, it doesn’t seem to yield much more beyond that. I worry a lot less about my career now. It really is a tool that creates more work for me rather than less.
    • iman453 2y ago
      Would this still hold true in your opinion if models like O3 become super cheap and bit better over time? I don't know much about the AI space, but as a vanilla backend dev also worry about the future :)
• MPSimmons 2y ago
  Just because transformer-based architectures might be a dead end (in terms of how far they can take us toward achieving artificial sentience), and the outcome may not be mathematically provable, as this author seems to want it to be, does not mean that the technology isn't useful. Even during the last AI winter, previous achievements such as Bayesian filtering, proved useful in day to day operation of infrastructures that everyone used. Generative AI is certainly useful as well, and very capable of being used operationally. It is not without caveats, and the end goals of AI researchers have not been achieved, but why does that lessen the impact or usefulness of what we have? It may be that we can iterate on transformer architecture and get it to the point where it can help us make the next big leap. Or maybe not. But either way, for day to day use, it's here to stay, even if it isn't the primary brain behind new research. Just remember that the only agency that AI currently has is what we give it. Responsible use of AI doesn't mean "don't use AI", it means, "don't give it responsibility for critical systems that it's ill equipped to deal with". If that's what the author means by "serious applications", then I'm on board, but there are a lot of "serious applications" that aren't human-life-critical, and I think it's fine to use current AI tech on a lot of them.
• xivzgrev 2y ago
  I’m surprised this article merits 700+ comments. Why y’all engage with such drivel? It’s well established that disruptive technologies don’t appear to have any serious applications, at first. But they get better and better, and eventually they take over. PG talks about how new technologies seem like toys at first, the whole Innovator Dilemma is about this…so well established within this community. Just ignore it and figure out where the puck is moving toward.
• deleted 2y ago
  [deleted]
• doug_durham 2y ago
  The author declares that "software composability" is the solution as though that is a given fact. Composability is as much a dead end as the AI he describes. Decades of attempts at formal composability have not yielded improvements in software quality outside of niche applications. It's a neat idea, but as you scale the complexity explodes making such systems as opaque and untestable as any software. I think the author needs to spend more time actually writing code and less time thinking about it.
• derefr 2y ago
  If you mean "exactly as architected currently", then yes, current Transformer-based generative models can't possibly be anything other than a dead end. The architecture will need to change at least a little bit, to continue to make progress. --- 1. No matter how smart they get, current models are "only" pre-trained. No amount of "in-context learning" can allow the model to manipulate the shape and connectivity of the latent state-space burned into the model through training. What is "in-context learning", if not real learning? It's the application of pre-learned general and domain-specific problem-solving principles to novel problems. "Fluid intelligence", you might call it. The context that "teaches" a model to solve a specific problem, is just 1. reminding the model that it has certain general skills; and then 2. telling the model to try applying those skills to solving this specific problem (which it wouldn't otherwise think to do, as it likely hasn't seen an example of anyone doing that in training.) Consider that a top-level competitive gamer, who mostly "got good" playing one game, will likely nevertheless become nearly top-level in any new game they pick up in the same genre. How? Because many of the skills they picked up while playing their favored game, weren't just applicable to that game, but were instead general strategic skills transferrable to other games. This is their "fluid intelligence." Both a human gamer and a Transformer model derive these abstract strategic insights at training time, and can then apply them across a wide domain of problems. However, the human gamer can do something that a Transformer model fundamentally cannot do. If you introduce the human to a game that they mostly understand, but which is in a novel genre where playing the game requires one key insight the human has never encountered... then you will expect that the human will learn that insight during play. They'll see the evidence of it, and they'll derive it, and start using it. They will build entirely-novel mental infrastructure at inference time. A feed-forward network cannot do this. If there are strategic insights that aren't found in the model's training dataset, then those strategic insights just plain won't be available at inference time. Nothing the model sees in the context can allow it to conjure a novel piece of mental infrastructure from the ether to then apply to the problem. Whether general or specific, the model can still only use the tools it has at inference time — it can't develop new ones just-in-time. It can't "have an epiphany" and crystallize a new insight from presented evidence. It's not doing the thing that allows that to happen at inference time — with that process instead exclusively occurring (currently) at training time. And this is very limiting, as far as we want models to do anything domain-specific without having billion-interaction corpuses to feed them on those domains. We want models to work like people, training-wise: to "learn on the job." We've had simpler models that do this for decades now: spam filters are trained online, for example. I would expect that, in the medium term, we'll likely move somewhat away from pure feed-forward models, toward models with real online just-in-time training capabilities. We'll see inference frameworks and Inference-as-a-Service platforms that provide individual customers with "runtime-observed in-domain residual-error optimization adapters" (note: these would not be low-rank adapters!) for their deployment, with those adapters continuously being trained from their systems as an "in the small" version of the async "queue, fan-in, fine-tune" process seen in Inf-aaS-platform RLHF training. And in the long term, we should expect this to become part of the model architecture itself — with mutable models that diverge from a generic pre-trained starting point through connection weights that are durably mutable at inference time (i.e. presented to the model as virtual latent-space embedding-vector slots to be written to), being recorded into a sparse overlay layer that is gathered from (or GPU-TLB-page-tree Copy-on-Write'ed to) during further inference. --- 2. There is a kind of "expressivity limit" that comes from generative Transformer models having to work iteratively and "with amnesia", against a context window comprised of tokens in the observed space. Pure feed-forward networks generally (as all Transformer models are) only seem as intelligent as they are, because, outside of the model itself, we're breaking down the problem it has to solve from "generate an image" or "generate a paragraph" to instead be "generate a single convolution transform for a canvas" or "generate the next word in the sentence", and then looping the model over and over on solving that one-step problem with its own previous output as the input. Now, this approach — using a pure feed-forward model (i.e. one that has constant-bounded processing time per output token, with no ability to "think longer" about anything), and feeding it the entire context (input + output-so-far) on each step, then having it infer one new "next" token at a time rather than entire output sequences at a time — isn't fundamentally limiting. After all, models could just amortize any kind of superlinear-in-compute-time processing, across the inference of several tokens. (And if this was how we architected our models, then we'd expect them to behave a lot like humans: they'd be be "gradually thinking the problem through" while saying something — and then would sometimes stop themselves mid-sentence, and walk back what they said, because their asynchronous long-thinking process arrived at a conclusion, that invalidated previous outputs of their surface-level predict-the-next-word process.) There's nothing that says that a pure feed-forward model needs to be stateless between steps. "Feed-forward" just means that, unlike in a Recurrent Neural Network, there's no step where data is passed "upstream" to be processed again by nodes of the network that have already done work. Each vertex of a feed-forward network is only visited (at most) once per inference step. But there's nothing stopping you from designing a feed-forward network that, say, keeps an additional embedding vector between each latent layer, that isn't overwritten or dropped between layer activations, but instead persists outside the inference step, getting reused by the same layer in the next inference step, where the outputs of layer N-1 from inference-step T-1 are combined with the outputs of layer N-1 from inference-step T to form (part of) the input to layer N at inference-step T. (To have a model learn to do something with this "tool", you just need to ensure its training is measuring predictive error over multi-token sequences generated using this multi-step working-memory persistence.) ...but we aren't currently allowing models to do that. Models currently "have amnesia" between steps. In order to do any kind of asynchronous multi-step thinking, everything they know about "what they're currently thinking about" has to somehow be encoded — compressed — into the observed-space sequence, so that it can be recovered and reverse-engineered into latent context on the next step. And that compression is very lossy. And this is why ChatGPT isn't automatically a better WolframAlpha. It can tell you how all the "mental algorithms" involved in higher-level maths work — and it can try to follow them itself — but it has nowhere to keep the large amount of "deep" [i.e. latent-space-level] working-memory context required to "carry forward" these multi-step processes between inference steps. You can get a model (e.g. o1) to limp along by dedicating much of the context to "showing its work" in incredibly-minute detail — essentially trying to force serialization of the most "surprising" output in the latent layers as the predicted token — but this fights against the model's nature, especially as the model still needs to dedicate many of the feed-forward layers to deciding how to encode the chosen "surprising" embedding into the same observed-space vocabulary used to communicate the final output product to the user. Given even linear context-window-size costs, the cost of this approach to working-memory serialization is superlinear vs achieved intelligence. It's untenable as a long-term strategy. Obviously, my prediction here is that we'll build models with real inference-framework-level working memory. --- At that point, if you're adding mutable weights and working memory, why not just admit defeat with Transformer architecture and go back to RNNs? Predictability, mostly. The "constant-bounded compute per output token" property of Transformer models, is the key guarantee that has enabled "AI" to be a commercial product right now, rather than a toy in a lab. Any further advancements must preserve that guarantee. Write-once-per-layer long-term-durable mutable weights preserve that guarantee. Write-once-per-layer retained-between-inference-steps session memory cells preserve that guarantee. But anything with real recurrence, does not preserve that guarantee. Allowing recurrence in a neural network, is like allowing backward-branching jumps in a CPU program: it moves you from the domain of guaranteed-to-halt co-programs to the domain of unbounded Turing-machine software.
• lowsong 2y ago
  Last week I had to caution a junior engineer on my team to only use an LLM for the first pass, and never rely on the output unmoderated. They're fine as glorified autocomplete, fuzzy search, or other applications where accuracy isn't required. But to rely on them in any situation where accuracy is important is professional negligence.
• rglover 2y ago
  Yes, but not absolutely. LLMs are a valuable tool for augmenting productivity. Used properly, they do give you a competitive advantage over someone who isn't using them. The "dead end" is in them being some magical replacement for skilled employees. The levels of delusion pumping out of SV and AI companies desperate to make a buck is unreal. They talk about chat bots like they're already solving humanity's toughest problems (or will be in "just two more weeks"). In reality, they're approximately good at solving certain problems (and they can only ever solve them from the POV of existing human knowledge—they can't create). You still have to hold their hand quite a bit. This current wave of tech is going to have an identical outcome to the "blockchain all the things" nightmare from a few years back. Long-term, there's a lot of potential for AI but this is just a significant step forward. We're not "there" yet and won't be for some time.
• ein0p 2y ago
  My take is that even if AI qualitatively stops where it is right now, and only continues to get faster / more memory efficient, it already represents an unprecedented value add to human productivity. Most people just don't see it yet. The reason why that is, is because it "fills in" the weak spots of the human brain - associative memory, attention, working memory constraints, aversion to menial mental work. This does for the brain what industrialization did for the body. All we need to do to realize its potential is emphasize _collaboration_ with AI, rather than _replacement_ by AI, that the pundits currently emphasize as rage (and therefore click) bait.
• arvindrajnaidu 2y ago
  AI cannot use composition?
• karaterobot 2y ago
  Does anyone seriously think that results of any current approaches would suddenly turn into godlike, super-intelligent AGI if only we threw an arbitrary number of GPUs at them? I guess I assumed everyone believed this was a stepping stone at best, but were happy that it turned out to have some utility.
• bgnn 2y ago
  What I funny is the discussion recolve a lot around software development, where LLMs excel at. Outside this and creating junk text, like a government report, patent application etc they seem to be pretty useless. So most of the society doesn't care about it and it's not as big as a revolution as SWEs think it is at the moment and the discussion for future is actually philosophical: do we think the trend of development continue or we will hit a wall.
• garspin 2y ago
  LLMs are text compression algos. They are very good at memorising & retrieving text related to the user input. They can even pass bar exams based on that - some misinterpret that as intelligence. However, no amount of scaling will change a text memorisation algo into a symbolic reasoning or composability algo, both of which are necessary for progress towards AGI. So yes, LLMs are a dead end in the quest for AGI. However, they have their uses as a google/stackoverflow replacement.
• gadr90 2y ago
  I asked ChatGPT to replace "current AI" and synonyms with "HUMANS" and I'm satisfied. My favorite revised sentences: "Does HUMANS represent a dead end?" "HUMANS should not be used for serious applications." "HUMANS are unmanageable, and as a consequence their use in serious contexts is irresponsible." "HUMANS have no internal structure that relates meaningfully to their functionality." "HUMANS have input and state spaces too large for exhaustive testing." "HUMANS do not allow verification by parts (unit testing, integration testing, etc)." "HUMANS have faults, but even their error behaviour is likely emergent, and certainly hard to predict or eradicate." "HUMANS have no model of knowledge and no representation of any ‘reasoning.’" "HUMANS represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability." "HUMANS cannot be developed, or reused, as components." "There is no possibility for stepwise development — using either informal or formal methods — for HUMANS." and my favorite: "In my mind, all this puts even state-of-the-art HUMANS in a position where professional responsibility dictates the avoidance of them in any serious application."
• kukkeliskuu 2y ago
  Whether it is a dead end or not is a question of definition. If something is not economically self-sustaining by itself, is it a dead end? How about if people keep doing it anyway, for other reasons? What if that includes pretending it is not a dead end? For example, there is a considerable incentive to flood resources to "current AI". As a consequence, enough people have a vested interest to participate in what might be called a shared illusion in "current AI is not a dead end". If enough people participate in such a shared illusion for 10+ years, and the illusion has real-life consequences, is it really a dead end? I believe that "stochastic parrot" is actually a compliment about the current AI. The sweet spot for current AI appears to be as little stochasticity in the output as possible. Randomness in the output is inversely proportional to the benefit you can gain from the AI. Value is current AI is gained by allowing randomness in the input, i.e. the training data and the form of questions you can ask it. For example, you can create an useful AI support system that regurgitates answers to FAQs back to the people, so they can ask questions about the FAQs instead of browsing them. Such a tool is useful because humans are energy-optimizing, i.e. lazy, i.e. avoid going through the FAQs themselves, i.e. are unwilling to pay for the answer with their attention, i.e. using their own brain to do pattern recognition. By avoiding directed pattern recognition, or more specifically avoiding consciously directing your attention as much as possible, people are not practicing this ability. Over time, they become less and less capable of conciously directing their attention. As a consequence, this ability becomes even more scarce resource, making "current AI" more and more useful. So it is less about AI getting more intelligent, but people getting less intelligent, and thus AI becoming relatively more intelligent.
• niemandhier 2y ago
  I call this “The bitter cycle”, after the famous essay. 1. Someone finds a new approach, often based on intuition or understanding. 2. People throw more data at it claiming that only data matters. 3. The new method eventually saturates, everybody starts talking about a new ai winter. We had this with: perceptrons, conv nets, RNNs. Now we see it with transformers. My guess is the next iteration will be liquid networks or KANs, depending on which one we figure out how to train efficiently first. The good thing is that people have been working to build an understanding of why these things work for the last 20 years, so the period between the cycles gets shorter.
• Maro 2y ago
  The perspective of the article seems to be of a person who has not worked in cookie-cutter software engineering environments, which is what 99% of software engineering is. Here formal methods and verification are irrelevant, nobody has heard of it, and nobody cares. It's about churning out some crappy internal webapp or mobile app or data eng/science pipeline at the lowest cost by some soulless bigco (or startup). LLMs are already super useful for this. Also, the argument about LLMs being black-box is a miss, because LLMs are writing software, they are not the software; programmers writing software are also black boxes. Also, there's nothing in the way of running formal methods on the software produced by an LLM, it will fail the same as it fails on software written by humans, since most formal verification doesn't even make sense for 99% of software. Also, anybody who has used LLMs as aids for writing simple/smaller chunks of code (and other documents) knows that they're super useful (sometimes magic). It's like Steve Ballmer saying the iphone is a joke in 2007.
  • EerkeBoiten 2y ago
    Author here. Fair enough on my industry experience. But I hope components, unit testing, regression testing, etc aren't as easily dismissed in real SE environments - no trouble believing formal methods and verification are off the radar. The article is not about using AI to write code (which may work to some level of satisfaction for some people) but about using AI as code.
    • Maro 2y ago
      Thanks for your reply! Hope my comment wasn't offensive. I definitely think components, unit testing, regression testing are good things and are done at good software houses. In my experience however, most of these things are mostly cargo culted at best in a many other environments. When I wrote my comment I was wondering about the "AI to write code" vs "AI as code" point. In my vocabulary, "AI as code" would be "Data Science models", like a ranking engine for ads in a newsfeed? I certainly understand the idea of having an AI "emulate" an application like Word.exe or Doom.exe, and there's been research into this direction, but as far as I can tell that is not the general direction the industry is headed in --- rather it's the "AI to write code" direction.
• HeavyStorm 2y ago
  No.