5 ms·

But OpenAI said LLMs can't innovate until human-level reasoning and long-term agenthood is solved. [1] Referring to their precious 5 stages to classify AI befor

But OpenAI said LLMs can't innovate until human-level reasoning and long-term agenthood is solved. [1] Referring to their precious 5 stages to classify AI before it reaches the scary "beyond" levels of intelligence... presumably at that point they get the feds involved to reg cap the field, so genuine is the fear of the pace they've set. It's clear OpenAI is a hype company knocking over glass bottle stacks at its own wonderful carnival stall. Obviously if you can reason you can reason about what is innovative and we don't need OpenAI to set up fake scary progress markers like an Automated Scientific Organization. Lets see if scientists even want this style of tech progress, it'd be sad to see each multitudes of AI papers having to be rebuilt from scratch and flushed down the toilet because associating with it is taboo. [1]: https://arstechnica.com/information-technology/2024/07/openai-reportedly-nears-breakthrough-with-reasoning-ai-reveals-progress-framework/ https://arstechnica.com/information-technology/2024/07/opena...

uniqueuid 2y ago
It would also be sad to see the scientific system destroyed by a wave of automatically generated papers that no human has the capacity to verify. It's not hard to generate ideas, it's hard to generate reliable and relevant ideas. Such AI science generators are destroying the grass they graze on unless they take science more seriously (and not as a toddler idea of "generating and testing ideas", which is only a small part of the story).
- foolfoolz 2y ago
  we already have a wave of papers that no human has the capacity to verify
  - uniqueuid 2y ago
    Maybe, maybe not. It's a tiered system - you get the deluge at the unfiltered bottom and a narrower selection the more prestigious and selective the outlets / conferences / journals are. Problem is, of course, that selection criteria are in large parts proxies, not measures of quality. With AI, those proxies become tainted and then you get an explosion of effort. If anyone has a good recommendation for scalable criteria to assess the quality of papers (beyond fame haha) I'm all ears.
    - staunton 2y ago
      If we had scalable criteria for quality of papers, that would be the end of science. The rest would be engineering...
    - PROMISE_237 2y ago
      [dead]
  - EGreg 2y ago
    Literally every AI discussion on HN has the same format > AI is awesome, regulation is stupid > I wouldnt want to see AI flooding the market with X we can’t verify > We already have (copy whatever was just said) For what it’s worth, when it comes to SCIENCE, I an actually in favor of AI, even giving it to everyone. Except possibly AI that would help engineer designer viruses. Because in science, people literally ARE just doing pattern matching and testing the patterns. Kepler just looked at a bunch of star charts and found a low-dimensional explanation with ellipses. You can throw that data at an AI and it will come up with physics laws involving 24,729 variables that predict things way more accurately. Including potentially chaotic systems like weather or a three body problem etc. So yeah, use AI for that, because you can actually check its predictions against data, and develop a reputation. We can’t really reason about theories and models of systems with 30,000 variables anyway. If AI churns out 25 scientific models per day, the proper venue isnt publishing them on arxiv or Nature magazine. It’s testing their predictions and putting them up the same way HuggingFace does. To amass reputation and be checked against real data by multiple parties. As a side question — is protein folding @ home dead now, since Google “solved it” with massive clusters and AI? What about SETI@home? Why not just run AI for SETI and solve the drake equation once and for all? :)
    - EvgeniyZh 2y ago
      > If AI churns out 25 scientific models per day, the proper venue isnt publishing them on arxiv or Nature magazine. It’s testing their predictions and putting them up the same way HuggingFace does. Unless the testing will be done by AI, I doubt human scientist will bother with testing tons of incomprehensible models, unless the accuracy of these models will be exceptionally high
      - jalman 2y ago
        The value of AI-mated research is results. This tool will aid engineering. It will offer a pinpointed research to resolve a particular issue at hand. It will close the verification loop naturally by proving that the research has indeed facilitated a useful result. Most of such useful research will never be publicly available. What you are complaining about is a legitimately brocken verification loop. Let's consider a case. An engineering department has a problem. It passes the problem to the research department. The research department slacks out, while produces junk. It makes tons of papers that are hard to prove, not to say apply. They drink champagne with other researchers, so they can publish the junk and defend the turf. AI-mated research will finish the racket and corruption.
  - mhuffman 2y ago
    I have also, unless I hallucinated it, read accusations on this very site of peer reviewed papers that were at least partly generated by LLMs.
ineedaj0b 2y ago
sam made comments on Twitter we hit level 2 - we’ll know more in the coming weeks if he’s right.
- nprateem 2y ago
  Highly unlikely since today's models can't consistently follow simple instructions. Eg "don't waffle, don't sound like you're writing an essay, don't use the fucking word delve, don't apologise" "Sorry about that, let's delve into this"
ben_w 2y ago
> It's clear OpenAI is a hype company Every other industry: "My new invention is safe, I swear" Public reaction: "You're biased, it's dangerous!" Almost the entire AI industry, including people who resign to speak more openly about the risks: "This may kill literally everyone, none of us knows what we're doing or what 'safe' even means" Public reaction: "You're biased, it's safe, just saying that to look cool!"
- sampo 2y ago
  20 to 30 years ago, activists firebombed university research labs (e.g. Michigan State University, University of Washington, Michigan Technological University [1]) because they believed genetically engineered plants are dangerous. Today, we don't have such serious activism against AI. So you are right, the public doesn't think AI is a danger. [1] https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable_attacks:_1998%E2%80%932009 https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable...
  - eddiewithzato 2y ago
    reminds me, I would’ve rather seen VCs fund more genetic engineering startups. Imagine the good it could do, from stem cells to nanobots to “hacking” human DNA itself. But I know the business model there can’t compete with software. So it will never reach the funding it needs.
- VieEnCode 2y ago
  I found this guy's take on the AI safety scene to be quite insightful. In summary, he feels the focus on sci-fi type existential risk to be a deliberate distraction from the AI industry's current and real legal and ethical harms: e.g. scraping copyrighted content for training without paying or attributing creators, not protecting those affected by the misuse of tools to create deepfake porn, the crashes and deaths attributed to Tesla's self-driving mode, AI resume screening bots messing up etc. https://www.youtube.com/watch?v=YsLf4lAG0xQ https://www.youtube.com/watch?v=YsLf4lAG0xQ
  - hoseja 2y ago

uniqueuid 2y ago

It would also be sad to see the scientific system destroyed by a wave of automatically generated papers that no human has the capacity to verify. It's not hard to generate ideas, it's hard to generate reliable and relevant ideas. Such AI science generators are destroying the grass they graze on unless they take science more seriously (and not as a toddler idea of "generating and testing ideas", which is only a small part of the story).

foolfoolz 2y ago
we already have a wave of papers that no human has the capacity to verify
- uniqueuid 2y ago
  Maybe, maybe not. It's a tiered system - you get the deluge at the unfiltered bottom and a narrower selection the more prestigious and selective the outlets / conferences / journals are. Problem is, of course, that selection criteria are in large parts proxies, not measures of quality. With AI, those proxies become tainted and then you get an explosion of effort. If anyone has a good recommendation for scalable criteria to assess the quality of papers (beyond fame haha) I'm all ears.
  - staunton 2y ago
    If we had scalable criteria for quality of papers, that would be the end of science. The rest would be engineering...
  - PROMISE_237 2y ago
    [dead]
- EGreg 2y ago
  Literally every AI discussion on HN has the same format > AI is awesome, regulation is stupid > I wouldnt want to see AI flooding the market with X we can’t verify > We already have (copy whatever was just said) For what it’s worth, when it comes to SCIENCE, I an actually in favor of AI, even giving it to everyone. Except possibly AI that would help engineer designer viruses. Because in science, people literally ARE just doing pattern matching and testing the patterns. Kepler just looked at a bunch of star charts and found a low-dimensional explanation with ellipses. You can throw that data at an AI and it will come up with physics laws involving 24,729 variables that predict things way more accurately. Including potentially chaotic systems like weather or a three body problem etc. So yeah, use AI for that, because you can actually check its predictions against data, and develop a reputation. We can’t really reason about theories and models of systems with 30,000 variables anyway. If AI churns out 25 scientific models per day, the proper venue isnt publishing them on arxiv or Nature magazine. It’s testing their predictions and putting them up the same way HuggingFace does. To amass reputation and be checked against real data by multiple parties. As a side question — is protein folding @ home dead now, since Google “solved it” with massive clusters and AI? What about SETI@home? Why not just run AI for SETI and solve the drake equation once and for all? :)
  - EvgeniyZh 2y ago
    > If AI churns out 25 scientific models per day, the proper venue isnt publishing them on arxiv or Nature magazine. It’s testing their predictions and putting them up the same way HuggingFace does. Unless the testing will be done by AI, I doubt human scientist will bother with testing tons of incomprehensible models, unless the accuracy of these models will be exceptionally high
    - jalman 2y ago
      The value of AI-mated research is results. This tool will aid engineering. It will offer a pinpointed research to resolve a particular issue at hand. It will close the verification loop naturally by proving that the research has indeed facilitated a useful result. Most of such useful research will never be publicly available. What you are complaining about is a legitimately brocken verification loop. Let's consider a case. An engineering department has a problem. It passes the problem to the research department. The research department slacks out, while produces junk. It makes tons of papers that are hard to prove, not to say apply. They drink champagne with other researchers, so they can publish the junk and defend the turf. AI-mated research will finish the racket and corruption.
- mhuffman 2y ago
  I have also, unless I hallucinated it, read accusations on this very site of peer reviewed papers that were at least partly generated by LLMs.

ineedaj0b 2y ago

sam made comments on Twitter we hit level 2 - we’ll know more in the coming weeks if he’s right.

nprateem 2y ago
Highly unlikely since today's models can't consistently follow simple instructions. Eg "don't waffle, don't sound like you're writing an essay, don't use the fucking word delve, don't apologise" "Sorry about that, let's delve into this"

ben_w 2y ago

> It's clear OpenAI is a hype company Every other industry: "My new invention is safe, I swear" Public reaction: "You're biased, it's dangerous!" Almost the entire AI industry, including people who resign to speak more openly about the risks: "This may kill literally everyone, none of us knows what we're doing or what 'safe' even means" Public reaction: "You're biased, it's safe, just saying that to look cool!"

sampo 2y ago
20 to 30 years ago, activists firebombed university research labs (e.g. Michigan State University, University of Washington, Michigan Technological University [1]) because they believed genetically engineered plants are dangerous. Today, we don't have such serious activism against AI. So you are right, the public doesn't think AI is a danger. [1] https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable_attacks:_1998%E2%80%932009 https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable...
- eddiewithzato 2y ago
  reminds me, I would’ve rather seen VCs fund more genetic engineering startups. Imagine the good it could do, from stem cells to nanobots to “hacking” human DNA itself. But I know the business model there can’t compete with software. So it will never reach the funding it needs.
VieEnCode 2y ago
I found this guy's take on the AI safety scene to be quite insightful. In summary, he feels the focus on sci-fi type existential risk to be a deliberate distraction from the AI industry's current and real legal and ethical harms: e.g. scraping copyrighted content for training without paying or attributing creators, not protecting those affected by the misuse of tools to create deepfake porn, the crashes and deaths attributed to Tesla's self-driving mode, AI resume screening bots messing up etc. https://www.youtube.com/watch?v=YsLf4lAG0xQ https://www.youtube.com/watch?v=YsLf4lAG0xQ
- hoseja 2y ago