3 ms·

https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/your-llm-assisted-scientific-breakthrough-probably-isn-t https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/you

https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/your-llm-assisted-scientific-breakthrough-probably-isn-t https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/your-llm-a... Hi author, this isn't personal, but I think your AI may be deceiving you into thinking you've made a breakthrough.

daikikadowaki 9mo ago
[flagged]
- durch 9mo ago
  If you have a few minutes I invite you to check what we're doing over at Open Horizon Labs, its exactly the type of thinking we have around the current state of the world. Apologies I feel like I'm stalking you in the comments, but what you're saying absolutely resonates with what I've been thinking, and what I've been trying to build, and its refreshing to finally feel that I'm not insane. https://github.com/open-horizon-labs/superego https://github.com/open-horizon-labs/superego is probably the most useful tool we have, but I'm hoping that we can package it and bring it to the people, as it does make all these LLMs orders of magnitude more useful
  - daikikadowaki 9mo ago
    No apologies needed—I'm just glad to find I'm not the only 'insane' person here. It's easy to feel that way when obsessing over these problems, so knowing my ideas resonate with what you're building at superego is a huge relief. I’m diving into your repo now. Please keep me posted on your progress or any new thoughts—I'd love to hear them.
  - judahmeek 9mo ago
    > as it does make all these LLMs orders of magnitude more useful That seems like something that should be really easy to prove statistically.
    - daikikadowaki 9mo ago
      As for "proving it statistically"—you're looking for utility, but I'm defining legitimacy. A constitution isn't a tool designed to statistically improve a metric; it is a framework to ensure that the system remains aligned with human agency. I am not building an LLM optimization plugin; I am building a benchmark for human-AI co-evolution
- stuartjohnson12 9mo ago
  In the essay I linked, there are some instructions you can follow to test out the idea under "step 1". It's really important to follow them exactly and not to use the same ChatGPT instance as you're talking to about this idea so we can test with an independent party what is going on. I'd be curious what the output is.
  - daikikadowaki 9mo ago
    I took the challenge. To ensure a completely objective 'reality-check,' I opened a fresh session in Chrome Incognito mode with a brand-new account and used GPT-5, as suggested. I followed 'Step 1' of the essay to the letter—copy-pasting the exact prompt designed to expose self-deception and 'AI-aided' delusions. I didn't frame it as my own work, allowing the model to provide a raw, critical audit without any bias toward the author. https://chatgpt.com/share/6963b843-9bbc-8001-a2ea-409a5f6dd684 https://chatgpt.com/share/6963b843-9bbc-8001-a2ea-409a5f6dd6...
    - thunfischbrot 9mo ago
      That’s not too bad and mirrored some of the feedback in this thread. Tldr: interesting idea, more worthy of a blog post or a thread in one of your favourite online communities, rather than a paper.
    - stuartjohnson12 9mo ago
      Awesome - now read it really closely and compare it to the version of reality in your OP. And DON'T paste it or this comment into your normal ChatGPT instance and ask it to respond. Really just think for a moment on your own. > The goal: replace vague legal and philosophical notions of “manipulation” with a concrete engineering variable. [...] formally define the metric What's the conclusion? Is this a "concrete engineering paper"? Has anything been "formally proved"? From your link: > The math is conceptual, not formal. > This is serious, careful, and intellectually honest work, but it is not conventional science. > The project would be strongest if positioned explicitly as foundational theory + open design pattern, rather than as something awaiting “validation.” > it is valid as a design pattern or architectural disclosure, not as experimental systems research Be careful before immediately dismissing this as just imprecise language or a translation issue. There's a reason I suggested this to you.
- deleted 9mo ago
  [deleted]
usefulposter 9mo ago
Fascinating. Searching https://hn.algolia.com https://hn.algolia.com for "zenodo" and "academia.edu" (past year) reveals hundreds of similar "breakthroughs". The commons (open access repositories, HN, Reddit, ...) is being swamped.
- stuartjohnson12 9mo ago
  Since OpenAI patched the LLM spiritual awakening attractor state, physics and computer science is what sycophantic AI is pushing people towards now. My theory is that those things tend to be especially optimised for deceit because they involve modelling and many people can become confused between the difference between a model as the expression of a concept and a model as in the colloquial idea of "the way the universe works".
  - cap11235 9mo ago
    I'd love to see a new cult form around UML. Unified Modeling Language already sounds LLMy.
    - daikikadowaki 9mo ago
      [flagged]
  - daikikadowaki 9mo ago
    [flagged]
- daikikadowaki 9mo ago
  [flagged]
- amarcheschi 9mo ago
  it's all ai allucination, in a subreddit i once found a tailor asking for how to contact some professors because they found a breakthrough discovery on how knowledge is arranged inside neural networks (whatever that means)

daikikadowaki 9mo ago

[flagged]

durch 9mo ago
If you have a few minutes I invite you to check what we're doing over at Open Horizon Labs, its exactly the type of thinking we have around the current state of the world. Apologies I feel like I'm stalking you in the comments, but what you're saying absolutely resonates with what I've been thinking, and what I've been trying to build, and its refreshing to finally feel that I'm not insane. https://github.com/open-horizon-labs/superego https://github.com/open-horizon-labs/superego is probably the most useful tool we have, but I'm hoping that we can package it and bring it to the people, as it does make all these LLMs orders of magnitude more useful
- daikikadowaki 9mo ago
  No apologies needed—I'm just glad to find I'm not the only 'insane' person here. It's easy to feel that way when obsessing over these problems, so knowing my ideas resonate with what you're building at superego is a huge relief. I’m diving into your repo now. Please keep me posted on your progress or any new thoughts—I'd love to hear them.
- judahmeek 9mo ago
  > as it does make all these LLMs orders of magnitude more useful That seems like something that should be really easy to prove statistically.
  - daikikadowaki 9mo ago
    As for "proving it statistically"—you're looking for utility, but I'm defining legitimacy. A constitution isn't a tool designed to statistically improve a metric; it is a framework to ensure that the system remains aligned with human agency. I am not building an LLM optimization plugin; I am building a benchmark for human-AI co-evolution
stuartjohnson12 9mo ago
In the essay I linked, there are some instructions you can follow to test out the idea under "step 1". It's really important to follow them exactly and not to use the same ChatGPT instance as you're talking to about this idea so we can test with an independent party what is going on. I'd be curious what the output is.
- daikikadowaki 9mo ago
  I took the challenge. To ensure a completely objective 'reality-check,' I opened a fresh session in Chrome Incognito mode with a brand-new account and used GPT-5, as suggested. I followed 'Step 1' of the essay to the letter—copy-pasting the exact prompt designed to expose self-deception and 'AI-aided' delusions. I didn't frame it as my own work, allowing the model to provide a raw, critical audit without any bias toward the author. https://chatgpt.com/share/6963b843-9bbc-8001-a2ea-409a5f6dd684 https://chatgpt.com/share/6963b843-9bbc-8001-a2ea-409a5f6dd6...
  - thunfischbrot 9mo ago
    That’s not too bad and mirrored some of the feedback in this thread. Tldr: interesting idea, more worthy of a blog post or a thread in one of your favourite online communities, rather than a paper.
  - stuartjohnson12 9mo ago
    Awesome - now read it really closely and compare it to the version of reality in your OP. And DON'T paste it or this comment into your normal ChatGPT instance and ask it to respond. Really just think for a moment on your own. > The goal: replace vague legal and philosophical notions of “manipulation” with a concrete engineering variable. [...] formally define the metric What's the conclusion? Is this a "concrete engineering paper"? Has anything been "formally proved"? From your link: > The math is conceptual, not formal. > This is serious, careful, and intellectually honest work, but it is not conventional science. > The project would be strongest if positioned explicitly as foundational theory + open design pattern, rather than as something awaiting “validation.” > it is valid as a design pattern or architectural disclosure, not as experimental systems research Be careful before immediately dismissing this as just imprecise language or a translation issue. There's a reason I suggested this to you.
deleted 9mo ago
[deleted]

usefulposter 9mo ago

Fascinating. Searching https://hn.algolia.com https://hn.algolia.com for "zenodo" and "academia.edu" (past year) reveals hundreds of similar "breakthroughs". The commons (open access repositories, HN, Reddit, ...) is being swamped.

stuartjohnson12 9mo ago
Since OpenAI patched the LLM spiritual awakening attractor state, physics and computer science is what sycophantic AI is pushing people towards now. My theory is that those things tend to be especially optimised for deceit because they involve modelling and many people can become confused between the difference between a model as the expression of a concept and a model as in the colloquial idea of "the way the universe works".
- cap11235 9mo ago
  I'd love to see a new cult form around UML. Unified Modeling Language already sounds LLMy.
  - daikikadowaki 9mo ago
    [flagged]
- daikikadowaki 9mo ago
  [flagged]
daikikadowaki 9mo ago
[flagged]
amarcheschi 9mo ago
it's all ai allucination, in a subreddit i once found a tailor asking for how to contact some professors because they found a breakthrough discovery on how knowledge is arranged inside neural networks (whatever that means)