4 ms·

Do they actually test these system prompts in a rigorous way? Or is this the modern version of the rain dance? I don't think you need to spell it out long-form

Do they actually test these system prompts in a rigorous way? Or is this the modern version of the rain dance? I don't think you need to spell it out long-form with fancy words like you're a lawyer. The LLM doesn't work that way.

mhmmmmmm 1y ago
They certainly do, and also offer the tooling to the public: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-improver https://docs.anthropic.com/en/docs/build-with-claude/prompt-... They also recommend to use it to iterate on your own prompts when using Claude Code for example
- otabdeveloper4 1y ago
  By "rigorous" I mean peeking under the curtain and actually quantifying the interactions between different system prompts and model weights. "Chain of thought" and "reasoning" is marketing bullshit.
  - int_19h 1y ago
    How would you quantify it? The LM is still a black box, we don't know what most of those weights actually do.
    - otabdeveloper4 1y ago
      Yes, that was my question.
Applejinx 1y ago
It doesn't matter whether they do or not. They're saying things like 'Claude does not hallucinate. When it doesn't know something, it always thinks harder about it and only says things that are like totally real man'. It doesn't KNOW. It's a really complicated network of associations, like WE ARE, and so it cannot know whether it is hallucinating, nor can it have direct experience in any way, so all they've done is make it hallucinate that it cares a lot about reality, but it doesn't 'know' what reality is either. What it 'knows' is what kind of talk is associated with 'speakers who are considered by somebody to be associated with reality' and that's it. It's gaslighting everybody including itself. I guess one interesting inference is that when LLMs work with things like code, that's text-based and can deliver falsifiable results which is the closest an LLM can get to experience. Our existence is more tangible and linked to things like the physical world, where in most cases the LLM's existence is very online and can be linked to things like the output of, say, xterms and logging into systems. Hallucinating that this can generalize to all things seems a mistake.
zahlman 1y ago
What humans are qualified to test whether Claude is correctly implementing "Claude should not be politically biased in any direction."?

mhmmmmmm 1y ago

They certainly do, and also offer the tooling to the public: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-improver https://docs.anthropic.com/en/docs/build-with-claude/prompt-... They also recommend to use it to iterate on your own prompts when using Claude Code for example

otabdeveloper4 1y ago
By "rigorous" I mean peeking under the curtain and actually quantifying the interactions between different system prompts and model weights. "Chain of thought" and "reasoning" is marketing bullshit.
- int_19h 1y ago
  How would you quantify it? The LM is still a black box, we don't know what most of those weights actually do.
  - otabdeveloper4 1y ago
    Yes, that was my question.

Applejinx 1y ago

It doesn't matter whether they do or not. They're saying things like 'Claude does not hallucinate. When it doesn't know something, it always thinks harder about it and only says things that are like totally real man'. It doesn't KNOW. It's a really complicated network of associations, like WE ARE, and so it cannot know whether it is hallucinating, nor can it have direct experience in any way, so all they've done is make it hallucinate that it cares a lot about reality, but it doesn't 'know' what reality is either. What it 'knows' is what kind of talk is associated with 'speakers who are considered by somebody to be associated with reality' and that's it. It's gaslighting everybody including itself. I guess one interesting inference is that when LLMs work with things like code, that's text-based and can deliver falsifiable results which is the closest an LLM can get to experience. Our existence is more tangible and linked to things like the physical world, where in most cases the LLM's existence is very online and can be linked to things like the output of, say, xterms and logging into systems. Hallucinating that this can generalize to all things seems a mistake.

zahlman 1y ago

What humans are qualified to test whether Claude is correctly implementing "Claude should not be politically biased in any direction."?