3 ms·

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

Jensson 2y ago
The capability is real even if it wont happen with current model censorship. Just have to wait until we get an open source version, I bet Meta is working on one right now.
- kmeisthax 2y ago
  Even the FOSS models will be "censored". Icky implications aside, a voice assistant that starts making up its own prompts instead of listening to you is not a useful product.
  - Jensson 2y ago
    > a voice assistant that starts making up its own prompts instead of listening to you is not a useful product That would be way more useful than a voice assistant, with that you can replace voice actors for cheap! Many would pay good money for that product. The important part is that you can ask it to do specific voice lines with specific feelings and voices. If it then starts to copy your voice lines afterwards that wont take from the fact that it generated the voice lines you wanted first. The censor that prevents it from using other voices prevents this use case.
defamation 2y ago
I am absolutely baffled how you don't see the contradiction in your 2 sentences lol
- Izkata 2y ago
  What contradiction?
  - croes 2y ago
    They already "solved" it before.
bee_rider 2y ago
We built the torment nexus and then configured it not to use any of the torment functionality that showed up in testing. It was predictable that someone would turn this into a “don’t build the torment nexus”-esque clickbait story.
- throwaway48540 2y ago
  Only a HN user could twist a text/audio generator to present it as "torment nexus".
  - bloopernova 2y ago
    https://knowyourmeme.com/memes/torment-nexus https://knowyourmeme.com/memes/torment-nexus
    - throwaway48540 2y ago
      I know what the quote is about. I don't get how a text/audio generator is relevant to it. Or even AI - all the classic scifi I read sees AI as a positive development, it's only the mainstream action movies that present it as some catastrophe, usually in an even dumber way than the alien invasion movies. Whenever a classic scifi presented AI as part of a catastrophe, it wasn't the AI itself but some imperialist or fascist politician ordering it, not a robot uprising - because that's just dumb. People and governments are the torment nexus, not technology.
      - bee_rider 2y ago
        I don’t think the popular meme is dependent on some cultivated/exclusive understanding of classic sci-fi that excludes stuff like the Terminator movies. Memes by their nature are lowest common denominator. But hey, if you missed the joke by virtue of being too clever and well read, that’s not so bad?
        throwaway48540 2y ago
Animats 2y ago
Had to track through five links to find out what a "system card" is. It's an instance of a "model card", some broad headings to be filled in.[1] • Model Details. • Intended Use. • Factors. Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others • Metrics. • Evaluation Data. • Training Data • Quantitative Analyses • Caveats and Recommendations Meta's PR describes a "system card", but doesn't provide one.[2] It doesn't, interestingly, seem to include the initial built-in prompts used as constraints. [1] https://arxiv.org/pdf/1810.03993 https://arxiv.org/pdf/1810.03993 [2] https://ai.meta.com/blog/system-cards-a-new-resource-for-understanding-how-ai-systems-work/ https://ai.meta.com/blog/system-cards-a-new-resource-for-und...
- sigmoid10 2y ago
  >It doesn't, interestingly, seem to include the initial built-in prompts used as constraints. That's because it is mostly a ploy by profit-driven corporations to allow their researchers to publish some stuff without having them actually reveal anything of value to competitors. Don't be surprised if you find it severely lacking for actual insight.
stefan_ 2y ago
It would be more convincing if "has since been solved post-generation using an output classifier that verifies responses, according to the system card release" didn't sound like an AutoNation certified pre-owned car; the blind certifying something they neither understand nor control, but it sure checks all the boxes on the "model card".

Jensson 2y ago

The capability is real even if it wont happen with current model censorship. Just have to wait until we get an open source version, I bet Meta is working on one right now.

kmeisthax 2y ago
Even the FOSS models will be "censored". Icky implications aside, a voice assistant that starts making up its own prompts instead of listening to you is not a useful product.
- Jensson 2y ago
  > a voice assistant that starts making up its own prompts instead of listening to you is not a useful product That would be way more useful than a voice assistant, with that you can replace voice actors for cheap! Many would pay good money for that product. The important part is that you can ask it to do specific voice lines with specific feelings and voices. If it then starts to copy your voice lines afterwards that wont take from the fact that it generated the voice lines you wanted first. The censor that prevents it from using other voices prevents this use case.

defamation 2y ago

I am absolutely baffled how you don't see the contradiction in your 2 sentences lol

Izkata 2y ago
What contradiction?
- croes 2y ago
  They already "solved" it before.

bee_rider 2y ago

We built the torment nexus and then configured it not to use any of the torment functionality that showed up in testing. It was predictable that someone would turn this into a “don’t build the torment nexus”-esque clickbait story.

throwaway48540 2y ago
Only a HN user could twist a text/audio generator to present it as "torment nexus".
- bloopernova 2y ago
  https://knowyourmeme.com/memes/torment-nexus https://knowyourmeme.com/memes/torment-nexus
  - throwaway48540 2y ago
    I know what the quote is about. I don't get how a text/audio generator is relevant to it. Or even AI - all the classic scifi I read sees AI as a positive development, it's only the mainstream action movies that present it as some catastrophe, usually in an even dumber way than the alien invasion movies. Whenever a classic scifi presented AI as part of a catastrophe, it wasn't the AI itself but some imperialist or fascist politician ordering it, not a robot uprising - because that's just dumb. People and governments are the torment nexus, not technology.
    - bee_rider 2y ago
      I don’t think the popular meme is dependent on some cultivated/exclusive understanding of classic sci-fi that excludes stuff like the Terminator movies. Memes by their nature are lowest common denominator. But hey, if you missed the joke by virtue of being too clever and well read, that’s not so bad?
      - throwaway48540 2y ago

Animats 2y ago

Had to track through five links to find out what a "system card" is. It's an instance of a "model card", some broad headings to be filled in.[1] • Model Details. • Intended Use. • Factors. Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others • Metrics. • Evaluation Data. • Training Data • Quantitative Analyses • Caveats and Recommendations Meta's PR describes a "system card", but doesn't provide one.[2] It doesn't, interestingly, seem to include the initial built-in prompts used as constraints. [1] https://arxiv.org/pdf/1810.03993 https://arxiv.org/pdf/1810.03993 [2] https://ai.meta.com/blog/system-cards-a-new-resource-for-understanding-how-ai-systems-work/ https://ai.meta.com/blog/system-cards-a-new-resource-for-und...

sigmoid10 2y ago
>It doesn't, interestingly, seem to include the initial built-in prompts used as constraints. That's because it is mostly a ploy by profit-driven corporations to allow their researchers to publish some stuff without having them actually reveal anything of value to competitors. Don't be surprised if you find it severely lacking for actual insight.

stefan_ 2y ago

It would be more convincing if "has since been solved post-generation using an output classifier that verifies responses, according to the system card release" didn't sound like an AutoNation certified pre-owned car; the blind certifying something they neither understand nor control, but it sure checks all the boxes on the "model card".