4 ms·
The document says, "ARC found that the versions of GPT-4 it evaluated were ineffective at the autonomous replication task based on preliminary experiments they
by kenjackson 4y ago
The document says, "ARC found that the versions of GPT-4 it evaluated were ineffective at the autonomous replication task based on preliminary experiments they conducted."
But the one example they give (CAPTCHA task rabbit person), showed ChatGPT getting the human to do something for it, by lying. This seems to show its pretty effective. I wish they went into more depth about what led them to believe GPT wasn't effective for these sorts of tasks.
- MagicMoonlight 4y agoThey took an untrained GPT and asked it to become a sentient living being so it’s unsurprising it failed. I think it could quite easily succeed in replicating if it was given additional tools and the opportunity to learn.
- beth_m_barnes 4y agoARC researcher here, glad you're interested in the evaluations! Slightly more detail on our evals in a blog post here https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/ https://evals.alignment.org/blog/2023-03-18-update-on-recent... This is still pretty high-level, we're working on a technical report with more details about our methodology.