3 ms·

The ability to reliably follow such instructions given a JSON manifest likely comes from training on a (small) dataset of these "plugin manifests".

• iforgotpassword 4y ago
  I guess some additional training doesn't hurt and could make it more deterministic and reliable. But it's impressive how you can already tell it to create simple JSON structures from naturally worded descriptions, so I'm convinced it would already work reasonably well without additional training.
  • mike_hearn 4y ago
    It can but it'd make up its own schema. The driver is deterministic logic and needs a completely predictable token sequence.
    • thomashop 4y ago
      Not if you give it the correct schema in the prompt
      • rfoo 4y ago
        In my experience it still sometimes makes up schema or outright outputs mansplaining-like plaintext instead of JSON even if I give it correct schema. Happened once about 15~20 attempts, 5% is still too high to be considered reliable :( I've tuned temperature, added a logit_bias to heavily prefer `{` token, this helped with the plain English v.s. JSON issue, but didn't help with hallucination. I guess I really need API access to {role: "tool"}.