4 ms·

Gemini is really odd in particular (even with reasoning). Chatgpt still uses a similar religion-influenced language but it's not as weird.

• gwerbin 8mo ago
  We were messing around at work last week building an AI agent that was supposed to only respond with JSON data. GPT and Sonnet more or less what we wanted, but Gemma insisted on giving us a Python code snippet.
  • otabdeveloper4 8mo ago
    > that was supposed to only respond with JSON data. You need to constrain token sampling with grammars if you actually want to do this.
    • written-beyond 8mo ago
      That reduces the quality of the response though.
      • debugnik 8mo ago
        As opposed to emitting non-JSON tokens and having to throw away the answer?
        • jgalt212 8mo ago
          Or just run json.dumps on the correct answer in the wrong format.
        • written-beyond 8mo ago
          Don't shoot the messenger
          • debugnik 7mo ago
            Whom's messenger? You didn't point us to anyone's research. I just don't see how sampling tokens constrained to a grammar can be worse than rejection-sampling whole answers against the same grammar. The latter needs to follow the same constraints naturally to not get rejected, and both can iterate in natural language before starting their structured answer. Under a fair comparison, I'd expect the former to provide answers at least just as good while being more efficient. Possibly better if top-whatever selection happened after the grammar constraint.
      • Der_Einzige 8mo ago
        THIS IS LIES: https://blog.dottxt.ai/say-what-you-mean.html https://blog.dottxt.ai/say-what-you-mean.html I will die on this hill and I have a bunch of other Arxiv links from better peer reviewed sources than yours to back my claim up (i.e. NeurIPS caliber papers with more citations than yours claiming it does harm the outputs) Any actual impact of structured/constrained generation on the outputs is a SAMPLER problem, and you can fix what little impact may exist with things like https://arxiv.org/abs/2410.01103 https://arxiv.org/abs/2410.01103 Decoding is intentionally nerfed/kept to top_k/top_p by model providers because of a conspiracy against high temperature sampling: https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb08b94bce20d5397 https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
        • iugtmkbdfil834 8mo ago
          I honestly would like to hope people were more up in arms over this, but.. based on historical human tendencies, convenience will win here.
        • otabdeveloper4 8mo ago
          I use LLMs for Actual Work (boring shit). I always set temperature to literally zero and don't sample.
  • cubefox 8mo ago
    Gemma≠Gemini