4 ms·

I don't understand how this is different from oai "structured output" (and whatever the similar paradigm was on Sonnet ~3.7 back then) which everyone moved on f

• mritchie712 16d ago
  in short: it's faster, cheaper, smart structured output. each "question" is answered in parallel instead of a sequential (like an LLM). so if you have an input like: {"is_it_hotdog": noul, "is_it_apple", noul} it answers is_it_hotdog and is_it_apple in parallel and gives a probability.
  • satvikpendem 16d ago
    Can't I just parallelize my LLM calls myself for each question?
    • orbital-decay 16d ago
      You can. It will be expensive, slow, and less reliable than a specialized model.
    • zwily 16d ago
      Anything you can do in Jev can be done with an LLM at much greater cost and latency.
      • mmnfrdmcx 16d ago
        Agree, except the probabilities for outcomes in the structured output. I don't think you can get those for most frontier LLMs (logprobas). You can get it for open source models but not frontier LLMs.
        • esafak 16d ago
          That number is a big deal, assuming it is well calibrated. Did they talk about calibration?
          • Matticus_Rex 16d ago
            I've seen them talk about it a bit on Twitter -- it seems to be fairly well-calibrated in general, but obviously you need to test it on your use case and dial it in comparison with known data for best results.
    • shados 14d ago
      Yup. LLMs can do almost anything. Can they do it at the speed, cost and confidence of a model like Jev is a different story. In Jev output tokens are straight up free because it's not doing text token generations.
• orbital-decay 16d ago
  It's a non-instruction-tuned classifier model trained on a confidence-aware RL variety that generates its own schema and follows it, with a confidence score output. Think BERT on crack, smart enough to be used as a decision maker (conceptually). They call it "not an LLM" because it's non-generative but of course it's a language model in the same way all non-instruction-tuned classifiers are.
  • ozgung 16d ago
    Isn't that the same transformer at the end of the day? It must be faster only because it generates a single token output, just one evaluation of the model. It takes the same input context and has the same O(n^2) attention blocks. It probably takes options as appended to the input and returns a probability over them instead of the whole dictionary. It's post-trained to do that specific job. If so what's the big deal?
    • orbital-decay 16d ago
      They say it's "parallelized". Whatever that means in reality, their demos are pretty good, their prices are extremely low compared to alternatives, and it responds in ~100ms which is pretty fast for what they do. Whether it holds for longer inputs, edge cases, etc. remains to be seen, but I can imagine the use cases for that, for example you can use it directly in the sampling layer of a normal generative model, or just as a generic decision maker/controller. They can (and will, in their words) do this for images too. I don't know if it's a big deal, but it's kind of a fresh perspective.
      • Topfi 16d ago
        Unless I misunderstood what they wrote, I read parallelized in the diffusion sense, akin to GemmaDiffusion and Inception Labs models. Incidentally, Mercury 2.5 is truly groundbreaking, giving it a try is highly recommended.
        • mohsen1 16d ago
• jLaForest 16d ago
  I'm in the middle of moving my app to openAI structured output. Could you please explain what you mean by "which everyone moved on from"?
  • slickytail 16d ago
    Structured decoding limits next-token probabilities to ensure valid JSON. The main issue is that if the model puts a substantial probability on an invalid token, then it was already confused, and in that case, you don't actually want whatever the next-most-likely valid token is: even if it's syntactically valid, it's likely semantically erroneous.
    • rhodysurf 16d ago
      So whats the alternative? Letting it codegen a file and the piping that? What a worse workflow
      • qeternity 14d ago
        Alternative is to fail the validation and regenerate the request, but I have no idea what this person is talking about. It is absolutely not the case that everyone has moved on from structured generation. Absurd claim.
    • slices 16d ago
      sounds like an argument in favor of structured output, is that right?
    • qeternity 14d ago
      Not sure why you think everyone has moved on from this. Valid JSON is only one aspect. It almost certainly has far more uptake today than ca. Sonnet 3.7. Grammar engines can enforce a lot of other things too. Models are actually really good at emitting valid JSON, but complex schemas effectively require it.
• Topfi 16d ago
  Jev is, as far as I understand, essentially very optimised for zero shot classification [0]. Something like BERT could be and has been tuned to provide similar "decision making" at a similar latency and cost advantage quite some time back. Advantage over full on LLMs is mainly the efficiency and of something like Jev over e.g. the encoder/decoder based classifier I had in front of an LLM to route to different prompts depending on the users likely needs, that Jev does perform at a more consistent level, allegedly roughly akin to GPT-5.6 Terra, but at the lower cost and latency. Currently testing that, but seems promising, if Jev classifies at or above Terra level, I see no reason not to leverage it. Can add that I tried using a heavily pruned mt0 based model for structured classification along with structured output for local tagging and simple renaming suggestions. While it does work, the balance is hard to get right for the machine I was targeting as a minimum spec (Macbook Neo), so that's on ice. Focusing on one of the tasks easily goes below 100mb with solid latency across all EU Latin script languages, but the second you add a few, it's simply not in the quality budget, so while LLMs can do anything Jev and similarly focused models can, it comes at a literal cost. Could maybe accomplish the goal with multiple models (BERT+mt0+...), but that get messy. In general just happy to see a bit of the millions flooding into the industry being used to improve on less flashy but immensely useful solutions. It's amazing that you can technically use LLMs for most tasks, but not every org has a near infinite budget and there is still a lot to gain from applying more recent learnings to old solutions along with just updating their training data to the current year. Also makes business sense, competition on frontier or mid-tier LLMs is vicious, focusing on an underserved niche with clear application is clever. [0] https://huggingface.co/tasks/zero-shot-classification https://huggingface.co/tasks/zero-shot-classification
• messh 16d ago
  In Jev you pass options in the input and its output just gives some probability for each. Oai structured output just follows a schema. The exact output is still generated and there is no probability
  • brokensegue 15d ago
    you can ask structured output for probabilities...not that they necessarily mean anything.
    • cheesecakegood 15d ago
      In theory Jev “calibrates” the probabilities, meaning a probability of 20% is optimized to happen near 20% of the time, which as you point out traditional models with schemas do NOT optimize for at all
• petesergeant 15d ago
  No idea at all what this particular project called openjev is doing, but https://github.com/TheoLeeCJ/openjev https://github.com/TheoLeeCJ/openjev and https://github.com/ekzhang/openjev-sglang https://github.com/ekzhang/openjev-sglang (neither of which I have any relation to) generate a single token, rather than JSON structured output. I wrote up this technique here: https://sgnt.ai/p/jev/ https://sgnt.ai/p/jev/
  • wuhhh 15d ago
    I came to say thanks for the link to sgnt.ai on Jev, then realised you're the author! Well, thank you so much, I feel informed :)