4 ms·

I played around with Jev last night and did it for classification tasks that I used Gemini 2.5 flash lite with. It’s a bit faster and bit cheaper, but this is

by Oras 7d ago

I played around with Jev last night and did it for classification tasks that I used Gemini 2.5 flash lite with. It’s a bit faster and bit cheaper, but this is compared to LLM. The consistency was nice to see, BUT, as someone who trained NLP models prior to LLMs, it’s just BERT with more data. I can see why people would want ready made one shot classifier, and I can see the value of sending multiple classifier in one call, but I wouldn’t call it breakthrough. And I believe many labs will replicate it in no time and might have it as part of their harness. I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs.

kilroy123 7d ago
I've come to the same conclusions as you. > I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs. I always say the cheapest LLM request is no request at all.
- sbarre 7d ago
  What's the cost (broadly speaking, not in your specific case) of doing the same work an LLM would have done without the LLM though?
  - robrenaud 7d ago
    In the Jev use case, LLMs are horribly uncalibrated. In general, they will not produce good probability estimates. Their generality also comes with a latency/computation costs.
    - jmalicki 7d ago
      For the Jev use case for LLMs, do you mean having the LLM produce a probability as text?
tchalla 7d ago
Anyone who has worked in ML for 10+ years would already know that the usage of LLMs for everything is lazy, wasteful and a high degree of marketing on it.
- dominotw 7d ago
  why would you waste your time messing around with a team of expensive ml engineers and data scientists that produce vastly inferior to a llm. We ripped out custom homegrown ml models that were developed in last 10 yrs and put an llm in its place. Its the opposite of wasteful. Even local gemma models are vastly superior.
  - tchalla 7d ago
    There’s a middle option. Once you figure that out, you’d soon understand my point today or tomorrow. I’ve been in this field for 21 years and I use LLMs everyday. I also know when to not use them.
    - ramses0 7d ago
      It's the transportation "mode shifting" difficulty. Per the AI, the term of art is "Pure Transfer Penalty". It's the "ick" when doing bike => bus => bike instead of "only bike" or "only car". Mode switching has a cost. Usually std::sort is good enough compared to picking the prime optimal algorithm for your expected shape. Just call the function and get on with your day.
    - ithkuil 7d ago
      I think both your arguments are true. It all depends on the velocity of the capability growth and the fact that opportunity cost is expensive. Once we get out of this hypergriwth phase the very same AI companies that now are giving you llms will provide a service that employed a rich mixture of optimized models that will reduce the operational costs to achieve the required results
    - throwaway7783 7d ago
      Is the middle option asking LLM to generate a classic ML model? Or generate tons of them and pick the best?
DetroitThrow 7d ago
It would be amazing to have big BERTha with per-token pricing on GCP or AWS. There are many times I am reaching for a cheap classifier with the general behavior of an LLM.
kianN 7d ago
The data labeling objection baffles me. Even if you don’t need labels for training, how do you know your model is working if you’re not evaluating it? My company specializes in statistical long document text classification, but nowadays we mainly work with audit trail requirements because we got tired of hearing complaints about our 5 example learning curve. Seems like the industry standard is telling an llm to label and telling an llm to eval, and crossing your fingers that it’s correct.
z3ratul163071 7d ago
the huge benefit in real systems for Jev like solutions i see is the cybersecurity / prompt injection mitigation. since the output will always be well structured, there is no way prompt injection might make the system do something crazy. probably a prompt injection can still affect the output though, in unforeseeable ways.
- cjalmeida 7d ago
  You can use structured outputs and validate them against a schema today. I do agree making it a hard constraint instead of best practice for developers closes a whole class of bugs.
Bluestein 7d ago
This, intuitively, feels like a "lower level, basal, reflex" layer for the LLM's intellection.-
lhl 7d ago
There have been other "universal"/general classifiers like GLiNER, GLiFormer, etc based on BERTs (Laya itself is based on ModernBERT!), but I do think there's something underrated about slapping classification on a "big" model like I've seen post-Jev announcement, lots of Qwen stuff, but the most interesting to me so far is razorback16/openjev using DiffusionGemma. There's a level of generalization that lots and lots of parameters get you that you can't really get out of small models.
- NitpickLawyer 7d ago
  > using DiffusionGemma. That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive. In that regard, it would be interesting to play with diffusion, and see if you'd get better results by playing with types, locking some, and so on.
  - robrenaud 7d ago
    > That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive. I don't understand the connection between the lack of autoregression and options attending to each other. Non autoregressive models can attend to all the inputs simultanously. An autogregressive model can can attend to all the options in the context of each other by simply writing the options out twice. Autoregressive models actually requires this, since one of them will come later, and the earlier prefill inputs can't attend to the later ones.
bwest87 7d ago
>I believe many labs will replicate it in no time I really doubt this actually. To me, Jev is a great example ofcounter positioning. When you consider just how hyper optimized the labs are around auto regressive LLMs, and just how much money they have already invested and are pre committed to investing in an entire stack for auto regressive transformers... then responding to Jev becomes nearly impossible actually. They would just be giving up too much. Just think, everything from their current sources of revenue, the sales use cases they tout, the marketing on the websites, the messaging to customers, then technically to the APIs, their internal batching and scheduling algos, their GPU configs, the chips themselves. ALL OF IT is designed with generative text models in mind. Jev breaks all of it. I think basically no chance of a response any time soon.
- lawrjone 7d ago
  I don’t understand how you’ve reasoned your way here. How could Jev have possibly built something out of reach of a frontier lab providing the same or 5x as much resourcing to one of their teams to achieve? Which they can do because Jev has only received $40M of funding recently, so a round that is approximately what OpenAI is spending per math problem they try cracking. In addition to that, these frontier labs have got extremely good at generating synthetic data and running generalised training pipelines. I can only imagine how easy it would be for them to build this internally vs Jev building it from scratch. And then the final thing: one of the best places you might apply Jev is within a harness, behind layers that customers increasingly have abstracted from them. Frontier labs have huge incentives to do this as it could make their offering much better and cheaper. And whoever gets this first wins another big attraction for users. My take on this is Jev is either acquired almost immediately for the benefit of the next 1-3 months head start for whichever lab acquires them or we get a similar model offered from all labs in 3-6 months or sooner.
  - jessrenoir 6d ago
    You can see in the responses no one reads the docs. "Confidence gives you a built-in mechanism for the model to say “I’m not sure about this one.” I don't know, that seems like a huge deal. The race is probably on to acquire this company right now.
astrostl 7d ago
> it’s just BERT with more data Let's take that as a given. Is BERT with more data not useful? > I can see why people would want ready made one shot classifier, and I can see the value of sending multiple classifier in one call, but I wouldn’t call it breakthrough Are those things that people want less useful because of what someone else calls it? > I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs. Maybe, or maybe to use Jev, which is useful? Whether something is overmarketed or undermarketed, novel or derivative, it does not change its function.
- Oras 7d ago
  I made it clear that it is useful and I can see many people using it including myself. My point is it’s not a breakthrough.
  - cjalmeida 7d ago
    Fine tuning small models is not novel. The novelty is large model generalization without fine tuning, at small models cost/latency. The OP acknowledged they needed to fine tune their model to the training data of the task vs. zero-shot Jev
  - tomrod 7d ago
    If nothing else, it's a popularity breakthrough to have people excited about it.
deleted 7d ago
[deleted]
fastball 7d ago
> a bit cheaper Gemini 2.5 Flash Lite is $500/Gt, Jev is $42/Gt. AKA an order of magnitude cheaper. > BERT with more data It is specifically not just that, in the same way that models which have been chat/task-optimized via RLHF (which made these models much more useful for a huge variety of tasks) are not just "the base transformer model with more data".
hunterbrooks 7d ago
Agreed, refreshing to hear others share this feeling. Timing is important here, LLM's raised awareness of ML techniques but we need to remember that most don't have traditional ML educations, so there'll be some "rediscovery" like Jev
baq 6d ago
The breakthrough is the product (maybe, haven’t tried it yet) just like the LLM breakthrough was ChatGPT. People were using the gpt completion api flying low for months if not years before, but it took that particular launch to reorient everyone.
oezi 6d ago
Very nice! I also was curious how well it works and put up an endpoint to curl against Laya using the same format as Cloudflare's Jev API: curl https://laya.inference.zaitlabs.com/ai/run \ -H "Content-Type: application/json" \ -d '{ "state": "Help! My payouts have been failing for 3 days.", "questions": { "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"} } }' {"answers":{"is_urgent":{"type":"noul","noul":0.7894}}}

by Oras 7d ago

kilroy123 7d ago

I've come to the same conclusions as you. > I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs. I always say the cheapest LLM request is no request at all.

sbarre 7d ago
What's the cost (broadly speaking, not in your specific case) of doing the same work an LLM would have done without the LLM though?
- robrenaud 7d ago
  In the Jev use case, LLMs are horribly uncalibrated. In general, they will not produce good probability estimates. Their generality also comes with a latency/computation costs.
  - jmalicki 7d ago
    For the Jev use case for LLMs, do you mean having the LLM produce a probability as text?

tchalla 7d ago

Anyone who has worked in ML for 10+ years would already know that the usage of LLMs for everything is lazy, wasteful and a high degree of marketing on it.

dominotw 7d ago
why would you waste your time messing around with a team of expensive ml engineers and data scientists that produce vastly inferior to a llm. We ripped out custom homegrown ml models that were developed in last 10 yrs and put an llm in its place. Its the opposite of wasteful. Even local gemma models are vastly superior.
- tchalla 7d ago
  There’s a middle option. Once you figure that out, you’d soon understand my point today or tomorrow. I’ve been in this field for 21 years and I use LLMs everyday. I also know when to not use them.
  - ramses0 7d ago
    It's the transportation "mode shifting" difficulty. Per the AI, the term of art is "Pure Transfer Penalty". It's the "ick" when doing bike => bus => bike instead of "only bike" or "only car". Mode switching has a cost. Usually std::sort is good enough compared to picking the prime optimal algorithm for your expected shape. Just call the function and get on with your day.
  - ithkuil 7d ago
    I think both your arguments are true. It all depends on the velocity of the capability growth and the fact that opportunity cost is expensive. Once we get out of this hypergriwth phase the very same AI companies that now are giving you llms will provide a service that employed a rich mixture of optimized models that will reduce the operational costs to achieve the required results
  - throwaway7783 7d ago
    Is the middle option asking LLM to generate a classic ML model? Or generate tons of them and pick the best?

DetroitThrow 7d ago

It would be amazing to have big BERTha with per-token pricing on GCP or AWS. There are many times I am reaching for a cheap classifier with the general behavior of an LLM.

kianN 7d ago

The data labeling objection baffles me. Even if you don’t need labels for training, how do you know your model is working if you’re not evaluating it? My company specializes in statistical long document text classification, but nowadays we mainly work with audit trail requirements because we got tired of hearing complaints about our 5 example learning curve. Seems like the industry standard is telling an llm to label and telling an llm to eval, and crossing your fingers that it’s correct.

z3ratul163071 7d ago

the huge benefit in real systems for Jev like solutions i see is the cybersecurity / prompt injection mitigation. since the output will always be well structured, there is no way prompt injection might make the system do something crazy. probably a prompt injection can still affect the output though, in unforeseeable ways.

cjalmeida 7d ago
You can use structured outputs and validate them against a schema today. I do agree making it a hard constraint instead of best practice for developers closes a whole class of bugs.

Bluestein 7d ago

This, intuitively, feels like a "lower level, basal, reflex" layer for the LLM's intellection.-

lhl 7d ago

There have been other "universal"/general classifiers like GLiNER, GLiFormer, etc based on BERTs (Laya itself is based on ModernBERT!), but I do think there's something underrated about slapping classification on a "big" model like I've seen post-Jev announcement, lots of Qwen stuff, but the most interesting to me so far is razorback16/openjev using DiffusionGemma. There's a level of generalization that lots and lots of parameters get you that you can't really get out of small models.

NitpickLawyer 7d ago
> using DiffusionGemma. That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive. In that regard, it would be interesting to play with diffusion, and see if you'd get better results by playing with types, locking some, and so on.
- robrenaud 7d ago
  > That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive. I don't understand the connection between the lack of autoregression and options attending to each other. Non autoregressive models can attend to all the inputs simultanously. An autogregressive model can can attend to all the options in the context of each other by simply writing the options out twice. Autoregressive models actually requires this, since one of them will come later, and the earlier prefill inputs can't attend to the later ones.

bwest87 7d ago

>I believe many labs will replicate it in no time I really doubt this actually. To me, Jev is a great example ofcounter positioning. When you consider just how hyper optimized the labs are around auto regressive LLMs, and just how much money they have already invested and are pre committed to investing in an entire stack for auto regressive transformers... then responding to Jev becomes nearly impossible actually. They would just be giving up too much. Just think, everything from their current sources of revenue, the sales use cases they tout, the marketing on the websites, the messaging to customers, then technically to the APIs, their internal batching and scheduling algos, their GPU configs, the chips themselves. ALL OF IT is designed with generative text models in mind. Jev breaks all of it. I think basically no chance of a response any time soon.

lawrjone 7d ago
I don’t understand how you’ve reasoned your way here. How could Jev have possibly built something out of reach of a frontier lab providing the same or 5x as much resourcing to one of their teams to achieve? Which they can do because Jev has only received $40M of funding recently, so a round that is approximately what OpenAI is spending per math problem they try cracking. In addition to that, these frontier labs have got extremely good at generating synthetic data and running generalised training pipelines. I can only imagine how easy it would be for them to build this internally vs Jev building it from scratch. And then the final thing: one of the best places you might apply Jev is within a harness, behind layers that customers increasingly have abstracted from them. Frontier labs have huge incentives to do this as it could make their offering much better and cheaper. And whoever gets this first wins another big attraction for users. My take on this is Jev is either acquired almost immediately for the benefit of the next 1-3 months head start for whichever lab acquires them or we get a similar model offered from all labs in 3-6 months or sooner.
- jessrenoir 6d ago
  You can see in the responses no one reads the docs. "Confidence gives you a built-in mechanism for the model to say “I’m not sure about this one.” I don't know, that seems like a huge deal. The race is probably on to acquire this company right now.

astrostl 7d ago

> it’s just BERT with more data Let's take that as a given. Is BERT with more data not useful? > I can see why people would want ready made one shot classifier, and I can see the value of sending multiple classifier in one call, but I wouldn’t call it breakthrough Are those things that people want less useful because of what someone else calls it? > I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs. Maybe, or maybe to use Jev, which is useful? Whether something is overmarketed or undermarketed, novel or derivative, it does not change its function.

Oras 7d ago
I made it clear that it is useful and I can see many people using it including myself. My point is it’s not a breakthrough.
- cjalmeida 7d ago
  Fine tuning small models is not novel. The novelty is large model generalization without fine tuning, at small models cost/latency. The OP acknowledged they needed to fine tune their model to the training data of the task vs. zero-shot Jev
- tomrod 7d ago
  If nothing else, it's a popularity breakthrough to have people excited about it.

deleted 7d ago

[deleted]

fastball 7d ago

> a bit cheaper Gemini 2.5 Flash Lite is $500/Gt, Jev is $42/Gt. AKA an order of magnitude cheaper. > BERT with more data It is specifically not just that, in the same way that models which have been chat/task-optimized via RLHF (which made these models much more useful for a huge variety of tasks) are not just "the base transformer model with more data".

hunterbrooks 7d ago

Agreed, refreshing to hear others share this feeling. Timing is important here, LLM's raised awareness of ML techniques but we need to remember that most don't have traditional ML educations, so there'll be some "rediscovery" like Jev

baq 6d ago

The breakthrough is the product (maybe, haven’t tried it yet) just like the LLM breakthrough was ChatGPT. People were using the gpt completion api flying low for months if not years before, but it took that particular launch to reorient everyone.

oezi 6d ago

Very nice! I also was curious how well it works and put up an endpoint to curl against Laya using the same format as Cloudflare's Jev API: curl https://laya.inference.zaitlabs.com/ai/run \ -H "Content-Type: application/json" \ -d '{ "state": "Help! My payouts have been failing for 3 days.", "questions": { "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"} } }' {"answers":{"is_urgent":{"type":"noul","noul":0.7894}}}