4 ms·

Yeah, real Jev got really weird, no benchmarking clause. Their Terms of Use (1(v)) and MCA (2.3(f)) both prohibit users from publishing "benchmarks or performan

Yeah, real Jev got really weird, no benchmarking clause. Their Terms of Use (1(v)) and MCA (2.3(f)) both prohibit users from publishing "benchmarks or performance information about the Services". No major AI has it; we are back to Oracle-style legal. Though Jev is original, it looks highly replicable.

sodimel 15d ago
I'm working on something from a crappy laptop, those numbers from jev can totally be matched: Local Latency: 0.1813 seconds
- cmrdporcupine 15d ago
  and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways. why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU/TPU/NPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever) the nice thing is that if you're skipping decode you're not as memory bandwidth bound.
- toasty228 15d ago
  I can also run a 0.6b model on my phone faster than openai can run astra, it doesn't mean my model is useful.
cmrdporcupine 15d ago
There's also prior art. Or probably, anyways. https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_literally_built_the_jev_architecture_one_year/ https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_liter... Not only is it replicable as you say, things like it already exist(ed). The important bit of course is in the actual implementation: a) models fine tuned to produce good results for these types of questions and b) runtimes optimized to do this quickly and at scale
ramoz 14d ago
There's no benchmarking because it's not very intelligent at all. Right now everybody's being hype-shotted into believing you can use it for intelligent decisions. https://backnotprop.com/blog/jev-poker/ https://backnotprop.com/blog/jev-poker/
- Culonavirus 14d ago
  > You have to evaluate every situation you want to use this in, one at a time, against an answer key you trust, and decide what a wrong call costs you there. Lmao, imagine a time when all of the talking heads are yapping about AGI and you come up with crap that goes the opposite direction and you call it progress... bold.

sodimel 15d ago

I'm working on something from a crappy laptop, those numbers from jev can totally be matched: Local Latency: 0.1813 seconds

cmrdporcupine 15d ago
and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways. why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU/TPU/NPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever) the nice thing is that if you're skipping decode you're not as memory bandwidth bound.
toasty228 15d ago
I can also run a 0.6b model on my phone faster than openai can run astra, it doesn't mean my model is useful.

cmrdporcupine 15d ago

There's also prior art. Or probably, anyways. https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_literally_built_the_jev_architecture_one_year/ https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_liter... Not only is it replicable as you say, things like it already exist(ed). The important bit of course is in the actual implementation: a) models fine tuned to produce good results for these types of questions and b) runtimes optimized to do this quickly and at scale

ramoz 14d ago

There's no benchmarking because it's not very intelligent at all. Right now everybody's being hype-shotted into believing you can use it for intelligent decisions. https://backnotprop.com/blog/jev-poker/ https://backnotprop.com/blog/jev-poker/

Culonavirus 14d ago
> You have to evaluate every situation you want to use this in, one at a time, against an answer key you trust, and decide what a wrong call costs you there. Lmao, imagine a time when all of the talking heads are yapping about AGI and you come up with crap that goes the opposite direction and you call it progress... bold.