3 ms·

I have to be honest. While this is obviously a smart and useful idea, it misses one of the core features of Jev: its confidence scores. Partial confidence could

I have to be honest. While this is obviously a smart and useful idea, it misses one of the core features of Jev: its confidence scores. Partial confidence could easily be mapped to fractional spaces, using unicode characters like U+2009: THIN SPACE. As it stands, this package is not harnessing the full power of Jev.

preommr 13d ago
> its confidence scores important to note that the "confidence" score is... maybe not what people think it is - kind of useless, and just a convenience step from the probabilities. from the docs: "confidence is a statistic computed from the probability distribution the answer already gives you." [0] I actually encourage people to visit the docs because it has a specific page on this with a little applet to really make this clear. [0] https://docs.typesafe.ai/confidence https://docs.typesafe.ai/confidence
- wongarsu 13d ago
  What else do people think it is? If Typesafe had found a way to measure arbitrary AI results against objective reality (past, future and present) they'd either be making a killing on the stock market or working for the NRO, not publishing that as a confidence value on their API
  - preommr 13d ago
    Well I think there's an expectation that it's similar to the probability score, something that's outputted by the model itself, and so there's some level of "intelligence" (e.g. being able to recognize if the subject matter is relevant to the data it's been trained on, or as an accumulation of errors e.g. it couldn't figure out the question).
danieltanfh95 13d ago
Man we're going full 2015 ML, telling normies that confidence scores mean NOTHING to alleviate false negatives/positives.
- bbor 13d ago
  Yeah I keep getting this weird sense that Jev is kinda poorly reinventing ML. I guess the graphs don't lie and theoretically I can replace luna with it, but I don't really use luna anyway. What is the use case for a classifier that works 90% of the time...? I feel like if I'm classifying something, I probably care enough that 90% ain't gonna cut it... I guess the answer is just agential stuff that effectively gets double checked by the LLM in the driver seat, anyway? That tracks, though it means that jev is mostly just for the people making harnesses. Which is all of us but still!
  - serbuvlad 13d ago
    I think the argument would be that the classifiers of classic ML can be very useful and that Jav is a geenral purpose classifier you can just use that doesn't need to be trained per-task.
  - torginus 13d ago
    I mean when you get your bloodwork done to check for an illness, the test you get will give the right result 90% of the time - and depending on the result, you doc might order more tests, which could be more expensive but no mrpe reliable than the first - but they are going to be statistically independent, and after 2 more, he can be 99.9% sure. Which begs the question, can Jev retest until it gets the right result? Can it tell how corellated two of its results are? 90% correct makes for a wonderful iterator, but a poor oracle.

preommr 13d ago

> its confidence scores important to note that the "confidence" score is... maybe not what people think it is - kind of useless, and just a convenience step from the probabilities. from the docs: "confidence is a statistic computed from the probability distribution the answer already gives you." [0] I actually encourage people to visit the docs because it has a specific page on this with a little applet to really make this clear. [0] https://docs.typesafe.ai/confidence https://docs.typesafe.ai/confidence

wongarsu 13d ago
What else do people think it is? If Typesafe had found a way to measure arbitrary AI results against objective reality (past, future and present) they'd either be making a killing on the stock market or working for the NRO, not publishing that as a confidence value on their API
- preommr 13d ago
  Well I think there's an expectation that it's similar to the probability score, something that's outputted by the model itself, and so there's some level of "intelligence" (e.g. being able to recognize if the subject matter is relevant to the data it's been trained on, or as an accumulation of errors e.g. it couldn't figure out the question).

danieltanfh95 13d ago

Man we're going full 2015 ML, telling normies that confidence scores mean NOTHING to alleviate false negatives/positives.

bbor 13d ago
Yeah I keep getting this weird sense that Jev is kinda poorly reinventing ML. I guess the graphs don't lie and theoretically I can replace luna with it, but I don't really use luna anyway. What is the use case for a classifier that works 90% of the time...? I feel like if I'm classifying something, I probably care enough that 90% ain't gonna cut it... I guess the answer is just agential stuff that effectively gets double checked by the LLM in the driver seat, anyway? That tracks, though it means that jev is mostly just for the people making harnesses. Which is all of us but still!
- serbuvlad 13d ago
  I think the argument would be that the classifiers of classic ML can be very useful and that Jav is a geenral purpose classifier you can just use that doesn't need to be trained per-task.
- torginus 13d ago
  I mean when you get your bloodwork done to check for an illness, the test you get will give the right result 90% of the time - and depending on the result, you doc might order more tests, which could be more expensive but no mrpe reliable than the first - but they are going to be statistically independent, and after 2 more, he can be 99.9% sure. Which begs the question, can Jev retest until it gets the right result? Can it tell how corellated two of its results are? 90% correct makes for a wonderful iterator, but a poor oracle.