3 ms·
> Astra for Law passed the evaluation’s overall correctness check on 54.0% of questions. How is this a product that you are selling?
by CobrastanJorji 8d ago
> Astra for Law passed the evaluation’s overall correctness check on 54.0% of questions.
How is this a product that you are selling?
- prodigycorp 8d agoHelp me understand your consternation.
- fragmede 8d ago54.0% isn't particularly high.
- prodigycorp 8d agoWhat I'm trying to say which doesnt seem obvious is that all models are x% correct at benchmarks until they get saturated, and then new benchmarks get made.
- jambalaya8 8d agowaiting for the actual Prodigy Corp. to show up and offer to buy your username out? <my own consternation, stated>
- calyhre 8d agoIt’s barely any better than a coin flip
- CobrastanJorji 6d agoWhat is a possible use case for a legal reasoning tool that is wrong 46% of the time?
- noisy_boy 8d agoOn the promise of potential cost savings by means of firing and/or not hiring as much.