5 ms·
I don't know. This paper [1] reports accuracies in the 97-98% range on a similar task with more powerful models. With Gemma 2 2b the accuracy will certainly be
by kaspermarstal 2y ago
I don't know. This paper [1] reports accuracies in the 97-98% range on a similar task with more powerful models. With Gemma 2 2b the accuracy will certainly be lower.
[1] https://www.medrxiv.org/content/10.1101/2024.10.01.24314702v1 https://www.medrxiv.org/content/10.1101/2024.10.01.24314702v...
- indolering 2y agoY'all definitely need to cross validate a small number of samples by hand. When I did this kind of research, I would hand validate to at least P < .01.
- kaspermarstal 2y agoShe and one other researcher has manually classified all 7000 papers as per standard protocol. Perhaps for the next article they will measure how this tool agreed with them against them and include it in the protocol if good enough.
- beernet 2y ago> I don't know. HN in a nutshell: I've built some cool tech but have no idea if it is helpful or even counter productive...
- rasmus1610 2y agoSometimes people just like to build stuff for the sake of it.
- jajko 2y agoAlmost like hackers, doing shit just for the heck of it because they can (mostly)
- corobo 2y agoReal HN in a nutshell: People who don't build stuff telling people who do build stuff that the thing they built is useless :P It's a hacker forum, let people hack! If anything have a dig at OP for posting the thread too soon before the parent commenter has had the chance to gather any data, haha
- greenavocado 2y agoJust because you can, doesn't mean you should
- corobo 2y agoIf you're building a dinosaur sanctuary sure
- stackghost 2y agoOr an Internet surveillance-capitalism panopticon.
- Breza 2y agoGreat attitude! I recently built a tool for my wife that uses an LLM to automate a task. Is it production ready? Definitely not. But it saves her time even in its current state.
- sidcool 2y agoSometimes it's the joy of creation. Utility and optimization come later. It's fun. Like a hobby.
- kaspermarstal 2y agoI am not going to claim or report any kind of accuracy, especially with such a small model and such a specific, context dependent use case. It is the user’s responsibility to cross validate if it’s accurate enough for their use case and upgrade model or use another approach if not.
- jbs789 2y agoA user buys a car because it gets them from point A to point B. I get what you’re saying though - we are earlier along the adoption curve for these models and more responsibility sits with the user. Over time the expectations will no doubt increase.
- dzamo_norton 2y agoOffer a 100% money back guarantee if the user finds that the software is not fit for purpose :)