4 ms·
Yeah, my thoughts exactly. If you're running 500k in tokens through through someone else's hallucination-prone computer and paying for the privilege, I want to
by specproc 3y ago
Yeah, my thoughts exactly. If you're running 500k in tokens through through someone else's hallucination-prone computer and paying for the privilege, I want to know why that's any better than something like SetFit.
All I saw were attempts to reproduce some chatgpt output.
- espe 3y ago+1 for setfit. a baseline that's hard to beat.
- hellovai 3y agoSetFit is fairly good, and we do help train SetFit like models for the results you get, however, the issue with SetFit is that its latency and cost benefits come at the price of flexibility. If you want to add a new class, to update an existing one, it requires training a new model. Sometimes this is ok and sometimes it's not. This is why we generally prefer a hybrid approach where some classes are using traditional models (BERT based) while others are determined by the LLM.
- specproc 3y agoI guess use case is everything. There are numerous reasons, not least of which confidentiality, why chatgpt is a no go for me. What I'd like to see more of is systematic comparison between chatgpt and classic models. I was hoping to see a bit of this in this article and was disappointed.
- hellovai 3y agoI appreciate the feedback, and also agree that chatgpt is a no-go for many use cases. We're working on putting together a better comparison specifically along the lines of accuracy between the LLMs (chatgpt, bard, falcon) and also traditional models. Hope that one hits the spot for you! Are their specific metrics you think might be interesting? We were primarily looking at f1/accuracy for this task, but also attempting to see what types of classes they work well in using semantic similarity.
- famouswaffles 3y agohttps://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921