4 ms·

I think tests like "can this LLM pass an English literature exam it's never seen before" are probably useful, but yeah there's a lot of silly stuff like math te

by TillE 3y ago

I think tests like "can this LLM pass an English literature exam it's never seen before" are probably useful, but yeah there's a lot of silly stuff like math tests. I suppose the question is where are they most commercially viable. I've found them fantastic for creative brainstorming, but that's sort of hard to test and maybe not a huge market.

TuringNYC 3y ago
>> I suppose the question is where are they most commercially viable. Fair point, though I'm not aiming to start a competing LLM SaaS service, rather i'm evaluating swapping out the TCO of Azure Cognitive Service OpenAI for the TCO of dedicated cloud compute running my own LLM -- to serve my own LLM calls currently being sent to a metered service (Azure Cognitive Service OpenAI) Evaluation points would be: output quality; meter vs fixed breakeven points; latency; cost of human labor to maintain/upgrade in most cases, i'd outsource and not think about it. BUT we're currently in some strange economics where the costs are off the charts for some services