3 ms·
How do you "just" accurately evaluate the error state space of an LLM relative to a real business process? Sounds approximately impossible to me. If you alread
by pphysch 3y ago
How do you "just" accurately evaluate the error state space of an LLM relative to a real business process? Sounds approximately impossible to me.
If you already have the business process robustly defined as code, then the utility of LLM is unclear. The value prop of LLM is in fuzzy business processes like parsing arbitrary helpdesk tickets.
- lukasb 3y agoYou evaluate it the way we've evaluated production ML for years, with cheap QC layers sampled and checked by more expensive layers (with humans on top.) LLMs didn't invent stochastic process steps.