2 ms·

Can you elaborate? I think you're right that LLMs are very good at classification and structuring, but that's not how I see them used. The experiments I have w

by bambax 3y ago

Can you elaborate? I think you're right that LLMs are very good at classification and structuring, but that's not how I see them used. The experiments I have witnessed resemble "hail Mary" attempts to replace human agents with a magic chatbot that would know it all and speak plainly. Those experiments almost always fail, because the bot hallucinates too much and/or because the training data is poor and not detailed enough to provide efficient fine-tuning. For example: a helpdesk handles incoming support requests for an internal, proprietary application. Support agents write the request down, then spend time investigating the problem and possible solutions, based on the symptoms. There is a weak link between the symptoms reported by the users and the actual underlying problem, so this takes time, esp. for newly hired support agents. One approach would be to build a semantic search engine to speed up the investigation phase, which would positively affect the productivity of the agents. But that's not what's been tried: the company hoped to get rid of support agents completely, and replace them with a bot that would interact with the users and solve their problems on the fly. When that doesn't work, everyone's disappointed and tends to simply give up.

d4rkp4ttern 3y ago
I agree with the approach of augment-not-replace human agents. For example there’s a contact center management company using Langroid[1] (the Multi-Agent LLM framework from ex-CMU/UW-Madison researchers) in exactly this way: they use a Langroid RAG agent to come up with top 3 possible answers, and the human agent then picks a good one. Interestingly, they index past successful conversations of experienced human agents, and the Langroid agent’s suggestions are informed from these as well as company docs. [1] https://GitHub.com/Langroid/Langroid https://GitHub.com/Langroid/Langroid
fnordpiglet 3y ago
So, imagine you need to evaluate applications for some line of credit by reading the documents associated and classifying the businesses in a taxonomy of business types. Depending on volume this could be millions of documents. You can prime an LLM context with the taxonomy and examples of each, then evaluate each document in a clean primed context asking for a classification if it’s clear and a request for review if it’s not clear. This works exceptionally well in practice, almost double the precision and recall of humans. Businesses are replete with similar relatively trivial classification problems that often require armies of low paid low skill workers but are fairly crucial to the ongoing operations of the business. Chat bots are the default thing people gravitate to with LLMs because they’re the example in chatgpt. But that’s a really uncreative use of the tech, and the domain they’re exposed to is pretty open ended leaving them subject to hallucination, prompt injection, etc. A more constrained domain with more business impact is where I see meaningful applications happening.

by bambax 3y ago

d4rkp4ttern 3y ago

I agree with the approach of augment-not-replace human agents. For example there’s a contact center management company using Langroid[1] (the Multi-Agent LLM framework from ex-CMU/UW-Madison researchers) in exactly this way: they use a Langroid RAG agent to come up with top 3 possible answers, and the human agent then picks a good one. Interestingly, they index past successful conversations of experienced human agents, and the Langroid agent’s suggestions are informed from these as well as company docs. [1] https://GitHub.com/Langroid/Langroid https://GitHub.com/Langroid/Langroid

fnordpiglet 3y ago

So, imagine you need to evaluate applications for some line of credit by reading the documents associated and classifying the businesses in a taxonomy of business types. Depending on volume this could be millions of documents. You can prime an LLM context with the taxonomy and examples of each, then evaluate each document in a clean primed context asking for a classification if it’s clear and a request for review if it’s not clear. This works exceptionally well in practice, almost double the precision and recall of humans. Businesses are replete with similar relatively trivial classification problems that often require armies of low paid low skill workers but are fairly crucial to the ongoing operations of the business. Chat bots are the default thing people gravitate to with LLMs because they’re the example in chatgpt. But that’s a really uncreative use of the tech, and the domain they’re exposed to is pretty open ended leaving them subject to hallucination, prompt injection, etc. A more constrained domain with more business impact is where I see meaningful applications happening.