4 ms·
In my day job we tried creating a credit assessor tool using LLM as the credit assessor. It did great, generated a report on the assessed business that was inc
by marcus_holmes 4mo ago
In my day job we tried creating a credit assessor tool using LLM as the credit assessor.
It did great, generated a report on the assessed business that was incredibly detailed and plausible.
Then I started running tests and getting into the details, and found that if you ran the same report on the same data, it generated completely different, still very plausible, results. I could run the same source data through the assessment process 10 times and get 10 very different results. We had to can the project and go a different route.
LLMs are designed to produce plausible results, not factual results. We can fix this when using them for software dev by using linters and tests (though we've all had the experience where the LLM invents an API endpoint). I would not trust raw LLM output in any situation where that kind of testing and verification capability isn't present.
- adamddev1 4mo agoLinters and tests help of course, but they cannot "fix" the problem since tests cannot prove the absence of bugs.
- marcus_holmes 4mo agoagree, and I think we'll see more use of formal methods with LLMs for this reason
- ryukoposting 4mo agoAt a certain point it just feels like we're reinventing the concept of programming languages from first principles.
- ncruces 4mo agoI'm using this new programming language: it's called LLM prompting, and everything is undefined behavior.
- marcus_holmes 4mo agoHardly the first time we've done this - we had to do it with compilers too
- Suppafly 4mo agoWhat's crazy is that there are ton of businesses building processes around LLMs that haven't done this exercise and fully believe the LLM is giving them accurate data.
- xbmcuser 4mo agoYup I use llm to write scripts for me to process data I don't ask the llm to process the data themselves. Even when I wrote something for my day trading I used llm write scripts that do all the processing and predict price movement from that the more data is pre processed the more all the llm come up with similar trades.
- yubblegum 4mo ago> LLMs are designed to produce plausible results, not factual results. They are true to their name: Language models. It is precisely the same problem in a language: a grammatically correct sentence is not necessarily true.
- bondarchuk 4mo agoIt's funny that if the LLMs had all given the same result each time (it sounds like) you would have considered it more valid, even though it might just be giving a single wrong answer more consistently.
- deleted 4mo ago[deleted]
- coderatlarge 4mo agosurely getting the same answer multiple ways suggests it is nore likely in the probability space?
- ShinyLeftPad 4mo agoSlavery used to be super likely in the probability space, doesn't make it right...
- coderatlarge 3mo agoif two llms independently cite the same law given a certain factual legal question then it generally makes it more likely that the law cited does, in fact, apply to the fact pattern in question ; i think you’re using “right” in a moral sense irrelevant to the comment i was responding to.
- ShinyLeftPad 3mo agoBut that's the thing, legal stuff is not fully divorced from moral stuff and those things evolve with our morals
- coderatlarge 3mo agoyour follow-up makes no difference to the main point under discussion : if you ask two different llms about a legal fact pattern and they independently reference the same law, then it is, in fact, more likely that the law is more relevant to the fact pattern than if only one llm mentioned it.
- greenail 4mo agoyou can set the "temperature" which is a lever on how stochastic the prediction is. If you are doing your own inference this is clear and easy. If you are consuming tokens this is outsourced.