2 ms·

That is impressive at first glance, no question. To stay with the example of the bookshelf, you would only follow this path for several or very many books, as i

That is impressive at first glance, no question. To stay with the example of the bookshelf, you would only follow this path for several or very many books, as in the example with the cookbooks. I have no idea how good the Geminis or GPTs of this world currently are, but let's optimistically assume a 3% error rate due to hallucinations or something. If I want to be sure that the results are correct, then I have to go through and check each entry manually. I want to rule out the possibility that there are titles listed in the 3% that would completely turn an outsider's world view of me upside down. So, even if data entry is incredibly fast, curation is still time-consuming. On balance, would it even be faster to capture the ISBN code of 100 books with a scanner app, assuming that the index lookup is correct, or to compare 100 JSON objects with title and author for correctness? The example is only partly serious. I just think that as long as hallucinations occur, Generative AI will only get part of my trust - and I don't know about you, but if I knew that a person was outright lying to me in 3% of all his statements, I wouldn't necessarily seek his proximity in things that are important to me...

jpc0 3y ago
This right here. I'm currently building out some code that should go in production in the next week or two and simply because of this we are using LLM to prefill data and then have a human look over it. For our use case the LLM prefilling the data is significantly faster but if it ever gets to the point of that not needing to happen it would take a task whichtakes about 3 hours ( now down to one hour ) and make it a task that takes 3 minutes. Will LLMs ever get to the point where it is perfectly reliable ( or at least with an error margin low enough for our use case ), I don't think so. It does make for a very cheap accelerator though.
simonw 3y ago
This isn't a problem that's unique to LLMs though. Pay a bunch of people to go through and index your book collection and you'll get some errors too. What's interesting about LLMs is they take tasks that were previously impossible - I'm not going to index my book collection, I do not have the time or willpower to do that - and turned them into things that I can get done to a high but not perfect standard of accuracy. I'll take a searchable index of my books that's 95% accurate over no searchable index at all.

jpc0 3y ago

This right here. I'm currently building out some code that should go in production in the next week or two and simply because of this we are using LLM to prefill data and then have a human look over it. For our use case the LLM prefilling the data is significantly faster but if it ever gets to the point of that not needing to happen it would take a task whichtakes about 3 hours ( now down to one hour ) and make it a task that takes 3 minutes. Will LLMs ever get to the point where it is perfectly reliable ( or at least with an error margin low enough for our use case ), I don't think so. It does make for a very cheap accelerator though.

simonw 3y ago

This isn't a problem that's unique to LLMs though. Pay a bunch of people to go through and index your book collection and you'll get some errors too. What's interesting about LLMs is they take tasks that were previously impossible - I'm not going to index my book collection, I do not have the time or willpower to do that - and turned them into things that I can get done to a high but not perfect standard of accuracy. I'll take a searchable index of my books that's 95% accurate over no searchable index at all.