3 ms·
> I read through your entire article and the three main points I took away from it were also contained in the gpt4o summary I then generated to compare afterwar
by Terr_ 2y ago
> I read through your entire article and the three main points I took away from it were also contained in the gpt4o summary I then generated to compare afterwards. So here's some empirical counter evidence.
But evidence of what, precisely? How do we measure failure and what failure rate is sufficient?
As is, would you be comfortable with doctors applying it to all of your medical records?
- Closi 2y agoThe failure rate that is “sufficient” will obviously change based on the use case. No, I might not want AI summarising my medical records right now, but I might be quite happy for it to summarise a blog post for me.
- ianbutler 2y agoEvidence it can summarize something correctly. Medical records are not my bar though, nor would I be comfortable giving my medical records to 99% of the human race. The bar these tools need to meet for me is far more innocuous (code and apparently summarizing fluff articles) and two things can happen at the same time: 1. Models, and not to be understated, the engineering around them can improve until they pass the point we are comfortable using them in high stakes situations like medical records. 2. People's expectations come down until we settle on the agreed upon tasks where LLMs provide real value. My perspective is 1 will happen in the long term and 2 is what we should be focusing on right now to provide real immediate value with an eye on 1 so we reap the continued improvements. For 2 calibrating people's expectations are going to be messy and people rarely estimate the utility of something correctly in the early days, either over or under which is the reason for the mess. LLMs have been in the works for almost a decade (building on research that's been around since at least the 70s), but their utility has only been apparent for about two years. We're still super early days for that settling period.