3 ms·

Having used GPT 4 for a while now I would say I trust its factual accuracy more than the average human you'd talk to on the street. The sheer volume of things w

Having used GPT 4 for a while now I would say I trust its factual accuracy more than the average human you'd talk to on the street. The sheer volume of things we make up on a daily basis through no malice of our own but bad memory and wrong associations is just astounding. That said, fact checking is still very much needed. Once someone figures out how to streamline and automate that process it'll be on Google's level of general reliability.

crabmusket 4y ago
It's interesting to imagine an "average human on the street" being put in the position that an LLM is in. If you asked someone on the street to summarise a page of React Native documentation for you, they'd probably look at you strangely and say "sorry I have a bus to catch, good luck!" But if you had someone in a room, sat them down with a pen and paper and instructions along the lines of: - answer the question on the page without diversion - do not ask for additional clarifying information - provide your answer in 3-5 paragraphs with a summary and didn't let them leave until they'd completed the task, then .. I'm sure the results would be "worse", and much more variable, than GPT output, while being conceptually similar. Of course, you don't ask a random person on the street to do this, probably because of social constraint, inconvenience, and the expectation that they will not do a better or faster job than you will yourself. But if you were stuck with a stranger, began to converse, and asked them e.g. "should I get a divorce?" I imagine the conversation would go very differently. If they weren't using their agency to end the conversation as quickly as possible, I imagine there'd be back and forth asking for more detail, exploring the context, sharing their own experience, etc. I'm rambling, but I just find it very diverting to compare not the output of LLMs versus the output of humans, but the context and situation.
- moffkalast 4y ago
  Indeed LLMs are currently better at some things and worse at others than people, but I was comparing in terms of accuracy of provided info (assuming they know the answer to whatever question was posed). An equal footing example would be something like seeing a statue of someone known and trying to remember when they were born (or what exactly they did) when the last time you heard about them was in high school years ago. Or like on which address is some restaurant, or to which pressure your bike or car tyres need inflating to. The type of data that both humans and LLMs are kind of bad at replicating perfectly. Also in terms of math I'd also definitely trust GPT 4 quite a bit and 3.5 not at all. For humans, well, it's definitely a range from one to the other.

crabmusket 4y ago

It's interesting to imagine an "average human on the street" being put in the position that an LLM is in. If you asked someone on the street to summarise a page of React Native documentation for you, they'd probably look at you strangely and say "sorry I have a bus to catch, good luck!" But if you had someone in a room, sat them down with a pen and paper and instructions along the lines of: - answer the question on the page without diversion - do not ask for additional clarifying information - provide your answer in 3-5 paragraphs with a summary and didn't let them leave until they'd completed the task, then .. I'm sure the results would be "worse", and much more variable, than GPT output, while being conceptually similar. Of course, you don't ask a random person on the street to do this, probably because of social constraint, inconvenience, and the expectation that they will not do a better or faster job than you will yourself. But if you were stuck with a stranger, began to converse, and asked them e.g. "should I get a divorce?" I imagine the conversation would go very differently. If they weren't using their agency to end the conversation as quickly as possible, I imagine there'd be back and forth asking for more detail, exploring the context, sharing their own experience, etc. I'm rambling, but I just find it very diverting to compare not the output of LLMs versus the output of humans, but the context and situation.

moffkalast 4y ago
Indeed LLMs are currently better at some things and worse at others than people, but I was comparing in terms of accuracy of provided info (assuming they know the answer to whatever question was posed). An equal footing example would be something like seeing a statue of someone known and trying to remember when they were born (or what exactly they did) when the last time you heard about them was in high school years ago. Or like on which address is some restaurant, or to which pressure your bike or car tyres need inflating to. The type of data that both humans and LLMs are kind of bad at replicating perfectly. Also in terms of math I'd also definitely trust GPT 4 quite a bit and 3.5 not at all. For humans, well, it's definitely a range from one to the other.