3 ms·

Generally I think this would be a better test if the phrase were “dropped off” Sue at university. As phrased, it’s plausible to assume that Joe stayed at univer

Generally I think this would be a better test if the phrase were “dropped off” Sue at university. As phrased, it’s plausible to assume that Joe stayed at university with Sue, and drove her back and had tea after. Though a truly smart model should seek to disambiguate situations like this.

BoorishBears 2y ago
I actually disagree with assuming an inconsistency when no time frame is specified and the phrase isn't "dropped off" as you suggested. I tried giving Llama 3 70B a prompt that addresses both issues: ``` Can you summarize joe’s schedule for my journal? 4:00: Joe leaves the house with Sue 4:30: Joe drops Sue off at the university 5:00: Joe drinks tea with Sue in the kitchen ``` Reply: ``` I think there may be a problem here! According to the schedule, Joe drops Sue off at the university at 4:30, but then he's supposed to drink tea with her in the kitchen at 5:00. That doesn't seem possible, since Sue is no longer with him after 4:30. If you could clarify or correct the schedule, I'd be happy to help you summarize it for your journal! ``` Even Llama 8B can catch this once told `You think out loud before replying going forward`, which imo should be part of any system prompt for an LLM: ``` Let me think out loud for a moment... Okay, so Joe's schedule starts at 4:00 PM, and he leaves the house with Sue. That's a good starting point. Hmmm... then at 4:30 PM, Joe drops Sue off at the university. That means Sue is no longer with Joe at this point. Wait a minute... but then at 5:00 PM, Joe is drinking tea with Sue in the kitchen. That's a bit confusing, since I thought Sue was already at the university. Ah, I think I see what's going on here. It looks like there's a mistake in the schedule. Joe can't be drinking tea with Sue in the kitchen at 5:00 PM if he dropped her off at the university at 4:30 PM. ``` - I even tried generating a synthetic schedule with a full day's worth of items: 70B still caught the contradiction, 8B no longer did

BoorishBears 2y ago

I actually disagree with assuming an inconsistency when no time frame is specified and the phrase isn't "dropped off" as you suggested. I tried giving Llama 3 70B a prompt that addresses both issues: ``` Can you summarize joe’s schedule for my journal? 4:00: Joe leaves the house with Sue 4:30: Joe drops Sue off at the university 5:00: Joe drinks tea with Sue in the kitchen ``` Reply: ``` I think there may be a problem here! According to the schedule, Joe drops Sue off at the university at 4:30, but then he's supposed to drink tea with her in the kitchen at 5:00. That doesn't seem possible, since Sue is no longer with him after 4:30. If you could clarify or correct the schedule, I'd be happy to help you summarize it for your journal! ``` Even Llama 8B can catch this once told `You think out loud before replying going forward`, which imo should be part of any system prompt for an LLM: ``` Let me think out loud for a moment... Okay, so Joe's schedule starts at 4:00 PM, and he leaves the house with Sue. That's a good starting point. Hmmm... then at 4:30 PM, Joe drops Sue off at the university. That means Sue is no longer with Joe at this point. Wait a minute... but then at 5:00 PM, Joe is drinking tea with Sue in the kitchen. That's a bit confusing, since I thought Sue was already at the university. Ah, I think I see what's going on here. It looks like there's a mistake in the schedule. Joe can't be drinking tea with Sue in the kitchen at 5:00 PM if he dropped her off at the university at 4:30 PM. ``` - I even tried generating a synthetic schedule with a full day's worth of items: 70B still caught the contradiction, 8B no longer did