3 ms·

Still lots of work to do to catch up, I feel. Falcon 40B: > I've got two cats named Buttons and Jackson. They both like to wear cute little ties. Buttons wear

by mabbo 3y ago

Still lots of work to do to catch up, I feel. Falcon 40B: > I've got two cats named Buttons and Jackson. They both like to wear cute little ties. Buttons wears a red tie. My other cat wears a blue tie. What is the name of the cat that wears a blue tie? > I'm sorry, but I do not have enough information to answer your question. Could you please provide me with more context or information? ChatGPT: > I've got two cats named Buttons and Jackson. They both like to wear cute little ties. Buttons wears a red tie. My other cat wears a blue tie. What is the name of the cat that wears a blue tie? > The cat that wears a blue tie is named Jackson.

CamperBob2 3y ago
The cat that wears a blue tie is named Jackson. "B...b...b...but it's just autocomplete." -- HN
- ithkuil 3y ago
  "Autoconplete is all you need"
  - CamperBob2 3y ago
    More like, "Autocomplete is all you are."
Aliabid94 3y ago
It correctly answered `The name of the cat that wears a blue tie is Jackson.` for me
orost 3y ago
You have to turn down temperature and/or p when you want accuracy. Otherwise you don't know if the model's read is bad, or if you just happened to get a low-probability outlier. With the playground's defaults of t=0.8, p=0.9, I got the right answer 7/10 times. With t=0.1, p=0.1, it's always exactly "The name of the cat that wears a blue tie is Jackson."
druskacik 3y ago
It's actually better in some of these simple logic tasks than ChatGPT. Try this: Prompt: When I was born, my biological mother was in Paris, and my father was in the United States. Where was I born? GPT-4: Based on the information you've provided, I can't definitively say where you were born. Your location of birth depends on various factors, such as where your mother was living around the time of your birth, or if she was traveling. If you were born when your mother was in Paris, then you were likely born in Paris. However, if she traveled to the United States or another location for your birth, you would have been born there. Additional information would be needed to determine your birth location more accurately. Falcon40B: You were born in Paris. Your biological mother was in Paris when she gave birth to you. This kind of evaluation is meaningless, it would be necessary to test the models on a bunch of different prompts to get some meaningful metrics.

by mabbo 3y ago

CamperBob2 3y ago

The cat that wears a blue tie is named Jackson. "B...b...b...but it's just autocomplete." -- HN

ithkuil 3y ago
"Autoconplete is all you need"
- CamperBob2 3y ago
  More like, "Autocomplete is all you are."

Aliabid94 3y ago

It correctly answered `The name of the cat that wears a blue tie is Jackson.` for me

orost 3y ago

You have to turn down temperature and/or p when you want accuracy. Otherwise you don't know if the model's read is bad, or if you just happened to get a low-probability outlier. With the playground's defaults of t=0.8, p=0.9, I got the right answer 7/10 times. With t=0.1, p=0.1, it's always exactly "The name of the cat that wears a blue tie is Jackson."

druskacik 3y ago

It's actually better in some of these simple logic tasks than ChatGPT. Try this: Prompt: When I was born, my biological mother was in Paris, and my father was in the United States. Where was I born? GPT-4: Based on the information you've provided, I can't definitively say where you were born. Your location of birth depends on various factors, such as where your mother was living around the time of your birth, or if she was traveling. If you were born when your mother was in Paris, then you were likely born in Paris. However, if she traveled to the United States or another location for your birth, you would have been born there. Additional information would be needed to determine your birth location more accurately. Falcon40B: You were born in Paris. Your biological mother was in Paris when she gave birth to you. This kind of evaluation is meaningless, it would be necessary to test the models on a bunch of different prompts to get some meaningful metrics.