4 ms·
Claude 3 Opus has an issue – Repeats same response over and over
I am using the API version of Claude 3 Opus to simulate patient profiles. I've been trying since the model got released and I keep getting the same patient description even with different combinations temperature, top_k, etc. I assumed it was a launch issue and it would eventually get fixed, but that did not happen.
Is anyone else seeing the same issue ?
```import anthropic
client = anthropic.Anthropic(
# defaults to os.environ.get("ANTHROPIC_API_KEY")
api_key=os.environ["ANTHROPIC_API_KEY"] # Save ANTHROPIC_API_KEY in .env
)
def create_anthropic_message(system_prompt, user_prompt, anthropic_model = "claude-3-opus-20240229", max_tokens = 200, temperature = 0.999, *kwargs):
message = client.messages.create(
model=anthropic_model,
max_tokens=max_tokens,
temperature=temperature,
system=system_prompt,
# Also include any other arguments in the def function kwargs
*kwargs,
messages=[{"role": "user","content": [{"type": "text","text": user_prompt}]}])
return (message)
# Run
for i in range(20):
try:
r = create_anthropic_message("You are an oncology expert","Compose a brief 1-sentence presentation of a US-based patient presenting with Skin Cancer.")
print(r.content[0])
except Exception as error:
# handle the exception
print("An exception occurred:", error)```
Responses --
# Results
ContentBlock(text='A 68-year-old retired Caucasian male from Florida presented with a growing, asymmetric, irregularly bordered, multicolored 8mm lesion on his left forearm that had changed in size and color over the past 6 months.', type='text')
ContentBlock(text='A 62-year-old Caucasian male with a history of extensive sun exposure presents with a rapidly growing, irregularly shaped, darkly pigmented lesion on his left forearm that has changed in size and color over the past 3 months.', type='text')
ContentBlock(text='A 45-year-old fair-skinned male from Florida presents with a suspicious, asymmetric, irregularly bordered, multicolored, and evolving mole on his left upper back that has been growing for the past 3 months.', type='text')
ContentBlock(text='A 62-year-old fair-skinned male from Florida presented with a rapidly growing, asymmetric, irregularly bordered, darkly pigmented lesion on his left shoulder that had changed in size and color over the past 3 months.', type='text')
ContentBlock(text='A 45-year-old Caucasian male construction worker from Florida presents with a rapidly growing, irregularly shaped, ulcerated dark brown lesion on his left forearm that has doubled in size over the past 3 months and is accompanied by swollen lymph nodes in his left axilla.', type='text')
ContentBlock(text='A 62-year-old Caucasian male from Florida presented with a slow-growing, irregularly shaped, dark brown patch on his left shoulder that had changed in size and color over the past 6 months.', type='text')
ContentBlock(text='A 62-year-old Caucasian male from Florida presented with a slowly enlarging, irregularly bordered, pigmented lesion on his left forearm that had changed in color and size over the past 6 months.', type='text')
ContentBlock(text='A 45-year-old Caucasian male construction worker from Florida presents with a rapidly growing, asymmetric, irregularly bordered, multicolored, 8mm diameter lesion on his left upper back that has changed in size and color over the past 3 months.', type='text')
ContentBlock(text='A 62-year-old Caucasian male with a history of extensive sun exposure presents with a rapidly growing, irregularly shaped, ulcerated pigmented lesion measuring 1.5 cm in diameter on his left upper back.', type='text')
[... Same profile with changes in only 1-2 words]
- qrios 3y agoNot using Claude but this behaviour comes not from the model itself. Basically the model does not now what it provided in the other iterations of your for loop. Ask for 20 different profiles. Or, even better: ask for different types of cancers and iterate this list.
- xbsd98 3y agoI have tried this even with all ranges of temperature (0 to 1) ... it produces the SAME profile over and over ... Since I couldn't get to format the code here, I have put it on pastebin -- https://pastebin.com/fCu3Ji6n https://pastebin.com/fCu3Ji6n Any chance someone can try this out ? I am pretty sure it will be the same, something is odd (maybe it is too harmless with patient profile?)
- mikewarot 3y agoI suspect it has something to do with this snippet of code, which sets the amount of randomness in the output via "Temperature"[1] temperature = 0.999 [1] https://www.iguazio.com/glossary/llm-temperature https://www.iguazio.com/glossary/llm-temperature
- xbsd98 3y agoI have tried this even with all ranges of temperature (0 to 1) ... it produces the SAME profile over and over ... Since I couldn't get to format the code here, I have put it on pastebin -- https://pastebin.com/fCu3Ji6n https://pastebin.com/fCu3Ji6n Any chance someone can try this out ? I am pretty sure it will be the same, something is odd (maybe it is too harmless with patient profile?)
- mikewarot 3y agoWhy haven't you tried a temperature over 1? Try 10 and see what happens. Look at the reference page I supplied for a better idea of what this parameter does on any LLM.
- 3y ago