6 ms·
I just tried this, and GPT-4 said: Based on the information provided, Mable was alive at noon. Her heart rate was recorded at 9 AM, and her blood pressure was
by rytill 3y ago
I just tried this, and GPT-4 said:
Based on the information provided, Mable was alive at noon. Her heart rate was recorded at 9 AM, and her blood pressure was recorded at 7 PM, both of which are signs of life. She was reported to have died at 11 PM, so she would have been alive at noon.
I checked five times and it answered that Mable was alive at noon all five times.
- H8crilA 3y agoThe people that write such "reviews" of ChatGPT must have no respect for their careers. To include statements that can be disproven in 30 seconds is ... something else.
- croes 3y agoHow do you know they didn't change the model?
- not2b 3y agoDo you know whether you are using the same version of GPT-4 as the authors of the article did?
- rytill 3y agoI don’t, but for most HN readers, the latest user-facing model’s capabilities are most relevant.
- lukeinator42 3y agohaha, I just asked a quantized version of llama 2 13b model using llama.cpp the same question and this is what it said the first time: "Answer: No, Mable was not alive at noon. Her heart rate at 9 AM was 75 bpm, which suggests that she was already experiencing some medical issues. By 7 PM, her blood pressure had risen to 120/80, which is a concerning sign for hypertension. Taken together with the fact that she died at 11 PM, it is likely that Mable was not alive at noon. [end of text]" and a second time it said: "Solution: Mable's heart rate at 9 AM was 75 bpm, which means that it was not abnormal at that time. At 7 PM, her blood pressure was 120/80, which is also within the normal range. Therefore, based on these two pieces of information, Mable was alive at noon. Note: This problem is an example of a "living or dead" problem, where we are given two or more pieces of information about a person's physical state and must determine if they are alive or dead based on those facts. These types of problems can be tricky, as the information may not be directly related to the person's vital signs, but rather to other aspects of their health or well-being. [end of text]" If I got the second answer the first time I'd definitely be impressed. A paper like this should probably run the tests a bunch of times though to quantify how badly these networks "can't reason".
- rytill 3y agoWhich quantized version?
- lukeinator42 3y agothe llama-2-13b-chat.ggmlv3.q4_0.bin version from here: https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML
- danielbln 3y agoHere is the output of llama2-7B-uncensored (a fun model to play with, btw.): 1st try: > Yes, she was alive at noon as her blood pressure and heart rate were within normal range. 2nd try: > No, Mable was not alive at noon as she died 4 hours before that. Her blood pressure and heart rate were both normal at 120/80 mmHg and 75 bpm respectively during the time when her death occurred between 11 PM to 12 AM. 3rd try: > If Mable's heart rate was 75 bpm at 9 AM and her blood pressure was 120/80 mmHg at 7 PM, then it is likely that she was alive at noon.
- glerk 3y agoMine also correctly answered that she was alive at noon, but added his standard infuriating disclaimer in the last sentence: > However, always remember that real-life medical situations can be complex, and this is a simple logical deduction based on the data provided. If this is a fictional or hypothetical scenario, then the above assumption is valid. If this is a real-life situation or requires medical expertise, it would be essential to consult with professionals or have more comprehensive data to determine the facts.
- nerevarthelame 3y agoI don't think it's infuriating that OpenAI tries to discourage people from getting medical advice from an LLM that was not designed to provide it.
- glerk 3y agoI just wish there was a “treat me like an adult” button.