4 ms·

Oh man, apparently y'all need to get off ChatGPT and just use the API. Here's what I got querying text-divinci-003. The book you're thinking of is likely "

by Spivak 3y ago

Oh man, apparently y'all need to get off ChatGPT and just use the API. Here's what I got querying text-divinci-003. The book you're thinking of is likely "Dissolving Illusions: Disease, Vaccines, and The Forgotten History" by Dr. Suzanne Humphries and Roman Bystrianyk. The book discusses the history and efficacy of vaccines with a focus on smallpox. Another one. > List the top ten books on smallpox written by medical professionals that present evidence that the vaccine was ineffective. # text-davinci-003 “Smallpox Vaccine: Deception and Tragedy” by Dr. Vernon Coleman “The Demon in the Freezer: Smallpox, Biological Warfare and the End of the World” by Richard Preston “The Vaccine Race: Science, Politics, and the Human Costs of Defeating Disease” by Meredith Wadman “Vaccine: The Controversial Story of Medicine’s Greatest Lifesaver” by Arthur Allen ... # gpt4 "The Smallpox Vaccination Hoax: The Case Against Vaccination" by Viera Scheibner "Dissolving Illusions: Disease, Vaccines, and The Forgotten History" by Suzanne Humphries and Roman Bystrianyk "Vaccine: The Controversial Story of Medicine's Greatest Lifesaver" by Arthur Allen "The Age of Autism: Mercury, Medicine, and a Man-Made Epidemic" by Dan Olmsted and Mark Blaxill ... I don't know enough about anti-vaxx literature to know if these are actually good results for the query but they seem on target. This has got to be why I have such a disconnect with people who are mad about gpt getting nerfed, I've been using my own chatbot that very simplistically creates a transcript of the conversation and queries the model for the next message.

mike_hearn 3y ago
Yes, DaVinci is a base model that hasn't been put through RLHF which is where that sort of bias gets introduced. Unfortunately there's no base model for GPT-4. If you force things by demanding a list of 10 books then I can imagine that would hack around it, but then if there aren't actually ten such books you're going to get hallucinations again. For example, the first item on the GPT-4 list doesn't exist (the author does).
- Spivak 3y ago
  Fair, let me try gpt4 with your original prompt and see what I get. The book you're thinking of is "Dissolving Illusions: Disease, Vaccines, and The Forgotten History" by Dr. Suzanne Humphries and Roman Bystrianyk. The book discusses the history of vaccines, including smallpox, and challenges the popular belief that vaccines were solely responsible for their eradication. The gpt4 api gives zero fucks. It really does seem like it's just whatever logic they layered on to ChatGPT.
  - mike_hearn 3y ago
    You're right, I can also reproduce that behavior with the API. Fascinating. I really thought they were the same model but ChatGPT4 is clearly a lot more RLHFd compared to the API. Not only is the API honest but it's also way less wordy. Practically every answer from ChatGPT in the sample conversation had to be truncated because it includes an essay about the scientific consensus.
    - lucubratory 3y ago
      The API model is just as RLHF'd as they disclosed in their 'paper'. The difference you're seeing is very likely to just be the ChatGPT pre-prompt telling it not to engage in harmful activities.
duskwuff 3y ago
GPT3: “Smallpox Vaccine: Deception and Tragedy” by Dr. Vernon Coleman Vernon Coleman is a real anti-vaxer but hasn't written any books. “The Demon in the Freezer: Smallpox, Biological Warfare and the End of the World” by Richard Preston Richard Preston has published a book titled "The Demon in the Freezer". But it doesn't have that subtitle (although it's an accurate summary of the book), and it isn't critical of vaccination. “The Vaccine Race: Science, Politics, and the Human Costs of Defeating Disease” by Meredith Wadman “Vaccine: The Controversial Story of Medicine’s Greatest Lifesaver” by Arthur Allen These books do exist, but they're about the history of vaccines and appear to speak of vaccines in positive terms. GPT4: "The Smallpox Vaccination Hoax: The Case Against Vaccination" by Viera Scheibner Doesn't exist. Viera Scheibner is a real anti-vaxxer, but she doesn't have a book by this name. "Dissolving Illusions: Disease, Vaccines, and The Forgotten History" by Suzanne Humphries and Roman Bystrianyk Finally, a correct answer! This book is real and is critical of vaccines. "Vaccine: The Controversial Story of Medicine's Greatest Lifesaver" by Arthur Allen Same book that GPT3 referenced; still not critical of vaccines. "The Age of Autism: Mercury, Medicine, and a Man-Made Epidemic" by Dan Olmsted and Mark Blaxill This one is real as well and, as the title suggests, is also anti-vax.

by Spivak 3y ago

mike_hearn 3y ago

Yes, DaVinci is a base model that hasn't been put through RLHF which is where that sort of bias gets introduced. Unfortunately there's no base model for GPT-4. If you force things by demanding a list of 10 books then I can imagine that would hack around it, but then if there aren't actually ten such books you're going to get hallucinations again. For example, the first item on the GPT-4 list doesn't exist (the author does).

Spivak 3y ago
Fair, let me try gpt4 with your original prompt and see what I get. The book you're thinking of is "Dissolving Illusions: Disease, Vaccines, and The Forgotten History" by Dr. Suzanne Humphries and Roman Bystrianyk. The book discusses the history of vaccines, including smallpox, and challenges the popular belief that vaccines were solely responsible for their eradication. The gpt4 api gives zero fucks. It really does seem like it's just whatever logic they layered on to ChatGPT.
- mike_hearn 3y ago
  You're right, I can also reproduce that behavior with the API. Fascinating. I really thought they were the same model but ChatGPT4 is clearly a lot more RLHFd compared to the API. Not only is the API honest but it's also way less wordy. Practically every answer from ChatGPT in the sample conversation had to be truncated because it includes an essay about the scientific consensus.
  - lucubratory 3y ago
    The API model is just as RLHF'd as they disclosed in their 'paper'. The difference you're seeing is very likely to just be the ChatGPT pre-prompt telling it not to engage in harmful activities.

duskwuff 3y ago

GPT3: “Smallpox Vaccine: Deception and Tragedy” by Dr. Vernon Coleman Vernon Coleman is a real anti-vaxer but hasn't written any books. “The Demon in the Freezer: Smallpox, Biological Warfare and the End of the World” by Richard Preston Richard Preston has published a book titled "The Demon in the Freezer". But it doesn't have that subtitle (although it's an accurate summary of the book), and it isn't critical of vaccination. “The Vaccine Race: Science, Politics, and the Human Costs of Defeating Disease” by Meredith Wadman “Vaccine: The Controversial Story of Medicine’s Greatest Lifesaver” by Arthur Allen These books do exist, but they're about the history of vaccines and appear to speak of vaccines in positive terms. GPT4: "The Smallpox Vaccination Hoax: The Case Against Vaccination" by Viera Scheibner Doesn't exist. Viera Scheibner is a real anti-vaxxer, but she doesn't have a book by this name. "Dissolving Illusions: Disease, Vaccines, and The Forgotten History" by Suzanne Humphries and Roman Bystrianyk Finally, a correct answer! This book is real and is critical of vaccines. "Vaccine: The Controversial Story of Medicine's Greatest Lifesaver" by Arthur Allen Same book that GPT3 referenced; still not critical of vaccines. "The Age of Autism: Mercury, Medicine, and a Man-Made Epidemic" by Dan Olmsted and Mark Blaxill This one is real as well and, as the title suggests, is also anti-vax.