3 ms·

It's not just lawyers who think that ChatGPT is a search engine. I've observed this many times in my vicinity, people from all walks of life think that Star Tre

It's not just lawyers who think that ChatGPT is a search engine. I've observed this many times in my vicinity, people from all walks of life think that Star Trek is here and computers now respond accurately to natural language queries. For non-techies, "just asking the computer" is so much more convenient than translating your question into traditional search queries. So I guarantee you that stuff like this is happening daily across all industries. Depending on the profession, people will lose money or get hurt as a result of someone blindly trusting this technology. I can't prove it, but statistically, that's basically a certainty. In my opinion, you can't overstate the importance of articles like this, which point out the limitations and highlight the dangers. I'm also against banning. But lay people need to be informed what ChatGPT is and is not, and OpenAI won't do it because they want to ride the hype train.

fennecfoxy 3y ago
Idk it's pretty reliable even now if you chuck a vector db of "knowledge" in and inform the GPT in the overall prompt that it is must not go outside the bounds of knowledge provided with the user's query, and that if no hard knowledge is provided that it should respond along the lines of "I don't know" (or search the internet and parse results for them). I imagine at some point this behaviour will be built in. People are treating GPTs like they're knowledge bases, but they're not. The fact that they can answer simple or complex questions correctly is only a byproduct of the transformer being taught how to string language together. It's like an artist who learns to draw by making paintings of raccoons. You can't then ask them "what do raccoons like to eat?" or "what foods are poisonous to a raccoon?" just because they learnt to draw by painting raccoons. This is how people are treating GPTs atm. They believe in it because they ask the artist "what colour fur do raccoons have?" and because the artist can answer that correctly, they assume all other answers are factual.
gpm 3y ago
In local-to-me politics we have a report on changing the admissions process for specialty programs in the public school system - which had a bunch of fake citations and people suspect was written with the "aid" of ChatGPT. https://www.thestar.com/news/gta/2023/05/26/tdsb-fires-researcher-for-allegedly-plagiarizing-parts-of-report-on-specialty-schools.html https://www.thestar.com/news/gta/2023/05/26/tdsb-fires-resea...
Xenoamorphous 3y ago
> It's not just lawyers who think that ChatGPT is a search engine. Let’s not forget that Google often puts incorrect information in their snippets/factboxes or whatever they call them.
deleted 3y ago
[deleted]
cratermoon 3y ago
Computer, what's the formula for transparent aluminum? Seriously, I got ChatGPT to spit out a scientific-seeming paper on the formula and manufacturing process for transparent aluminum. It did note that there's a real thing, aluminum oxynitride, which is the closest thing we have to the Star Trek material. It even wrote the following abstract, based on my prompt: > This scientific description provides an overview of the formula and manufacturing process of transparent aluminum, a material used in applications where both structural strength and transparency are required. Transparent aluminum finds extensive use in diverse fields, including public aquaria, where it allows for the display of large marine organisms. The description outlines the chemical composition, key properties, and the manufacturing steps involved in creating transparent aluminum. Whether or not the six-step manufacturing process it came up with is correct or not, I haven't the expertise to say.
- codeflo 3y ago
  > Whether or not the six-step manufacturing process it came up with is correct or not, I haven't the expertise to say. I mean, you can try it and perhaps get rich. Or, probably more likely, die in a horrible explosion.
  - cratermoon 3y ago
    [the material is] placed in a high-pressure vessel, and temperature and pressure are simultaneously increased to near the melting point of aluminum. I don't need to try that to know that molten metal under high pressure at over 1000C is something I'm not ready to deal with.

fennecfoxy 3y ago

Idk it's pretty reliable even now if you chuck a vector db of "knowledge" in and inform the GPT in the overall prompt that it is must not go outside the bounds of knowledge provided with the user's query, and that if no hard knowledge is provided that it should respond along the lines of "I don't know" (or search the internet and parse results for them). I imagine at some point this behaviour will be built in. People are treating GPTs like they're knowledge bases, but they're not. The fact that they can answer simple or complex questions correctly is only a byproduct of the transformer being taught how to string language together. It's like an artist who learns to draw by making paintings of raccoons. You can't then ask them "what do raccoons like to eat?" or "what foods are poisonous to a raccoon?" just because they learnt to draw by painting raccoons. This is how people are treating GPTs atm. They believe in it because they ask the artist "what colour fur do raccoons have?" and because the artist can answer that correctly, they assume all other answers are factual.

gpm 3y ago

In local-to-me politics we have a report on changing the admissions process for specialty programs in the public school system - which had a bunch of fake citations and people suspect was written with the "aid" of ChatGPT. https://www.thestar.com/news/gta/2023/05/26/tdsb-fires-researcher-for-allegedly-plagiarizing-parts-of-report-on-specialty-schools.html https://www.thestar.com/news/gta/2023/05/26/tdsb-fires-resea...

Xenoamorphous 3y ago

> It's not just lawyers who think that ChatGPT is a search engine. Let’s not forget that Google often puts incorrect information in their snippets/factboxes or whatever they call them.

deleted 3y ago

[deleted]

cratermoon 3y ago

Computer, what's the formula for transparent aluminum? Seriously, I got ChatGPT to spit out a scientific-seeming paper on the formula and manufacturing process for transparent aluminum. It did note that there's a real thing, aluminum oxynitride, which is the closest thing we have to the Star Trek material. It even wrote the following abstract, based on my prompt: > This scientific description provides an overview of the formula and manufacturing process of transparent aluminum, a material used in applications where both structural strength and transparency are required. Transparent aluminum finds extensive use in diverse fields, including public aquaria, where it allows for the display of large marine organisms. The description outlines the chemical composition, key properties, and the manufacturing steps involved in creating transparent aluminum. Whether or not the six-step manufacturing process it came up with is correct or not, I haven't the expertise to say.

codeflo 3y ago
> Whether or not the six-step manufacturing process it came up with is correct or not, I haven't the expertise to say. I mean, you can try it and perhaps get rich. Or, probably more likely, die in a horrible explosion.
- cratermoon 3y ago
  [the material is] placed in a high-pressure vessel, and temperature and pressure are simultaneously increased to near the melting point of aluminum. I don't need to try that to know that molten metal under high pressure at over 1000C is something I'm not ready to deal with.