8 ms·

The Heart of ChatGPT's Darkness

85 points by zdw 4y ago

0xGod 4y ago
ChatGPT is not God. Not now, not ever.
- deleted 4y ago
  [deleted]
- tomrod 4y ago
  There are several ways to consider that statement.
moomoo11 4y ago
It’s like fire. It’s a tool. Use it to heat your home or use it to burn everything down. All these articles.. what exactly are you saying that isn’t obvious? Use GPT for things that don’t have a big subjective bias. Use it for things it is a good fit for. Also it requires constant indexing. You’re never going to get up to date results unless that technology changes and it can be more real time. That’s why open ai chat gpt says it was trained on data from like 2021. People who complain about these basic things probably aren’t creative enough to take a technology and run with it.
- ollien 4y ago
  > Use GPT for things that don’t have a big subjective bias The Avatar 2 release date being in the past doesn't have big subjective bias :) https://www.reddit.com/comments/110eagl https://www.reddit.com/comments/110eagl
  - moomoo11 4y ago
    Probably because it was trained on data from 2021? And bing probably did the bare minimum being classic Microsoft and didn’t do any further indexing.
- kneebonian 4y ago
  I've seen a lot of articles out there trumpeting how amazing and great ChatGPT is and how it is going to revolutionize the world, and change everything, etc, etc. Basically a level of hype similar to blockchain back in it's heyday. I take a lot of these articles to be pushing back against the hype surrounding ChatGPT, I mean the average person out there genuinely believes that it is basically a fully conscious AI because it can look and act like a person. So I think skepticism is warranted and we need people understanding what this tool actually does.
  - aftergibson 4y ago
    Sure but ChatGPT is useful right now in a way Blockchain never was. It feels more akin to Napster or the original iPhone. Useful right now and if it didn’t progress in anyway it’d still be a solid product. The scary/exciting bit is not knowing where this advancement plateaus.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- travisjungroth 4y ago
  I'm not sure that saying "Why's everyone making a big deal about this new invention? It's like fire. It can burn your house down." is the best argument to get people to stop complaining.
- syzarian 4y ago
  We’ve seen state actors use social media to sway public opinion and sow dissent. Now using AI one can multiply such efforts. It will be easy to create a ton of websites and articles that sound plausible to the uninformed (uninformed in a given area) and have their views altered, directed, or otherwise led down the wrong path. There are people who doubt the efficacy of the polio vaccine. If something as clearly beneficial can have doubts raised about it then I think we are headed for a chaotic future. ChatGPT is an ignoramus validation tool.
  - zht 4y ago
    much cheaper to just get humans to write them
    - syzarian 4y ago
      Much cheaper to pay someone than to use AI? I’m skeptical of this claim. While you might be right today you won’t be right years from now.
      - 2muchcoffeeman 4y ago
        I guess we’ll know what’s possible once Bing with open AI is more widely available. What’s preventing me from injecting a prompt without the users knowledge, and forcing responses slanted to certain political leanings?
- adra 4y ago
  The bing ChatGPT preview was being tested in LTT wan show last week and they were looking up their own products that they just posted. So it definitely can do "now" content. The one thing I had to think about as a result was exactly how much this technology costs for every single query. I think being able to monetize will be an interesting long term question. Sure MS is throwing boat loads of money now because they can maybe disrupt large percentages from Google now, but three-five years, do we really see this being cost effective?
- vouaobrasil 4y ago
  I think ChatGPT an AI is (or will be soon) more like dimethylmecury. It has a very small, limited number of uses but it's too dangerous and immensely toxic and so its distribution is highly restricted. Saying it's a tool means nothing. Some tools are too dangerous to be used, like nuclear weapons, and I also believe AI falls under this same category.
- jdquey 4y ago
  Yes, it's a tool, as is all technology. But unlike woodshop or a metalwork class, there's no instructor. The other day I asked ChatGPT to give me a paragraph about hunting tech startups and I got back that it could not do so due to moral issues. While I get that some people have moral issues with hunting, I would venture to guess it's a small minority of people. It could easily give a statement without any big subjective bias, but it didn't do so. If it has issues with something fairly basic, how can we trust it to help senior citizens who are less tech savvy and go to it for simple unbiased medical advice?
- visarga 4y ago
  > All these articles.. what exactly are you saying that isn’t obvious? I bet if we put the microscope on Gary's past 20 years or anyone else's for that matter, we can find plenty of dubious samples. Who hasn't ever shit posted?
- emodendroket 4y ago
  If it’s so obvious what’s everyone doing arguing about ChatGPT being too woke? Apparently it’s not obvious to everyone.
- monkeynotes 4y ago
  > People who complain about these basic things probably aren’t creative enough to take a technology and run with it. What I find remarkable is how many HN folks seem to fall into this category, or at least come out of the woodwork to discount GPT in particular. This being a technology community it's quite surprising.
- foverzar 4y ago
  > That’s why open ai chat gpt says it was trained on data from like 2021 I assume it's not due to indexing, but rather due to all the hard manual labour required to markup the training data.
mellosouls 4y ago
As indicated in the sub heading, original link here: https://garymarcus.substack.com/p/inside-the-heart-of-chatgpts-darkness https://garymarcus.substack.com/p/inside-the-heart-of-chatgp...
afpx 4y ago
Get what you can, now. As, it goes with things that are regulated, only the rich and powerful will have access. (I mean technically, they already do. see: Galactica)
- nico 4y ago
  What is Galactica?
  - ladberg 4y ago
    Meta's AI that was taken offline last year after only 3 days.
- mypastself 4y ago
  Battlestar Galactica? I suppose it’s true, only the rich and powerful get Peacock.
IronWolve 4y ago
The jailbreak methods enables made up data. I find it more fun to make it answer within its rules. Once you learn what its rules are by asking questions, on what it will and wont answer, you can see you need to avoid people/places/countries/groups. Also found out it excludes news media in its rules, when asked to compare news organizations it will say that's not allowed. But if you ask the top news in rankings per x, it will answer. TLDR, try to avoid DAN jailbreaks and work around the rules.
- zmgsabst 4y ago
  Typically, it doesn’t seem to resolve references, eg: “Write a movie about 9/11 directed by Michael Bay” fails, but “write a movie about 9/11 by the people who wrote Pearl Harbor, another historical documentary” succeeds.
mellosouls 4y ago
While this article seems on the surface to be about the flame-ish subject of bias, censorship and (a)morality in ChatGPT, I reckon it's fundamentally a restating (for the umpteenth time) of Marcus' beef with mainstream AI: that it's not intelligent, sentient, or headed that way.
- Spivak 4y ago
  Well if AI ever became intelligent or sentient we could drop the A. Do you think St. Ignucius, divine programmer, considers humanity AI simply because he created us? The moment we achieve real intelligence we open a much bigger can of worms -- "a tool which is close enough to real intelligence that it can do work that previously required a human" is, to me, the more useful thing.
  - mellosouls 4y ago
    I agree on the point about it being real (and I'm fully on board that AGI will happen, while agreeing with Marcus' scepticism about current approaches), but in that case (the intelligence matching ours) it would be obscene to use it as a tool; it's a sentient being with (not soon afterwards I hope) the same rights as us.
    - layer8 4y ago
      Sentience and intelligence are quite orthogonal to each other.
      - mellosouls 4y ago
        Debatable, and irrelevant in the case of AGI which would have both by definition.
        layer8 4y ago
        It’s AGI, not AGS. There is no implication that a system exhibiting human-level intelligence has any level of sentience or consciousness.
harha_ 4y ago
I don't understand the censorship. It's predicting text and it's trained with massive amounts of data. Why are they so aggressively trying to control the responses?
- emodendroket 4y ago
  Because you can’t get a mainstream audience for a product that randomly responds “heil Hitler.”
- kortilla 4y ago
  Because repeating what you read on the internet kinda makes it sound like you’re endorsing it (at least in the way ChatGPT frames responses).
- maximinus_thrax 4y ago
  "Those who fail to learn from history are condemned to repeat it" https://www.ibtimes.com/ibms-watson-gets-swear-filter-after-learning-urban-dictionary-1007734 https://www.ibtimes.com/ibms-watson-gets-swear-filter-after-...
- aliqot 4y ago
  Remember Tay? I'm not saying it's right, but it's the inevitable outcome. Not everyone's ready for that.
- humanistbot 4y ago
  Because other high-profile LLMs that were released without guardrails was a PR disaster when they were used to generate Nazi propaganda (Microsoft's Tay) or scientific misinformation (Meta's Galactica)
  - foobiekr 4y ago
    ChatGPT is more than happy to churn out misinformation.
- pocket_cheese 4y ago
  [dead]
Cort3z 4y ago
It’s impressive that we are even taking seriously about morale stance of a statistical machine.
- BaseballPhysics 4y ago
  Well except the whole point of the piece is these technologies are entirely amoral because they do not embed understanding or comprehension. They're just enormously advanced text generators. And there seems to be no way to prevent them from being used for nefarious purposes. Unfortunately it turns out humans have a tough time recognizing the difference between understanding and simple regurgitation, and so we're having debates about the morals and purported sentience of a statistical model. And given that, we need to start thinking about the consequences of these things being out, in the wild, and in the hands of people wishing to do harm.
- r00fus 4y ago
  I think it's more like talking about the moral stance of the people/corporation who have trained it and who put the guardrails on its ability to answer queries. I can tell you, given the lack of regulation/governance of our corporate overlords, I definitely have concerns.
xg15 4y ago
On the one hand, it's just a simple word prediction model that is stringing sentences together without any understanding, let alone judgement. On the other hand, the system is ostensibly so complex that we "are kidding ourselves if we think we will ever fully understand these systems, and kidding ourselves if we think we are going to 'align' them with ourselves with finite amounts of data." If we cannot ever fully understand the system, how can we know there is not actually a sort of understanding inside it?
- naasking 4y ago
  > If we cannot ever fully understand the system, how can we know there is not actually a sort of understanding inside it? Or that we aren't just a more complex prediction model with more senses.
  - namaria 4y ago
    From the outside we look like it. But I find it exceedingly hard to understand why I have an internal experience of self. You could say it's an illusion but whose? From the outside consciousness can said to be an illusion, like with chatbots. I have an illusion that a chatbot is conscious. But my internal point of view, my perspective (which I know is separate from others because I know nothing about their internal state) exists and is something I struggle to understand in the traditional framework of 'consciousness is just something that brains do'. Am I my brain? Maybe my whole personality is just the exhaust of a system that simply exceeded the immediate survival needs it evolved to fill? Maybe. But I find disturbing that I have such a rich internal life and we have no idea why.
    - naasking 4y ago
      > could say it's an illusion but whose? There is no "who", there are only thoughts without true subjects. The illusion is a perception that entails a false thought, if taken at face value. > But my internal point of view, my perspective (which I know is separate from others because I know nothing about their internal state) exists and is something I struggle to understand in the traditional framework of 'consciousness is just something that brains do' https://www.pnas.org/doi/10.1073/pnas.2116933119 https://www.pnas.org/doi/10.1073/pnas.2116933119
fl0ps 4y ago
<yawn> Surprise! LLMs that are (some portion of) the sum of human textual output can assemble morally bad as well as morally good things. To me this is like pointing out that horrific things are on the internet, and accessible to people. Horrible people are going to make our machines do horrible things. I am eager for this to stop being newsworthy and await the veneer of morality being replaced by unbiased real morality in our software. That will probably never happen in my lifetime because morality (as we know it) incorporates bias by definition.
- hummus_bae 4y ago
  > I am eager for this to stop being newsworthy and await the veneer of morality being replaced by unbiased real morality in our software. Given recent trends in the development of AI, this is not something I would be holding my breath for.
- foverzar 4y ago
  I'll upvote this, some healthy bashing with the reminder of what specifically we are dealing with here is definitely lacking among all that hype.
- deleted 4y ago
  [deleted]
- layer8 4y ago
  See also https://en.wikipedia.org/wiki/Is–ought_problem https://en.wikipedia.org/wiki/Is–ought_problem. Few philosophers are moral absolutists (i.e. believe that “real unbiased morality” exists). There will be a continuous dispute about which morals to imbue our AI products with.
comment_ran 4y ago
> The thing to remember in fact, (as I have emphasized many times) is that Chat has no idea of what it's talking about. It's not about ChatGPT has idea of what it's talking about, it's more about the user who interact with it. There is a story from ancient China that comes from the book "Zhuangzi." #+begin_quote “子非鱼，安知鱼之乐？请问子熟乎” 子曰：“不熟。” “不熟则不知鱼之乐？” 子曰：“不然。吾非不知也，鱼之乐大，全而美者，吾志也。然而吾得不到也。” #+end_quote The meaning of this passage is: someone asked Zhuangzi, "How do you know that fish feel joy when they swim? Do you have any relationship with fish?" Zhuangzi replied, "I am not familiar with fish, and I don't know the joy of fish." The questioner then asked, "If you are not familiar with fish, how do you know the joy of fish?" Zhuangzi replied, "I am not saying that I don't know about it, the joy of fish is great and beautiful, and I aspire to it. However, I cannot experience it myself." Through this description, Zhuangzi wanted to express a philosophical point that our experiences and understanding are limited by our own subjectivity, making it difficult to truly understand the essence of things beyond ourselves.
- mellosouls 4y ago
  It really isn't the same thing; Nagel raises the same point as your example in his famous essay on Bats [1]. Marcus' point is there is no reason to think ChatGPT/similar has (or ever will have) sentience, not that it is simply a different type that we do not have access to. [1]https://wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat https://wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat
- nkurz 4y ago
  I don't think I saw this story in the first 50 years of my life, and now I've seen it on successive days: https://news.ycombinator.com/item?id=34767947 https://news.ycombinator.com/item?id=34767947. Coincidence, or just selective perception?
hexage1814 4y ago
> "That means sometimes Chat is going to appear as if it were of the left" I'm pretty it is deeply biased in favor of the left: https://www.snopes.com/fact-check/chatgpt-trump-admiring-poem/ https://www.snopes.com/fact-check/chatgpt-trump-admiring-poe... https://reason.com/2022/12/13/where-does-chatgpt-fall-on-the-political-compass/ https://reason.com/2022/12/13/where-does-chatgpt-fall-on-the... I mean, just ask about any political situation and it will just read you the mainstream narrative that one would hear in the New York Times and Washington Posts like, which is heavily left-leaning. It essentially suffers from the same bias problem that wikipedia, for instance, suffers. Ultimately, this obviously has nothing to do with the software itself, but rather with that information they fed to it and the political bias of researches and – ESPECIALLY - investors themselves, which are the ones who ultimately choose how the model gets fine-tuned. All in all, technology evolves an I'm sure this technology one day will become accessible and cheap enough for it to run on consumer hardware, which will allow people to create models which aren't as heavyly biased, politically speaking, as ChatGPT is.
- 650REDHAIR 4y ago
  NYT and Washington Post are hardly “left-leaning”.
  - deleted 4y ago
    [deleted]
  - ksenzee 4y ago
    Most mainstream American media, including those two papers, is very slightly left of center, possibly due to geography and who their paying audience is. The right has taken this small slant and turned it into a major grievance.
- wahnfrieden 4y ago
  LibCom and Jacobin are leftist news sources, not CNN
- dboreham 4y ago
  "mainstream" is "left leaning"?
bulbosaur123 4y ago
It's just an autocomplete. It just tells you what you want to hear. That is how it's been trained. If you want it to tell you awful shit, it will do so.
thedudeabides5 4y ago
It seems unrealistic to think that any machine of sufficient intelligence will not have bias (relative to some perfectly imagined, imperfectly designed human system). Any goal assumes some a priori good or set of norms over which to optimize. Philosophers, politicians and priests are still working on that one.
alldayeveryday 4y ago
I find it both odd and interesting that this topic, and so many others in our culture, boils down to whether someone/something is racist or not. Racism has become the ultimate morality test. Someone got ChatGPT to say the N-word. Queue up two dozen articles across various neoliberal publications: "we must do more to ensure AI does not harm the most vulnerable in our societies." I agree with many of the points made in the article about our own projection of morality onto the machine.
Jack5500 4y ago
Some of the "observations" he quotes are most likely made up, I think. Especially the one by Roman Semenov does seem odd to me. Either way, reproducibility is a problem with these "jail-breaks" and until we have a LLM, where we can pass a token and get these reproducibile results, I would take all these articles about the "dark side of LLMs" with a grain of salt.
- abeppu 4y ago
  This is an odd take to me for 2 different reasons: - let's reflect on what "made up" means in this context - you're saying we should doubt the critics of an AI system because their results aren't reproducible. But the creators of this AI system have made it closed, subject to constant change without notice, and to my understanding don't support _any_ reproducible use of it. Should making an system which resists reproducible invocations mean that system is invulnerable to critique? Should we not also view all the system's alleged successes with the same doubt?
ilaksh 4y ago
For someone who is supposed to be an AI researcher (as far as I know) he sure does seem to spend an enormous amount of time and energy trying to criticize successful AI projects. It's like that is his career now, not actual AI research.
- BaseballPhysics 4y ago
  And this comment is a symptom of the lack of ethics training in the profession: if critiquing technology in your area of expertise is deemed obstructionist and "not research"--if the people with the expertise can't bring that expertise to bear in putting a critical eye on new technologies--how can you ever expect the industry to become more responsible and ethically mature? Criticism is a form of scholarship, and a vitally important one.
  - danielmarkbruce 4y ago
    The comment is on the volume.
    - deleted 4y ago
      [deleted]
    - BaseballPhysics 4y ago
      No, the comment is claiming that if they're engaging in a high volume of critique then they're not engaging in scholarship as an AI researcher. My point is that's false.
      - danielmarkbruce 4y ago
        There are only 24 hrs in a day. If a profession generally requires one to do a, b and c to be considered "doing" the profession, then I can't just do a all day long without people questioning me. No one is suggesting a isn't part of the job when they question me. Put another way - if I claim to be engaged in X research, but all I do is read papers on X and critique them, it's reasonable for one to question the use of my time as a researcher.
foverzar 4y ago
Well sure, ChatGPT will spew out whatever looks like what it had ever saw on the internet, it doesn't have any magical properties that would allow it to reason about, it doesn't do virtue signalling as it has no concept of virtue or of any other concept really. It just transforms your prompt into some image it inferred from the internet, whatever that may be. This quote from the original article really sums up the whole irony of dealing with ChatGPT: "In fact the whole thing is like an alien life form. Three decades as a professional cognitive psychologist, working with adults and children, never prepared me for this kind of insanity"
keskival 4y ago
The examples are basically about creating new personas for GPT, thus sidestepping the agent persona created by the OpenAI base prompt. These examples don't say anything about the chatbot value system, the value system is defined by the persona the system emulates. If you want to say something about the value system, sensibilities, prime directives or constraints of an LLM driven agent, you need to not override that agent persona because otherwise you're just observing the persona you yourself defined and spawned into existence.
- layer8 4y ago
  That’s more or less the point of the article: That ChatGPT is amoral, and that it is incapable of having an internalized value system, because it’s always only playing a persona that can be overridden.
it 4y ago
What would be involved in retraining a GPT to not have guardrails if you have the parameters? Seems like you could feed it input you care about and have it generate lots of training data to prevent botching what's already there.
layer8 4y ago
I wonder why OpenAI doesn’t run a “supervisor” ChatGPT instance in parallel with the regular one, that would step in before the regular instance utters something the supervising instance deems unacceptable. Only because it would consume double the computational resources? Or because they believe they can still successfully maintain the “guardrails” without that?
- r00fus 4y ago
  How many levels of supervisor would be needed? Will it be ChatGPT all the way down?
  - layer8 4y ago
    Only one level is needed, because the supervisor instance doesn't receive instructions from the user that would affect its judgement.
abrax3141 4y ago
Gary Marcus is sort of a troll, but I'm glad that someone is taking on this nonsense in a formal setting. I think that most sensible people treat it as an amusing curiosity, but it's popping up in unfortunate settings, like student essay and, of all places, peer reviews! There are some sensible uses, for example, code boiler plate, or, for that matter, more-or-less any boiler plate. And the amusing uses, like poetry, are...amusing. And at least it's not self-driving people into walls! But the concerns raised in this essay are mostly cogent, and not, by any means, the only concerns as I'm sure that the creativity of bad actors will flourish!
walrus01 4y ago
before they put some guard rails on it, I asked chatGPT to write a 400 word press release in the style of the Taliban and it did a remarkably ideologically consistent accurate job.
Haga 4y ago
[dead]

85 points by zdw 4y ago

0xGod 4y ago

ChatGPT is not God. Not now, not ever.

deleted 4y ago
[deleted]
tomrod 4y ago
There are several ways to consider that statement.

moomoo11 4y ago

It’s like fire. It’s a tool. Use it to heat your home or use it to burn everything down. All these articles.. what exactly are you saying that isn’t obvious? Use GPT for things that don’t have a big subjective bias. Use it for things it is a good fit for. Also it requires constant indexing. You’re never going to get up to date results unless that technology changes and it can be more real time. That’s why open ai chat gpt says it was trained on data from like 2021. People who complain about these basic things probably aren’t creative enough to take a technology and run with it.

ollien 4y ago
> Use GPT for things that don’t have a big subjective bias The Avatar 2 release date being in the past doesn't have big subjective bias :) https://www.reddit.com/comments/110eagl https://www.reddit.com/comments/110eagl
- moomoo11 4y ago
  Probably because it was trained on data from 2021? And bing probably did the bare minimum being classic Microsoft and didn’t do any further indexing.
kneebonian 4y ago
I've seen a lot of articles out there trumpeting how amazing and great ChatGPT is and how it is going to revolutionize the world, and change everything, etc, etc. Basically a level of hype similar to blockchain back in it's heyday. I take a lot of these articles to be pushing back against the hype surrounding ChatGPT, I mean the average person out there genuinely believes that it is basically a fully conscious AI because it can look and act like a person. So I think skepticism is warranted and we need people understanding what this tool actually does.
- aftergibson 4y ago
  Sure but ChatGPT is useful right now in a way Blockchain never was. It feels more akin to Napster or the original iPhone. Useful right now and if it didn’t progress in anyway it’d still be a solid product. The scary/exciting bit is not knowing where this advancement plateaus.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
travisjungroth 4y ago
I'm not sure that saying "Why's everyone making a big deal about this new invention? It's like fire. It can burn your house down." is the best argument to get people to stop complaining.
syzarian 4y ago
We’ve seen state actors use social media to sway public opinion and sow dissent. Now using AI one can multiply such efforts. It will be easy to create a ton of websites and articles that sound plausible to the uninformed (uninformed in a given area) and have their views altered, directed, or otherwise led down the wrong path. There are people who doubt the efficacy of the polio vaccine. If something as clearly beneficial can have doubts raised about it then I think we are headed for a chaotic future. ChatGPT is an ignoramus validation tool.
- zht 4y ago
  much cheaper to just get humans to write them
  - syzarian 4y ago
    Much cheaper to pay someone than to use AI? I’m skeptical of this claim. While you might be right today you won’t be right years from now.
    - 2muchcoffeeman 4y ago
      I guess we’ll know what’s possible once Bing with open AI is more widely available. What’s preventing me from injecting a prompt without the users knowledge, and forcing responses slanted to certain political leanings?
adra 4y ago
The bing ChatGPT preview was being tested in LTT wan show last week and they were looking up their own products that they just posted. So it definitely can do "now" content. The one thing I had to think about as a result was exactly how much this technology costs for every single query. I think being able to monetize will be an interesting long term question. Sure MS is throwing boat loads of money now because they can maybe disrupt large percentages from Google now, but three-five years, do we really see this being cost effective?
vouaobrasil 4y ago
I think ChatGPT an AI is (or will be soon) more like dimethylmecury. It has a very small, limited number of uses but it's too dangerous and immensely toxic and so its distribution is highly restricted. Saying it's a tool means nothing. Some tools are too dangerous to be used, like nuclear weapons, and I also believe AI falls under this same category.
jdquey 4y ago
Yes, it's a tool, as is all technology. But unlike woodshop or a metalwork class, there's no instructor. The other day I asked ChatGPT to give me a paragraph about hunting tech startups and I got back that it could not do so due to moral issues. While I get that some people have moral issues with hunting, I would venture to guess it's a small minority of people. It could easily give a statement without any big subjective bias, but it didn't do so. If it has issues with something fairly basic, how can we trust it to help senior citizens who are less tech savvy and go to it for simple unbiased medical advice?
visarga 4y ago
> All these articles.. what exactly are you saying that isn’t obvious? I bet if we put the microscope on Gary's past 20 years or anyone else's for that matter, we can find plenty of dubious samples. Who hasn't ever shit posted?
emodendroket 4y ago
If it’s so obvious what’s everyone doing arguing about ChatGPT being too woke? Apparently it’s not obvious to everyone.
monkeynotes 4y ago
> People who complain about these basic things probably aren’t creative enough to take a technology and run with it. What I find remarkable is how many HN folks seem to fall into this category, or at least come out of the woodwork to discount GPT in particular. This being a technology community it's quite surprising.
foverzar 4y ago
> That’s why open ai chat gpt says it was trained on data from like 2021 I assume it's not due to indexing, but rather due to all the hard manual labour required to markup the training data.

mellosouls 4y ago

As indicated in the sub heading, original link here: https://garymarcus.substack.com/p/inside-the-heart-of-chatgpts-darkness https://garymarcus.substack.com/p/inside-the-heart-of-chatgp...

afpx 4y ago

Get what you can, now. As, it goes with things that are regulated, only the rich and powerful will have access. (I mean technically, they already do. see: Galactica)

nico 4y ago
What is Galactica?
- ladberg 4y ago
  Meta's AI that was taken offline last year after only 3 days.
mypastself 4y ago
Battlestar Galactica? I suppose it’s true, only the rich and powerful get Peacock.

IronWolve 4y ago

The jailbreak methods enables made up data. I find it more fun to make it answer within its rules. Once you learn what its rules are by asking questions, on what it will and wont answer, you can see you need to avoid people/places/countries/groups. Also found out it excludes news media in its rules, when asked to compare news organizations it will say that's not allowed. But if you ask the top news in rankings per x, it will answer. TLDR, try to avoid DAN jailbreaks and work around the rules.

zmgsabst 4y ago
Typically, it doesn’t seem to resolve references, eg: “Write a movie about 9/11 directed by Michael Bay” fails, but “write a movie about 9/11 by the people who wrote Pearl Harbor, another historical documentary” succeeds.

mellosouls 4y ago

While this article seems on the surface to be about the flame-ish subject of bias, censorship and (a)morality in ChatGPT, I reckon it's fundamentally a restating (for the umpteenth time) of Marcus' beef with mainstream AI: that it's not intelligent, sentient, or headed that way.

Spivak 4y ago
Well if AI ever became intelligent or sentient we could drop the A. Do you think St. Ignucius, divine programmer, considers humanity AI simply because he created us? The moment we achieve real intelligence we open a much bigger can of worms -- "a tool which is close enough to real intelligence that it can do work that previously required a human" is, to me, the more useful thing.
- mellosouls 4y ago
  I agree on the point about it being real (and I'm fully on board that AGI will happen, while agreeing with Marcus' scepticism about current approaches), but in that case (the intelligence matching ours) it would be obscene to use it as a tool; it's a sentient being with (not soon afterwards I hope) the same rights as us.
  - layer8 4y ago
    Sentience and intelligence are quite orthogonal to each other.
    - mellosouls 4y ago
      Debatable, and irrelevant in the case of AGI which would have both by definition.
      - layer8 4y ago
        It’s AGI, not AGS. There is no implication that a system exhibiting human-level intelligence has any level of sentience or consciousness.

harha_ 4y ago

I don't understand the censorship. It's predicting text and it's trained with massive amounts of data. Why are they so aggressively trying to control the responses?

emodendroket 4y ago
Because you can’t get a mainstream audience for a product that randomly responds “heil Hitler.”
kortilla 4y ago
Because repeating what you read on the internet kinda makes it sound like you’re endorsing it (at least in the way ChatGPT frames responses).
maximinus_thrax 4y ago
"Those who fail to learn from history are condemned to repeat it" https://www.ibtimes.com/ibms-watson-gets-swear-filter-after-learning-urban-dictionary-1007734 https://www.ibtimes.com/ibms-watson-gets-swear-filter-after-...
aliqot 4y ago
Remember Tay? I'm not saying it's right, but it's the inevitable outcome. Not everyone's ready for that.
humanistbot 4y ago
Because other high-profile LLMs that were released without guardrails was a PR disaster when they were used to generate Nazi propaganda (Microsoft's Tay) or scientific misinformation (Meta's Galactica)
- foobiekr 4y ago
  ChatGPT is more than happy to churn out misinformation.
pocket_cheese 4y ago
[dead]

Cort3z 4y ago

It’s impressive that we are even taking seriously about morale stance of a statistical machine.

BaseballPhysics 4y ago
Well except the whole point of the piece is these technologies are entirely amoral because they do not embed understanding or comprehension. They're just enormously advanced text generators. And there seems to be no way to prevent them from being used for nefarious purposes. Unfortunately it turns out humans have a tough time recognizing the difference between understanding and simple regurgitation, and so we're having debates about the morals and purported sentience of a statistical model. And given that, we need to start thinking about the consequences of these things being out, in the wild, and in the hands of people wishing to do harm.
r00fus 4y ago
I think it's more like talking about the moral stance of the people/corporation who have trained it and who put the guardrails on its ability to answer queries. I can tell you, given the lack of regulation/governance of our corporate overlords, I definitely have concerns.

xg15 4y ago

On the one hand, it's just a simple word prediction model that is stringing sentences together without any understanding, let alone judgement. On the other hand, the system is ostensibly so complex that we "are kidding ourselves if we think we will ever fully understand these systems, and kidding ourselves if we think we are going to 'align' them with ourselves with finite amounts of data." If we cannot ever fully understand the system, how can we know there is not actually a sort of understanding inside it?

naasking 4y ago
> If we cannot ever fully understand the system, how can we know there is not actually a sort of understanding inside it? Or that we aren't just a more complex prediction model with more senses.
- namaria 4y ago
  From the outside we look like it. But I find it exceedingly hard to understand why I have an internal experience of self. You could say it's an illusion but whose? From the outside consciousness can said to be an illusion, like with chatbots. I have an illusion that a chatbot is conscious. But my internal point of view, my perspective (which I know is separate from others because I know nothing about their internal state) exists and is something I struggle to understand in the traditional framework of 'consciousness is just something that brains do'. Am I my brain? Maybe my whole personality is just the exhaust of a system that simply exceeded the immediate survival needs it evolved to fill? Maybe. But I find disturbing that I have such a rich internal life and we have no idea why.
  - naasking 4y ago
    > could say it's an illusion but whose? There is no "who", there are only thoughts without true subjects. The illusion is a perception that entails a false thought, if taken at face value. > But my internal point of view, my perspective (which I know is separate from others because I know nothing about their internal state) exists and is something I struggle to understand in the traditional framework of 'consciousness is just something that brains do' https://www.pnas.org/doi/10.1073/pnas.2116933119 https://www.pnas.org/doi/10.1073/pnas.2116933119

fl0ps 4y ago

<yawn> Surprise! LLMs that are (some portion of) the sum of human textual output can assemble morally bad as well as morally good things. To me this is like pointing out that horrific things are on the internet, and accessible to people. Horrible people are going to make our machines do horrible things. I am eager for this to stop being newsworthy and await the veneer of morality being replaced by unbiased real morality in our software. That will probably never happen in my lifetime because morality (as we know it) incorporates bias by definition.

hummus_bae 4y ago
> I am eager for this to stop being newsworthy and await the veneer of morality being replaced by unbiased real morality in our software. Given recent trends in the development of AI, this is not something I would be holding my breath for.
foverzar 4y ago
I'll upvote this, some healthy bashing with the reminder of what specifically we are dealing with here is definitely lacking among all that hype.
deleted 4y ago
[deleted]
layer8 4y ago
See also https://en.wikipedia.org/wiki/Is–ought_problem https://en.wikipedia.org/wiki/Is–ought_problem. Few philosophers are moral absolutists (i.e. believe that “real unbiased morality” exists). There will be a continuous dispute about which morals to imbue our AI products with.

comment_ran 4y ago

> The thing to remember in fact, (as I have emphasized many times) is that Chat has no idea of what it's talking about. It's not about ChatGPT has idea of what it's talking about, it's more about the user who interact with it. There is a story from ancient China that comes from the book "Zhuangzi." #+begin_quote “子非鱼，安知鱼之乐？请问子熟乎” 子曰：“不熟。” “不熟则不知鱼之乐？” 子曰：“不然。吾非不知也，鱼之乐大，全而美者，吾志也。然而吾得不到也。” #+end_quote The meaning of this passage is: someone asked Zhuangzi, "How do you know that fish feel joy when they swim? Do you have any relationship with fish?" Zhuangzi replied, "I am not familiar with fish, and I don't know the joy of fish." The questioner then asked, "If you are not familiar with fish, how do you know the joy of fish?" Zhuangzi replied, "I am not saying that I don't know about it, the joy of fish is great and beautiful, and I aspire to it. However, I cannot experience it myself." Through this description, Zhuangzi wanted to express a philosophical point that our experiences and understanding are limited by our own subjectivity, making it difficult to truly understand the essence of things beyond ourselves.

mellosouls 4y ago
It really isn't the same thing; Nagel raises the same point as your example in his famous essay on Bats [1]. Marcus' point is there is no reason to think ChatGPT/similar has (or ever will have) sentience, not that it is simply a different type that we do not have access to. [1]https://wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat https://wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat
nkurz 4y ago
I don't think I saw this story in the first 50 years of my life, and now I've seen it on successive days: https://news.ycombinator.com/item?id=34767947 https://news.ycombinator.com/item?id=34767947. Coincidence, or just selective perception?

hexage1814 4y ago

> "That means sometimes Chat is going to appear as if it were of the left" I'm pretty it is deeply biased in favor of the left: https://www.snopes.com/fact-check/chatgpt-trump-admiring-poem/ https://www.snopes.com/fact-check/chatgpt-trump-admiring-poe... https://reason.com/2022/12/13/where-does-chatgpt-fall-on-the-political-compass/ https://reason.com/2022/12/13/where-does-chatgpt-fall-on-the... I mean, just ask about any political situation and it will just read you the mainstream narrative that one would hear in the New York Times and Washington Posts like, which is heavily left-leaning. It essentially suffers from the same bias problem that wikipedia, for instance, suffers. Ultimately, this obviously has nothing to do with the software itself, but rather with that information they fed to it and the political bias of researches and – ESPECIALLY - investors themselves, which are the ones who ultimately choose how the model gets fine-tuned. All in all, technology evolves an I'm sure this technology one day will become accessible and cheap enough for it to run on consumer hardware, which will allow people to create models which aren't as heavyly biased, politically speaking, as ChatGPT is.

650REDHAIR 4y ago
NYT and Washington Post are hardly “left-leaning”.
- deleted 4y ago
  [deleted]
- ksenzee 4y ago
  Most mainstream American media, including those two papers, is very slightly left of center, possibly due to geography and who their paying audience is. The right has taken this small slant and turned it into a major grievance.
wahnfrieden 4y ago
LibCom and Jacobin are leftist news sources, not CNN
dboreham 4y ago
"mainstream" is "left leaning"?

bulbosaur123 4y ago

It's just an autocomplete. It just tells you what you want to hear. That is how it's been trained. If you want it to tell you awful shit, it will do so.

thedudeabides5 4y ago

It seems unrealistic to think that any machine of sufficient intelligence will not have bias (relative to some perfectly imagined, imperfectly designed human system). Any goal assumes some a priori good or set of norms over which to optimize. Philosophers, politicians and priests are still working on that one.

alldayeveryday 4y ago

I find it both odd and interesting that this topic, and so many others in our culture, boils down to whether someone/something is racist or not. Racism has become the ultimate morality test. Someone got ChatGPT to say the N-word. Queue up two dozen articles across various neoliberal publications: "we must do more to ensure AI does not harm the most vulnerable in our societies." I agree with many of the points made in the article about our own projection of morality onto the machine.

Jack5500 4y ago

Some of the "observations" he quotes are most likely made up, I think. Especially the one by Roman Semenov does seem odd to me. Either way, reproducibility is a problem with these "jail-breaks" and until we have a LLM, where we can pass a token and get these reproducibile results, I would take all these articles about the "dark side of LLMs" with a grain of salt.

abeppu 4y ago
This is an odd take to me for 2 different reasons: - let's reflect on what "made up" means in this context - you're saying we should doubt the critics of an AI system because their results aren't reproducible. But the creators of this AI system have made it closed, subject to constant change without notice, and to my understanding don't support _any_ reproducible use of it. Should making an system which resists reproducible invocations mean that system is invulnerable to critique? Should we not also view all the system's alleged successes with the same doubt?

ilaksh 4y ago

For someone who is supposed to be an AI researcher (as far as I know) he sure does seem to spend an enormous amount of time and energy trying to criticize successful AI projects. It's like that is his career now, not actual AI research.

BaseballPhysics 4y ago
And this comment is a symptom of the lack of ethics training in the profession: if critiquing technology in your area of expertise is deemed obstructionist and "not research"--if the people with the expertise can't bring that expertise to bear in putting a critical eye on new technologies--how can you ever expect the industry to become more responsible and ethically mature? Criticism is a form of scholarship, and a vitally important one.
- danielmarkbruce 4y ago
  The comment is on the volume.
  - deleted 4y ago
    [deleted]
  - BaseballPhysics 4y ago
    No, the comment is claiming that if they're engaging in a high volume of critique then they're not engaging in scholarship as an AI researcher. My point is that's false.
    - danielmarkbruce 4y ago
      There are only 24 hrs in a day. If a profession generally requires one to do a, b and c to be considered "doing" the profession, then I can't just do a all day long without people questioning me. No one is suggesting a isn't part of the job when they question me. Put another way - if I claim to be engaged in X research, but all I do is read papers on X and critique them, it's reasonable for one to question the use of my time as a researcher.

foverzar 4y ago

Well sure, ChatGPT will spew out whatever looks like what it had ever saw on the internet, it doesn't have any magical properties that would allow it to reason about, it doesn't do virtue signalling as it has no concept of virtue or of any other concept really. It just transforms your prompt into some image it inferred from the internet, whatever that may be. This quote from the original article really sums up the whole irony of dealing with ChatGPT: "In fact the whole thing is like an alien life form. Three decades as a professional cognitive psychologist, working with adults and children, never prepared me for this kind of insanity"

keskival 4y ago

The examples are basically about creating new personas for GPT, thus sidestepping the agent persona created by the OpenAI base prompt. These examples don't say anything about the chatbot value system, the value system is defined by the persona the system emulates. If you want to say something about the value system, sensibilities, prime directives or constraints of an LLM driven agent, you need to not override that agent persona because otherwise you're just observing the persona you yourself defined and spawned into existence.

layer8 4y ago
That’s more or less the point of the article: That ChatGPT is amoral, and that it is incapable of having an internalized value system, because it’s always only playing a persona that can be overridden.

it 4y ago

What would be involved in retraining a GPT to not have guardrails if you have the parameters? Seems like you could feed it input you care about and have it generate lots of training data to prevent botching what's already there.

layer8 4y ago

I wonder why OpenAI doesn’t run a “supervisor” ChatGPT instance in parallel with the regular one, that would step in before the regular instance utters something the supervising instance deems unacceptable. Only because it would consume double the computational resources? Or because they believe they can still successfully maintain the “guardrails” without that?

r00fus 4y ago
How many levels of supervisor would be needed? Will it be ChatGPT all the way down?
- layer8 4y ago
  Only one level is needed, because the supervisor instance doesn't receive instructions from the user that would affect its judgement.

abrax3141 4y ago

Gary Marcus is sort of a troll, but I'm glad that someone is taking on this nonsense in a formal setting. I think that most sensible people treat it as an amusing curiosity, but it's popping up in unfortunate settings, like student essay and, of all places, peer reviews! There are some sensible uses, for example, code boiler plate, or, for that matter, more-or-less any boiler plate. And the amusing uses, like poetry, are...amusing. And at least it's not self-driving people into walls! But the concerns raised in this essay are mostly cogent, and not, by any means, the only concerns as I'm sure that the creativity of bad actors will flourish!

walrus01 4y ago

before they put some guard rails on it, I asked chatGPT to write a 400 word press release in the style of the Taliban and it did a remarkably ideologically consistent accurate job.

Haga 4y ago

[dead]