14 ms·

One, I think overlooked issue with AI as ML, is how radically it affords the done-and-finished past over a creative future. When faced with a blank page some p

One, I think overlooked issue with AI as ML, is how radically it affords the done-and-finished past over a creative future. When faced with a blank page some percentage of people will not copy a known solution, they'll draw on their experiences and eg., write a book about them. For example, I could write a book about software development without research only inventorying my opinions, arguing for them, illustrating them, and so on. If I were to start with ChatGPT I would, necessarily, only say what has been said. There are so many dangers with the Eliza-effect, and this is one of them. I think the narrative on how these systems work needs more strongly wrestled from their salesmen -- and the real downsides exposed. A client of mine seemed quite insistent that ChatGPT was "creative". I hope in a few years that opinion will be rare, as everyone ends up writing the same article over-and-over. And likewise, in development, using earlier versions of libraries with the most "stackoverflow training data"; using dying languages with the most github repos; and so on. In other words: what we have written the most about is not what we now need to write the most about. And the former is the training data for modern AI. It is not in the world with us, with our concerns: it is a statistical average over "done and dead" cases.

chii 4y ago
> inventorying my opinions but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. > If I were to start with ChatGPT I would, necessarily, only say what has been said. And i would also imagine that this isn't too different from someone with a vast experience behind them. And these ML models don't just repeat what has been said - i think there's enough complexity in the model where they could genuinely extract knowledge (or patterns) across different areas, and have it be applied to specific circumstances.
- iudqnolq 4y ago
  I'm less confident it innovates based on my practical experience with Copilot. It writes boring, verbose code. And it assumes APIs are written in that style as well. That effect is worse the less it knows about an API. So if you want it to be able to complete based on the rest of your code in your project you'd have to tailor your APIs to it, which I won't do. TabNine has a much more primitive model, but I like that they weigh the code on your hard drive much heavier.
  - piva00 4y ago
    Boring code is not bad code, overly verbose might be but descriptive verbose code is pretty damn good when you are maintaining something for 5-10 years.
    - saurik 4y ago
      In my experience maintaining things for even longer than that, if your code is verbose you are way more likely to make a mistake many years later because you don't remember all of the rules required to correctly edit or generate new code in the same form. You are much better off compressing that code using domain-specific abstractions that force all development to fully follow every mental model you ascribed to the model, even if it means you have to load more mental state and it takes you a bit longer to "get back up to speed" in the project you haven't edited in a while, as it helps guarantee you actually load the correct state.
    - iudqnolq 4y ago
      It depends on the specifics. I think a correct AI should look at the relevant context and give code with matching verbosity.
- Etherlord87 4y ago
  > but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. I think the OP's argument is, the ChatGPT can only average inputs into an output, as opposed to a human mind that can extrapolate. Regardless, the obvious difference is we humans build bigger understanding, a complex structure of ideas based on our inputs, and so not only we have a lot more ideas to combine, but also we use acquired ideas to rule which ideas we preliminarily choose to combine and how we combine them. So as to quote the meme > We are not the same.
  - lordnacho 4y ago
    But this is only an effect of reward maximization. If I write a poem the way everyone writes it, I won't be found out as a fraud. Which is what we've told AI to do, just do whatever you need to get a passing grade like all the human kids in the class. Why couldn't we create truly novel things by allowing for some failures? It's a bit like how there was a bunch of new musical styles after the war. Terrible flops mixed in there, but we got some interesting things as well.
    - throwaway4aday 4y ago
      I think this is something OpenAI is working on, updating their RLHF practices so that there is a wider variety of input shaping the selected responses. There's a huge difference between 3.5 and 4 due to this. Sam Altman has mentioned it a few times in podcasts he's appeared on so they are aware that it is a problem to have too narrow of a selection of reviewers. Ideally, you'd have a really broad selection of people from many different areas of expertise and tacit knowledge train a generalist model. You could then have more specialist models as well that were trained by different professions although you'd probably still want some amount of training from other areas for cross-pollination. It's kind of crazy how the model learns to act this way, we funnel in more knowledge than any one person would ever be exposed to in their lifetime and it learns some latent structures in language and knowledge from this, then we teach it to interact with humans in a more natural way. It's both different and similar to how humans learn language and then social skills. Makes me wonder what else we could teach it to do through interacting with it.
- mjburgess 4y ago
  Modern AI is just a variation on `mean()` -- the apparent intelligence comes from what it's averaging. (In particular, it's just an `Expectation_weightDistribution[TrainingData|Query]` ) If you wanted to be, I think pathologically cynical about animal intelligence, and reduce it down to the same operation, then at the very least the "TrainingData" and the "weightDistribution" is specialised to the individual -- and so different from the average. However, we cannot really be this extremely cynical (I find this pseudoscience) -- animals acquire novel training data by changing their environments. Animals are a source of data. So even on this cynical reading we have to at least go towards, while(Alive && ...) MyExperiences = ChangeTheWorld(MyExperiences|Goal) MyIdeas = Expectation_mySenseOfRelevance[MyExperiences|Goal] ... Which makes `MyIdeas` quite radically different from the `mean()`ing of the Ai.
  - mitjam 4y ago
    Interesting viewpoint, following this, it would probably be beneficial to have many different personal AIs as compared to a few AIs that serve many.
    - caddemon 4y ago
      Yeah this is a good point, independent of AI (but probably not the internet) things are already becoming more homogeneous IMO. If that's expedited by everybody using e.g. ChatGPT I don't think that would be good.
      - pas 4y ago
        How many degrees of freedom these networks have? There's a lot of room in high dimensional space :) Sure, same prompts same answers, but people are pretty creative already in their questions.
    - CuriouslyC 4y ago
      It makes more sense to have a really strong base model that is "general" and then apply LoRAs to it to "specialize" it.
- koonsolo 4y ago
  ChatGPT for sure can be creative, but it lacks 1 thing: real life world experience. For example, as a senior software developer, I'm not convinced pure unit testing is always the way to go. I was thinking about writing my own alternative and how to weigh the benefits against the drawbacks. I was querying ChatGPT about it, but it's just able to restate the current "common knowledge". So yes, ChatGPT can come up with new concepts. It however, cannot find out the problems, issues and unexpected consequences of applying it in the real world. Somehow, these things will have to be fed into it again, in either the form of text, images or video. 99% of current writing can be replaced by ChatGPT. But this 1% of things only people with real world experience can come up with, it cannot do.
  - edwardso 4y ago
    Yeah, with my interaction with ChatGPT it just spits out general concepts that almost make it seem like it exhaustively covers the prompt, but from those concepts sometimes there'll be 1 or 2 that just won't be applicable to the reality of the real world. Large language model lacks the acuteness to anticipate any unintended consequences and that 1% of edge case is where things might truly matter.
  - pillefitz 4y ago
    ChatGPT lists (common) downsides, remedies and situations when to avoid unit testing. Do you have the expected answer in mind, and could chatGPT have possibly deduced this from your prompt?
    - koonsolo 4y ago
      My problem and solution is probably too wide to explain here, so let's take an easier example. TDD where you write the test before the code, is a widespread concept now. Imagine a time before this was the case. Would ChatGPT be able to come up with such a concept? Probably. Would it be able to really understand all the consequences of such a setup? I highly doubt it. Even if it did, would we trust it? In my opinion, such things can only be tested in the real world, where you work with real projects, developers and testers, and gather all the impacts. I don't think a LLM could derive such things.
- rcme 4y ago
  > but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. Of course opinions are partially formed by preexisting knowledge learnt by your past self. But the fundamental difference between yourself and an ML model, which TFA touches on, is that an ML model stops learning after training whereas your past self did learn and your current self continues to learn. “Learn” here means something akin to updating the model weights of your brain. And you see this in practice. Ask ChatGPT about something not in the training set and it doesn’t answer well whereas a human is capable of learning about a topic and incorporating that knowledge in the future.
- tomxor 4y ago
  > but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. Knowledge != experience, there is overlap for sure, but they are qualitatively different things. And that's before we even get into the underlying differences in the process: when GPT writes a book it's predicting desired output based on what it's learned from it's training data, but it's only as deep as language, coherant logic that appears deeper is incidental and only as common as it is due to the sheer amount of data it's consumed and statistical correlation - all of the amusement we get from ChatGPT is when this lack of depth is revealed. When a human writes a book they are using the same mechanisms for sure, predicting desired output, but that's only the language level, bellow that is a more abstract process driving the direction and ideation, introspection and interrogating experiences, exploring and validating them before attempting to express them in language. I think part of what makes these LLMs appear to be smarter and more creative than they have any right to be, is due to the inhuman level of input... they are extremely shallow models, but have massive bandwidth, way more than a human could consume - as a result clever stuff emerges more though the sheer volume than something intrinsic to the model itself.
cowl 4y ago
>If I were to start with ChatGPT I would, necessarily, only say what has been said. You would but not chatGPT. I have been very vocal about cautioning people that LLM are just a parrot, but that is in the sense that there is no real meaning or "reasoning" behind it. However Its also wrong to say that there is no Creativity. ChatGPT could very well produce a creative solution just as a byproduct of the sheer amount of weights. When you consider what to write you draw on your past experiences but your ability to make connection between things (and the amount of your experiences) is very limited. ChatPGT could very well stumble upon an obscure connection between two words that is improbable for humans to make because of the sheer dimensions of connectivity of the graph. That still wouldn't mean that the model has "learned" that inferrence though, it would still be more like the result of the million monkeys typing and writing a shakespeare play.
- illegalmemory 4y ago
  > ChatPGT could very well stumble upon an obscure connection between two words that is improbable for humans to make But is there any guarantee that ChatGPT will consistently arrive at same results over and over ?
  - parineum 4y ago
    What would that be a test/condition for?
  - cowl 4y ago
    No, and that's why I also say that it has not "learned" or "reasoned". it may very well make a connection between A and B and miss the same type of connection between C and D. or also miss the same A - B connection in the presence of other variables F,G,H. So the LLM cannot guarantee the generalization of the problem although it may solve a particular instance of that problem.
- mjburgess 4y ago
  People often talk about AI apart from its optimization objective -- as-if it "could" find anything in the data... as-if it could find what we could find it in. No! The AI is radically imprisoned by its optimization objective. What it must necessarily find is whatever "pattern of associations" maximises "correlations between the items in the data". Necessarily, an AI looking at the sky can never find newton's law of gravity. Newton (et al.) had to disregard what they were looking at to place Earth in the centre and F equal to GMm/r2 etc. etc. The AI isnt "finding patterns in data", it's taking data and maximising a single objective. A very very very strange one, which necessarily treats its data as profoundly irrationally superstitious person would. Are there useful such patterns? Sure. But an AI isnt "finding patterns" as we might -- it's finding one specific kind of pattern which we can characterise quite easily. We know how useful those kinds of patterns are.
abigail95 4y ago
Does a well read human produce more, or less creative work than one that is less well read?
- froggychairs 4y ago
  What a fun question! Obviously the answer is that its a non-question. The amount an author reads doesn’t correlate to success. Plenty of authors are voracious readers, others rarely pick up a book. What do you see with a lot of creative authors (but not all) is a dedication to the craft. Putting in hours of writing. Often editing or just tossing everything. I have noticed though. People on my team best capable of writing technical documentation. Read a lot of technical documentation. This is the exact domain GPT can Excel at, and one where generic language is accepted.
  - pieix 4y ago
    > The amount an author reads doesn’t correlate to success. This is, in practice, not true — good writing doesn’t emerge spontaneously, it’s nurtured by reading other good writing. As Cormac McCarthy put it, "the ugly fact is books are made out of books."
    - notahacker 4y ago
      Nobody's saying reading other good writing doesn't help in any way, they're saying that quantity read doesn't particularly correlate with quality of output. I mean, I doubt Shakespeare read as much good quality English writing as the average human today, but the quality of his visual imagery and wordplay is generally considered to be on another level (and at the other end of the scale, his plots were heavily borrowed from the relatively small corpus of histories and dramas he was exposed to, whereas many writers far more widely read and educated in tropes and genre conventions resolved not to use them and came up with more imaginative settings and storylines... ). Many very well read people are competent enough in the use of language but will never write prose other people particularly want to read.
- mjburgess 4y ago
dr_dshiv 4y ago
> If I were to start with ChatGPT I would, necessarily, only say what has been said. That is simply untrue. I think you vastly underestimate the space of possibility.
- caddemon 4y ago
  Eh I think it's untrue prose wise, and I can buy a non-anthropomorphizing argument that ChatGPT can be creative with language use. However I don't think it's untrue when it comes to being truly creative with ideas. If you ask it about less well established research (but still well known about within whatever scientific community circa 2021) it utterly fails to reason about it or be creative about it at all. It often says wrong things but forgetting that it just doesn't coherently put together different ideas that weren't in its training set IME. Like well below the bar for a conversation with an average lay person about whatever biology topic. Tbf I haven't tried this much with ChatGPT4, but initial impressions weren't great and I played around with the original release quite a bit. I'd say I was impressed with the language but very rarely with the content.
  - dr_dshiv 4y ago
    I have soo many counter examples… maybe just keep trying?
    - caddemon 4y ago
      Care to share any?
  - fauxpause_ 4y ago
    I talked to a kid about research he hadn’t seen before and I wasn’t impressed by the language or the content. TBF, he was a kid and he didn’t know why I was talking about. But I think it’s ok to generalize to all humans.
DharmaPolice 4y ago
If someone takes samples from a bunch of songs and mixes them together in way that sounds like a new track - that's creative, right? Wouldn't you apply that to some of what ChatGPT does? You can ask for a story with oddly specific elements and you'll get a story which has never been written before. It's becoming harder and harder to say it's not creative (even if you include the prompter as part of the creative process).
- layer8 4y ago
  If you look at what songs are being written today, it’s quite different from what was written a hundred years ago. ChatGPT, on the other hand, would still create the same stuff a hundred years from now that it is creating now. (Same for Midwinter, etc.) What current AIs are missing, at the very least, is learning and exploration, and learning from its own learning and exploration.
- pbhjpbhj 4y ago
  People often argued that a Jackson Pollock style of artwork is not art because they just threw paint on a canvas. Of course, Pollock did think about the process, there's more form to his works than that. Similarly, if you just arbitrarily mash music samples together you're unlikely to get anything that's really listenable music ... you need a little more for it too be creative. Indeed, we often describe work as "derivative" when it's not particularly creative, it lacks spark. That seems like the definition of generative models. But then probably most human artwork is substantially derivative (in the art-critic sense). If you provide "oddly specific input", then you've provided the creative seed, growing that seed is impressive work, but it is work with a different fundamental character.
mejutoco 4y ago
My personal take on this: only humans are creative. If a human is using ChatGPT as a tool then that is creativity. Same as using a sampler or DJing or saying a urinal is art (found objects). If the process does not involve a human, we simply do not call it creative, even if it is the same (which it is not clear it is).
- psychphysic 4y ago
  I have no strong opinion on the matter but what is clear is that humans struggle to make use of AI. This is not a novel position. Ghost writers and biography writers exist. Why not make ChatGPT you ghost writer? Tell it what to talk about. Tell it to adjust it to reflect you. And then do the final draft yourself. Obviously if you ask ChatGPT to just write you a book the result will be pants.
  - mejutoco 4y ago
    > A client of mine seemed quite insistent that ChatGPT was "creative" I was addressing this point of OP. I agree my POV is not novel, but I believe it is consistent (and simple). > Why not make ChatGPT you ghost writer? Why not. As long as a human is driving the tool, the output will be creative. If the human has nothing to say the output will not be very interesting anyway.
  - chongli 4y ago
    I'm skeptical of how useful ChatGPT would be for writing. Truly great writing is a painstaking process. Writers go through many drafts to craft great sentences. Read Hemingway and it's not at all apparent why. But try to write Hemingway? Forget about it! Asking ChatGPT to rewrite a particular sentence over and over again, until it's perfect, does not sound like a particularly efficient or meaningful use of one's time. You've replaced the process of crafting a sentence with the process of crafting a prompt and the relationship between the two is non-obvious. Now I suppose someone will actually try to do it, as some kind of writing exercise. Like that book written without the letter 'e' [1]. [1] https://en.wikipedia.org/wiki/A_Void https://en.wikipedia.org/wiki/A_Void
mnky9800n 4y ago
In my experience, ChatGPT is only as clever or knowledgeable about a topic as I am. It is a multiplier of my ability, not an addition.
lifeisstillgood 4y ago
Just wanted to say thanks for expressing something I did not realise I was worried about. Yes it's great to have something that can summarise all human knowledge - kinda. But what we need is to move on. I suppose it's a bit like the "read the papers" research prior to starting a phd.
throwaway22032 4y ago
A better way of describing this might be to say that right now, GPT learns from what others have written. I can learn from what others have written, but I can also learn by actually running scripts in a terminal and viewing their output. By actually laying floor tiles and iterating on the process. etc.
- PeterisP 4y ago
  With its plugin system, ChatGPT and/or similar models can also learn by actually running scripts in a terminal (or looking at recent webcrawl document, or querying an API) and viewing its output. The components are there, it works in principle, and this integration is currently ongoing.
AndrewThrowaway 4y ago
This is a fundamental truth everybody needs to understand. Not my idea, heard it somewhere. That the crucial difference between a human being and AI is that if you show a 3 year old kid one picture of a cat, a kid can recognize all other cats. Was it a lion or a tiger. You can feed ML 5000 pictures of cats and it can recognize a cat in a picture with something like 95% confidence.
- lm28469 4y ago
  And if you fold a napkin in a weird way with the correct lighting it might flag it as a tiger too
  - MayeulC 4y ago
    People watch clouds for that reason too!
- TheUndead96 4y ago
  This is an important distinction but it simplifies the situation in my opinion. A 3 year old may only see one cat, but it has probably seen many other things in its life already. Humans likely also have prewired neurology to recognise eyes and other common features. So the analogy is seem more to me like one-shot or few-shot learning with an already partially trained model.
- jhanschoo 4y ago
  > That the crucial difference between a human being and AI is that if you show a 3 year old kid one picture of a cat, a kid can recognize all other cats. Was it a lion or a tiger. I don't understand what you mean with this, as that was certainly not me as a child. As a child I thought cats and dogs might be differently sexed animals of the same species. I also thought that the big cats could be related to each other, though how any of them was related to the housecat was beyond me, given the size difference.
  - coldtea 4y ago
    >As a child I thought cats and dogs might be differently sexed animals of the same species. Sounds irrelevant to the parent's point. Which isn't that you knew what a cat was (with regards to taxonomy or whatever), but that you could tell one from a dog or a fire hydrant.
Extasia785 4y ago
> If I were to start with ChatGPT I would, necessarily, only say what has been said. This is not true though, you can test it for yourself: Prompt: Invent a new word, which has never been written before. When I type it in Google, I should get zero results. I'm completely transparent, I tried this query 3 times because of my own curiosity and will provide all 3 answers (I added --- for formatting): --- ChatGPT (answer 1): Sure, how about "flumtraz"? --- ChatGPT (answer 2): Here's a new word: "Ardidio" It is a combination of "ardor" (meaning passion or enthusiasm) and "audio" (meaning sound or hearing). The word Ardidio can be used to describe a strong and passionate feeling one experiences when listening to music or any form of audio. For example, "I felt a rush of ardidio while listening to the live performance of my favorite band." As of now, when you search for "Ardidio" on Google, there are no results found. --- ChatGPT (answer 3): "Flumixen" - a verb that means to mix or blend something in a clumsy or awkward manner. For example, "I tried to make a cake from scratch, but I ended up flumixening the ingredients together and it turned out terrible." --- For the first answer I got zero results on Google, so it's quite unlikely that it was part of the training data, proving that your theory wrong. It can say things that have never been said. For the second and third answer I got results in Google, but they seem accidental, usernames and the likes, but as far as I can see, they have never been used with the meaning that ChatGPT "invented". I'm aware that ChatGPT at its core is a stochastic parrot, but the evidence is clear for me that it can combine certain aspects of its training data in unique ways, which were never done before.
- anentropic 4y ago
  these are great words!
  - oxfordmale 4y ago
    Wordastic!
- aerhardt 4y ago
  I ran your exact prompt, this is what I got: > Zykluptogibbera > Quixilantrofen > Zylprenostim Satisfies the prompt sure enough, but they're hardly creative. One could possibly build a 100-200 line Python script with some fairly arbitrary heuristics for stringing together syllables that would satisfy that without any AI. Appended: "Make it sound as plausible as possible" > Flarishimble > Quindextrove > Crombifilate Still not amazed. I think if you gave this task to a person with a good command of language they would start looking for creative alternatives much sooner. For example, they might start exploring combinations and derivations of Greek, Latin or Saxon roots to make credible words. I'm not denying it's not creative, for the record. I'm just skeptical, and I don't think this task proves it. Edit: for reference, ran all of the above on ChatGPT4.
amcoastal 4y ago
[dead]
ekianjo 4y ago
> or example, I could write a book about software development without research only inventorying my opinions, arguing for them, illustrating them, and so on. You have a great deal of confidence if you think that you have a lot of original thoughts that have never been expressed by anyone before
golol 4y ago
> If I were to start with ChatGPT I would, necessarily, only say what has been said. Perhaps with ChatGPT you would actually go ahead and write that book, adding those crucial 10% of key ideas which are yours and letting it help you fill out the 90% of standard but necessary content. > If I were to start with ChatGPT I would, necessarily, only say what has been said. This is a bold claim I don't aee any evidence for. If you sample a model outaude its data (which happens all the time(, you expect to get results outside of its data.
boringuser2 4y ago
This is just an absurd claim. Of course AI draws on past experiences to build new things. This is also what humans do. There is no invisible well of "creativity" that humans draw from that AI can't. In fact, in areas that AI has "solved", such as chess, human experts describe AI as extremely "creative" beyond their understanding. "Creative" is just a word for connecting existing concepts in ways that seem novel. AI does this when it solves a new problem. AI clearly has the potential to make better connections, much faster than humans, in pretty much all domains. There's only one piece "missing" that is already technically possible, but expensive: giving an LLM the ability to "learn". In the near future, all creative work will be produced by AI. This will likely come before all work is produced by AI, as creative work is typically less precise.
- sudosysgen 4y ago
  Humans can seek out new experiences optimized for what they want to learn, which is fundamentally different. Creativity is not just connecting existing concepts. The pinnacle of creativity is inventing new concepts ab nihilo, which is something that every human does through self-directed interaction with the environment.
  - boringuser2 4y ago
    That's simply not true and ties strongly to a metaphysical claim that isn't substantiated.
    - sudosysgen 4y ago
      Instead of giving a trite dismissal, give a substantiaded rebuttal. I don't see any metaphysical claim in my comment. It's objectively true that humans will seek out or create experiences that allow them to obtain better understanding, and it's also objectively true that humans can create entirely original concepts, so long as you agree that there is such a thing. If you don't think that exists, then your entire comment is pointless, because then you're arguing by hidden definition, which I'd rather assume you aren't.
simplyluke 4y ago
Several people have observed that in a world where LLMs are widely influential, original creative thought will become even more of a valued differentiator than it already is.
freehorse 4y ago
Really spot-on, I agree this is the most important issue with inherent limitations of LLMs: not LLMs themselves and what they are supposed to do, but how a lot of people do not understand the limits of language and equating human knowledge and experience with language and "what has already been written". Tbh I find it quite disappointing from an intellectual/philosophical perspective that so many people think that the realm of human experience can be reduced to communicative language (and even more to "what has already been written in the internet") thus equating human creativity with LLMs' creativity. My fear is that people may not realise/care about this issue till it is too late (eg when journalism degrades to "the same article is written over and over"). The reason is that the bulk of "human creativity" is in fact reducible to what LLMs do: building a standard website, a lot of "creative writing" and art in general is to a big part not "innovation" that comes from experience (and in general from non-written-linguistic domains), at least in a lot of fields. I am afraid that because the bulk of work for a lot of fields is like that, especially exactly that kind of boring, repeatitive work that gives most of the money to survive and AI is better than humans at (ie cheaper), we may even risk having degradation on many fields that we may not even be able to notice.
1vuio0pswjnm7 4y ago
What about so-called synthetic training data. For example, does the synthesis ever involve creativity. Note: The term "creativity" might not be the best term for this discussion. Unless I am mistaken, the issue is actually originality.
cubefox 4y ago
> A client of mine seemed quite insistent that ChatGPT was "creative". I hope in a few years that opinion will be rare, as everyone ends up writing the same article over-and-over. It seems rather probable that in a few years GPTs would become far more powerful, including more creative than most humans.
cratermoon 4y ago
The so-called Stochastic Parrot paper mentions some specific harms that the "done-and-finished" aspect. in Section 4.2, "Static Data/Changing Social Views", saying, > A central aspect of social movement formation involves using language strategically to destabilize dominant narratives and call attention to underrepresented social perspectives. Social movements produce new norms, language, and ways of communicating. This adds challenges to the deployment of LMs, as methodologies reliant on LMs run the risk of ‘value-lock’, where the LM-reliant technology reifies older, less-inclusive understandings.

chii 4y ago

> inventorying my opinions but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. > If I were to start with ChatGPT I would, necessarily, only say what has been said. And i would also imagine that this isn't too different from someone with a vast experience behind them. And these ML models don't just repeat what has been said - i think there's enough complexity in the model where they could genuinely extract knowledge (or patterns) across different areas, and have it be applied to specific circumstances.

iudqnolq 4y ago
I'm less confident it innovates based on my practical experience with Copilot. It writes boring, verbose code. And it assumes APIs are written in that style as well. That effect is worse the less it knows about an API. So if you want it to be able to complete based on the rest of your code in your project you'd have to tailor your APIs to it, which I won't do. TabNine has a much more primitive model, but I like that they weigh the code on your hard drive much heavier.
- piva00 4y ago
  Boring code is not bad code, overly verbose might be but descriptive verbose code is pretty damn good when you are maintaining something for 5-10 years.
  - saurik 4y ago
    In my experience maintaining things for even longer than that, if your code is verbose you are way more likely to make a mistake many years later because you don't remember all of the rules required to correctly edit or generate new code in the same form. You are much better off compressing that code using domain-specific abstractions that force all development to fully follow every mental model you ascribed to the model, even if it means you have to load more mental state and it takes you a bit longer to "get back up to speed" in the project you haven't edited in a while, as it helps guarantee you actually load the correct state.
  - iudqnolq 4y ago
    It depends on the specifics. I think a correct AI should look at the relevant context and give code with matching verbosity.
Etherlord87 4y ago
> but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. I think the OP's argument is, the ChatGPT can only average inputs into an output, as opposed to a human mind that can extrapolate. Regardless, the obvious difference is we humans build bigger understanding, a complex structure of ideas based on our inputs, and so not only we have a lot more ideas to combine, but also we use acquired ideas to rule which ideas we preliminarily choose to combine and how we combine them. So as to quote the meme > We are not the same.
- lordnacho 4y ago
  But this is only an effect of reward maximization. If I write a poem the way everyone writes it, I won't be found out as a fraud. Which is what we've told AI to do, just do whatever you need to get a passing grade like all the human kids in the class. Why couldn't we create truly novel things by allowing for some failures? It's a bit like how there was a bunch of new musical styles after the war. Terrible flops mixed in there, but we got some interesting things as well.
  - throwaway4aday 4y ago
    I think this is something OpenAI is working on, updating their RLHF practices so that there is a wider variety of input shaping the selected responses. There's a huge difference between 3.5 and 4 due to this. Sam Altman has mentioned it a few times in podcasts he's appeared on so they are aware that it is a problem to have too narrow of a selection of reviewers. Ideally, you'd have a really broad selection of people from many different areas of expertise and tacit knowledge train a generalist model. You could then have more specialist models as well that were trained by different professions although you'd probably still want some amount of training from other areas for cross-pollination. It's kind of crazy how the model learns to act this way, we funnel in more knowledge than any one person would ever be exposed to in their lifetime and it learns some latent structures in language and knowledge from this, then we teach it to interact with humans in a more natural way. It's both different and similar to how humans learn language and then social skills. Makes me wonder what else we could teach it to do through interacting with it.
mjburgess 4y ago
Modern AI is just a variation on `mean()` -- the apparent intelligence comes from what it's averaging. (In particular, it's just an `Expectation_weightDistribution[TrainingData|Query]` ) If you wanted to be, I think pathologically cynical about animal intelligence, and reduce it down to the same operation, then at the very least the "TrainingData" and the "weightDistribution" is specialised to the individual -- and so different from the average. However, we cannot really be this extremely cynical (I find this pseudoscience) -- animals acquire novel training data by changing their environments. Animals are a source of data. So even on this cynical reading we have to at least go towards, while(Alive && ...) MyExperiences = ChangeTheWorld(MyExperiences|Goal) MyIdeas = Expectation_mySenseOfRelevance[MyExperiences|Goal] ... Which makes `MyIdeas` quite radically different from the `mean()`ing of the Ai.
- mitjam 4y ago
  Interesting viewpoint, following this, it would probably be beneficial to have many different personal AIs as compared to a few AIs that serve many.
  - caddemon 4y ago
    Yeah this is a good point, independent of AI (but probably not the internet) things are already becoming more homogeneous IMO. If that's expedited by everybody using e.g. ChatGPT I don't think that would be good.
    - pas 4y ago
      How many degrees of freedom these networks have? There's a lot of room in high dimensional space :) Sure, same prompts same answers, but people are pretty creative already in their questions.
  - CuriouslyC 4y ago
    It makes more sense to have a really strong base model that is "general" and then apply LoRAs to it to "specialize" it.
koonsolo 4y ago
ChatGPT for sure can be creative, but it lacks 1 thing: real life world experience. For example, as a senior software developer, I'm not convinced pure unit testing is always the way to go. I was thinking about writing my own alternative and how to weigh the benefits against the drawbacks. I was querying ChatGPT about it, but it's just able to restate the current "common knowledge". So yes, ChatGPT can come up with new concepts. It however, cannot find out the problems, issues and unexpected consequences of applying it in the real world. Somehow, these things will have to be fed into it again, in either the form of text, images or video. 99% of current writing can be replaced by ChatGPT. But this 1% of things only people with real world experience can come up with, it cannot do.
- edwardso 4y ago
  Yeah, with my interaction with ChatGPT it just spits out general concepts that almost make it seem like it exhaustively covers the prompt, but from those concepts sometimes there'll be 1 or 2 that just won't be applicable to the reality of the real world. Large language model lacks the acuteness to anticipate any unintended consequences and that 1% of edge case is where things might truly matter.
- pillefitz 4y ago
  ChatGPT lists (common) downsides, remedies and situations when to avoid unit testing. Do you have the expected answer in mind, and could chatGPT have possibly deduced this from your prompt?
  - koonsolo 4y ago
    My problem and solution is probably too wide to explain here, so let's take an easier example. TDD where you write the test before the code, is a widespread concept now. Imagine a time before this was the case. Would ChatGPT be able to come up with such a concept? Probably. Would it be able to really understand all the consequences of such a setup? I highly doubt it. Even if it did, would we trust it? In my opinion, such things can only be tested in the real world, where you work with real projects, developers and testers, and gather all the impacts. I don't think a LLM could derive such things.
rcme 4y ago
> but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. Of course opinions are partially formed by preexisting knowledge learnt by your past self. But the fundamental difference between yourself and an ML model, which TFA touches on, is that an ML model stops learning after training whereas your past self did learn and your current self continues to learn. “Learn” here means something akin to updating the model weights of your brain. And you see this in practice. Ask ChatGPT about something not in the training set and it doesn’t answer well whereas a human is capable of learning about a topic and incorporating that knowledge in the future.
tomxor 4y ago
> but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. Knowledge != experience, there is overlap for sure, but they are qualitatively different things. And that's before we even get into the underlying differences in the process: when GPT writes a book it's predicting desired output based on what it's learned from it's training data, but it's only as deep as language, coherant logic that appears deeper is incidental and only as common as it is due to the sheer amount of data it's consumed and statistical correlation - all of the amusement we get from ChatGPT is when this lack of depth is revealed. When a human writes a book they are using the same mechanisms for sure, predicting desired output, but that's only the language level, bellow that is a more abstract process driving the direction and ideation, introspection and interrogating experiences, exploring and validating them before attempting to express them in language. I think part of what makes these LLMs appear to be smarter and more creative than they have any right to be, is due to the inhuman level of input... they are extremely shallow models, but have massive bandwidth, way more than a human could consume - as a result clever stuff emerges more though the sheer volume than something intrinsic to the model itself.

cowl 4y ago

>If I were to start with ChatGPT I would, necessarily, only say what has been said. You would but not chatGPT. I have been very vocal about cautioning people that LLM are just a parrot, but that is in the sense that there is no real meaning or "reasoning" behind it. However Its also wrong to say that there is no Creativity. ChatGPT could very well produce a creative solution just as a byproduct of the sheer amount of weights. When you consider what to write you draw on your past experiences but your ability to make connection between things (and the amount of your experiences) is very limited. ChatPGT could very well stumble upon an obscure connection between two words that is improbable for humans to make because of the sheer dimensions of connectivity of the graph. That still wouldn't mean that the model has "learned" that inferrence though, it would still be more like the result of the million monkeys typing and writing a shakespeare play.

illegalmemory 4y ago
> ChatPGT could very well stumble upon an obscure connection between two words that is improbable for humans to make But is there any guarantee that ChatGPT will consistently arrive at same results over and over ?
- parineum 4y ago
  What would that be a test/condition for?
- cowl 4y ago
  No, and that's why I also say that it has not "learned" or "reasoned". it may very well make a connection between A and B and miss the same type of connection between C and D. or also miss the same A - B connection in the presence of other variables F,G,H. So the LLM cannot guarantee the generalization of the problem although it may solve a particular instance of that problem.
mjburgess 4y ago
People often talk about AI apart from its optimization objective -- as-if it "could" find anything in the data... as-if it could find what we could find it in. No! The AI is radically imprisoned by its optimization objective. What it must necessarily find is whatever "pattern of associations" maximises "correlations between the items in the data". Necessarily, an AI looking at the sky can never find newton's law of gravity. Newton (et al.) had to disregard what they were looking at to place Earth in the centre and F equal to GMm/r2 etc. etc. The AI isnt "finding patterns in data", it's taking data and maximising a single objective. A very very very strange one, which necessarily treats its data as profoundly irrationally superstitious person would. Are there useful such patterns? Sure. But an AI isnt "finding patterns" as we might -- it's finding one specific kind of pattern which we can characterise quite easily. We know how useful those kinds of patterns are.

abigail95 4y ago

Does a well read human produce more, or less creative work than one that is less well read?

froggychairs 4y ago
What a fun question! Obviously the answer is that its a non-question. The amount an author reads doesn’t correlate to success. Plenty of authors are voracious readers, others rarely pick up a book. What do you see with a lot of creative authors (but not all) is a dedication to the craft. Putting in hours of writing. Often editing or just tossing everything. I have noticed though. People on my team best capable of writing technical documentation. Read a lot of technical documentation. This is the exact domain GPT can Excel at, and one where generic language is accepted.
- pieix 4y ago
  > The amount an author reads doesn’t correlate to success. This is, in practice, not true — good writing doesn’t emerge spontaneously, it’s nurtured by reading other good writing. As Cormac McCarthy put it, "the ugly fact is books are made out of books."
  - notahacker 4y ago
    Nobody's saying reading other good writing doesn't help in any way, they're saying that quantity read doesn't particularly correlate with quality of output. I mean, I doubt Shakespeare read as much good quality English writing as the average human today, but the quality of his visual imagery and wordplay is generally considered to be on another level (and at the other end of the scale, his plots were heavily borrowed from the relatively small corpus of histories and dramas he was exposed to, whereas many writers far more widely read and educated in tropes and genre conventions resolved not to use them and came up with more imaginative settings and storylines... ). Many very well read people are competent enough in the use of language but will never write prose other people particularly want to read.
mjburgess 4y ago

dr_dshiv 4y ago

> If I were to start with ChatGPT I would, necessarily, only say what has been said. That is simply untrue. I think you vastly underestimate the space of possibility.

caddemon 4y ago
Eh I think it's untrue prose wise, and I can buy a non-anthropomorphizing argument that ChatGPT can be creative with language use. However I don't think it's untrue when it comes to being truly creative with ideas. If you ask it about less well established research (but still well known about within whatever scientific community circa 2021) it utterly fails to reason about it or be creative about it at all. It often says wrong things but forgetting that it just doesn't coherently put together different ideas that weren't in its training set IME. Like well below the bar for a conversation with an average lay person about whatever biology topic. Tbf I haven't tried this much with ChatGPT4, but initial impressions weren't great and I played around with the original release quite a bit. I'd say I was impressed with the language but very rarely with the content.
- dr_dshiv 4y ago
  I have soo many counter examples… maybe just keep trying?
  - caddemon 4y ago
    Care to share any?
- fauxpause_ 4y ago
  I talked to a kid about research he hadn’t seen before and I wasn’t impressed by the language or the content. TBF, he was a kid and he didn’t know why I was talking about. But I think it’s ok to generalize to all humans.

DharmaPolice 4y ago

If someone takes samples from a bunch of songs and mixes them together in way that sounds like a new track - that's creative, right? Wouldn't you apply that to some of what ChatGPT does? You can ask for a story with oddly specific elements and you'll get a story which has never been written before. It's becoming harder and harder to say it's not creative (even if you include the prompter as part of the creative process).

layer8 4y ago
If you look at what songs are being written today, it’s quite different from what was written a hundred years ago. ChatGPT, on the other hand, would still create the same stuff a hundred years from now that it is creating now. (Same for Midwinter, etc.) What current AIs are missing, at the very least, is learning and exploration, and learning from its own learning and exploration.
pbhjpbhj 4y ago
People often argued that a Jackson Pollock style of artwork is not art because they just threw paint on a canvas. Of course, Pollock did think about the process, there's more form to his works than that. Similarly, if you just arbitrarily mash music samples together you're unlikely to get anything that's really listenable music ... you need a little more for it too be creative. Indeed, we often describe work as "derivative" when it's not particularly creative, it lacks spark. That seems like the definition of generative models. But then probably most human artwork is substantially derivative (in the art-critic sense). If you provide "oddly specific input", then you've provided the creative seed, growing that seed is impressive work, but it is work with a different fundamental character.

mejutoco 4y ago

My personal take on this: only humans are creative. If a human is using ChatGPT as a tool then that is creativity. Same as using a sampler or DJing or saying a urinal is art (found objects). If the process does not involve a human, we simply do not call it creative, even if it is the same (which it is not clear it is).

psychphysic 4y ago
I have no strong opinion on the matter but what is clear is that humans struggle to make use of AI. This is not a novel position. Ghost writers and biography writers exist. Why not make ChatGPT you ghost writer? Tell it what to talk about. Tell it to adjust it to reflect you. And then do the final draft yourself. Obviously if you ask ChatGPT to just write you a book the result will be pants.
- mejutoco 4y ago
  > A client of mine seemed quite insistent that ChatGPT was "creative" I was addressing this point of OP. I agree my POV is not novel, but I believe it is consistent (and simple). > Why not make ChatGPT you ghost writer? Why not. As long as a human is driving the tool, the output will be creative. If the human has nothing to say the output will not be very interesting anyway.
- chongli 4y ago
  I'm skeptical of how useful ChatGPT would be for writing. Truly great writing is a painstaking process. Writers go through many drafts to craft great sentences. Read Hemingway and it's not at all apparent why. But try to write Hemingway? Forget about it! Asking ChatGPT to rewrite a particular sentence over and over again, until it's perfect, does not sound like a particularly efficient or meaningful use of one's time. You've replaced the process of crafting a sentence with the process of crafting a prompt and the relationship between the two is non-obvious. Now I suppose someone will actually try to do it, as some kind of writing exercise. Like that book written without the letter 'e' [1]. [1] https://en.wikipedia.org/wiki/A_Void https://en.wikipedia.org/wiki/A_Void

mnky9800n 4y ago

In my experience, ChatGPT is only as clever or knowledgeable about a topic as I am. It is a multiplier of my ability, not an addition.

lifeisstillgood 4y ago

Just wanted to say thanks for expressing something I did not realise I was worried about. Yes it's great to have something that can summarise all human knowledge - kinda. But what we need is to move on. I suppose it's a bit like the "read the papers" research prior to starting a phd.

throwaway22032 4y ago

A better way of describing this might be to say that right now, GPT learns from what others have written. I can learn from what others have written, but I can also learn by actually running scripts in a terminal and viewing their output. By actually laying floor tiles and iterating on the process. etc.

PeterisP 4y ago
With its plugin system, ChatGPT and/or similar models can also learn by actually running scripts in a terminal (or looking at recent webcrawl document, or querying an API) and viewing its output. The components are there, it works in principle, and this integration is currently ongoing.

AndrewThrowaway 4y ago

This is a fundamental truth everybody needs to understand. Not my idea, heard it somewhere. That the crucial difference between a human being and AI is that if you show a 3 year old kid one picture of a cat, a kid can recognize all other cats. Was it a lion or a tiger. You can feed ML 5000 pictures of cats and it can recognize a cat in a picture with something like 95% confidence.

lm28469 4y ago
And if you fold a napkin in a weird way with the correct lighting it might flag it as a tiger too
- MayeulC 4y ago
  People watch clouds for that reason too!
TheUndead96 4y ago
This is an important distinction but it simplifies the situation in my opinion. A 3 year old may only see one cat, but it has probably seen many other things in its life already. Humans likely also have prewired neurology to recognise eyes and other common features. So the analogy is seem more to me like one-shot or few-shot learning with an already partially trained model.
jhanschoo 4y ago
> That the crucial difference between a human being and AI is that if you show a 3 year old kid one picture of a cat, a kid can recognize all other cats. Was it a lion or a tiger. I don't understand what you mean with this, as that was certainly not me as a child. As a child I thought cats and dogs might be differently sexed animals of the same species. I also thought that the big cats could be related to each other, though how any of them was related to the housecat was beyond me, given the size difference.
- coldtea 4y ago
  >As a child I thought cats and dogs might be differently sexed animals of the same species. Sounds irrelevant to the parent's point. Which isn't that you knew what a cat was (with regards to taxonomy or whatever), but that you could tell one from a dog or a fire hydrant.

Extasia785 4y ago

> If I were to start with ChatGPT I would, necessarily, only say what has been said. This is not true though, you can test it for yourself: Prompt: Invent a new word, which has never been written before. When I type it in Google, I should get zero results. I'm completely transparent, I tried this query 3 times because of my own curiosity and will provide all 3 answers (I added --- for formatting): --- ChatGPT (answer 1): Sure, how about "flumtraz"? --- ChatGPT (answer 2): Here's a new word: "Ardidio" It is a combination of "ardor" (meaning passion or enthusiasm) and "audio" (meaning sound or hearing). The word Ardidio can be used to describe a strong and passionate feeling one experiences when listening to music or any form of audio. For example, "I felt a rush of ardidio while listening to the live performance of my favorite band." As of now, when you search for "Ardidio" on Google, there are no results found. --- ChatGPT (answer 3): "Flumixen" - a verb that means to mix or blend something in a clumsy or awkward manner. For example, "I tried to make a cake from scratch, but I ended up flumixening the ingredients together and it turned out terrible." --- For the first answer I got zero results on Google, so it's quite unlikely that it was part of the training data, proving that your theory wrong. It can say things that have never been said. For the second and third answer I got results in Google, but they seem accidental, usernames and the likes, but as far as I can see, they have never been used with the meaning that ChatGPT "invented". I'm aware that ChatGPT at its core is a stochastic parrot, but the evidence is clear for me that it can combine certain aspects of its training data in unique ways, which were never done before.

anentropic 4y ago
these are great words!
- oxfordmale 4y ago
  Wordastic!
aerhardt 4y ago
I ran your exact prompt, this is what I got: > Zykluptogibbera > Quixilantrofen > Zylprenostim Satisfies the prompt sure enough, but they're hardly creative. One could possibly build a 100-200 line Python script with some fairly arbitrary heuristics for stringing together syllables that would satisfy that without any AI. Appended: "Make it sound as plausible as possible" > Flarishimble > Quindextrove > Crombifilate Still not amazed. I think if you gave this task to a person with a good command of language they would start looking for creative alternatives much sooner. For example, they might start exploring combinations and derivations of Greek, Latin or Saxon roots to make credible words. I'm not denying it's not creative, for the record. I'm just skeptical, and I don't think this task proves it. Edit: for reference, ran all of the above on ChatGPT4.

amcoastal 4y ago

[dead]

ekianjo 4y ago

> or example, I could write a book about software development without research only inventorying my opinions, arguing for them, illustrating them, and so on. You have a great deal of confidence if you think that you have a lot of original thoughts that have never been expressed by anyone before

golol 4y ago

> If I were to start with ChatGPT I would, necessarily, only say what has been said. Perhaps with ChatGPT you would actually go ahead and write that book, adding those crucial 10% of key ideas which are yours and letting it help you fill out the 90% of standard but necessary content. > If I were to start with ChatGPT I would, necessarily, only say what has been said. This is a bold claim I don't aee any evidence for. If you sample a model outaude its data (which happens all the time(, you expect to get results outside of its data.

boringuser2 4y ago

This is just an absurd claim. Of course AI draws on past experiences to build new things. This is also what humans do. There is no invisible well of "creativity" that humans draw from that AI can't. In fact, in areas that AI has "solved", such as chess, human experts describe AI as extremely "creative" beyond their understanding. "Creative" is just a word for connecting existing concepts in ways that seem novel. AI does this when it solves a new problem. AI clearly has the potential to make better connections, much faster than humans, in pretty much all domains. There's only one piece "missing" that is already technically possible, but expensive: giving an LLM the ability to "learn". In the near future, all creative work will be produced by AI. This will likely come before all work is produced by AI, as creative work is typically less precise.

sudosysgen 4y ago
Humans can seek out new experiences optimized for what they want to learn, which is fundamentally different. Creativity is not just connecting existing concepts. The pinnacle of creativity is inventing new concepts ab nihilo, which is something that every human does through self-directed interaction with the environment.
- boringuser2 4y ago
  That's simply not true and ties strongly to a metaphysical claim that isn't substantiated.
  - sudosysgen 4y ago
    Instead of giving a trite dismissal, give a substantiaded rebuttal. I don't see any metaphysical claim in my comment. It's objectively true that humans will seek out or create experiences that allow them to obtain better understanding, and it's also objectively true that humans can create entirely original concepts, so long as you agree that there is such a thing. If you don't think that exists, then your entire comment is pointless, because then you're arguing by hidden definition, which I'd rather assume you aren't.

simplyluke 4y ago

Several people have observed that in a world where LLMs are widely influential, original creative thought will become even more of a valued differentiator than it already is.

freehorse 4y ago

Really spot-on, I agree this is the most important issue with inherent limitations of LLMs: not LLMs themselves and what they are supposed to do, but how a lot of people do not understand the limits of language and equating human knowledge and experience with language and "what has already been written". Tbh I find it quite disappointing from an intellectual/philosophical perspective that so many people think that the realm of human experience can be reduced to communicative language (and even more to "what has already been written in the internet") thus equating human creativity with LLMs' creativity. My fear is that people may not realise/care about this issue till it is too late (eg when journalism degrades to "the same article is written over and over"). The reason is that the bulk of "human creativity" is in fact reducible to what LLMs do: building a standard website, a lot of "creative writing" and art in general is to a big part not "innovation" that comes from experience (and in general from non-written-linguistic domains), at least in a lot of fields. I am afraid that because the bulk of work for a lot of fields is like that, especially exactly that kind of boring, repeatitive work that gives most of the money to survive and AI is better than humans at (ie cheaper), we may even risk having degradation on many fields that we may not even be able to notice.

1vuio0pswjnm7 4y ago

What about so-called synthetic training data. For example, does the synthesis ever involve creativity. Note: The term "creativity" might not be the best term for this discussion. Unless I am mistaken, the issue is actually originality.

cubefox 4y ago

> A client of mine seemed quite insistent that ChatGPT was "creative". I hope in a few years that opinion will be rare, as everyone ends up writing the same article over-and-over. It seems rather probable that in a few years GPTs would become far more powerful, including more creative than most humans.

cratermoon 4y ago

The so-called Stochastic Parrot paper mentions some specific harms that the "done-and-finished" aspect. in Section 4.2, "Static Data/Changing Social Views", saying, > A central aspect of social movement formation involves using language strategically to destabilize dominant narratives and call attention to underrepresented social perspectives. Social movements produce new norms, language, and ways of communicating. This adds challenges to the deployment of LMs, as methodologies reliant on LMs run the risk of ‘value-lock’, where the LM-reliant technology reifies older, less-inclusive understandings.