12 ms·

> Published content will be later used to train subsequent models, and being able to distinguish AI from human input may be very valuable going forward I find

> Published content will be later used to train subsequent models, and being able to distinguish AI from human input may be very valuable going forward I find this to be a particularly interesting problem in this whole debacle. Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? I think it's a particular risk for text generation. I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. Now imagine a myriad fake blogs using ChatGPT under the hood to generate blog entries explaining how to solve often wanted problems, and that then being spidered and fed into ChatGPT 2.0. Such things could end up creating a downwards trend in quality, as more and more of such junk gets posted, absorbed into the model and amplified further. I think image generation should be less vulnerable to this since all images need tagging to be useful, "ai generated" is a common tag that can be used to exclude reingesting old outputs, and also because with artwork precision doesn't matter so much. If people like the results, then it doesn't matter that much that something isn't drawn realistically.

r3trohack3r 4y ago
I feel like we are nearing peak "uncurated" content, both for humans and machines. Humans are still grappling with our novel abundance problems. As we move forward, suspect we will see an increase in curation services and AI models will do more with less. You can bootstrap a productive adult human on an almost infintismal slice of the training sets we are using for the current gen of AI, can't imagine future approaches are going to need such an unbounded input to get better results - but might be wrong! If content is curated for its quality, whether or not it's AI generated (or assisted) doesn't matter.
- joshspankit 4y ago
  We’d have to adjust capitalism to deal with the “novel abundance” problems. Most of the drive for novel content/audiences is simply to decide which people get a cut of the revenue (and/or audience). If we focused on quality and stopped caring about who gets paid for what I suspect that not only would we have better quality overall but we’d also push the boundaries much faster thus making things even more interesting.
  - r3trohack3r 4y ago
    Strong disagree. Capitalism (can argue whether that's what we have or not) uses capital to respond to markets. Markets are signals of what people want. The scarcity mindset in an the era of information abundance isn't being perpetuated by capitalism, it's being perpetuated by people making their own decisions one transaction at a time.
    - joshspankit 4y ago
      You make some strong logical cases so I'd like to use some of your language to try to give another perspective on my points: Markets in our current system are (IMO) a distorted view of what people want. For example TVs: How much of the market is people who simply want a nice clean picture and how much is people who are feature-chasing based on the TV marketing? And then how much of the 'market' is actually smoke and mirrors accounting to increase the value of companies? Some things that I think are only common because of capitalism (or "capitalism") and could therefore be phased out if we transitioned to an AI-supported system: - "Keeping up with the Jones'" - Needing short hits of artificial 'excitement' (such as the kind of novelty drip-fed to us by the movie industry) - Keeping ideas secret as a form of Intellectual Property protection, and therefore the scarcity mindset that there are no good solutions and no new problems to solve (since we'd all be exposed to the latest of those if we chose) - Capital being used for poor-quality projects (since many of these again are just about trying to grab more money for the people making them) - People making their own decisions one transaction at a time. This is definitely underrated. Sure some people will continue to think short-term just because that's how their brains are wired, but I suspect the majority will find themselves able to breathe slowly for the first time and they will naturally unwind in to longer and longer-term thinking.
messe 4y ago
Hmm, forgetting natural language for a moment and instead considering programming languages: it’s pretty easy to generate nonsense but semi-plausible looking ASTs without the help of AI. Could this be used to attack GitHub’s copilot? Step 1. Release a tool that generates nonsense code across a thousand repositories, and allow anybody to publish crap to GitHub. Step 2. Copilot trains on those nonsensical repositories because it can’t distinguish them from the real thing. Step 3. Copilot begins to generate similar crap.
- empyrrhicist 4y ago
  You'd probably also have to botspam stars, issues, pull requests.
- theRealMe 4y ago
  Imagine this as a security attack vector. Instead of nonsense, spam a bunch of repos with code that does a specific thing but in a very hard to understand way. Then add in a small piece of very hard to understand, but legit looking malicious code. Copilot trains on it and then starts feeding it to developers around the world. Probably easier ways to achieve this, but interesting to think about.
rgrieselhuber 4y ago
As someone in SEO, I've been pretty disgusted by the desire for site owners to want to use AI-generated content. There are various opinions on this, of course, but I got into SEO out of interest in the "organic web" vs. everything being driven by ads. Love the idea of having AI-Free declarations of content as it could / should help to differentiate organic content from generated content. It would be very interesting if companies and site owners wished to self-certify their site as organic with something like an /ai-free.txt.
- dale_glass 4y ago
  I don't see the point. There's lots of old content out there that won't get tagged, so lacking the tag doesn't mean it's AI generated. Meanwhile people abusing AI for profit (eg, generating AI driven blogs to stick ads on them) wouldn't want to tag their sites in a way that might get them ignored. And what are the consequences for lying?
  - rgrieselhuber 4y ago
    It would basically depend on how serious the search engines are about wanting created vs. generated content. Generated content is ultimately going to be a regurgitation.
  - westurner 4y ago
    Does use of a search engine violate the "No AI" covenant with oneself? Variation on the Turning Test: prove that it's not a human claiming to be a computer. Modeling premises and Meta-analysis are again necessary elements for critical reasoning about Sources and Methods and superpositions of Ignorance and Malice.
    - jhbadger 4y ago
      Maybe this could encourage the recreation of the original Yahoo! (If you don't remember, Yahoo! started out not as a search engine in the Google sense but as a collection of human curated links to websites about various topics)
jhbadger 4y ago
>I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. Yes! I remember generating a seemingly reasonable R script except that the library that it called to do most of the work didn't exist! It was like code from an alternate dimension!
- PartiallyTyped 4y ago
  I have had the same experience with Typescript and Python.
- bamboozled 4y ago
  I asked it to generate sample aws step function config and as far as I could tell it made up configuration parameters. I know it’s a language model.
- Kelteseth 4y ago
  Can confirm that this happened to me when I asked ChatGPT to generate a parallax effect in Qt/QML. It simply used a QML Elemened with the name Parallax.
- jay-barronville 4y ago
  > It was like code from an alternate dimension! Maybe it was? Haha. ChatGPT has seen some things and knows something we don’t!
- gattilorenz 4y ago
  It’s as if ChatGPT was behaving like a language model with no real connection or understanding of R… hmmmmm…
  - alpos 4y ago
    This take strikes me as a little off. Programming languages are language. Unlike natural languages they are also based on context-free grammar. So an understanding of programming languages should actually be easier for even a general language model to incorporate than natural languages. We can expect a bot like this to not really get context clues in natural language, although they seem to be getting better at that, but context is not necessary to have a true and functional understanding of a programming language. That was the point of creating such languages. Using an API that doesn't exist but logically should once the use cases are demonstrated is not an example of lacking understanding, it is an example of advanced insight. A human might have invented the necessary functions inline with the rest of the project but if they are expressing functionality that is commonly applicable, then a common API for those functions is what the humans would eventually converge upon to clean up the code from the initial inline implementation, making it more consistent and readable.
noduerme 4y ago
>> a downwards trend in quality Have you googled for reviews on toaster ovens recently?
- A4ET8a8uTh0 4y ago
  To your point, anecdotally, the system is heavily gamed. The other day, I saw reviews for a restaurant pop up that did not quite open yet. Either reviewers got a sneak peek behind the chef's curtain or those reviews are not quite true. Sadly, word of mouth again becomes defacto the only semi-reliable way to separate crap from non-crap and even that comes with its own set of issues and attempts at gaming.
CuriouslyC 4y ago
This isn't an issue, because it's possible to add prose quality and content accuracy scores to training data and train the model to predict those quantities during generation, which would allow you to condition the generation on high prose quality/accuracy. It just requires a small update to the model, and a shit ton of data set annotation time. Likewise, images can be scored for aesthetics and consistency and models updated to predict and condition in the same way.
- rcme 4y ago
  How would you score them at scale without training some model to differentiate real vs. AI content? If you need to train such a model, where would you get the data from?
  - CuriouslyC 4y ago
    We don't need to differentiate AI vs Human, just accurate and well written vs not. We'd do that the same way we've scored stuff at scale so far - grad students and crowdsourcing.
    - rcme 4y ago
      The scale of data for these LLMs is well beyond the scale producible via crowdsourcing.
      - CuriouslyC 4y ago
        That just isn't true. It's expensive, but entirely doable. Also, it's perfectly normal to perform initial model training on a large data set to capture the statistical properties of language, then perform a second stage of model training on more curated data to cause the model to actually do what you want.
        rcme 4y ago
        Current LLMs are trained on as much data that can be scraped from the public internet. It’s simply not possible to annotate that much data, even with crowdsourcing. It’s not even a matter of cost. You’d basically need to duplicate the amount of data on the internet. I don’t think you’re appreciating the scale of the data involved in training these models.
foobarbecue 4y ago
It reminds me of a dog eating its own vomit.
- euroderf 4y ago
  Excellent analogy. Filed for future use.
  - foobarbecue 4y ago
    Heh, if we work together we can ruin the term "dogfooding"
    - euroderf 4y ago
      "WOOF?" "BRRAAALFF!!"
visarga 4y ago
> Published content will be later used to train subsequent models, and being able to distinguish AI from human input may be very valuable going forward Not necessarily all AI contents are bad and all human contents are good. We need a way to separate the good from the bad, not AI from human, and it might be impossible to do 100% correct anyway.
- A4ET8a8uTh0 4y ago
  I think I would compare it to Stack Overflow. Some of the solutions do exist there, but not all are applicable to the use case or the exact circumstances the person asks there and yet the prompt used by AI would remain the same. SO has its rating system, but it has the same issue as the sentence above. From that perspective, we have identified potentially good human output ( assuming it wasn't already pollinated with AI output, which seems less and less likely ) that should only be accessible by humans and we would need a separate forum for bad AI output ( that should be verified by humans as bad but maybe only be accessible by AI once verified ). I am just spitballing. I do not really have a solution in mind. It just sounds like an interesting problem going forward.
alpos 4y ago
> I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. So the next question has to be: Was this still the right answer? I've personally had plenty of instances in my programming career where the code I was working on really needed functions which were best shopped out to a common API. To avoid interrupting my flow and to better inform the API I'd be designing for this, I just continued to write as if the API did exist. Then I went on to implement the functions that were needed. Perhaps the bot was right to presume that there should be an API for this. You might even be able to then prompt ChatGPT to create each of the functions in that API.
- lolinder 4y ago
  Well, this is ChatGPT, not Copilot, so I'd assume that OP was looking for a snippet using a public library rather than an internal API. In that context, suggesting you use an API that doesn't exist is just wrong. I've definitely done this with Copilot, though—it will suggest an API that doesn't actually exist but logically should in order to be consistent, and I'll go create it.
  - alpos 4y ago
    That seems more like misplaced expectations. Someone may have given you the impression that copilot was supposed to do things like that where that expectation seems to not be present for you in relation to ChatGPT. However, as far as I know, the OpenAI team has not made it a goal to have ChatGPT only produce functional code using existing APIs. So I'm not sure we can call that an incorrect answer based on context. If the API it demonstrated using logically should exist, it seems like the right answer is still to just go create it.
    - lolinder 4y ago
      If I asked a coworker how to do X in framework Y, giving me the name of a function that should exist but doesn't is not a correct answer. If they told me "well, that's the function that should exist, you should just go submit a PR to framework Y", I would stop asking that coworker for help. The difference I was drawing between ChatGPT and Copilot wasn't that Copilot has functionality ChatGPT doesn't, it's that it has context ChatGPT doesn't, so it suggests things related to internal APIs. In a conversation with ChatGPT it would be very difficult to get help with internal APIs, hence my assumption that OP wasn't referring to APIs they have any control over.
eloisius 4y ago
> Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? I think it's a particular risk for text generation. This is exactly what I don't like about Copilot, maybe even more than the IP ethics of it. If it really succeeds, it's going to have a feedback loop that amplifies its own code suggestions. The same boilerplate-ish kind of code that developers generate over and over will get calcified, even if it's suboptimal or buggy. If we really need robots to help write our code, to me that means we don't yet have expressive enough languages.
- sillysaurusx 4y ago
  Devs can use their noggins to check whether copilot’s output is decent or not. It’s not a given that people will use it automatically.
  - boringg 4y ago
    That and I bet the generic boilerplate code that co-pilot produces is on average much better than the boiler plate code that the average dev might use. So it removes the lower level work.
    - spacephysics 4y ago
      I think that can surely be the case, but just like any AI there may need to be manual review to asses what is the most optimal way to go about task X, and retrain. I can see this go both ways: boilerplate code being a great generic solution to a set of problems, but a more seasoned programmer may say “that works, but for our use case the trade offs don’t make sense” Or alternatively, “this code wasn’t something I knew I could do in language X, and it’s far more efficient”
    - brookst 4y ago
      It is. There are a lot of problems with copilot, but one magical thing is the way it rewards best practices like thinking through what a function is supposed to do before starting to write it. If you write a good comment describing name, inputs, outputs, logic, and exceptions, the "generate code from comment" capability is kind of amazing. I'm a terrible, hacky programmer, and it has wholly converted me to documentation-first.
bogdanoff_2 4y ago
>Such things could end up creating a downwards trend in quality, as more and more of such junk gets posted, absorbed into the model and amplified further. I feel like a similar thing already happened with YouTube recommendations
shadowgovt 4y ago
There will likely be selective pressure from human interaction with the data to curate good content above bad. After all, we had the issue of millions of auto-generated bad pages in the web 1.0 SEO days. Search engines addressed it by figuring out how to rely more heavily on human behavior signals as an indication of value of data.
user3939382 4y ago
I brought this exact issue up recently https://news.ycombinator.com/item?id=34252938 https://news.ycombinator.com/item?id=34252938
welshwelsh 4y ago
I think that in addition to influencing future models, AI content will also influence how humans think and write. People will start ironically and unironically copying GPT's style in their own writing, causing human produced content to increasing resemble AI content. High school students that are prohibited from using AI for their essays will have a bad time. Even if they don't use AI chatbots themselves, they will unknowingly cite sources that were written by AI, or were written by someone who learned about the topic by asking ChatGPT.
kaetemi 4y ago
Those blogs already exist. Pretty much 90% of the results I see in Google for non-technical household related queries. Just incoherent rambling that sounds plausible but is complete nonsense.
luxcem 4y ago
> Published content will be later used to train subsequent models, and being able to distinguish AI from human input may be very valuable going forward Every discussion on AI take the example of ChatGPT and its inherent flaws but AI-generated content doesn't have to be dull and low quality. One question that bother me is does it really matter? If AI-generated content is on par with Human-made or even better does it matter anymore that an AI generated it? Maybe it's the sentimental value, empathy, fidelity? If an AI had written Mozart's Requiem would it lessen its interest, its beauty?
- lolinder 4y ago
  > Every discussion on AI take the example of ChatGPT and its inherent flaws but AI-generated content doesn't have to be dull and low quality. To get away from that we'd have to dramatically change our approach. The LLMs we have are trained on as much content as possible and essentially average out the style of their training data. What it writes reads like a B-grade high school essay because that is what you get when you average all the writing on the internet. It's not obvious to me that a creative approach that boils down to "pick the most likely next word given the context so far" can avoid sounding bland.
- layer8 4y ago
  It’s not clear when and how we would reach that level of quality. It doesn’t seem very relevant to the present state of affairs.
- DennisP 4y ago
  I don't think AI has to be low-quality for GP's concern to be valid. Humans get inputs from a large variety of sources, but if an AI's input is just text, then there's the potential for AI's input to mostly consist of its prior output. Iterate this, and its model could gradually diverge from the real world. The equivalent in human society is groupthink, where members of a subgroup get most of their information from the same subgroup and end up believing weird things. We can counter that by purposely getting inputs from outside of our group. For a text-model AI, this means identifying text that wasn't produced by AI, as the article suggests.
layer8 4y ago
Information quality and authenticity will become a cat-and-mouse game just like information security.
davidkunz 4y ago
I assume at some point, ChatGPT needs some kind of text ranking. Popular texts are usually correct (content and presentation) and useful, so they should rank higher. At some point, low-quality texts are filtered out. Personally, I don't care if a text is written by a human or a machine as long as it's good.
- joshspankit 4y ago
  Popular as defined by who? Because I think we all know that popular as defined by Pagerank, sales, or visits has a lot of issues.
roughly 4y ago
Yes! There’s a common problem where people think that an ecosystem is infinite, or at least sufficiently large, when it’s not. We’ve done similar with dumping in the ocean, and now we’ve all got plastics in our blood, and we assumed soil quality was a given over time, too. AI content released into the wild will be consumed by AI; how can it not be? You’ve got a system which can produce content at a rate several orders of magnitude higher a human, of course the content ecosystem will be dominated by AI-generated content, so of course the quality of content generated by AI systems, which rely on non-AI-generated content to train, will go down over time.
jjtheblunt 4y ago
> Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? I think it's a particular risk for text generation. I just had my eyes opened reading that, because humans also do exactly that, inadvertently.
- jjtheblunt 4y ago
  another afterthought : does that mean humans are artificial AI?
orbital-decay 4y ago
> Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? No, because models are already trained like that. Datasets for large models are too vast for a human to even know what's inside, let alone label them manually. So instead they are processed (labeled, cropped, etc) by other models, with humans overseeing and tweaking the process. Often it's a chain with several models training each other, bootstrapped from whatever manual data you have, and curated by humans in key points of the process. So it's actually the opposite - the hybrid bootstrapping approach that combines human curation and ML labeling of bulk low-quality data typically delivers far better results than training on a small but 100% manual dataset.
- wincy 4y ago
  Okay for image models I think humans could help a lot more than we give them credit for. We can read and parse images WAY faster than you might think. What if we just crowdsource and have a new Folding@home protein thing but this time it’s for classifying data sets? LAION-5B has 5 billion image text pairs, if we got 10,000 people together that’d just be… 100,000 per person which would take… awhile but not forever. Humans can notice discrepancies super quickly. Like a slide show display the image and the text pair at a speed set by the user, and pause and tweak ones that are outright wrong or shitty. Boom, refined image set. Maybe? I’m looking at the LAION-5B example sets on their website and it seems to literally be this simple. A lot of the images seemed pretty poorly tagged. You get a gigantic manually tagged data set, at least for image classification.
- visarga 4y ago
  > They are processed by other models, humans overseeing and tweaking the process. Often it's a chain with several models training each other, bootstrapped from whatever manual data you have, and curated by humans in key points of the process. A great description of what actually happens when you deal with massive datasets. One way to inspect a large dataset is to cluster it, and then look at just a few samples from each cluster, to get an overview.
joshspankit 4y ago
The thing that concerns me is that we may end up with a downward trend in accuracy. If AI writes the bulk of the content, how long will it be before people simply do not put in the work to make sure things are true or put in the work to discover and understand new true things?

r3trohack3r 4y ago

I feel like we are nearing peak "uncurated" content, both for humans and machines. Humans are still grappling with our novel abundance problems. As we move forward, suspect we will see an increase in curation services and AI models will do more with less. You can bootstrap a productive adult human on an almost infintismal slice of the training sets we are using for the current gen of AI, can't imagine future approaches are going to need such an unbounded input to get better results - but might be wrong! If content is curated for its quality, whether or not it's AI generated (or assisted) doesn't matter.

joshspankit 4y ago
We’d have to adjust capitalism to deal with the “novel abundance” problems. Most of the drive for novel content/audiences is simply to decide which people get a cut of the revenue (and/or audience). If we focused on quality and stopped caring about who gets paid for what I suspect that not only would we have better quality overall but we’d also push the boundaries much faster thus making things even more interesting.
- r3trohack3r 4y ago
  Strong disagree. Capitalism (can argue whether that's what we have or not) uses capital to respond to markets. Markets are signals of what people want. The scarcity mindset in an the era of information abundance isn't being perpetuated by capitalism, it's being perpetuated by people making their own decisions one transaction at a time.
  - joshspankit 4y ago
    You make some strong logical cases so I'd like to use some of your language to try to give another perspective on my points: Markets in our current system are (IMO) a distorted view of what people want. For example TVs: How much of the market is people who simply want a nice clean picture and how much is people who are feature-chasing based on the TV marketing? And then how much of the 'market' is actually smoke and mirrors accounting to increase the value of companies? Some things that I think are only common because of capitalism (or "capitalism") and could therefore be phased out if we transitioned to an AI-supported system: - "Keeping up with the Jones'" - Needing short hits of artificial 'excitement' (such as the kind of novelty drip-fed to us by the movie industry) - Keeping ideas secret as a form of Intellectual Property protection, and therefore the scarcity mindset that there are no good solutions and no new problems to solve (since we'd all be exposed to the latest of those if we chose) - Capital being used for poor-quality projects (since many of these again are just about trying to grab more money for the people making them) - People making their own decisions one transaction at a time. This is definitely underrated. Sure some people will continue to think short-term just because that's how their brains are wired, but I suspect the majority will find themselves able to breathe slowly for the first time and they will naturally unwind in to longer and longer-term thinking.

messe 4y ago

Hmm, forgetting natural language for a moment and instead considering programming languages: it’s pretty easy to generate nonsense but semi-plausible looking ASTs without the help of AI. Could this be used to attack GitHub’s copilot? Step 1. Release a tool that generates nonsense code across a thousand repositories, and allow anybody to publish crap to GitHub. Step 2. Copilot trains on those nonsensical repositories because it can’t distinguish them from the real thing. Step 3. Copilot begins to generate similar crap.

empyrrhicist 4y ago
You'd probably also have to botspam stars, issues, pull requests.
theRealMe 4y ago
Imagine this as a security attack vector. Instead of nonsense, spam a bunch of repos with code that does a specific thing but in a very hard to understand way. Then add in a small piece of very hard to understand, but legit looking malicious code. Copilot trains on it and then starts feeding it to developers around the world. Probably easier ways to achieve this, but interesting to think about.

rgrieselhuber 4y ago

As someone in SEO, I've been pretty disgusted by the desire for site owners to want to use AI-generated content. There are various opinions on this, of course, but I got into SEO out of interest in the "organic web" vs. everything being driven by ads. Love the idea of having AI-Free declarations of content as it could / should help to differentiate organic content from generated content. It would be very interesting if companies and site owners wished to self-certify their site as organic with something like an /ai-free.txt.

dale_glass 4y ago
I don't see the point. There's lots of old content out there that won't get tagged, so lacking the tag doesn't mean it's AI generated. Meanwhile people abusing AI for profit (eg, generating AI driven blogs to stick ads on them) wouldn't want to tag their sites in a way that might get them ignored. And what are the consequences for lying?
- rgrieselhuber 4y ago
  It would basically depend on how serious the search engines are about wanting created vs. generated content. Generated content is ultimately going to be a regurgitation.
- westurner 4y ago
  Does use of a search engine violate the "No AI" covenant with oneself? Variation on the Turning Test: prove that it's not a human claiming to be a computer. Modeling premises and Meta-analysis are again necessary elements for critical reasoning about Sources and Methods and superpositions of Ignorance and Malice.
  - jhbadger 4y ago
    Maybe this could encourage the recreation of the original Yahoo! (If you don't remember, Yahoo! started out not as a search engine in the Google sense but as a collection of human curated links to websites about various topics)

jhbadger 4y ago

>I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. Yes! I remember generating a seemingly reasonable R script except that the library that it called to do most of the work didn't exist! It was like code from an alternate dimension!

PartiallyTyped 4y ago
I have had the same experience with Typescript and Python.
bamboozled 4y ago
I asked it to generate sample aws step function config and as far as I could tell it made up configuration parameters. I know it’s a language model.
Kelteseth 4y ago
Can confirm that this happened to me when I asked ChatGPT to generate a parallax effect in Qt/QML. It simply used a QML Elemened with the name Parallax.
jay-barronville 4y ago
> It was like code from an alternate dimension! Maybe it was? Haha. ChatGPT has seen some things and knows something we don’t!
gattilorenz 4y ago
It’s as if ChatGPT was behaving like a language model with no real connection or understanding of R… hmmmmm…
- alpos 4y ago
  This take strikes me as a little off. Programming languages are language. Unlike natural languages they are also based on context-free grammar. So an understanding of programming languages should actually be easier for even a general language model to incorporate than natural languages. We can expect a bot like this to not really get context clues in natural language, although they seem to be getting better at that, but context is not necessary to have a true and functional understanding of a programming language. That was the point of creating such languages. Using an API that doesn't exist but logically should once the use cases are demonstrated is not an example of lacking understanding, it is an example of advanced insight. A human might have invented the necessary functions inline with the rest of the project but if they are expressing functionality that is commonly applicable, then a common API for those functions is what the humans would eventually converge upon to clean up the code from the initial inline implementation, making it more consistent and readable.

noduerme 4y ago

>> a downwards trend in quality Have you googled for reviews on toaster ovens recently?

A4ET8a8uTh0 4y ago
To your point, anecdotally, the system is heavily gamed. The other day, I saw reviews for a restaurant pop up that did not quite open yet. Either reviewers got a sneak peek behind the chef's curtain or those reviews are not quite true. Sadly, word of mouth again becomes defacto the only semi-reliable way to separate crap from non-crap and even that comes with its own set of issues and attempts at gaming.

CuriouslyC 4y ago

This isn't an issue, because it's possible to add prose quality and content accuracy scores to training data and train the model to predict those quantities during generation, which would allow you to condition the generation on high prose quality/accuracy. It just requires a small update to the model, and a shit ton of data set annotation time. Likewise, images can be scored for aesthetics and consistency and models updated to predict and condition in the same way.

rcme 4y ago
How would you score them at scale without training some model to differentiate real vs. AI content? If you need to train such a model, where would you get the data from?
- CuriouslyC 4y ago
  We don't need to differentiate AI vs Human, just accurate and well written vs not. We'd do that the same way we've scored stuff at scale so far - grad students and crowdsourcing.
  - rcme 4y ago
    The scale of data for these LLMs is well beyond the scale producible via crowdsourcing.
    - CuriouslyC 4y ago
      That just isn't true. It's expensive, but entirely doable. Also, it's perfectly normal to perform initial model training on a large data set to capture the statistical properties of language, then perform a second stage of model training on more curated data to cause the model to actually do what you want.
      - rcme 4y ago
        Current LLMs are trained on as much data that can be scraped from the public internet. It’s simply not possible to annotate that much data, even with crowdsourcing. It’s not even a matter of cost. You’d basically need to duplicate the amount of data on the internet. I don’t think you’re appreciating the scale of the data involved in training these models.

foobarbecue 4y ago

It reminds me of a dog eating its own vomit.

euroderf 4y ago
Excellent analogy. Filed for future use.
- foobarbecue 4y ago
  Heh, if we work together we can ruin the term "dogfooding"
  - euroderf 4y ago
    "WOOF?" "BRRAAALFF!!"

visarga 4y ago

> Published content will be later used to train subsequent models, and being able to distinguish AI from human input may be very valuable going forward Not necessarily all AI contents are bad and all human contents are good. We need a way to separate the good from the bad, not AI from human, and it might be impossible to do 100% correct anyway.

A4ET8a8uTh0 4y ago
I think I would compare it to Stack Overflow. Some of the solutions do exist there, but not all are applicable to the use case or the exact circumstances the person asks there and yet the prompt used by AI would remain the same. SO has its rating system, but it has the same issue as the sentence above. From that perspective, we have identified potentially good human output ( assuming it wasn't already pollinated with AI output, which seems less and less likely ) that should only be accessible by humans and we would need a separate forum for bad AI output ( that should be verified by humans as bad but maybe only be accessible by AI once verified ). I am just spitballing. I do not really have a solution in mind. It just sounds like an interesting problem going forward.

alpos 4y ago

> I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. So the next question has to be: Was this still the right answer? I've personally had plenty of instances in my programming career where the code I was working on really needed functions which were best shopped out to a common API. To avoid interrupting my flow and to better inform the API I'd be designing for this, I just continued to write as if the API did exist. Then I went on to implement the functions that were needed. Perhaps the bot was right to presume that there should be an API for this. You might even be able to then prompt ChatGPT to create each of the functions in that API.

lolinder 4y ago
Well, this is ChatGPT, not Copilot, so I'd assume that OP was looking for a snippet using a public library rather than an internal API. In that context, suggesting you use an API that doesn't exist is just wrong. I've definitely done this with Copilot, though—it will suggest an API that doesn't actually exist but logically should in order to be consistent, and I'll go create it.
- alpos 4y ago
  That seems more like misplaced expectations. Someone may have given you the impression that copilot was supposed to do things like that where that expectation seems to not be present for you in relation to ChatGPT. However, as far as I know, the OpenAI team has not made it a goal to have ChatGPT only produce functional code using existing APIs. So I'm not sure we can call that an incorrect answer based on context. If the API it demonstrated using logically should exist, it seems like the right answer is still to just go create it.
  - lolinder 4y ago
    If I asked a coworker how to do X in framework Y, giving me the name of a function that should exist but doesn't is not a correct answer. If they told me "well, that's the function that should exist, you should just go submit a PR to framework Y", I would stop asking that coworker for help. The difference I was drawing between ChatGPT and Copilot wasn't that Copilot has functionality ChatGPT doesn't, it's that it has context ChatGPT doesn't, so it suggests things related to internal APIs. In a conversation with ChatGPT it would be very difficult to get help with internal APIs, hence my assumption that OP wasn't referring to APIs they have any control over.

eloisius 4y ago

> Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? I think it's a particular risk for text generation. This is exactly what I don't like about Copilot, maybe even more than the IP ethics of it. If it really succeeds, it's going to have a feedback loop that amplifies its own code suggestions. The same boilerplate-ish kind of code that developers generate over and over will get calcified, even if it's suboptimal or buggy. If we really need robots to help write our code, to me that means we don't yet have expressive enough languages.

sillysaurusx 4y ago
Devs can use their noggins to check whether copilot’s output is decent or not. It’s not a given that people will use it automatically.
- boringg 4y ago
  That and I bet the generic boilerplate code that co-pilot produces is on average much better than the boiler plate code that the average dev might use. So it removes the lower level work.
  - spacephysics 4y ago
    I think that can surely be the case, but just like any AI there may need to be manual review to asses what is the most optimal way to go about task X, and retrain. I can see this go both ways: boilerplate code being a great generic solution to a set of problems, but a more seasoned programmer may say “that works, but for our use case the trade offs don’t make sense” Or alternatively, “this code wasn’t something I knew I could do in language X, and it’s far more efficient”
  - brookst 4y ago
    It is. There are a lot of problems with copilot, but one magical thing is the way it rewards best practices like thinking through what a function is supposed to do before starting to write it. If you write a good comment describing name, inputs, outputs, logic, and exceptions, the "generate code from comment" capability is kind of amazing. I'm a terrible, hacky programmer, and it has wholly converted me to documentation-first.

bogdanoff_2 4y ago

>Such things could end up creating a downwards trend in quality, as more and more of such junk gets posted, absorbed into the model and amplified further. I feel like a similar thing already happened with YouTube recommendations

shadowgovt 4y ago

There will likely be selective pressure from human interaction with the data to curate good content above bad. After all, we had the issue of millions of auto-generated bad pages in the web 1.0 SEO days. Search engines addressed it by figuring out how to rely more heavily on human behavior signals as an indication of value of data.

user3939382 4y ago

I brought this exact issue up recently https://news.ycombinator.com/item?id=34252938 https://news.ycombinator.com/item?id=34252938

welshwelsh 4y ago

I think that in addition to influencing future models, AI content will also influence how humans think and write. People will start ironically and unironically copying GPT's style in their own writing, causing human produced content to increasing resemble AI content. High school students that are prohibited from using AI for their essays will have a bad time. Even if they don't use AI chatbots themselves, they will unknowingly cite sources that were written by AI, or were written by someone who learned about the topic by asking ChatGPT.

kaetemi 4y ago

Those blogs already exist. Pretty much 90% of the results I see in Google for non-technical household related queries. Just incoherent rambling that sounds plausible but is complete nonsense.

luxcem 4y ago

> Published content will be later used to train subsequent models, and being able to distinguish AI from human input may be very valuable going forward Every discussion on AI take the example of ChatGPT and its inherent flaws but AI-generated content doesn't have to be dull and low quality. One question that bother me is does it really matter? If AI-generated content is on par with Human-made or even better does it matter anymore that an AI generated it? Maybe it's the sentimental value, empathy, fidelity? If an AI had written Mozart's Requiem would it lessen its interest, its beauty?

lolinder 4y ago
> Every discussion on AI take the example of ChatGPT and its inherent flaws but AI-generated content doesn't have to be dull and low quality. To get away from that we'd have to dramatically change our approach. The LLMs we have are trained on as much content as possible and essentially average out the style of their training data. What it writes reads like a B-grade high school essay because that is what you get when you average all the writing on the internet. It's not obvious to me that a creative approach that boils down to "pick the most likely next word given the context so far" can avoid sounding bland.
layer8 4y ago
It’s not clear when and how we would reach that level of quality. It doesn’t seem very relevant to the present state of affairs.
DennisP 4y ago
I don't think AI has to be low-quality for GP's concern to be valid. Humans get inputs from a large variety of sources, but if an AI's input is just text, then there's the potential for AI's input to mostly consist of its prior output. Iterate this, and its model could gradually diverge from the real world. The equivalent in human society is groupthink, where members of a subgroup get most of their information from the same subgroup and end up believing weird things. We can counter that by purposely getting inputs from outside of our group. For a text-model AI, this means identifying text that wasn't produced by AI, as the article suggests.

layer8 4y ago

Information quality and authenticity will become a cat-and-mouse game just like information security.

davidkunz 4y ago

I assume at some point, ChatGPT needs some kind of text ranking. Popular texts are usually correct (content and presentation) and useful, so they should rank higher. At some point, low-quality texts are filtered out. Personally, I don't care if a text is written by a human or a machine as long as it's good.

joshspankit 4y ago
Popular as defined by who? Because I think we all know that popular as defined by Pagerank, sales, or visits has a lot of issues.

roughly 4y ago

Yes! There’s a common problem where people think that an ecosystem is infinite, or at least sufficiently large, when it’s not. We’ve done similar with dumping in the ocean, and now we’ve all got plastics in our blood, and we assumed soil quality was a given over time, too. AI content released into the wild will be consumed by AI; how can it not be? You’ve got a system which can produce content at a rate several orders of magnitude higher a human, of course the content ecosystem will be dominated by AI-generated content, so of course the quality of content generated by AI systems, which rely on non-AI-generated content to train, will go down over time.

jjtheblunt 4y ago

> Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? I think it's a particular risk for text generation. I just had my eyes opened reading that, because humans also do exactly that, inadvertently.

jjtheblunt 4y ago
another afterthought : does that mean humans are artificial AI?

orbital-decay 4y ago

> Could we end up having AI quality trend downwards due to AI ingesting its own old outputs and reinforcing bad habits? No, because models are already trained like that. Datasets for large models are too vast for a human to even know what's inside, let alone label them manually. So instead they are processed (labeled, cropped, etc) by other models, with humans overseeing and tweaking the process. Often it's a chain with several models training each other, bootstrapped from whatever manual data you have, and curated by humans in key points of the process. So it's actually the opposite - the hybrid bootstrapping approach that combines human curation and ML labeling of bulk low-quality data typically delivers far better results than training on a small but 100% manual dataset.

wincy 4y ago
Okay for image models I think humans could help a lot more than we give them credit for. We can read and parse images WAY faster than you might think. What if we just crowdsource and have a new Folding@home protein thing but this time it’s for classifying data sets? LAION-5B has 5 billion image text pairs, if we got 10,000 people together that’d just be… 100,000 per person which would take… awhile but not forever. Humans can notice discrepancies super quickly. Like a slide show display the image and the text pair at a speed set by the user, and pause and tweak ones that are outright wrong or shitty. Boom, refined image set. Maybe? I’m looking at the LAION-5B example sets on their website and it seems to literally be this simple. A lot of the images seemed pretty poorly tagged. You get a gigantic manually tagged data set, at least for image classification.
visarga 4y ago
> They are processed by other models, humans overseeing and tweaking the process. Often it's a chain with several models training each other, bootstrapped from whatever manual data you have, and curated by humans in key points of the process. A great description of what actually happens when you deal with massive datasets. One way to inspect a large dataset is to cluster it, and then look at just a few samples from each cluster, to get an overview.

joshspankit 4y ago

The thing that concerns me is that we may end up with a downward trend in accuracy. If AI writes the bulk of the content, how long will it be before people simply do not put in the work to make sure things are true or put in the work to discover and understand new true things?