4 ms·

Since you’re releasing the code to GitHub, do you think you’ll eventually run into issues with the training data including prior versions of the game?

tdy_err 2y ago
The implied scenario being that the memory of its own output would result in the model producing degraded future output? Why is that a given?
- Chaosvex 2y ago
  Read about model collapse. The TL;DR is garbage in, garbage out. https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse
- mythrwy 2y ago
  Probably the same reason that close relatives marrying each other for generations produces genetic problems.
  - Etherlord87 2y ago
    Not the same reason at all. In genetics the reason is that you're losing gene variety and eventually recessive genes aren't suppressed anymore. In case of LLM it's just error accumulation.
    - mythrwy 2y ago
      It's a few days late but "losing gene variety" isn't the cause. What happens is genetic errors compound and are more likely to be expressed. I.E. "error accumulation".
      - Etherlord87 2y ago
        You're wrong. You clearly have the Internet, I don't understand why won't you just google it and learn about it instead of claiming stuff that is bs.
        mythrwy 2y ago
        How about a number of grad level genetics courses? Does that beat your google search? Because that is what I have. And what I am telling you is what happens. This is really easily searched (as you said). You might read up on it if interested. Check out why inbreeding can lead to expression of genetic defects. What is the mechanism? (hint: it's not "losing gene diversity" or "suppression").
        Etherlord87 2y ago
        Very bad courses then. https://biology.stackexchange.com/questions/58769/what-are-the-consequences-of-inbreeding https://biology.stackexchange.com/questions/58769/what-are-t...
        mythrwy 2y ago
        Without getting into the validity of the source, let's look at what it says: Here is the first sentence from the top answer: `You are right. Inbreeding strongly increases overall homozygosity which subjects inbred individuals to diseases caused by rare recessive alleles.`. Let's see what homozygosity means shall we? https://www.genome.gov/genetics-glossary/homozygous https://www.genome.gov/genetics-glossary/homozygous `Homozygous, as related to genetics, refers to having inherited the same versions (alleles) of a genomic marker from each biological parent. Thus, an individual who is homozygous for a genomic marker has two identical versions of that marker. By contrast, an individual who is heterozygous for a marker has two different versions of that marker.` In other words, errors can accumulate and are more likely to be expressed. Not "gene diversity" (this is a topic relating to evolutionary fitness, selection potential etc.), not "suppression". Error accumulation. Which is the exact analogy I made initially.
        2y ago

tdy_err 2y ago

The implied scenario being that the memory of its own output would result in the model producing degraded future output? Why is that a given?

Chaosvex 2y ago
Read about model collapse. The TL;DR is garbage in, garbage out. https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse
mythrwy 2y ago
Probably the same reason that close relatives marrying each other for generations produces genetic problems.
- Etherlord87 2y ago
  Not the same reason at all. In genetics the reason is that you're losing gene variety and eventually recessive genes aren't suppressed anymore. In case of LLM it's just error accumulation.
  - mythrwy 2y ago
    It's a few days late but "losing gene variety" isn't the cause. What happens is genetic errors compound and are more likely to be expressed. I.E. "error accumulation".
    - Etherlord87 2y ago
      You're wrong. You clearly have the Internet, I don't understand why won't you just google it and learn about it instead of claiming stuff that is bs.
      - mythrwy 2y ago
        How about a number of grad level genetics courses? Does that beat your google search? Because that is what I have. And what I am telling you is what happens. This is really easily searched (as you said). You might read up on it if interested. Check out why inbreeding can lead to expression of genetic defects. What is the mechanism? (hint: it's not "losing gene diversity" or "suppression").
        Etherlord87 2y ago
        Very bad courses then. https://biology.stackexchange.com/questions/58769/what-are-the-consequences-of-inbreeding https://biology.stackexchange.com/questions/58769/what-are-t...
        mythrwy 2y ago
        Without getting into the validity of the source, let's look at what it says: Here is the first sentence from the top answer: `You are right. Inbreeding strongly increases overall homozygosity which subjects inbred individuals to diseases caused by rare recessive alleles.`. Let's see what homozygosity means shall we? https://www.genome.gov/genetics-glossary/homozygous https://www.genome.gov/genetics-glossary/homozygous `Homozygous, as related to genetics, refers to having inherited the same versions (alleles) of a genomic marker from each biological parent. Thus, an individual who is homozygous for a genomic marker has two identical versions of that marker. By contrast, an individual who is heterozygous for a marker has two different versions of that marker.` In other words, errors can accumulate and are more likely to be expressed. Not "gene diversity" (this is a topic relating to evolutionary fitness, selection potential etc.), not "suppression". Error accumulation. Which is the exact analogy I made initially.
        2y ago