3 ms·

I have asked dang to comment on this issue specifically in the context of this post/thread. The “opposite policy” is sort of the current status quo, per dang:

I have asked dang to comment on this issue specifically in the context of this post/thread. The “opposite policy” is sort of the current status quo, per dang: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20%22generated%20comments%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... See this thread for my own reasoning on the issue (as well as dang’s), as it was raised recently: https://news.ycombinator.com/item?id=41937993 https://news.ycombinator.com/item?id=41937993 You’ll need showdead enabled on your profile to see the whole thread, which speaks to the controversial nature of this issue on HN. I agree that your mention of “encouraging accusatory behavior” is a point well-taken, and in the absence of evidence, such accusations themselves would likely run afoul of the Guidelines, but it’s worth noting that dang has said that LLM output itself is generally against the Guidelines, which could lead to a feedback loop of disinterested parties posting LLM content, only to be confronted with interested parties posting uninteresting takedowns of said LLM content and posters of it. No easy answers here, I’m afraid.

benatkin 2y ago
From the thread with see this thread > There are lot of grey areas; for example, your GP comment wasn't just generated—it came with an annotation that you're a lawyer and thought it was sound. That's better than a completely pasted comment. But it was probably still on the wrong side of the line. We want comments that the commenters actually write, as part of curious human conversation. This doesn't leave much room for AI non-slop: > We want comments that the commenters actually write, as part of curious human conversation. I think HN is trying to be good at being HN, not just to provide the most utility to its users in general. So those wanting something like HN if it started in 2030, may want to try and build a new site.
refulgentis 2y ago
Law is hard! In general, the de facto status quo is: 1. For whatever reason*, large swaths of LLM output copy-pasted is easily detectable. 2. If you're restrained, polite, with an accurate signal on this, you can indicate you see this, and won't get downvoted heavily. (ex. I'll post "my internal GPT detector went off, [1-2 sentence clipped version of why I think its wrong even if it wasn't GPT]") 3. People tend to downvote said content, as an ersatz vote. In general, I don't think there needs to be a blanket ban against it, in the sense of I have absolutely no problem with LLM output per se, just lazy invocation of it, ex. large entry-level arguments that were copy-pasted. i.e. I've used an LLM to sharpen my already-written rushed poor example, which didn't result in low-perplexity, standard-essay-formatted, content. Additionally, IMHO it's not bad, per se, if someone invests in replying to an LLM. The fact they are replying indicates its an argument worth furthering with their own contribution. * a strong indicator that a fundamental goal other than perplexity minimization may increase perceived quality
- famouswaffles 2y ago
  The reason is not strange or unknown. The text completion GPT-3 from 2020 often sounds more natural than 4. The reason is the post training processes. Models are more or less being trained to sound like that during RLHF. Stilted, robotic, like a good little assistant. Open AI, Anthropic have said as much. It's not a limitation of the loss function or even state of the art.
  - refulgentis 2y ago
    I can't give in to misguided pessimism - "Open AI, Anthropic have said as much" is especially not something I can support! I'm hearing some of the ideas on my corner of llm x creativity Twitter expressed clunkily and if its some irrevocable thing. You're right the default is to speak like an assistant. You're wrong that its forced and immutable and a consequence of RLHF and the companies say its so. https://x.com/jpohhhh/status/1784077479730090346 https://x.com/jpohhhh/status/1784077479730090346 You're especially wrong that RLHF is undesirable https://x.com/jpohhhh/status/1819549737835528555 https://x.com/jpohhhh/status/1819549737835528555 https://x.com/jpohhhh/status/1819550145522160044 https://x.com/jpohhhh/status/1819550145522160044. It's also nigh-trivial to get the completion model back https://x.com/jpohhhh/status/1776434608403325331 https://x.com/jpohhhh/status/1776434608403325331 I don't know when I'll stop seeing surface-level opinions disguised as cold technological claims on this subject. I would have thought, by now, people doing that would wonder why the wide open lane hasn't been taken, at least once.
    - famouswaffles 2y ago
      I don't understand what you're getting at here. No idea why you've put tweets from a random? person to make your point. Yes these guys have all noted on the effects of post-training on the models. "We want people to know that they’re interacting with a language model and not a person." This is literally a goal of post-training for all these companies. Even when they are training it to have a character, it mustn't sound like a person. It's no surprise they don't sound as natural as their base counterparts. https://www.anthropic.com/research/claude-character https://www.anthropic.com/research/claude-character >You're wrong that its forced and immutable and a consequence of RLHF and the companies say its so. I never said it was immutable. I said it was a consequence of post-training and it is. All the base models speak more naturally with much less effort. >You're especially wrong that RLHF is undesirable I don't understand what point you're trying to make here. I didn't say it was undesirable. I said it was heavily affecting how natural the models sounded. >It's also nigh-trivial to get the completion model back Try getting GPT-4o to write a story with villains that doesn't end with everyone singing Kumbaya and you'll see how much post-training affects the outputs of these models.

benatkin 2y ago

From the thread with see this thread > There are lot of grey areas; for example, your GP comment wasn't just generated—it came with an annotation that you're a lawyer and thought it was sound. That's better than a completely pasted comment. But it was probably still on the wrong side of the line. We want comments that the commenters actually write, as part of curious human conversation. This doesn't leave much room for AI non-slop: > We want comments that the commenters actually write, as part of curious human conversation. I think HN is trying to be good at being HN, not just to provide the most utility to its users in general. So those wanting something like HN if it started in 2030, may want to try and build a new site.

refulgentis 2y ago

Law is hard! In general, the de facto status quo is: 1. For whatever reason*, large swaths of LLM output copy-pasted is easily detectable. 2. If you're restrained, polite, with an accurate signal on this, you can indicate you see this, and won't get downvoted heavily. (ex. I'll post "my internal GPT detector went off, [1-2 sentence clipped version of why I think its wrong even if it wasn't GPT]") 3. People tend to downvote said content, as an ersatz vote. In general, I don't think there needs to be a blanket ban against it, in the sense of I have absolutely no problem with LLM output per se, just lazy invocation of it, ex. large entry-level arguments that were copy-pasted. i.e. I've used an LLM to sharpen my already-written rushed poor example, which didn't result in low-perplexity, standard-essay-formatted, content. Additionally, IMHO it's not bad, per se, if someone invests in replying to an LLM. The fact they are replying indicates its an argument worth furthering with their own contribution. * a strong indicator that a fundamental goal other than perplexity minimization may increase perceived quality

famouswaffles 2y ago
The reason is not strange or unknown. The text completion GPT-3 from 2020 often sounds more natural than 4. The reason is the post training processes. Models are more or less being trained to sound like that during RLHF. Stilted, robotic, like a good little assistant. Open AI, Anthropic have said as much. It's not a limitation of the loss function or even state of the art.
- refulgentis 2y ago
  I can't give in to misguided pessimism - "Open AI, Anthropic have said as much" is especially not something I can support! I'm hearing some of the ideas on my corner of llm x creativity Twitter expressed clunkily and if its some irrevocable thing. You're right the default is to speak like an assistant. You're wrong that its forced and immutable and a consequence of RLHF and the companies say its so. https://x.com/jpohhhh/status/1784077479730090346 https://x.com/jpohhhh/status/1784077479730090346 You're especially wrong that RLHF is undesirable https://x.com/jpohhhh/status/1819549737835528555 https://x.com/jpohhhh/status/1819549737835528555 https://x.com/jpohhhh/status/1819550145522160044 https://x.com/jpohhhh/status/1819550145522160044. It's also nigh-trivial to get the completion model back https://x.com/jpohhhh/status/1776434608403325331 https://x.com/jpohhhh/status/1776434608403325331 I don't know when I'll stop seeing surface-level opinions disguised as cold technological claims on this subject. I would have thought, by now, people doing that would wonder why the wide open lane hasn't been taken, at least once.
  - famouswaffles 2y ago
    I don't understand what you're getting at here. No idea why you've put tweets from a random? person to make your point. Yes these guys have all noted on the effects of post-training on the models. "We want people to know that they’re interacting with a language model and not a person." This is literally a goal of post-training for all these companies. Even when they are training it to have a character, it mustn't sound like a person. It's no surprise they don't sound as natural as their base counterparts. https://www.anthropic.com/research/claude-character https://www.anthropic.com/research/claude-character >You're wrong that its forced and immutable and a consequence of RLHF and the companies say its so. I never said it was immutable. I said it was a consequence of post-training and it is. All the base models speak more naturally with much less effort. >You're especially wrong that RLHF is undesirable I don't understand what point you're trying to make here. I didn't say it was undesirable. I said it was heavily affecting how natural the models sounded. >It's also nigh-trivial to get the completion model back Try getting GPT-4o to write a story with villains that doesn't end with everyone singing Kumbaya and you'll see how much post-training affects the outputs of these models.