3 ms·

This is pop culture type nonsense. Firstly, paying someone to do a job is not "exploiting". I mean, I guess it is if exploit is pared down to just mean "use fo

This is pop culture type nonsense. Firstly, paying someone to do a job is not "exploiting". I mean, I guess it is if exploit is pared down to just mean "use for a benefit", but it is a pejorative in this context, meant to clutch some pearls and virtue signal (in the weird "I'd rather they don't get paid or be a part of this" way). I'm Canadian and to Americans getting skilled work done in Canada is usually cheaper, and I guarantee you we don't see it as exploitation. But then it argues that "their" language in Nigeria/Kenya -- you know, English -- is being exploited by being "pushed back" on them. Bizarre. Not only does the contention make zero sense from an RLHF perspective, it's spurious and backed by nothing. Some random guy on YouTube said "Oh yeah this word must come from Kenya". Delve? I see this word regularly, and have for decades. The other words -- “Explore”, “tapestry”, “testament” and “leverage” -- are in no universe rare words. They may appear more frequently than random Facebook comments because OpenAI weighted the inputs and put more emphasis on high quality sources where the grade level is a bit higher, which makes infinitely more sense than it being some manifestation of Kenya or Nigeria. Similarly words and phrases are viral, especially in niches. That words come into or out of favour in some realm like papers on PubMed by itself proves little and isn't particularly demonstrative. We all stand on the shoulders of giants, and we're all aping what we read and what we see, etc.

mzs 2y ago
Also there is no world-wide authority on English. I'm glad more regional niches are entering training data.
- itishappy 2y ago
  TIL that English is the official language of 53 countries. This list does not include the UK, the US, Australia, or New Zealand. https://en.wikipedia.org/wiki/List_of_countries_and_territories_where_English_is_an_official_language https://en.wikipedia.org/wiki/List_of_countries_and_territor...
wrs 2y ago
As a counterexample, English speakers in India use the word “revert” in email a lot more often than Americans do, because there it can mean “reply”, as in “I will revert with that information later today.” I would think that is a much more frequently expressed concept than any of the other meanings of “revert”, so Indian English (at least Indian business email English) is going to have a higher frequency of “revert” than American English. Not that this exact phenomenon applies to “delve”, but it shows how word frequencies can vary across cultures regardless of “grade level”.
- brightlancer 2y ago
  > English speakers in India use the word “revert” in email a lot more often than Americans do, because there it can mean “reply”, as in “I will revert with that information later today.” That's not just an issue of frequency but redefinition. I read about English-speaking Indians using the word "prepone" to mean "pushed forward" (as an inverse of "postpone"), and they were shocked when non-Indian English speakers didn't know what they were saying -- because it's not a word in British or American English, and AFAIK it's not a word in among any other native speaking English societies. Neologisms and re-definitions aren't inherently bad, but it can make communication more difficult.
viccis 2y ago
Delve is indeed a word that I've seen for decades, but never in grade grubbing emails from students until about a year ago. "I would like to delve into why I think my 45% should be a B- ...", etc. I would imagine that those other words (which seem very common to me) are being used in ways that would seem idiosyncratic to an American English speaker. Like I see tapestry all the time in very flowery language ("The country has a rich tapestry of influences ..."), but I would be amused if I read it being used to describe contents of a backlog at work. I will disagree strongly with this: >That words come into or out of favour in some realm like papers on PubMed by itself proves little and isn't particularly demonstrative. This is ignoring what appears to be an overwhelming effect (an increase of around 1250%) that kicked off with ChatGPT's inception. I don't think you outlined adequate grounds for ignoring this as unrelated, given the timing, the magnitude, and the ready explanation of "delve" being used frequently in countries where the model was trained.
- llm_nerd 2y ago
  > being used in ways that would seem idiosyncratic The argument is not that these words are being misused or abused. They're comparing the frequency on the web in general versus the frequency in GPT output, which is not useful or illuminating. For all those words, including delve, the usage is completely cromulent and grammatically correct, but someone needed to spin it into some egregious exploitation. > an overwhelming effect Delve was on an upswing for years. Again, it takes just one popular paper to use the phrase "delve into the results" and an army of new papers are going to use the same phrase because we're all stochastic parrots to some degree. Just as Attention Is All You Need has massively influenced countless subsequent papers. Again, there is zero evidence that ChatGPT even overuses "delve". The guy's evidence that GPTs overuse delve is that it used it a few times and he thinks that's overuse. Okay. Similarly, the evidence that Kenya and/or Nigeria use delve more is just...well it just works with the whole tepid argument okay? > "delve" being used frequently in countries where the model was trained The model was trained on TBs of internet data. A small amount of, essentially, "tone" training is done via RLHF, but that wouldn't suddenly make local dialect surface.

mzs 2y ago

Also there is no world-wide authority on English. I'm glad more regional niches are entering training data.

itishappy 2y ago
TIL that English is the official language of 53 countries. This list does not include the UK, the US, Australia, or New Zealand. https://en.wikipedia.org/wiki/List_of_countries_and_territories_where_English_is_an_official_language https://en.wikipedia.org/wiki/List_of_countries_and_territor...

wrs 2y ago

As a counterexample, English speakers in India use the word “revert” in email a lot more often than Americans do, because there it can mean “reply”, as in “I will revert with that information later today.” I would think that is a much more frequently expressed concept than any of the other meanings of “revert”, so Indian English (at least Indian business email English) is going to have a higher frequency of “revert” than American English. Not that this exact phenomenon applies to “delve”, but it shows how word frequencies can vary across cultures regardless of “grade level”.

brightlancer 2y ago
> English speakers in India use the word “revert” in email a lot more often than Americans do, because there it can mean “reply”, as in “I will revert with that information later today.” That's not just an issue of frequency but redefinition. I read about English-speaking Indians using the word "prepone" to mean "pushed forward" (as an inverse of "postpone"), and they were shocked when non-Indian English speakers didn't know what they were saying -- because it's not a word in British or American English, and AFAIK it's not a word in among any other native speaking English societies. Neologisms and re-definitions aren't inherently bad, but it can make communication more difficult.

viccis 2y ago

Delve is indeed a word that I've seen for decades, but never in grade grubbing emails from students until about a year ago. "I would like to delve into why I think my 45% should be a B- ...", etc. I would imagine that those other words (which seem very common to me) are being used in ways that would seem idiosyncratic to an American English speaker. Like I see tapestry all the time in very flowery language ("The country has a rich tapestry of influences ..."), but I would be amused if I read it being used to describe contents of a backlog at work. I will disagree strongly with this: >That words come into or out of favour in some realm like papers on PubMed by itself proves little and isn't particularly demonstrative. This is ignoring what appears to be an overwhelming effect (an increase of around 1250%) that kicked off with ChatGPT's inception. I don't think you outlined adequate grounds for ignoring this as unrelated, given the timing, the magnitude, and the ready explanation of "delve" being used frequently in countries where the model was trained.

llm_nerd 2y ago
> being used in ways that would seem idiosyncratic The argument is not that these words are being misused or abused. They're comparing the frequency on the web in general versus the frequency in GPT output, which is not useful or illuminating. For all those words, including delve, the usage is completely cromulent and grammatically correct, but someone needed to spin it into some egregious exploitation. > an overwhelming effect Delve was on an upswing for years. Again, it takes just one popular paper to use the phrase "delve into the results" and an army of new papers are going to use the same phrase because we're all stochastic parrots to some degree. Just as Attention Is All You Need has massively influenced countless subsequent papers. Again, there is zero evidence that ChatGPT even overuses "delve". The guy's evidence that GPTs overuse delve is that it used it a few times and he thinks that's overuse. Okay. Similarly, the evidence that Kenya and/or Nigeria use delve more is just...well it just works with the whole tepid argument okay? > "delve" being used frequently in countries where the model was trained The model was trained on TBs of internet data. A small amount of, essentially, "tone" training is done via RLHF, but that wouldn't suddenly make local dialect surface.