4 ms·

This is precisely the case where ChatGPT (or any LLM) will have trouble that a layperson would have trouble validating: Toki Pona has a limited document corpus

This is precisely the case where ChatGPT (or any LLM) will have trouble that a layperson would have trouble validating: Toki Pona has a limited document corpus and requires contextual interpretation obvious to experts. In short, expect hallucinations and other lies.

coder543 3y ago
In this case, I validated against the hand-translated example in the main link. It did fine. This is probably easy mode for it, given the tiny vocabulary at hand, even though it has surely seen very little Toki Pona in the wild. Large language models are extremely good at translation. I’ve personally tested ChatGPT-4 against Google Translate and other translators on some hard translations (languages that are not similar to English, such as Chinese), and it has consistently provided better translations than Google Translate in my opinion. As an example, for validation purposes, news stories that have been reported worldwide make things easier, since you can establish the facts of the translation by hand, and then compare what comes out. What does your comment actually add to the discussion? Yes, ChatGPT can be wrong. So can Google Translate! If you’d like to provide a clear example of ChatGPT-4 (not ChatGPT-3.5) making up a Toki Pona translation that is completely disconnected from reality, please do. That would be legitimately interesting conversational material. General cynicism isn’t. ChatGPT absolutely has limitations, and translation is a hard problem, even for the industry-standard tools. Crossvalidation with multiple translation tools and ideally humans is important for any translations that actually matter.
- NoZebra120vClip 3y ago
  Since LLMs' only valid uses involve entertainment and creativity, and Toki Pona is more or less a recreational language without serious application, I think they go together like two peas in a pod.
  - MostlyStable 3y ago
    >only valid use... Well. That's an opinion I suppose. -edit- After posting, I decided this was too snarky/low value on it's own. So to add: I personally get quite a bit of value out of it as an ad-hoc German tutor and as one-among-many tools for research and learning on new topics for which I am not already an expert. I've also used it to help me make and troubleshoot very simple code in languages that I am not an expert in. I personally have trivially found lots of uses that probably all cluster around "educational" to one degree or another. Is it perfect? Absolutely not. Does it hallucinate/get things wrong? Absolutely. Is it useful despite those limitations? Emphatically yes. This idea that its "only valid use" is entertainment doesn't even make sense to me. "Entertainment" is perhaps where I find it the most deficient relative to other current options.
- dotancohen 3y ago
  > In this case, I validated against the hand-translated example in the main link. It did fine. Presumably that hand-translated example was also part of the training data.
  - coder543 3y ago
    Not likely. The article says it was posted 1 year ago. ChatGPT’s training data cutoff is September 2021, so no, that would not make sense.
  - mistercow 3y ago
    I've done a fair amount of translating Toki Pona based on my own sentences, and based on recent online posts. None of that would have been in the training set. It does a good job. Since Toki Pona translation is "lossy", you can even prompt it to give you multiple interpretations and discussion.
    - deleted 3y ago
      [deleted]
- janAkali 3y ago
  Another challenge, is that toki pona requires a lot of context, not only of previous sentences, but visual and communal. For example, I can say 'soweli lili' and point to a cat, then in all further conversations 'soweli lili' will mean exactly a 'cat' until specified. I could use sentence that could mean literally hundred thousand of different things, but if I explain it properly once, you are expected to keep it as context. ChatGPT4 will struggle to keep all context as it will surely accumulate.
kordlessagain 3y ago
Lies are typically done on purpose and in this example used to indicate intent, when there is none.
- clankyclanker 3y ago
  Nope: https://news.ycombinator.com/item?id=35483823 https://news.ycombinator.com/item?id=35483823

coder543 3y ago

In this case, I validated against the hand-translated example in the main link. It did fine. This is probably easy mode for it, given the tiny vocabulary at hand, even though it has surely seen very little Toki Pona in the wild. Large language models are extremely good at translation. I’ve personally tested ChatGPT-4 against Google Translate and other translators on some hard translations (languages that are not similar to English, such as Chinese), and it has consistently provided better translations than Google Translate in my opinion. As an example, for validation purposes, news stories that have been reported worldwide make things easier, since you can establish the facts of the translation by hand, and then compare what comes out. What does your comment actually add to the discussion? Yes, ChatGPT can be wrong. So can Google Translate! If you’d like to provide a clear example of ChatGPT-4 (not ChatGPT-3.5) making up a Toki Pona translation that is completely disconnected from reality, please do. That would be legitimately interesting conversational material. General cynicism isn’t. ChatGPT absolutely has limitations, and translation is a hard problem, even for the industry-standard tools. Crossvalidation with multiple translation tools and ideally humans is important for any translations that actually matter.

NoZebra120vClip 3y ago
Since LLMs' only valid uses involve entertainment and creativity, and Toki Pona is more or less a recreational language without serious application, I think they go together like two peas in a pod.
- MostlyStable 3y ago
  >only valid use... Well. That's an opinion I suppose. -edit- After posting, I decided this was too snarky/low value on it's own. So to add: I personally get quite a bit of value out of it as an ad-hoc German tutor and as one-among-many tools for research and learning on new topics for which I am not already an expert. I've also used it to help me make and troubleshoot very simple code in languages that I am not an expert in. I personally have trivially found lots of uses that probably all cluster around "educational" to one degree or another. Is it perfect? Absolutely not. Does it hallucinate/get things wrong? Absolutely. Is it useful despite those limitations? Emphatically yes. This idea that its "only valid use" is entertainment doesn't even make sense to me. "Entertainment" is perhaps where I find it the most deficient relative to other current options.
dotancohen 3y ago
> In this case, I validated against the hand-translated example in the main link. It did fine. Presumably that hand-translated example was also part of the training data.
- coder543 3y ago
  Not likely. The article says it was posted 1 year ago. ChatGPT’s training data cutoff is September 2021, so no, that would not make sense.
- mistercow 3y ago
  I've done a fair amount of translating Toki Pona based on my own sentences, and based on recent online posts. None of that would have been in the training set. It does a good job. Since Toki Pona translation is "lossy", you can even prompt it to give you multiple interpretations and discussion.
  - deleted 3y ago
    [deleted]
janAkali 3y ago
Another challenge, is that toki pona requires a lot of context, not only of previous sentences, but visual and communal. For example, I can say 'soweli lili' and point to a cat, then in all further conversations 'soweli lili' will mean exactly a 'cat' until specified. I could use sentence that could mean literally hundred thousand of different things, but if I explain it properly once, you are expected to keep it as context. ChatGPT4 will struggle to keep all context as it will surely accumulate.

kordlessagain 3y ago

Lies are typically done on purpose and in this example used to indicate intent, when there is none.

clankyclanker 3y ago
Nope: https://news.ycombinator.com/item?id=35483823 https://news.ycombinator.com/item?id=35483823