38 ms·
Teaching ChatGPT to speak my son’s invented language
- replwoacause 4y agoI'm more impressed by your son than I am ChatGPT....TBH.
- drooby 4y agoYeah I was think yesterday maybe we can start translating dolphin language. Someone get on that
- pricklybear 4y agoSomeone is on that! https://time.com/6240144/aza-raskin-ai-animals-social-media/ https://time.com/6240144/aza-raskin-ai-animals-social-media/
- syntaxing 4y agoSuper curious, would fine tuning with LoRa on a LLaMa/Alpaca model work better?
- Tepix 3y agoYes, that's what i was thinking, please give fine-tuning a try!
- JCharante 4y agoI would like to see this expanded, I think it's a bit unfair to assess its abilities with so few examples. My hypothesis is that a rosetta stone with a thousand examples with a vector database hooked up to it so you don't hit the 32k token context limit would lead to much better performance.
- szopa 4y agoWe'd love to see that too! However, I'm afraid that creating a substantial number of examples would transform this delightful family activity into something akin to punishment. Kłeti is quite the challenge for us Indo-Europeans, and it seems that even its creator isn't immune to the struggle.
- afro88 4y agoBoth GPT-3.5 and GPT-4 versions of ChatGPT are limited to 4k tokens, even though GPT-4 is capable of 32k. This leads me to believe that part of the reason for some of the mediocre results OP saw was because they hit the token limit and ChatGPT started "forgetting" earlier parts of the conversation.
- szopa 4y agoNo, I was explicitly watching for this. In one of the sessions where we asked it to generate Kłeti sentences and the conversation passed the token limit it started inserting characters like ı (the Turkish dotless i). A week earlier I was playing with interpreting go positions, and at some point the model switched to talking about Chess (a bit less subtle than inserting unusual characters).
- knome 4y agoGPT-4 allows you to use 8k of context in their current beta, if you're using the chat api directly. It will be interesting ( and probably expensive, lol ) when they open it to a full 32k.
- Baeocystin 4y agoI'm really looking forward to being able to use a personalized LoRa on top of a GPT-4+ class model. I want to be able to train on all of may writing over the past few decades and interrogate the history of my ideas, and I think this would be tremendously valuable for writers of all kinds. Heck, think of the value of training (with their blessing) on something like /r/AskHistorians, or other deep-dive, high quality fora.
- Name_Chawps 4y agoThough unfortunately it will cost like $20 per 32k completion...
- M4v3R 4y agoMore like $1,92 (32 * 0,12) for a 32k prompt, or twice that for a 32k completion. Still not cheap though.
- Imnimo 4y agoThe vector database would be good for retrieving vocabulary, but could it be expected to do things like retrieve sentences with similar syntax or tenses? It feels like it would be hard to successfully retrieve examples that were important for reasons other than semantic content.
- rhn_mk1 4y agoNot trusting the models's self-assessment is the right call, considering that the actual score summed up to 7.5 compared to the self-reported 6.5 :)
- szopa 4y agoAs the author of the piece I feel that your comment triggered a great teachable moment :)
- famouswaffles 4y agoIn context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching arithmetic as an algorithm to be performed and taught similarly to how it's done with people, you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after.
- Buttons840 4y agoBeing able to learn within context, without updating weights is amazing. Imagine how much more efficient and/or powerful it could be if we found a way to update the weights in real time.
- joshribakoff 4y agoMaybe even more powerful would be reducing the number of examples needed to learn, eg less than one shot https://www.technologyreview.com/2020/10/16/1010566/ai-machine-learning-with-tiny-data/ https://www.technologyreview.com/2020/10/16/1010566/ai-machi... updating weights in real time is useless if each update basically does nothing because it takes an insurmountable amount of training, on the other hand if i can give my model a succinct “lesson” i’d then be very willing to wait a while for it to “process”
- dTal 4y agoAs I understand, that's basically what fine-tuning is?
- monkeydust 4y agoWhat sort of performance difference can you expect from in context Vs fine tuning, any relevant papers on this ?
- vintermann 4y agoOnce again illustrating that the powerful thing about ChatGPT is that no matter what you do, it does its best to play along. Its eyes do not glaze over.
- chankstein38 4y agoOne of the things that always gives me a little hit of hype is when I tell it to do something ridiculous and it just dutifully starts spitting out the result without complaining or questioning lol
- felipemnoa 4y agoI wonder if that is how our brain produces dreams? The guardrails are down so it will just start producing ridiculous and/or implausible things. Edit: It almost seems like you are anthropomorphizing it. It is just a program doing what it's supposed to be doing: to predict the next token based on its weights. Nothing more, nothing less. It does give the illusion of intelligence. Pretty soon, though, we may not be able to tell the difference.
- JKCalhoun 4y ago> It is just a program doing what it's supposed to be doing: to predict the next token based on its weights. Nothing more, nothing less. Every time I see a comment along these lines it gives me pause: there is a built-in assumption that each of us is somehow doing something more than this. I'm not convinced. I've heard people refer to some of our instinctive behaviors as due to "our lizard brain", suggesting that our brains are hierarchical, or comprised of a series of evolutionary steps, a more evolved order of brain wrapping the more primitive. I increasingly suspect that ChatGPT has more or less nailed one of those layers.
- hnfong 3y agoTo add to your point, note that a language model that "predicts the next token" accurately (as in predicting what human text would say) would pass the Turing test by definition. One may argue what passing the Turing test means, but at least by definition it is mimicking human intelligence in some way.
- marcodiego 4y agoI don't have access to ChatGPT4, but in my tests I could observe that it can't do some very simple tasks: - It can't play tic-tac-toe, - It can't play hangman, - It insists that winning on stone-paper-scissor using the chat (playing before me) is a matter of probability. It was also demonstrated that it can't reverse strings. Actually a transformer doesn't accesses 'strings', all it processes are tokens which are then mapped to vectors by whatever embedding is applied. I think it will be extremely difficult for a transformer to do any of these tasks correctly until a successor model is adopted. I don't have much hope of any reasonably complex symbolic processing of anything that it was not trained on. Some of these tasks are easy for a human to perform with paper and pencil and a set of rules; of course a human may get confused, but for that you write programs. Write code is one of GPT's skills but It is not "that" good with code for problems that are not mere small modification of problems it was trained on. EDIT: Could have expressed myself better: I don't have access to chatGPT4; I tested using the "available" chatGPT, I think it is 3.5. A transcript of me trying to play tic-tac-toe with it: https://pastebin.com/V1CW5hpt https://pastebin.com/V1CW5hpt
- serverholic 4y ago[dead]
- simonw 4y agoHow did you prompt it to play tic-tac-toe? I'm surprised that didn't work, it feels like something it should be able to handle really well. Hangman and stone-paper-scissors though are entirely unsuited to a language model, at least one with a chat interface like ChatGPT, because they both require it to be able to store a secret. ChatGPT has no ability to do this: each time it returns a response by evaluating the previous conversation. You could build a system that COULD play those games via an LLM but you'd have to write extra code to do it.
- ollien 4y agoWell, for hangman at least, if the human knows the secret, it should be possible for the LLM to handle that, no?
- dfxm12 4y agoDid it actually speak the language or did it just translate text? I'm not trying to be pedantic; these are two very different tasks.
- TeMPOraL 4y agoIt could not speak because it has no mouth, but as far as the translation go, I'd say somewhere in between. AFAIU, there's been some indication that GPT-4 works with concepts (so e.g. if it gets extra training for a specific task in one language, its performance on that task improves in other languages as well), GPT-3.5 probably does too, to a lesser extent.
- fcatalan 4y agoI've been trying a few things, some are very interesting. For example it understands Europanto* perfectly, but when I asked it to produce some it was germanic-only Europanto: English, German, Danish, Swedish... I told it to use more romance words and he came up with pure French. After some more prodding he achieved a decent mix. I also tried to get it to behave like an ersatz Duolingo for Basque and it sorta worked, but it would need some clever working on the prompts to really be usable. (*) Europanto is a joke language that uses random European language vocabulary on top of a generally English grammar.
- anon84873628 4y ago>All of these differences can make it surprising and challenging for someone with an Indo-European language background to learn and use Kłeti. Ironically, Proto-Indo-European is believed to be far more complex than its modern descendants, as described by Wikipedia: >PIE is believed to have had an elaborate system of morphology that included inflectional suffixes (analogous to English child, child's, children, children's) as well as ablaut (vowel alterations, as preserved in English sing, sang, sung, song) and accent. PIE nominals and pronouns had a complex system of declension, and verbs similarly had a complex system of conjugation. So maybe a PIE speaker would have an easier time with Kłeti than we :-)
- samus 4y agoSeveral of its modern descendants are not that much simpler :) Most famously, Baltic and Slavic languages have retained large parts of the case system. Some of them even the dual forms of nouns. Their verbal system has become even more sophisticated. Germanic languages retain the Ablaut system, even though it is no longer productive and has decayed into a bunch of irregular verbs.
- jutrewag 3y ago[dead]
- int_19h 3y agoWhat I find especially amusing with Baltic and Slavic languages is that they also preserved much of the original corpora for bodily parts and associated activities, just as swear / taboo words. https://en.wiktionary.org/wiki/Reconstruction:Proto-Indo-European/p%C3%ADsdeh%E2%82%82 https://en.wiktionary.org/wiki/Reconstruction:Proto-Indo-Eur... https://en.wiktionary.org/wiki/Reconstruction:Proto-Indo-European/h%E2%82%83y%C3%A9b%CA%B0eti https://en.wiktionary.org/wiki/Reconstruction:Proto-Indo-Eur...
- wizzwizz4 4y ago> as well as ablaut (vowel alterations, as preserved in English sing, sang, sung, song) Interestingly, English is gaining instances of ablaut. For example, dived seems to be being replaced by dove.
- dgritsko 4y agoThe idea of asking it to produce an "ouroboros prompt" that can be fed back into itself summarizing everything already learned is very clever; definitely going to use that in future ChatGPT sessions of my own.
- deleted 4y ago[deleted]
- zenlikethat 4y agoIt's surprisingly good at compressing and decompressing even sophisticated information if you ask it to! Makes you realize how much of our words are pretty much just fancy padding.
- b800h 4y agoBut it seems like it didn't work in this example..?
- m3kw9 4y agoNot sure if ChatGPT is correct but it does sound good
- GPTforfree 4y ago[flagged]
- robga 4y agoI am curious if the advent of GPT and LLMs allows linguistic theorists to adjudicate where we are with understanding the language instinct and settling the Chomsky vs Pinker vs Others debate. Perhaps it is entirely irrelevant as GLT has learned through billions of examples a child never could. Or perhaps it is totally relevant as it can synthesise billions of examples better than any linguist.
- fernly 4y agoOh I wish I had time to train it on one of my old hobbies, Lojban! https://lojban.io/ https://lojban.io/ https://mw.lojban.org/papri/Lojban https://mw.lojban.org/papri/Lojban
- JeromeLon 4y agoChatGPT already speaks Lojban, or at least enough to fool me.
- fernly 3y agoIt appears not: "vaguely grammatical and has some of the right words" according to someone who actually knows: https://www.reddit.com/r/lojban/comments/12i0d0i/chatgpt_appears_to_speak_fluent_lojban/ https://www.reddit.com/r/lojban/comments/12i0d0i/chatgpt_app... Not surprising, given it would have seen many orders of magnitude less Lojban training data than its English input (basically two books and maybe a few megabytes of web pages).
- szopa 3y agoThe word by word translation sounds like it's trying to say that it isn't very competent at lojban, but that it can try to learn lojban if you provide it with parallel examples. All this said in broken lojban, as expected. Quite reasonable, actually.
- int_19h 3y agoI gave much larger snippets from the translation of Alice in Wonderland, and it "translated" them without complaints, but it was about half gibberish. Same thing with modern takes on Old Norse.
- moffkalast 4y agoAny language existing prior to 2021 isn't gonna be very useful for testing its improv abilities, since they're likely all in the training data.
- crdrost 4y agoWow, they asked the model to self-evaluate and it just outright cheated: He has three cats. Proposed: h’io’ngkiltrikumrikumrikumri’nguuy Correct: h’io’ngkiltri’ngkumrikumri’nguuy Points: 1 Hypothesis: N/A (Other comments observe that it accidentally compensated for this by getting the sum wrong, haha, d'oh) I have had similar problems with trying to get ChatGPT to do nontrivial things, "here are the rules for this game, do you understand this game, great, let's play it." And then it's like herding cats. "No that's wrong, the game pieces cannot leave the game board," "Oh my apologies you are entirely correct, here is the revised board (proceeds to dump the exact same state of the game board that I told it was wrong)." Eventually it will lie about its own capacities, "As an AI language model I am incapable of selecting a move to play next"... But you have done several already!!! This is literally the ONLY thing you have been doing right and now you refuse? Some other prompts are more successful but it does seem to have a sing-song high school book review style that inclines it to be boring... Very uncanny valley.
- sharkweek 4y agoI was trying for 20 minutes to get it to spit out all 50 state capitals with the city names in alphabetical order and it kept doing two things: 1) It'd put the list in alphabetical order by state, but it'd include all the correct capitals 2) It'd list 49 of the 50 capitals, in alphabetical order this time, but duplicating Madison, WI. I'd ask it to try and figure out what it did wrong in both cases, and it'd correctly identify the mistake, but then repeat it. Not sure how I got there eventually, but on about the 7th or 8th attempt, it got it right.
- Daneel_ 4y agoNot here to one-up you, but currently this is just down to how you ask. I came up with this in about a minute: "Please list all 50 US state capital cities, with the list sorted alphabetically starting at the first letter of each line of your response. Please do not create sections for each letter." This returned: - Albany, New York - Annapolis, Maryland - Atlanta, Georgia - Augusta, Maine - Austin, Texas - Baton Rouge, Louisiana - Bismarck, North Dakota - ... My gut feeling is that to get what you want from it you need to have a solid understanding of how to manipulate search engines and other fuzzy input systems. On self reflection I find it interesting that I wrote "Please" at the start of each sentence, as if that would give me a better output. Heh.
- arps18 4y agoThis is a super amazing stuff! Just blown away with the power of AI in tbe future! I wonder if that still exist to make some of the languages spoken by humans which are on the verge of extinction?
- i-use-nixos-btw 4y agoThis sounds like a lot of fun! It’s also great that you’re encouraging this - having a passion is great, and following it through to this level is fantastic for development. I have a question. When you ask an LLM to summarise the rules of the language in a concise manner, and it does so, is it… disheartening? I mean, if you put a lot of work into creating something, and it gets described in terms that basically make it feel like a configuration - given a glossary and a set of toggles and fill in the blank settings (e.g. word boundaries), out pops a language. Does seeing that make it feel somewhat less exciting?
- deleted 4y ago[deleted]
- graypegg 4y agoUnrelated to the actual work here (which is interesting!), I just wanted to mention how much of an awesome impact you must be having on your son. You two seem like such a fun curious pair! I spent a lot of time tinkering away alone in my room as a kid and that sucked. The opportunities you’re giving him now to explore are going to inspire a lifetime of creativity. I know you don’t need someone on the internet to tell you that, but I thought I’d give kudos where it’s due :)
- isoprophlex 4y agoI second that. I remember finding a manual for programming BASIC in the library once. Overjoyed I was. Then, I spent an entire afternoon trying to type BASIC commands... into the MS-DOS prompt. If only someone could have told me to go find QBASIC.EXE first, haha
- 6510 4y agoMine is a silent language. I look at people and wait until they guess what I'm thinking or understand what I mean. Sometimes I show them an imaginary sphere that I hold with both hands. edit: I had one more OT thought: This article convinced me even further that children should learn machine code. Adults (we) should stop pretending we are anywhere near as sophisticated at learning languages as they are. I've seen 5 year olds play together and create a hybrid dutch/russian language in 5 minutes. They just kept talking. Within an hour they had a pretty good understanding, within a day it was good enough for what they needed.
- DonHopkins 4y agoPrompt: Write a poem in Simlish. ChatGPT-4: While it's important to remember that Simlish is a nonsensical language without a proper grammar or vocabulary, I can still attempt to create a playful poem using some known Simlish phrases and sounds. Keep in mind that this poem will not have any real meaning: Sul sul! Vadeesh, plurna zib, Badeesh! Dag dag, cribna drib, Froobi flazzle, noogit fraz, Glarbni weeb, frizzle maz. Lerpna gort, jibbly zor, Slooble wib, kwebna dor, Skibbi plurt, pleeba narn, Grubble veeb, shazzle darn. Wabble gret, minzi glarn, Flartle doo, zibba yarn, Yibbs frazzle, dribna glorp, Vorpti zound, greeble snorp Enjoy the playful sounds and rhythm of the poem, even though it lacks any real meaning!
- DonHopkins 4y agoPrompt: Write some gangsta rap lyrics in Simlish. ChatGTP-4: Creating gangsta rap lyrics in Simlish presents a unique challenge, as Simlish is a nonsensical language, and rap lyrics often rely on meaningful content and wordplay. However, I can attempt to create a playful verse using Simlish phrases and sounds while maintaining a rhythm typical of rap. Keep in mind that these lyrics won't carry any real meaning: Verse 1: Yo, sul sul, zibba zoob, Froodle grubble, wabba woob, Glarbni skib, dag dag, dret, Cribna drib, yibbs, noogit zet. Chorus: Gangsta Sim, plurna fraz, Vadeesh, plurna, zibba maz, Slooble wib, kwebna dor, Lerpna gort, jibbly zor. Verse 2: Flartle doo, minzi glarn, Yibbs frazzle, dribna darn, Badeesh! Noogit, greeble snorp, Vorpti zound, plurt, weeb, glorp. Chorus: Gangsta Sim, plurna fraz, Vadeesh, plurna, zibba maz, Slooble wib, kwebna dor, Lerpna gort, jibbly zor. Remember, these lyrics are meant to be playful and fun, as they do not have any real meaning or convey any specific message.
- DonHopkins 4y agoPrompt: Write an a cappella cover of a Philip Glass song in Simlish. ChatGPT-4: Creating an a cappella cover of a Philip Glass song in Simlish presents an interesting challenge. Philip Glass is known for his minimalist compositions that often consist of repetitive structures, and Simlish is a nonsensical language. Since Glass's compositions are largely instrumental, I will create an a cappella adaptation inspired by his musical style while using Simlish-like sounds. Verse 1: Doo doo zib, doo doo zib, doo doo zib, doo doo zib, Bada bada zoob, bada bada zoob, bada bada zoob, bada bada zoob, Glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb, Sul sul sul, sul sul sul, sul sul sul, sul sul sul. Chorus: Froodle froodle, zibba zibba, Dag dag, drib drib, yibbs yibbs, Slooble slooble, wib wib, dor dor, Lerpna lerpna, gort gort, zor zor. Verse 2: Doo doo zib, doo doo zib, doo doo zib, doo doo zib, Bada bada zoob, bada bada zoob, bada bada zoob, bada bada zoob, Glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb, Sul sul sul, sul sul sul, sul sul sul, sul sul sul. Chorus: Froodle froodle, zibba zibba, Dag dag, drib drib, yibbs yibbs, Slooble slooble, wib wib, dor dor, Lerpna lerpna, gort gort, zor zor. Remember, this adaptation is meant to be playful and fun, as it doesn't have any real meaning or convey any specific message. The focus here is on capturing the essence of Philip Glass's minimalist compositions using Simlish-like sounds in an a cappella context.
- sage11 4y agoYou might enjoy creating personalised bedtime stories for your kid: www.mybabystory.co
- lloeki 4y ago> For example, Kłeti has a sentence structure that is SOV (Subject-Object-Verb), while all the languages I know have a Subject-Verb-Object (SVO) sentence structure. "all the languages I know" caveat alright, still Japanese has the verb at the end, and my German is far far away but I seem to recall it has something like that as well (at least the meaningful "verb-like" semantic part of the sentence often being at the end)
- OJFord 4y agoThat was an 'example' of how: > the language is very different from the Indo-European languages that I am familiar with. So it's not even a caveat really, nevermind misleading or suggesting the author thinks all languages have that order.
- snickerer 4y agoWhat a wonderful story and a great family. Congratulations! You inspired me to ask GPT to do it the other way around. I asked it to invent a successor to Esperanto, with influences from Slavic, Germanic, and Latin languages. It called its language Euroglossa and wrote a short story in it. Who can understand it? Un tag, en froliko vilaž, un jun chico namen Tomas trovat un misterioz mapo v star bibliotek. Na mapo, skribet: "Skarb de Tri Montes." Tomas decidet da sledit la mapo in aventuro, sperante da otkriti grand skarb. Tomas paket svoi rukzak s neobkhodim stvari, inkluziv kompas, binaukli, i nutrimento. Nachet svoi putovanje, iz vilaž, do la Tri Montes. Po nekoliko dni, on prishel k bazen na pervoj monte. Tam, on otkril zagadka, ki je klyuch za dalsi koraki: "Kogda solntse küsst la luna, dvigat kamen i vstretit un oko." Tomas wartet geduldig bis la sonne küsst la luna in noktchielo. Kvando moment venit, er raskt dvigat un gros stein u otkryvajet secret passaž. Ingressante, on sledit un dunkel tunel, portant un torča por iluminar svoi put. La tunel führt tief in monte, bis Tomas entdeckt un hider kammer s ančient skulpturen i un glänzend tresor. V la hider kammer, Tomas najde un skarbnica poln von stran objekte, inkluziv un animat statuo von un banane, ki parlat und tanzt. Zaskočen, ale smejšeći, Tomas realisat ke la glänzend tresor je ver la amikeco i rado ke li povas dividi kon svoi vilaž. La legenda "Skarb de Tri Montes" je veritabile un symbol za unite de diversa kulturi i lingvoj en un mondo plen da froliko i humor.
- isaacfrond 4y agoprompt and output of gpt4: Someone created a successor to Esperanto, with influences from Slavic, Germanic, and Latin languages. It's called Euroglossa. Please, try to translate the following short story to English. One day, in a playful village, a young boy named Tomas found a mysterious map in an old library. On the map, it was written: "Treasure of the Three Mountains." Tomas decided to follow the map on an adventure, hoping to discover a great treasure. Tomas packed his backpack with necessary items, including a compass, binoculars, and food. He began his journey, from the village to the Three Mountains. After a few days, he arrived at the base of the first mountain. There, he discovered a riddle, which was the key to the next steps: "When the sun kisses the moon, move the stone and meet an eye." Tomas waited patiently until the sun kissed the moon in the night sky. When the moment came, he quickly moved a large stone and uncovered a secret passage. Entering, he followed a dark tunnel, carrying a torch to light his way. The tunnel led deep into the mountain, until Tomas discovered a hidden chamber with ancient sculptures and a shiny treasure. In the hidden chamber, Tomas found a treasure chest full of strange objects, including an animated statue of a banana that spoke and danced. Surprised but laughing, Tomas realized that the shiny treasure was truly the friendship and joy that he could share with his village. The legendary "Treasure of the Three Mountains" was genuinely a symbol of the unity of diverse cultures and languages in a world full of playfulness and humor.
- jakkos 4y agobookmark
- huseyinkeles 4y agoYou can use the "favorite" link instead.
- DonHopkins 3y agoThe Future of GPT4 1337 Programming: CyberCode Tutorial with <=\> cYber-C0mb1n4t0r !!! </=> I asked GPT-4 to implement QuickSort and the Y-Combinator in my own personal programming language that I designed — who can guess the influences (besides cannabis and shrooms)? https://donhopkins.medium.com/the-future-of-gpt4-programming-cybercode-tutorial-with-cyber-c0mb1n4t0r-ddfdf82a84d https://donhopkins.medium.com/the-future-of-gpt4-programming...