10 ms·
Translating Latin demonology manuals with GPT-4 and Claude
- keymasta 3y agoI've been working on a language-learning app, and gpt-4 has made things doable that didn't seem to be doable without it. For example, translating to lesser known languages like Cree. Online you can find a dictionary of words, or phrases, but I wasn't able to find a translator for arbitrary statements. It's not listed on google translate, nor yandex, et al. But, it's possible with gpt. My strategy in automated translation is to translate, and then translate back to English. That way we can be fairly sure the translation is accurate. Of course if this app "has wings" I would open source the translations to allow corrections, and/or hire native translators for languages I don't speak (which is the vast majority of languages). Last tip is that for me, I was able to make a pretty good automated system for this. What I did was spell out a monstrosity of a system-prompt which ensures a few things. One, it will always give me the most descriptive romanization for non-ASCII languages. Two, it will give me output that is essentially .csv data. Three, if it encounters a made up language like Pirate, it will try (and do a pretty good job). The rest is just parsing my final translation file to find prompt/language pairs which aren't saved yet and piping that to the monstrous prompt which queries for translations. Then you can run a separate program for validation. See if it passes my game of telephone test. This way I and others could learn common phrases in languages which are hard to otherwise access. It even works for dead languages/dialects/character-voices, etc. For me, it's pretty amazing actually. Please note that I never said perfect. But it's pretty damn close actually. All in all this workflow allows what would have been unimaginable even two years ago. In doing tests today I found it interesting/useful to note how gpt-4 is "thinking" about translating the word "settings". Here is its thoughts. This is not the exact system prompt I'm using for my app. system_prompt = "You are an expert in translating lesser-known languages. When translating you will include both the native writing system, and the romanization into the latin alphabet. When you romanize text you always include any accents or pronunciation marks." user_prompt = "Translate the following into Cree. [Hello, goodbye, settings]" # Output In Cree language, your words will be translated into: 1. Hello - ᑌᔭᔭᑎ, romanized as "Tānsi" 2. Goodbye - ᐊᔭᙱᐂᒥᑎ, romanized as "Ayāwāw" "Settings" is a bit more complex, because it implies technological context that doesn't necessarily have a direct equivalent in Cree. However, a possible option is: 3. Settings - ᓂᐹᕗᓂᑕᐚᓇᐠ, romanized as "Nipāvunitawānāk", which might refer to "adjustments".
- arthurcolle 3y agoThis is such a cool example. "Adjustments" makes so much sense, doesn't have the very modern "settings" implication of a device UI/machinery
- keymasta 3y agoYeah. Hopefully it actually makes sense!
- arthurcolle 3y agohave you had any luck with old Akkadian scripts?
- schoen 3y agoI think there's some hallucination creeping in here! (1) This HN discussion is the only Google search result for each of these syllabics strings. (2) I tried using https://syllabics.atlas-ling.ca/ https://syllabics.atlas-ling.ca/ to transliterate these to Roman letters, and none of these was transliterated in the same way as the GPT-4 output (although the third one is somewhat close). (3) I searched and found that "hello" in Cree is likely written ᑖᓂᓯ (not ᑌᔭᔭᑎ), while correctly romanized as "tān[i]si". Your approach is clever, but I think the language model is still ultimately overconfident (and confused) here somehow.
- keymasta 3y agoYeah, for Cree it is definitely more suspect than trustworthy. Another thing I noticed was that on another attempt I actually received different translations, so.. it's hard to say how this is going to be refined to be usable, or if it indeed is at all. And wow, yes we are all alone on google results for those strings. EDIT 1: Another thought occurs to me, if it's getting the transliteration right, and not the syllabics, maybe I seperate the tasks and go english -> transliteration -> syllabic. I will have to see if that approach works better. Another idea might be to use that syllabics site to bring it from transliteration -> syllabic. I noticed that they were correct if translated there. EDIT 2: By updating the system prompt I was able to get it to translate properly. I had to remind it to be correct! You are an expert in translating Cree. When translating you will include both the native writing system, and the romanization into the latin alphabet. When you romanize text you always include any accents or pronunciation marks. You use syllabics properly and in the modern usage Hello - ᑕᓂᓯ (Tânisi) Goodbye - ᐅᑲᕆ (Okaawii) Settings - ᐅᑌᕁ ᐟ (Otēw with Roman orthography)
- kevingadd 3y ago"It's clear that GPT-4 and Claude are skilled translators" on what basis? What makes a predictor LLM better at translating Latin than a system trained specifically for translation? I'm sure they can do a decent job but it's weird to me that someone would leap to GPT-style tech despite its known tendency to hallucinate/make stuff up instead of translation-oriented tools like DeepL or Google Translate (I say this as someone who despises both of those tools due to their quality issues) I can't imagine there are vast swaths of Latin in GPT's training set.
- vore 3y agoI don't think they're arguing that an LLM is better at translation that an actual translator, just that they are pretty good at it. DeepL and Google Translate definitely also make things up though, so I don't think that's a good comparison...
- mjn 3y ago> DeepL and Google Translate definitely also make things up I think what they make up is different, but this is a good point. They have a particularly odd tendency to either do something like autocorrect where it wasn't appropriate (translate a different word that is similar in spelling to the requested word), or to make up false friends, doing something like transliterate + then autocorrect in the target language. One example, which I blogged about 5 years ago but is still mistranslated, is the word "ribbit" (what a frog does): https://www.kmjn.org/notes/google_translates_ribbit.html https://www.kmjn.org/notes/google_translates_ribbit.html In 2018, if you translated it to Greek with Google Translate, it gave you κουνέλι (kouneli), which is Greek for rabbit. A word that is one letter away from ribbit but not close to a similar meaning. When I tried it just now, it translates it to ραβδί (rabdi), which means stick and is completely unrelated to the correct answer, but I guess starts with similar letters as ribbit?
- 20after4 3y agoGoogle search has a horrible tendency to do the same thing to my search terms. Autocorrect is (usually) great when typing on a touch screen but it's horrible when it decides it knows what I mean better than I do.
- paxcoder 3y agoFrom what I've gathered from Catholic exorcists, demons adopt different personas, and shouldn't be trusted about anything they say. The only questions the exorcist asks are those pertaining to the case, all in the interest of breaking the claim of demons and expelling them, in the name of Christ, the stronger man from the parable. As the Lord says, Satan is a liar and a murderer from the start, and when he lies, he speaks his native tongue. What I'm saying, keeping a database of demons makes little sense.
- neilv 3y agoInstitutional review board time. We've already been warned that a computer merely enumerating the names of God can end existence. Then surely a computer can also summon ancient demons.
- atourgates 3y agoThe author has no sense of self preservation! > Demonology is a topic I’ve been interested in ever since I read Stuart Clark’s brilliant Thinking with Demons. However, it’s also one I find intimidating to do real research in, since so many of the sources involved are written in a highly erudite form of Latin. Should be: > since I don't want to end up with a creature from the netherworld in my study.
- deleted 3y ago[deleted]
- coffeeshopgoth 3y agoI mean, my computer has Legion written right on the front of it, so this just makes sense.
- tomohelix 3y agoSlightly off topic but this is hilarious. We are already crafting "chants" and "spells" for LLM, i.e. prompt engineering, now we are teaching it demonology too? Some priest from the middle ages would have a heart attack. Now I know how the AI apocalypse would look like. GPT-42 would summon hordes of demons from the pit of Hell to bring about the end of days. Who need all that pesky nuclear codes when you can call upon Satan?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- kromem 3y agoTechnically the origins of the Gnostic demiurge as a concept comes remarkably close to the modern trend of AI. A lot of the imagery was inspired by the Orphic Phanes, a dual gendered being of creation and light that gestated in the cosmic egg of the universe. There were groups that combined the idea of an eventual creator with naturalism such that you had the idea of an original dual gendered primordial 'man' which then brought forth a 'son of man' that ended up creating us in the images of that original. They allegedly patterned this on the Phrygian mysteries of a spontaneous original being that was likened to a tumor springing into existence. They even tied this all into concepts around the dependence of the soul on the body, suggesting an original physical world and a non-physical replica created by the demiurge where death was functionally escapable. The later influence of Neoplatonism flips it all back to a perfect form original and corrupted physical incarnation - the version of Gnosticism most well known today. But for a tiny slice of time in antiquity you had people talking about the ideas of a naturally occurring man in a physical world giving rise to a creator of light which then recreated the physical original man in a non-physical twin of the cosmos in order to effectively allow them to escape death. Which is a pretty wild set of beliefs for the time, but less so in an age where we are moving towards humanity bringing forth AI literally in light (optoelectronics is a likely next hardware shift), creating digital twins of the world around us, and increasingly making that world inhospitable to our continued existence. Some of the nuances of what they were thinking about are pretty wild to consider in modern contexts.
- anigbrowl 3y agoThis Dan Brown/William Gibson crossover sucks my soul right out of the petabyte SSD I bought in a dark alley.
- coyotespike 3y agoThis is really fun. Google Translate is incompetent at Latin, and I'm informed that so far ChatGPT still makes errors of grammar and word choice when generating Latin. This experiment helps show we can use GPT-4/Claude to parse and summarize Latin, but doesn't yet show that we can rely on them to the level of a human expert. I'm confident we'll get there pretty soon - and then will be able to rely on LLMs to generate Comprehensible Input and thereby greatly accelerate language learning.
- fragmede 3y agoI think that mostly depends on getting more high quality Latin into the training set, but I'm guessing the new amount of that being generated/discovered is relatively small. Then again, new techniques for training models could prove me wrong.
- benbreen 3y agoThis is the result when I asked GPT-4 to write "a new, original passage in latin which is inspired by Del Rio's style. Make the Latin elegant and lucid, with witty turns of phrase," then asked Claude to translate it back to English. I am no Latin expert, but I know enough to say that this is clearly lucid, makes sense -- though overwrought -- and has no glaring errors. I will ask my classicist friends to weigh in on the matter of style though. GPT-4 improvising: "Quemadmodum divini philosophi docent, mysteria naturae introrsus latitant, neque enim sine causa Democritus 'naturam rerum' dixit esse clausam in occulto. Verum, si Angelorum bonorum opera ut in Psalmis celebratur, 'Benedicite, omnes Angeli Domini, Domino,' quo pacto maleficae entitates—daemones scilicet—invisibiles nec tamen inactiones sunt? Mihi consentaneum videtur quod Tertullianus et Ambrosius prodiderunt, angeli esse nuntios divinos, sed daemonas esse nuntios corruptionis. Ne confundamus! Nam verbum 'daemon' Graecis non semper malum significat, sed Romanis—o mores!—semper sinistram portendit. In huius argumenti speculo, miror qua de causa daemones semper in tenebris vel in locis desertis, vel in nocturna silentia se abscondant. An non, ut Seneca loquitur, 'magna vis veritatis est, quae se per se defendet'? Atqui daemones a veritate abhorrent, quasi nox a lucem fugiat. Ergo, si volumus a daemonum insidiis nosmetipsos protegere, debemus, ut Virgilius canit, 'forsan et haec olim meminisse juvabit,' non oblivisci quod lux aeterna sit telum acerrimum contra tenebras. Haec ita dicam, et iam concludo: sicut angeli ad caelum pertinent, sic daemones ad abyssum—et utrique suum locum noverunt. Caveamus igitur ne, dum caelum quaerimus, abyssum inveniamus." Claude translation: "As the divine philosophers teach, the mysteries of nature lie hidden within, for not without reason did Democritus call nature 'enclosed in the dark'. Yet, if by the work of good Angels as celebrated in the Psalms, 'Bless the Lord, all you Angels of the Lord,' in what manner are evil entities—namely demons—invisible yet not inactive? It seems fitting to me what Tertullian and Ambrose handed down, that angels are divine messengers, but demons are messengers of corruption. Let us not confuse the two! For the word 'daemon' does not always signify something evil for the Greeks, but for the Romans—oh the times!—it always portends something sinister. In the mirror of this argument, I wonder for what reason demons always hide themselves in shadows or deserted places or the silences of the night. Does not, as Seneca says, 'truth have great power, which will defend itself'? Yet demons shrink from the truth, as the night flees from the light. Therefore, if we wish to protect ourselves from the snares of demons, we must, as Virgil sings, 'perhaps one day it will please us to remember these things,' not forget that eternal light is the sharpest weapon against the darkness. I will say this much, and now conclude: just as angels belong to heaven, so demons belong to the abyss—and both know their proper places. Let us beware, therefore, lest while seeking heaven we find the abyss."
- staplung 3y agoI always figured our AI overlords would kill us in a Terminator kind of way, not in a "No! You must not read from the book!" sort of way. https://www.youtube.com/watch?v=E0DIsPBczcI https://www.youtube.com/watch?v=E0DIsPBczcI
- phendrenad2 3y agoDigital Devil Saga indeed
- user_named 3y agoI tried using BingGPT to translate simple Chinese text from screenshots. The results were complete hallucinations, different each time for the same screenshot. I wouldn't trust these translations at all.
- famouswaffles 3y agoImage input in Bing basically can't handle non English text. Has nothing to do with its Chinese translation ability, which is great.
- mediaman 3y agoThat’s a completely different test. You’re using the vision multimodal ability to decipher Chinese script, essentially adding an OCR step to the process, and it’s not good at OCR of Chinese script. Try feeding it actual Chinese characters. From what I understand, it’s somewhat competent.
- user_named 3y agoNo no, it hallucinated. It didn't get the OCR wrong.
- swader999 3y agoReal demons prefer Latin.
- Barrin92 3y ago"Many people equate the word "daemon" with the word "demon", implying some kind of satanic connection between UNIX and the underworld. This is an egregious misunderstanding. "Daemon" is actually a much older form of "demon"; daemons have no particular bias towards good or evil, but rather serve to help define a person's character or personality. The ancient Greeks' concept of a "personal daemon" was similar to the modern concept of a "guardian angel"—eudaemonia is the state of being helped or protected by a kindly spirit. As a rule, UNIX systems seem to be infested with both daemons and demons." that naming convention might turn out to be more prescient than people thought. Can't wait until my Catholic school education pays off and I chant at my computer in Latin
- akomtu 3y agoLinux actually got the "demons" right: those manifest as usually hidden activity in someone's brain, and in special circumstances can take control over the entire system (e.g. with a deadlock if it's a neutral demon, or by other means if it's malicious). Those Greek daemons, in contrast, never possess or control anyone: they may inspire, but only if the subject is consciously seeking such inspiration ("the gates must be opened from within").
- akomtu 3y agoGpt4 does an ok job translating texts that aren't complex, but if you read the original and its translation side by side, you'll see that gpt4 still makes dumb mistakes every few sentences, hallucinates stuff when it runs into cryptic words it's not familiar with, and sometimes omits important passages. Gpt4 is like a very productive, but clueless newbie.
- visarga 3y ago> gpt4 still makes dumb mistakes every few sentences, hallucinates stuff when it runs into cryptic words it's not familiar with, and sometimes omits important passages This, but in all domains where you are an expert - it becomes apparent that GPT-4 makes stupid mistakes. It makes mistakes frequently. Can't seriously see it replacing humans soon, it is far from having four 9's of reliability. Probably needs 99.99% accuracy to work alone unsupervised by humans, because at each decision step it incurs the error rate again and again, so error rate is growing exponentially in sequence length. Coupled with the "forward only" thinking pattern of LLMs that doesn't allow backtracking and planning this error rate kills autonomy. Recently, GPT-4-Vision was found to have poor OCR accuracy. Smart but stupid, same story.
- SideburnsOfDoom 3y agoAnd a demon summoning is one of the worst imaginable places to have dumb mistakes in the instructions and incantations.
- szundi 3y agoOh no, more demons summoned! AI ruins the world.
- iamnotsure 3y agoLatin demonology manuals were about GPT-T and Calude in the first place.
- Animats 3y agoComing soon, layoffs in medieval history departments.
- padraic7a 3y agoWhy? You think scholars are mostly translating?
- vintermann 3y agoIf only. There are far too many documents to read, transcribe and translate compared to the number of people paid to do that sort of thing!
- klntsky 3y agoEveryone is excited about LLM abilities to help with language learning, while completely ignoring the fact that for most people LLMs will make the learning unneeded. There will be less experts in the field, and therefore we will lose the part of language and foreign literature understanding not captured by statmodels. Which is a huge part (subtle contexts in poetry, etc)
- Drakim 3y agoYou aren't wrong, but this has been a dilemma with every new technology. The camera had that effect, modern metalworking had that effect, even tractors had that effect. It's definitely a problem we should be talking about, but we can't go back in time or remain frozen, the genie never goes back in the bottle. We have to move forward towards the future while salvaging the parts of the past we want to bring with us.
- HPsquared 3y agoThis line of thought goes back to Socrates and his supposed views on writing. (That it weakens the memory)
- PartiallyTyped 3y agoThe major difference is that in the event that there's a catastrophic event, we won't be able to build the tech we need because the intermediate steps will be lost. It is also how we know there hasn't been a civilization more advanced than us that left without a trace. All the oil that was easy to extract has been extracted.
- sebzim4500 3y agoI don't think that follows, it is easy to imagine an alternative history without all the hissyfits about nuclear fission or with real investment into nuclear fusion starting in the 70s.
- 3y ago
- hoseja 3y agoWhat could possibly go wrong.
- lofaszvanitt 3y agoI always wondered why these texts are so difficult to interpret... ... ... why certain symbols, like a crow have ambivalent meanings. In some cultures the crow is evil, while in others, it's benevolent. GPT4 to the rescue, let's see what'll happen if everyone has the means to summon demons, curse others and the like.
- tiborsaas 3y agoI envisioned future hacking will be like whispering magic poems and spells aka. prompts to AI systems. I know about prompt injection, however this would raise things to a new level :)
- bluelightning2k 3y agoI remember the moment I first saw ChatGPT. "Finally I can translate all my Latin demonology manuals"
- ralfd 3y agoI lost three hours reading through the rest of this wonderful blog.
- emeth 3y agoI've used ChatGPT 3.5 (not 4, too expensive) to translate most of the Latin writings of Jerome, Ambrose, and Ambrosiaster (from Migne's Patrologia Latina) - the translations have been put in this repo in the public domain: https://github.com/HistoricalChristianFaith/Writings-Database https://github.com/HistoricalChristianFaith/Writings-Databas... Some takeaways: - ChatGPT did excellent with about 3 sentences max at a time. Exceeding 3 sentences would cause it to often truncate the response (e.g. translating 3ish of 5 sentences, or hallucinating more). - ChatGPT would originally return the translation, sometimes randomly prefixed with a variant of "The translation is" and sometimes wrapped in quotes, othertimes not. Using the function interface to ChatGPT eliminated this problem. - When it comes to quotations from Bible verses, ChatGPT sometimes "embellished" (not sure what else to call it). E.g. if part of Ephesians 2:7 is quoted in Latin, in the English ChatGPT would sometimes insert Ephesians 2:7-8 in full.
- idopmstuff 3y agoI don't really understand what the value in posting these kinds of takeaways about using GPT3.5 here is. GPT4 is significantly better, and improved models are coming. There's just not a lot of point to benchmarking 3.5 when likely every issue you've pointed out is solved by 4.
- emeth 3y agoAverage cost to translate an entire work in that repo via GPT3.5 = ~$2 Average cost to translate an entire work in that repo via GPT4 = ~$40 For a side-project, one is feasible to scale, and one ain't.
- telotortium 3y agoCould also try Claude-2, as the OP did.
- keskival 3y agoIt makes a few mistakes still, and that makes it "just a tool"? Ok, how about in a few weeks when it stops making mistakes?
- bayindirh 3y agoCompilers, calculators, hammers, spreadsheet software, watches don’t make mistakes either, and they are “just tools”.
- smegger001 3y agoSure give the AI barely constrained by alignment prompts that flip off when given the wrong prompt arcane rites to summon demons that will end well...