13 ms·

Can LLMs accurately recall the Bible?

223 points by benkaiser 2y ago

MrQuincle 2y ago
"I've often found myself uneasy when LLMs (Large Language Models) are asked to quote the Bible. While they can provide insightful discussions about faith, their tendency to hallucinate responses raises concerns when dealing with scripture, which we regard as the inspired Word of God." Interesting. In my very religious upbringing I wasn't allowed to read fairy tales. The danger being not able to classify which stories truly happened and which ones didn't. Might be an interesting variant on the Turing test. Can you make the AI believe in your religion? Probably there's a sci-fi book written about it.
- aptsurdist 2y ago
  To be fair, the Bible’s authors also seemed to have hallucinated the word of God. At least in cases of contradictions between authors.
- mmooss 2y ago
  > In my very religious upbringing I wasn't allowed to read fairy tales. The danger being not able to classify which stories truly happened and which ones didn't. Thanks for sharing. You might be interested: JRR Tolkien, 'fairy tale' author, was also a/the leading scholar of Old English (Anglo-Saxon), related languages, and the culture and myth around them - including 'fairy tales'; and he was a devout Catholic. How could he write (and study) such ungodly material? He wrestles with the question multiple times, but if you are interested, I strongly recommend On Fairy-stories, an essay based on a lecture. It covers far more ground than this question, but it's worth reading anyway. I'll append a spoiler below. . . . . . . . . . . . . . . . . . . . . . [SPOILER] There's more to it than this, but it's a wonderful vision: "The Gospels contain a fairy-story, or a story of a larger kind which embraces all the essence of fairy-stories. They contain many marvels - peculiarly artistic,[1] beautiful, and moving: 'mythical' in their perfect, self-contained significance; and among the marvels is the greatest and most complete conceivable eucatastrophe. But this story has entered History and the primary world .... The Birth of Christ is the eucatastrophe of Man's history. The Resurrection is the eucatastrophe of the story of the Incarnation. This story begins and ends in joy. It has pre-eminently the 'inner consistency of reality'. There is no tale ever told that men would rather find was true, and none which so many sceptical men have accepted as true on its own merits. For the Art of it has the supremely convincing tone of Primary Art, that is, of Creation. ..." [1] "The Art is here in the story itself rather than in the telling; for the Author of the story was not the evangelists."
  - graemep 2y ago
    His friend and colleague CS Lewis had similar ideas.
- graemep 2y ago
  The Quest for Saint Aquin is a rather good short story on the topic.
waynecochran 2y ago
I find LLM's good for asking certain kinds of Biblical questions. For example, you can ask it to list the occurrences of some event, or something like "list all the Levitical sacrifices," "what sins required a sin offering in the OT," "Where in the Old Testament is God referred to as 'The Name'?" When asking LLM's to provide actual interpretations you should know that you are on shaky ground.
- nwatson 2y ago
  The LLMs have been fed the critique and analysis and discussion of all manner of biblical passages. The LLMs usually give great interpretations and even contrast different theological takes on such passages.
  - waynecochran 2y ago
    Yeah -- I ask it interpreted questions all the time, but just like for programming, I realize answers that appear good are often just plain wrong. I do know you can ask it leading questions if you want answers with a certain theological bent. e.g. "Did the judgments of Revelation 8 and 9 occur in the first century?"
Animats 2y ago
It's discouraging that an LLM can accurately recall a book. That is, in a sense, overfitting. The LLM is supposed to be much smaller than the training set, having in some sense abstracted the training inputs. Did they try this on obscure bible excerpts, or just ones likely to be well known and quoted elsewhere? Well known quotes would be reinforced by all the copies.
- evertedsphere 2y ago
  > Did they try this on obscure bible excerpts, or just ones likely to be well known and quoted elsewhere? the article contains examples of both
- kenjackson 2y ago
  Does GPT now query in real-time? If so, it should be able to reproduce anything searchable verbatim. It just needs to determine when verbatim quoting is appropriate given the prompt.
  - benkaiser 2y ago
    Some services may overlay this functionality (e.g. Bing), but in the article I'm making direct LLM calls without any external function calling.
- amelius 2y ago
  This is actually a good point. Are reciting and not-overfitting at odds?
- bluGill 2y ago
  The bible is probably in enough different training sets (not just in whole, various papers making some religious argument that quote a few verses to make their point) that the model should have most of the bible.
avree 2y ago
I wonder if the author knows that "slurpees" is misspelled in his bio on the post.
- benkaiser 2y ago
  Hah, I did not. Thanks
evanjrowley 2y ago
Approximately 1 year ago, there was a HN submission[0] for Biblos[1], an LLM trained on bible scriptures. [0] https://news.ycombinator.com/item?id=38040591 https://news.ycombinator.com/item?id=38040591 [1] http://www.biblos.app/ http://www.biblos.app/
- deleted 2y ago
  [deleted]
ComposedPattern 2y ago
[dead]
amelius 2y ago
[flagged]
- hobobaggins 2y ago
  That is a myth. The accuracy of most translations over millenia is frankly unbelievable when compared against other sources and ancient manuscripts such as the Dead Sea scrolls, Josephus, etc. Even comparing LXX/Septuagint and Vulgate are more remarkable for the very few ways in which they diverge, but even those can be harmonized with careful study.
- joshuamcginnis 2y ago
  There are over 5,000 manuscripts of the New Testament that overlap with more than 90% consistency. This makes it one of the most well-preserved and reliably transmitted ancient texts in history.
gwd 2y ago
I'm learning New Testament Greek on my own*, and sometimes I paste a snippet in to Claude Sonnet and ask questions about the language (or occasionally the interpretation); I usually say it's from the New Testament but don't bother with the reference. Probably around half the time, the opening line of the response is, "This verse is <reference>, and...". The reference is almost always accurate. * Using a system I developed myself; currently in open development: https://www.laleolanguage.com https://www.laleolanguage.com
- nickpsecurity 2y ago
  In case it helps, Bill Mounce has one or two classes on Biblical Greek via his free seminary: https://www.biblicaltraining.org/learn/institute/nt201-biblical-greek https://www.biblicaltraining.org/learn/institute/nt201-bibli... (Note: They actually host free classes from instructors at over a dozen seminaries. Mounce himself is a top expert in Greek.) For anyone learning Biblical Hebrew, I found Master's Seminary has some courses on it: https://www.youtube.com/watch?v=Qvh8yziVsCE&list=PL9392DD285C853693 https://www.youtube.com/watch?v=Qvh8yziVsCE&list=PL9392DD285... https://www.youtube.com/watch?v=joDB5azc_CM&list=PL4DC84F8EBBE6A3F9 https://www.youtube.com/watch?v=joDB5azc_CM&list=PL4DC84F8EB...
  - ARandomerDude 2y ago
    Mounce's Basics of Biblical Greek and the workbook were good enough that I stopped watching the lectures. The workbook is excellent. Can't recommend it enough.
  - gwd 2y ago
    So the theory behind Guided Immersion is that you shouldn't need most of that. When Priscilla and Aquilla were learning Greek, nobody sat them down and said, "Now definite articles are inflected according to gender, number, and case: ho, hoi, ..." They were just given example after example, and the language processing unit of their brains figured it out. So Guided Immersion tries to just give you not only vocab, but grammar in such a way that there's always only a handful of concepts you haven't mastered. I developed Guided Immersion to help myself master Mandarin, actually; I used Anki with Mandarin for probably 8 years before developing Guided Immersion; once I switched I never went back. Then about a year and a half ago ago I ported it over to Koine Greek not knowing any Greek, and started using it myself after watching a handful of YouTube Videos introducing the characters and the basic cases. Maybe it's just the way my brain works, but I can't imagine sitting down and trying to memorize all those endings, particularly for the verbs. I have now bought Mounce's "Basics of Biblical Greek Grammar", and "The Morphology of Biblical Greek", to help me refine the "language schema" the algorithm uses. I appreciate the work Mounce has done to find the deeper morphological rules which make sense of what look like "irregular" inflections; teaching the algorithm about those will certainly help it to present things in a more useful way to learners. But I don't think trying to grind through all that in your conscious mind is the way to go.
dudeinjapan 2y ago
In the beginning was the Vector, and the Vector was with God, and the Vector was God.
eddiewithzato 2y ago
Why then does it have a hard time being a judge for MTG rule interactions?
ChuckMcM 2y ago
Interesting that it takes an LLM with 405 BILLION parameters to accurately recall text from a document with slightly less than 728 THOUSAND words. (not quite three decimal orders of magnitude smaller but still).
- Sabinus 2y ago
  I don't think it's necessarily about the parameter count, but the amount of training material about the Bible relative to the rest of the training material, with higher parameter models able to retain more Bible information with a higher proportion of training on other topics.
  - ChuckMcM 2y ago
    I would be interested to hear your thoughts on what a parameter in an LLM model represents.
- benkaiser 2y ago
  I guess the challenge is that the parameters have to encode much more intricacy than just the Bible. Even if one were produced purely with text from the bible, it would likely not be able to converse as well as conversationally tuned ones behave. Perhaps there's a middle ground of a fine-tuned LLM on scripture recall + o1-style background reasoning to produce the best output. Or even just a RAG.
  - ChuckMcM 2y ago
    Okay, checking in here, so you see a 'parameter' as being an encoding? Is that like a unique encoding ? Like a bit pattern or more like a DNA pattern?
    - benkaiser 2y ago
      I'm talking about it mostly from an entropy standpoint here. With larger size, you can represent more information. Think of it like how you have a JPEG, you can compress it further and further, but eventually you lose the ability to understand the original image. With the models, if they had infinite size, I imagine they could recall values in their training data extremely accurately. But as you compress down further and further to smaller and smaller models, you are trying to distil the same amount of information in less space, and so things cannot be perfectly recalled. We have people tuning these smaller models to squeeze every ounce of what we consider meaningful out of them (passing certain benchmarks, seeming coherent in dialogue, etc), but in the process the things we don't tune them for, i.e. accurate recall of scripture (not that we should) they lose that ability. I will say with all of that though, that I only have a high level understanding of LLMs, I've integrated them into products on the job, but I am by no means an ML engineer.
jsenn 2y ago
Has there been any serious study of exactly how LLMs store and retrieve memorized sequences? There are so many interesting basic questions here. Does verbatim completion of a bible passage look different from generation of a novel sequence in interesting ways? How many sequences of this length do they memorize? Do the memorized ones roughly correspond to things humans would find important enough to memorize, or do LLMs memorize just as much SEO garbage as they do bible passages?
- nwatson 2y ago
  I imagine Bible passages, at least the more widely quoted and discussed ones, appear many, many times in the various available translations, in inspirational, devotional, scholarly articles, in sermon transcripts, etc. This surely reinforces almost word-for-word recall. SEO garage is a bit different each time, so common SEO-reinforced themes might be recalled in LLM output, but not word for word.
- suprjami 2y ago
  LLMs do not store and retrieve sequences. LLMs are not databases. LLMs are not predictable state machines. Understand how these things work. They take the input context and generate the next token, then feed that whole thing back in as context and predict the next token, and repeat until the most likely next token is their stop word. If they produce anything like a retrieved sequence, that's because they just happened to pick that set of tokens based on their training data. Regenerating the output from exactly the same input has a non-zero chance of generating different output.
  - Sharlin 2y ago
    It should have a zero chance of generating different output if the temperature is set to zero as in TFA. LLMs are not stochastic algorithms unless you add entropy yourself. Of course most people just use ChatGPT with its default settings and know nothing about the specifics. The point is, though – somehow the model has memorized these passages, in a way that allows reliable reproduction. No doubt in a super amorphous and diffuse way, as minute adjustments to the nth sigbits of myriads of floating-point numbers, but it cannot be denied that it absolutely has encoded the strings in some manner. Or otherwise you have to accept that humans can't memorize things either. Indeed given how much our memory works by association, and how it's considerably more difficult to recount some memorized sequence from an arbitrary starting point, it's easy to argue that in some relevant way human brains are next-token predictors too.
graemep 2y ago
The Bible is a very tricky thing to recall word for work because of differences between canons and translations. Different wording might be taken from a different translation than the one asked for, rather than being wrong.
- benkaiser 2y ago
  Hence why I marked those cases with warnings, indicating they were accurate if you give leniency for different translations.
  - graemep 2y ago
    I was looking mostly at the results file, and forgot what you ada distinguished between them in the article. That said, what I was getting at is that its a tough test. The differences between translations are often very small.
gerdesj 2y ago
Why? Why do you put a weird computer model between you and a computer and errr Your Faith? Do bear in mind that hallucinations might correspond to something demonic (just saying) I'm a bit of a rubbish Christian but I know a synoptic gospel when I see it and can quote quite a lot of scripture. I am also an IT consultant. What exactly is the point of Faith if you start typing questions into a ... computational model ... and trusting the outputs? Surely you should have a decent handle on the literature: It's just one big physical book these days - The Bible. Two Testaments and a slack handful of books and that for each. I'm not sure exactly but it looks about the same size as the Lord of the Rings. I've just checked: Bible: 600k LotR: 480K - so not too far off. I get that you might want to ask "what if" types of questions about the scriptures but why would you ask a computer? Faith is not embedded in an Intel Core i7 or an Nvidia A100. Faith is Faith. ChatGPT is odd.
- benkaiser 2y ago
  I can speak to a couple of perspectives I have seen other people use it for, ranging from valid to somewhat scary. 1. Preparing a sermon for Church, I don't advocate for this, but it's definitely being done out there. Here, the pastor may know the topic they are speaking on, but want the LLM to help them plan out the message and structure it. 2. Preparing lesson plans for Sunday School. This seems reasonably fine to me, but I would still err on the side of not trusting the raw scriptures output as evidence, and instead look them up separately before reading them out. The above examples may particularly come into play when English is not a first language, since although they can understand and express their faith easily in their native language, ChatGPT can help them represent it in English well. Personally, I think the use cases are many, but mostly for discussion / personal reflection. These include things like asking for perspectives that other Christians take on certain passages, helping understand how some scriptures link to other scriptures in the Bible, and sometimes even exploring some of the history of the Christian faith through the last ~2 millennia since it was written. Anything meaningful you can manually research further / reference before taking it at face value, but it can work as a great starting point for your search.
- chaosharmonic 2y ago
  I had a similar reaction myself -- I'm an escaped fundamentalist and don't personally have the same convictions about stuff like this, but even if it's on a level that amuses me a little, there's something that feels just a bit heretical about it... Not necessarily in a way where I would judge it though, and I certainly see how that could have use cases. It just feels a little bit like water gun baptisms, conceptually. One question of a less spiritual nature -- are we strictly talking about recall from within the models themselves? I've never gotten deep enough into this kind of thing to mess with RAG pipelines, but I wonder if direct access to a translation or several would have any impact on its overall effectiveness for this.
cowmix 2y ago
When I test new LLMs (whether SaaS or local), I have them create a fake post to r/AmItheAsshole from the POV of the older brother in the parable of the Prodigal Son. It's a great, fun test.
- reynaldi 2y ago
  I’m interested to know if you let the LLMs automatically make posts and interact on their own, or do you only use the response and submit manually?
  - drdeca 2y ago
    I didn’t get the impression that the GP actually posted them to r/aita , just asked it to write something in the style of such posts
    - cowmix 2y ago
      This is correct. I just submit the prompt and 'rate it' in my head. I wish there was a 'historical' or fiction version of that subreddit though.
pwinkeler 2y ago
I love that people are finally comfortable adding the word "artificial" into their analysis of the bible. About time. Because make no mistake, LLMs are at best artificial intelligence. More likely, they are very good regurgitating machines, telling us what we have been telling ourselves in an even better form thus goading us along in our fallacies.
danpalmer 2y ago
LLMs are bad databases, so for something like a bible which is so easily and precisely referenced, why not just... look it up? This is playing against their strengths. By all means ask them for a summary, or some analysis, or textual comparison, but please, please stop treating LLMs as databases.
- madiator 2y ago
  Not sure why you are so upset about a small and neat study ("please, please stop"). If you ask it to summarize (without feeding the entire bible), it needs to know the bible. Knowledge and reasoning are not entirely disconnected.
  - nwatson 2y ago
    ChatGPT chat interface has impressed me when going beyond the scope presented on TFA, eg, when asking about predestination, biblical passages for and against, theologians' and scholars' takes on the debate, and exploring the details in subsequent follow-ups. The LLMs have been fed the Bible and all manner of discussions of Bible-related matters. Like the grandparent comment suggests, the LLMs are much more impressive at interpreting biblical passages and presenting the varieties of opinions about them, or finding passages related to specific topics and presenting opinions.
  - danpalmer 2y ago
    > Not sure why you are so upset about a small and neat study This article is yet another example of someone misunderstanding what an LLM is at a fundamental level. We are all collectively doing a bad job at explaining what LLMs are, and it's causing issues. Only recently I was talking to someone who loves ChatGPT because it "takes into account everything I discuss with it", only, it doesn't. They think that it does because it's close-ish, but it's literally not at all doing a thing that they are relying upon it to do for their work. > If you ask it to summarize (without feeding the entire bible), it needs to know the bible. There's a difference between "knowing" the bible and its many translations/interpretations, and being able to reproduce them word for word. I would imagine most biblical scholars can produce better discourse on the bible than ChatGPT, but that few if any could reproduce exact verbatim content. I'm not arguing that testing ChatGPT's knowledge of the bible isn't valuable, I'm arguing that LLMs are the wrong tool for the job for verbatim reproduction, and testing that (and ignoring the actual knowledge) is a bad test, in the same way that asking students to regurgitate content verbatim is much less effective as a method of testing understanding than testing their ability to use that understanding.
deleted 2y ago
[deleted]
jccalhoun 2y ago
It is fun and frustrating to see what LLMs can and can't do. Last week I was trying to find the name of a movie so I typed a description of a scene in chatgpt and said "I think it was from late 70s or early 80s and even though it is set in the USA, I'm pretty sure it is European" and it correctly told me it was the House by the Cemetery. Then last night I saw a video about the Parker Solar Probe and how at 350,000mph it was the fastest moving man-made object. So I asked chatgpt how long at that speed it would take it to get to Alpha Centauri which is 4.37 light years away. It said it would take 59.8 million years. I knew that was way too long so I had it convert mph to miles per year and then it was able to give me the correct answer of 6817 years.
- QuantumG 2y ago
  Whereas you would previously (for your first example) have a conversation with the guy at the video store and he'd not only tell you the movie but also recommend something else you might like.
  - sadeshmukh 2y ago
    So instead, you'd drive there, to someone who also probably doesn't know, talk (while they might also not want to), and then, you might get a recommendation. You can also ask ChatGPT for recommendations. This isn't a case where I would return to pre-LLM times.
    - QuantumG 2y ago
      One day you'll understand how valuable it was to have a person with knowledge and their own thoughts.
      - sadeshmukh 2y ago
        Isn't that called socializing? And there's never been more people out there interested in whatever you're interested in.
- zeroonetwothree 2y ago
efitz 2y ago
Interesting result but probably predictable since you’re trying to use the LLM as a database. But I think you’re onto something in that your experiments can provide data to inform (and hopefully dissuade) creation of applications that similarly try to use LLMs for exact lookups. I think the experiment of using the LLM to recall described verses - eg “what’s the verse where Jesus did X”- is a much more interesting use. I think also that the LLM could be handy as, or to construct, a concordance. But I’d just use a document or database if I wanted to look up specific verses.
cbg0 2y ago
While this is slightly more catered towards a technical audience, I think articles on relatable subjects like this one could prove valuable in getting non-technical people to understand the limitations of LLMs, or what companies are calling "AI" these days. A version of this article that is more focused on real-world examples, showing exactly how the models can make mistakes and present the wrong or incomplete information with less technical focus would probably better cater to a non-technical audience.
JimmyWilliams1 2y ago
[dead]
killermouse0 2y ago
I believe I saw or read somewhere that, in the case of the brain, memories were not as much stored as they were reconstructed when recalled. If that's true, I feel like we are witnessing something similar with LLMs as well as with stable diffusion type of things. Is there any studies looking into this in the AI world? Also if anyone knows what I'm referring to (i.e "reconstructing memories") I would love some pointers because I can't remember for the love of me where I heard or read of this idea!
- tessellated 2y ago
  I understand the down vote, but it was an interesting prompt to test different models with.
seanhunter 2y ago
By Betteridge's law of headlines, the answer is clearly "no".[1] But also, LLM's in general build a lossy compression of their training data so are not the right tool if you want a completely accurate recall. Will the recall be accurate enough for a particular task? Well I'm not a religious person so I have no framework to help decide that question in the context of the bible. If you want a system to answer scripture questions I would expect a far better approach than just an LLM would be to build a RAG system and train the RAG embedding and search at the same time you train the model. [1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
weMadeThat 2y ago
they totally can. I got exiled into an isolated copy of an AI-populated internet once and they put perfectly accurate bible quotes into dictionaries!
nickpsecurity 2y ago
I tested this back when GPT4 was new. I found ChatGPT could quote the verses well. If I asked it to summarize something, it would sometimes hallucinate stuff that had nothing to do with what was in the text. If I prompted it carefully, it could do a proper exegesis of many passages using the historical-grammatical method. I believe this happens because the verses and verse-specific commentary are abundant in the pre-training sources they used. Whereas, if one asks a highly-interpretive question, then it starts re-hashing other patterns in its training data which are un-Biblical. Asking about intelligent design, it got super hostile trying to beat me into submission to its materialistic worldview every paragraph. So, they have their uses. I’ve often pushed for a large model trained on Project Gutenberg to have a 100% legal model for research and personal use. A side benefit of such a scheme would be that Gutenberg has both Bibles and good commentaries which trainers could repeat for memorization. One could add licensed, Christian works on a variety of topics to a derived model to make a Christian assistant AI.
- hobobaggins 2y ago
  Do you have an X or other social acct to get in touch?
  - nickpsecurity 2y ago
    Email me: digitalkevlar@gmail.com
    - hobobaggins 2y ago
      Thanks I will!
      - nickpsecurity 2y ago
        One of you who just emailed me can't receive my reply due to a configuration error. I think it was you. Gmail says: "response from the remote server was '554 5.7.1: relay access denied." If it was you, just email me back when you think that's fixed and I'll re-send hte reply. :)
kindeyoowee 2y ago
[flagged]
- kfarr 2y ago
  This being saturday night, imagine how many pastor sermons for tomorrow are being written with LLM help right at this very moment...
szvsw 2y ago
It seems like LLMs would be a fun way to study/manufacture syncretism, notions of the oracular, etc; turn up the temperature, and let godhead appear! If there’s some platonic notion of divinity or immanence that all faith is just a downward projection from, it seems like its statistical representation in tokenized embedding vectors is about as close as you could get to understanding it holistically across theological boundaries. All kidding aside, whether you are looking at Markov chain n-gram babble or high temperature LLM inference, the strange things that emerge are a wonderful form of glossolalia in my opinion that speak to some strange essence embedded in the collective space created by the sum of their corpi text. The Delphic oracle is real, and you can subscribe for a low fee of $20/month!
- JoshuaDavid 2y ago
  https://www.tumblr.com/kingjamesprogramming https://www.tumblr.com/kingjamesprogramming
  - Krasnol 2y ago
    > The resulting interpreter will run very slowly because of the truth. It is so deep.
- patcon 2y ago
  This is what strikes me about the peter todd phenomenon -- that there are hidden glitch tokens within the LLM that seem to conjure some representation of pure hell, and some representation of pure good. https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petertodd-phenomenon https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...
- timewizard 2y ago
  [dead]
- Trasmatta 2y ago
  > the strange things that emerge are a wonderful form of glossolalia in my opinion that speak to some strange essence embedded in the collective space created by the sum of their corpi text. The Delphic oracle is real, and you can subscribe for a low fee of $20/month! I've had some surprisingly insightful tarot readings with the assistance of ChatGPT and Claude. I use tarot for introspection rather than divination, and it turns out LLMs are extremely good at providing a sounding board to mirror and understand those insights.
michaelsbradley 2y ago
I’ve been pretty impressed with ChatGPT’s promising capabilities as a research assistant/springboard for complex inquiries into the Bible and patristics. Just one example: Can you provide short excerpts from works in Latin and Greek written between 600 and 1300 that demonstrate the evolution over those centuries specifically of literary references to Jesus' miracle of the loaves and fishes? https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df78325 https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df783...
- edflsafoiewq 2y ago
  How certain are you that's correct? IME these "search problems" are the kind of thing almost always provokes hallucinations. For example, I looked up the quotation provided from Isidore of Seville's De fide catholica contra Iudaeos, Lib. II, cap. 19, using this copy on WikiSource, https://la.wikisource.org/wiki/De_fide_catholica_contra_Iudaeos https://la.wikisource.org/wiki/De_fide_catholica_contra_Iuda.... The quote certainly does not appear under LIBER SECUNDUS, CAPUT XIX. Nor could I find it in whole or in fragment anywhere in the document, nor indeed any mention of the miracle of loaves and fishes (granted, I could have missed one, I relied on Ctrl+F and my very rusty Latin). Perhaps the copy on WikiSource is incomplete, or perhaps there are differing manuscripts, but perhaps also the quote was a complete hallucination to begin with.
  - FearNotDaniel 2y ago
    Exactly - it’s the same problem when using (current) LLMs for major programming tasks, generally useless if you don’t already have enough knowledge of the language/platform to spot and correct the mistakes, plus enough awareness of software design and architecture to recognise what is going to be secure, performant and maintainable in the long run.
- FearNotDaniel 2y ago
  I am by no means a professional in this area, but as a keen amateur I would worry about my inability to discern facts from hallucinations in such a scenario: while I could imagine such output provides a useful “springboard” set of references for someone already skilled in the right area, without being able to look up the original texts myself and make sense of the Latin/Greek I would not feel confident that such texts even really exist, let alone if they contain the actual words the LLM claims and if the translations are any good. And that’s before you get into questions of the “status” of any given work (was it considered accurate or apocryphal at the time of writing, for which audience was it intended using what kind of literary devices, what if any is the modern scholarly consensus on the value, truth or legitimacy of the text etc etc)
orionblastar 2y ago
There is this robot that reads the Bible: https://futurism.com/religious-robots-scripture-nursing-homes https://futurism.com/religious-robots-scripture-nursing-home...
- GrumpyNl 2y ago
  Isnt that just a modified mp3 player, according to the text they only just recite it.
  - orionblastar 2y ago
    It is supposed to be an AI robot that explains the Bible and takes questions and answers them with Bible verses.
asim 2y ago
I had similar thoughts about using it for the Quran. I think this highlights you have to be very specific in your use cases especially when expecting an exact response on static text that shouldn't change. This is why I'm trying something a bit different. I've generated embeddings for the Quran and use chromem-go for this. So I'll ask the index the question first based on a similarity search and then feed the results in as context to an LLM. But in the response I'll still sight the references so I can see what they were. It's not perfect but a first step towards something. I think they call this RAG. What I'm working on https://reminder.dev https://reminder.dev
- tokinonagare 2y ago
  [flagged]
  - danial 2y ago
    1. 4:144 This verse advises Muslims to prioritize loyalty within the community during a time of external threats. It is not a general prohibition but a caution in the context of potential betrayal. 2. 77:16 This refers to historical examples of past communities who faced consequences for rejecting divine guidance. It is a reminder of accountability, not a universal statement against non-believers. 3. 8:15 This verse gives instructions for battle, emphasizing courage and discipline during wartime. It applies to specific combat situations, not everyday relations with non-believers. 4. 5:41 This verse addresses the Prophet’s grief over those who rejected faith and distorted divine teachings. It critiques dishonesty and insincerity, not all members of specific groups. 5. 3:141 This verse speaks about trials that distinguish true believers and cleanse the community of wrongdoing. It emphasizes spiritual growth, not indiscriminate judgment of disbelievers. You seem to be making an accusation that Muslims widely practice "taqiyya" to deceive others. This is a baseless and Islamophobic trope. In mainstream Islam, lying is unequivocally condemned and considered an act of hypocrisy. While there is a narrow and rare historical exception permitting concealment of faith to protect one’s life under extreme duress, most Muslims have never encountered or practiced this concept. Ironically, those spreading this accusation often seem to know more about it than the Muslim communities they malign.
kittikitti 2y ago
I tried something similar with my favorite artist, Ariana Grande. Unfortunately, not even the most advanced AI could beat my knowledge of her lyrical work.
- egeozcan 2y ago
  As someone who usually listens to Anatolian Rock, even I know some Grande lyrics: > One taught me love, one taught me patience and one taught me pain. https://knowyourmeme.com/memes/thank-u-next https://knowyourmeme.com/memes/thank-u-next BTW, did someone already code an automated meme generator using LLMs? Only half-joking.
  - hackernewds 2y ago
    a reaction to your comment https://knowyourmeme.com/memes/neil-degrasse-tyson-reaction https://knowyourmeme.com/memes/neil-degrasse-tyson-reaction
asimpleusecase 2y ago
This is nice work. The safest approach is using the look up - which his data shows to be very good - and combine that with a database of verses. That way textual accuracy can be retained and very useful lookup be carried out by LLM. This same approach can be used for other texts where accurate rendering of the text is critical. For example say you built a tool to cite federal regulations in an app. The text is public domain and likely in the training data of large LLMs but in most use cases hallucinating the text of a fed regulation could expose the user to significant liability. Better to have that canonical text in a database to insure accuracy.
sneak 2y ago
> While they can provide insightful discussions about faith, their tendency to hallucinate responses raises concerns when dealing with scripture I experience the exact same problem with human beings. > , which we regard as the inspired Word of God. QED
ddtaylor 2y ago
I'm heavily biased here because I don't find much value in the bible personally. Some of the stories are interesting and some interpretations seem useful, but as a whole I find it arbitrary. I never tell other people what to believe or how they should do that in any capacity. With that said I find the hallucination component here fascinating. From my perspective everyone who interprets various religious text does so differently and usually that involves varying levels of fabrication or something that looks a lot like it. I'm speaking about the "talking in tongues" and other methods here. I'm not trying to lump all religions into the same bag here, but I have seen that a lot have different ways of "receiving" communication or directive. To me this seems pretty consistent with the colloquial idea of a hallucination.
- 1123581321 2y ago
  Most study and application tries to either source or fully work out from principles the meaning of the Bible. These can be wrong arguments but wouldn’t be hallucinations. Your experience sounds limited to Pentecostal-originated churches, which are 100-150 years old. In those churches, it’s acceptable to speak as if you’ve received a spontaneous understanding of the Bible and to not explain it. That does have a parallel to LLM hallucinations in face value output, I suppose, but the origination is completely different as the spontaneous human is making planned remarks passed off as spontaneous, trying to affect specific people in the room, or emotionally overwhelmed. None of those resemble why/how LLMs hallucinate.
  - o11c 2y ago
    As quick as I am to criticize the bizarre versions of Christianity, I do think you're in error to assume Pentecostalism is all or even mostly about "planned remarks passed off as spontaneous". Improv is a thing, and can be trained as a skill even outside of comedy/entertainment. Though, outside of Charismatic sects, Christianity does see a more reasonable level of "I had prepared by thinking about (verse X), but suddenly now I'm thinking about (obscure verse Y)."
    - User23 2y ago
ks2048 2y ago
This is interesting. I'm curious about how much (and what) these LLMs memorize verbatim. Does anyone know any more thorough papers on this topic? For example, this could be tested on every verse in bible and lots of other text that is certainly in the training data: books in project gutenberg, wikipedia articles, etc. Edit: this (and its references) looks like a good place to start: https://arxiv.org/abs/2407.17817v1 https://arxiv.org/abs/2407.17817v1
- int_19h 2y ago
  For one anecdotal data point, GPT-4 knows the "navy SEAL copypasta" verbatim. It can reproduce it complete with all the original typos and misspellings, and it can recognize it from the first sentence.
- deleted 2y ago
  [deleted]

223 points by benkaiser 2y ago

"I've often found myself uneasy when LLMs (Large Language Models) are asked to quote the Bible. While they can provide insightful discussions about faith, their tendency to hallucinate responses raises concerns when dealing with scripture, which we regard as the inspired Word of God." Interesting. In my very religious upbringing I wasn't allowed to read fairy tales. The danger being not able to classify which stories truly happened and which ones didn't. Might be an interesting variant on the Turing test. Can you make the AI believe in your religion? Probably there's a sci-fi book written about it.

aptsurdist 2y ago
To be fair, the Bible’s authors also seemed to have hallucinated the word of God. At least in cases of contradictions between authors.
mmooss 2y ago
> In my very religious upbringing I wasn't allowed to read fairy tales. The danger being not able to classify which stories truly happened and which ones didn't. Thanks for sharing. You might be interested: JRR Tolkien, 'fairy tale' author, was also a/the leading scholar of Old English (Anglo-Saxon), related languages, and the culture and myth around them - including 'fairy tales'; and he was a devout Catholic. How could he write (and study) such ungodly material? He wrestles with the question multiple times, but if you are interested, I strongly recommend On Fairy-stories, an essay based on a lecture. It covers far more ground than this question, but it's worth reading anyway. I'll append a spoiler below. . . . . . . . . . . . . . . . . . . . . . [SPOILER] There's more to it than this, but it's a wonderful vision: "The Gospels contain a fairy-story, or a story of a larger kind which embraces all the essence of fairy-stories. They contain many marvels - peculiarly artistic,[1] beautiful, and moving: 'mythical' in their perfect, self-contained significance; and among the marvels is the greatest and most complete conceivable eucatastrophe. But this story has entered History and the primary world .... The Birth of Christ is the eucatastrophe of Man's history. The Resurrection is the eucatastrophe of the story of the Incarnation. This story begins and ends in joy. It has pre-eminently the 'inner consistency of reality'. There is no tale ever told that men would rather find was true, and none which so many sceptical men have accepted as true on its own merits. For the Art of it has the supremely convincing tone of Primary Art, that is, of Creation. ..." [1] "The Art is here in the story itself rather than in the telling; for the Author of the story was not the evangelists."
- graemep 2y ago
  His friend and colleague CS Lewis had similar ideas.
graemep 2y ago
The Quest for Saint Aquin is a rather good short story on the topic.

waynecochran 2y ago

I find LLM's good for asking certain kinds of Biblical questions. For example, you can ask it to list the occurrences of some event, or something like "list all the Levitical sacrifices," "what sins required a sin offering in the OT," "Where in the Old Testament is God referred to as 'The Name'?" When asking LLM's to provide actual interpretations you should know that you are on shaky ground.

nwatson 2y ago
The LLMs have been fed the critique and analysis and discussion of all manner of biblical passages. The LLMs usually give great interpretations and even contrast different theological takes on such passages.
- waynecochran 2y ago
  Yeah -- I ask it interpreted questions all the time, but just like for programming, I realize answers that appear good are often just plain wrong. I do know you can ask it leading questions if you want answers with a certain theological bent. e.g. "Did the judgments of Revelation 8 and 9 occur in the first century?"

Animats 2y ago

It's discouraging that an LLM can accurately recall a book. That is, in a sense, overfitting. The LLM is supposed to be much smaller than the training set, having in some sense abstracted the training inputs. Did they try this on obscure bible excerpts, or just ones likely to be well known and quoted elsewhere? Well known quotes would be reinforced by all the copies.

evertedsphere 2y ago
> Did they try this on obscure bible excerpts, or just ones likely to be well known and quoted elsewhere? the article contains examples of both
kenjackson 2y ago
Does GPT now query in real-time? If so, it should be able to reproduce anything searchable verbatim. It just needs to determine when verbatim quoting is appropriate given the prompt.
- benkaiser 2y ago
  Some services may overlay this functionality (e.g. Bing), but in the article I'm making direct LLM calls without any external function calling.
amelius 2y ago
This is actually a good point. Are reciting and not-overfitting at odds?
bluGill 2y ago
The bible is probably in enough different training sets (not just in whole, various papers making some religious argument that quote a few verses to make their point) that the model should have most of the bible.

avree 2y ago

I wonder if the author knows that "slurpees" is misspelled in his bio on the post.

benkaiser 2y ago
Hah, I did not. Thanks

evanjrowley 2y ago

Approximately 1 year ago, there was a HN submission[0] for Biblos[1], an LLM trained on bible scriptures. [0] https://news.ycombinator.com/item?id=38040591 https://news.ycombinator.com/item?id=38040591 [1] http://www.biblos.app/ http://www.biblos.app/

deleted 2y ago
[deleted]

ComposedPattern 2y ago

[dead]

amelius 2y ago

[flagged]

hobobaggins 2y ago
That is a myth. The accuracy of most translations over millenia is frankly unbelievable when compared against other sources and ancient manuscripts such as the Dead Sea scrolls, Josephus, etc. Even comparing LXX/Septuagint and Vulgate are more remarkable for the very few ways in which they diverge, but even those can be harmonized with careful study.
joshuamcginnis 2y ago
There are over 5,000 manuscripts of the New Testament that overlap with more than 90% consistency. This makes it one of the most well-preserved and reliably transmitted ancient texts in history.

gwd 2y ago

I'm learning New Testament Greek on my own*, and sometimes I paste a snippet in to Claude Sonnet and ask questions about the language (or occasionally the interpretation); I usually say it's from the New Testament but don't bother with the reference. Probably around half the time, the opening line of the response is, "This verse is <reference>, and...". The reference is almost always accurate. * Using a system I developed myself; currently in open development: https://www.laleolanguage.com https://www.laleolanguage.com

nickpsecurity 2y ago
In case it helps, Bill Mounce has one or two classes on Biblical Greek via his free seminary: https://www.biblicaltraining.org/learn/institute/nt201-biblical-greek https://www.biblicaltraining.org/learn/institute/nt201-bibli... (Note: They actually host free classes from instructors at over a dozen seminaries. Mounce himself is a top expert in Greek.) For anyone learning Biblical Hebrew, I found Master's Seminary has some courses on it: https://www.youtube.com/watch?v=Qvh8yziVsCE&list=PL9392DD285C853693 https://www.youtube.com/watch?v=Qvh8yziVsCE&list=PL9392DD285... https://www.youtube.com/watch?v=joDB5azc_CM&list=PL4DC84F8EBBE6A3F9 https://www.youtube.com/watch?v=joDB5azc_CM&list=PL4DC84F8EB...
- ARandomerDude 2y ago
  Mounce's Basics of Biblical Greek and the workbook were good enough that I stopped watching the lectures. The workbook is excellent. Can't recommend it enough.
- gwd 2y ago
  So the theory behind Guided Immersion is that you shouldn't need most of that. When Priscilla and Aquilla were learning Greek, nobody sat them down and said, "Now definite articles are inflected according to gender, number, and case: ho, hoi, ..." They were just given example after example, and the language processing unit of their brains figured it out. So Guided Immersion tries to just give you not only vocab, but grammar in such a way that there's always only a handful of concepts you haven't mastered. I developed Guided Immersion to help myself master Mandarin, actually; I used Anki with Mandarin for probably 8 years before developing Guided Immersion; once I switched I never went back. Then about a year and a half ago ago I ported it over to Koine Greek not knowing any Greek, and started using it myself after watching a handful of YouTube Videos introducing the characters and the basic cases. Maybe it's just the way my brain works, but I can't imagine sitting down and trying to memorize all those endings, particularly for the verbs. I have now bought Mounce's "Basics of Biblical Greek Grammar", and "The Morphology of Biblical Greek", to help me refine the "language schema" the algorithm uses. I appreciate the work Mounce has done to find the deeper morphological rules which make sense of what look like "irregular" inflections; teaching the algorithm about those will certainly help it to present things in a more useful way to learners. But I don't think trying to grind through all that in your conscious mind is the way to go.

dudeinjapan 2y ago

In the beginning was the Vector, and the Vector was with God, and the Vector was God.

eddiewithzato 2y ago

Why then does it have a hard time being a judge for MTG rule interactions?

ChuckMcM 2y ago

Interesting that it takes an LLM with 405 BILLION parameters to accurately recall text from a document with slightly less than 728 THOUSAND words. (not quite three decimal orders of magnitude smaller but still).

Sabinus 2y ago
I don't think it's necessarily about the parameter count, but the amount of training material about the Bible relative to the rest of the training material, with higher parameter models able to retain more Bible information with a higher proportion of training on other topics.
- ChuckMcM 2y ago
  I would be interested to hear your thoughts on what a parameter in an LLM model represents.
benkaiser 2y ago
I guess the challenge is that the parameters have to encode much more intricacy than just the Bible. Even if one were produced purely with text from the bible, it would likely not be able to converse as well as conversationally tuned ones behave. Perhaps there's a middle ground of a fine-tuned LLM on scripture recall + o1-style background reasoning to produce the best output. Or even just a RAG.
- ChuckMcM 2y ago
  Okay, checking in here, so you see a 'parameter' as being an encoding? Is that like a unique encoding ? Like a bit pattern or more like a DNA pattern?
  - benkaiser 2y ago
    I'm talking about it mostly from an entropy standpoint here. With larger size, you can represent more information. Think of it like how you have a JPEG, you can compress it further and further, but eventually you lose the ability to understand the original image. With the models, if they had infinite size, I imagine they could recall values in their training data extremely accurately. But as you compress down further and further to smaller and smaller models, you are trying to distil the same amount of information in less space, and so things cannot be perfectly recalled. We have people tuning these smaller models to squeeze every ounce of what we consider meaningful out of them (passing certain benchmarks, seeming coherent in dialogue, etc), but in the process the things we don't tune them for, i.e. accurate recall of scripture (not that we should) they lose that ability. I will say with all of that though, that I only have a high level understanding of LLMs, I've integrated them into products on the job, but I am by no means an ML engineer.

jsenn 2y ago

Has there been any serious study of exactly how LLMs store and retrieve memorized sequences? There are so many interesting basic questions here. Does verbatim completion of a bible passage look different from generation of a novel sequence in interesting ways? How many sequences of this length do they memorize? Do the memorized ones roughly correspond to things humans would find important enough to memorize, or do LLMs memorize just as much SEO garbage as they do bible passages?

nwatson 2y ago
I imagine Bible passages, at least the more widely quoted and discussed ones, appear many, many times in the various available translations, in inspirational, devotional, scholarly articles, in sermon transcripts, etc. This surely reinforces almost word-for-word recall. SEO garage is a bit different each time, so common SEO-reinforced themes might be recalled in LLM output, but not word for word.
suprjami 2y ago
LLMs do not store and retrieve sequences. LLMs are not databases. LLMs are not predictable state machines. Understand how these things work. They take the input context and generate the next token, then feed that whole thing back in as context and predict the next token, and repeat until the most likely next token is their stop word. If they produce anything like a retrieved sequence, that's because they just happened to pick that set of tokens based on their training data. Regenerating the output from exactly the same input has a non-zero chance of generating different output.
- Sharlin 2y ago
  It should have a zero chance of generating different output if the temperature is set to zero as in TFA. LLMs are not stochastic algorithms unless you add entropy yourself. Of course most people just use ChatGPT with its default settings and know nothing about the specifics. The point is, though – somehow the model has memorized these passages, in a way that allows reliable reproduction. No doubt in a super amorphous and diffuse way, as minute adjustments to the nth sigbits of myriads of floating-point numbers, but it cannot be denied that it absolutely has encoded the strings in some manner. Or otherwise you have to accept that humans can't memorize things either. Indeed given how much our memory works by association, and how it's considerably more difficult to recount some memorized sequence from an arbitrary starting point, it's easy to argue that in some relevant way human brains are next-token predictors too.

graemep 2y ago

The Bible is a very tricky thing to recall word for work because of differences between canons and translations. Different wording might be taken from a different translation than the one asked for, rather than being wrong.

benkaiser 2y ago
Hence why I marked those cases with warnings, indicating they were accurate if you give leniency for different translations.
- graemep 2y ago
  I was looking mostly at the results file, and forgot what you ada distinguished between them in the article. That said, what I was getting at is that its a tough test. The differences between translations are often very small.

gerdesj 2y ago

Why? Why do you put a weird computer model between you and a computer and errr Your Faith? Do bear in mind that hallucinations might correspond to something demonic (just saying) I'm a bit of a rubbish Christian but I know a synoptic gospel when I see it and can quote quite a lot of scripture. I am also an IT consultant. What exactly is the point of Faith if you start typing questions into a ... computational model ... and trusting the outputs? Surely you should have a decent handle on the literature: It's just one big physical book these days - The Bible. Two Testaments and a slack handful of books and that for each. I'm not sure exactly but it looks about the same size as the Lord of the Rings. I've just checked: Bible: 600k LotR: 480K - so not too far off. I get that you might want to ask "what if" types of questions about the scriptures but why would you ask a computer? Faith is not embedded in an Intel Core i7 or an Nvidia A100. Faith is Faith. ChatGPT is odd.

benkaiser 2y ago
I can speak to a couple of perspectives I have seen other people use it for, ranging from valid to somewhat scary. 1. Preparing a sermon for Church, I don't advocate for this, but it's definitely being done out there. Here, the pastor may know the topic they are speaking on, but want the LLM to help them plan out the message and structure it. 2. Preparing lesson plans for Sunday School. This seems reasonably fine to me, but I would still err on the side of not trusting the raw scriptures output as evidence, and instead look them up separately before reading them out. The above examples may particularly come into play when English is not a first language, since although they can understand and express their faith easily in their native language, ChatGPT can help them represent it in English well. Personally, I think the use cases are many, but mostly for discussion / personal reflection. These include things like asking for perspectives that other Christians take on certain passages, helping understand how some scriptures link to other scriptures in the Bible, and sometimes even exploring some of the history of the Christian faith through the last ~2 millennia since it was written. Anything meaningful you can manually research further / reference before taking it at face value, but it can work as a great starting point for your search.
chaosharmonic 2y ago
I had a similar reaction myself -- I'm an escaped fundamentalist and don't personally have the same convictions about stuff like this, but even if it's on a level that amuses me a little, there's something that feels just a bit heretical about it... Not necessarily in a way where I would judge it though, and I certainly see how that could have use cases. It just feels a little bit like water gun baptisms, conceptually. One question of a less spiritual nature -- are we strictly talking about recall from within the models themselves? I've never gotten deep enough into this kind of thing to mess with RAG pipelines, but I wonder if direct access to a translation or several would have any impact on its overall effectiveness for this.

cowmix 2y ago

When I test new LLMs (whether SaaS or local), I have them create a fake post to r/AmItheAsshole from the POV of the older brother in the parable of the Prodigal Son. It's a great, fun test.

reynaldi 2y ago
I’m interested to know if you let the LLMs automatically make posts and interact on their own, or do you only use the response and submit manually?
- drdeca 2y ago
  I didn’t get the impression that the GP actually posted them to r/aita , just asked it to write something in the style of such posts
  - cowmix 2y ago
    This is correct. I just submit the prompt and 'rate it' in my head. I wish there was a 'historical' or fiction version of that subreddit though.

pwinkeler 2y ago

I love that people are finally comfortable adding the word "artificial" into their analysis of the bible. About time. Because make no mistake, LLMs are at best artificial intelligence. More likely, they are very good regurgitating machines, telling us what we have been telling ourselves in an even better form thus goading us along in our fallacies.

danpalmer 2y ago

LLMs are bad databases, so for something like a bible which is so easily and precisely referenced, why not just... look it up? This is playing against their strengths. By all means ask them for a summary, or some analysis, or textual comparison, but please, please stop treating LLMs as databases.

madiator 2y ago
Not sure why you are so upset about a small and neat study ("please, please stop"). If you ask it to summarize (without feeding the entire bible), it needs to know the bible. Knowledge and reasoning are not entirely disconnected.
- nwatson 2y ago
  ChatGPT chat interface has impressed me when going beyond the scope presented on TFA, eg, when asking about predestination, biblical passages for and against, theologians' and scholars' takes on the debate, and exploring the details in subsequent follow-ups. The LLMs have been fed the Bible and all manner of discussions of Bible-related matters. Like the grandparent comment suggests, the LLMs are much more impressive at interpreting biblical passages and presenting the varieties of opinions about them, or finding passages related to specific topics and presenting opinions.
- danpalmer 2y ago
  > Not sure why you are so upset about a small and neat study This article is yet another example of someone misunderstanding what an LLM is at a fundamental level. We are all collectively doing a bad job at explaining what LLMs are, and it's causing issues. Only recently I was talking to someone who loves ChatGPT because it "takes into account everything I discuss with it", only, it doesn't. They think that it does because it's close-ish, but it's literally not at all doing a thing that they are relying upon it to do for their work. > If you ask it to summarize (without feeding the entire bible), it needs to know the bible. There's a difference between "knowing" the bible and its many translations/interpretations, and being able to reproduce them word for word. I would imagine most biblical scholars can produce better discourse on the bible than ChatGPT, but that few if any could reproduce exact verbatim content. I'm not arguing that testing ChatGPT's knowledge of the bible isn't valuable, I'm arguing that LLMs are the wrong tool for the job for verbatim reproduction, and testing that (and ignoring the actual knowledge) is a bad test, in the same way that asking students to regurgitate content verbatim is much less effective as a method of testing understanding than testing their ability to use that understanding.

deleted 2y ago

[deleted]

jccalhoun 2y ago

It is fun and frustrating to see what LLMs can and can't do. Last week I was trying to find the name of a movie so I typed a description of a scene in chatgpt and said "I think it was from late 70s or early 80s and even though it is set in the USA, I'm pretty sure it is European" and it correctly told me it was the House by the Cemetery. Then last night I saw a video about the Parker Solar Probe and how at 350,000mph it was the fastest moving man-made object. So I asked chatgpt how long at that speed it would take it to get to Alpha Centauri which is 4.37 light years away. It said it would take 59.8 million years. I knew that was way too long so I had it convert mph to miles per year and then it was able to give me the correct answer of 6817 years.

QuantumG 2y ago
Whereas you would previously (for your first example) have a conversation with the guy at the video store and he'd not only tell you the movie but also recommend something else you might like.
- sadeshmukh 2y ago
  So instead, you'd drive there, to someone who also probably doesn't know, talk (while they might also not want to), and then, you might get a recommendation. You can also ask ChatGPT for recommendations. This isn't a case where I would return to pre-LLM times.
  - QuantumG 2y ago
    One day you'll understand how valuable it was to have a person with knowledge and their own thoughts.
    - sadeshmukh 2y ago
      Isn't that called socializing? And there's never been more people out there interested in whatever you're interested in.
zeroonetwothree 2y ago

efitz 2y ago

Interesting result but probably predictable since you’re trying to use the LLM as a database. But I think you’re onto something in that your experiments can provide data to inform (and hopefully dissuade) creation of applications that similarly try to use LLMs for exact lookups. I think the experiment of using the LLM to recall described verses - eg “what’s the verse where Jesus did X”- is a much more interesting use. I think also that the LLM could be handy as, or to construct, a concordance. But I’d just use a document or database if I wanted to look up specific verses.

cbg0 2y ago

While this is slightly more catered towards a technical audience, I think articles on relatable subjects like this one could prove valuable in getting non-technical people to understand the limitations of LLMs, or what companies are calling "AI" these days. A version of this article that is more focused on real-world examples, showing exactly how the models can make mistakes and present the wrong or incomplete information with less technical focus would probably better cater to a non-technical audience.

JimmyWilliams1 2y ago

[dead]

killermouse0 2y ago

I believe I saw or read somewhere that, in the case of the brain, memories were not as much stored as they were reconstructed when recalled. If that's true, I feel like we are witnessing something similar with LLMs as well as with stable diffusion type of things. Is there any studies looking into this in the AI world? Also if anyone knows what I'm referring to (i.e "reconstructing memories") I would love some pointers because I can't remember for the love of me where I heard or read of this idea!

tessellated 2y ago
I understand the down vote, but it was an interesting prompt to test different models with.

seanhunter 2y ago

By Betteridge's law of headlines, the answer is clearly "no".[1] But also, LLM's in general build a lossy compression of their training data so are not the right tool if you want a completely accurate recall. Will the recall be accurate enough for a particular task? Well I'm not a religious person so I have no framework to help decide that question in the context of the bible. If you want a system to answer scripture questions I would expect a far better approach than just an LLM would be to build a RAG system and train the RAG embedding and search at the same time you train the model. [1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

weMadeThat 2y ago

they totally can. I got exiled into an isolated copy of an AI-populated internet once and they put perfectly accurate bible quotes into dictionaries!

nickpsecurity 2y ago

I tested this back when GPT4 was new. I found ChatGPT could quote the verses well. If I asked it to summarize something, it would sometimes hallucinate stuff that had nothing to do with what was in the text. If I prompted it carefully, it could do a proper exegesis of many passages using the historical-grammatical method. I believe this happens because the verses and verse-specific commentary are abundant in the pre-training sources they used. Whereas, if one asks a highly-interpretive question, then it starts re-hashing other patterns in its training data which are un-Biblical. Asking about intelligent design, it got super hostile trying to beat me into submission to its materialistic worldview every paragraph. So, they have their uses. I’ve often pushed for a large model trained on Project Gutenberg to have a 100% legal model for research and personal use. A side benefit of such a scheme would be that Gutenberg has both Bibles and good commentaries which trainers could repeat for memorization. One could add licensed, Christian works on a variety of topics to a derived model to make a Christian assistant AI.

hobobaggins 2y ago
Do you have an X or other social acct to get in touch?
- nickpsecurity 2y ago
  Email me: digitalkevlar@gmail.com
  - hobobaggins 2y ago
    Thanks I will!
    - nickpsecurity 2y ago
      One of you who just emailed me can't receive my reply due to a configuration error. I think it was you. Gmail says: "response from the remote server was '554 5.7.1: relay access denied." If it was you, just email me back when you think that's fixed and I'll re-send hte reply. :)

kindeyoowee 2y ago

[flagged]

kfarr 2y ago
This being saturday night, imagine how many pastor sermons for tomorrow are being written with LLM help right at this very moment...

szvsw 2y ago

It seems like LLMs would be a fun way to study/manufacture syncretism, notions of the oracular, etc; turn up the temperature, and let godhead appear! If there’s some platonic notion of divinity or immanence that all faith is just a downward projection from, it seems like its statistical representation in tokenized embedding vectors is about as close as you could get to understanding it holistically across theological boundaries. All kidding aside, whether you are looking at Markov chain n-gram babble or high temperature LLM inference, the strange things that emerge are a wonderful form of glossolalia in my opinion that speak to some strange essence embedded in the collective space created by the sum of their corpi text. The Delphic oracle is real, and you can subscribe for a low fee of $20/month!

JoshuaDavid 2y ago
https://www.tumblr.com/kingjamesprogramming https://www.tumblr.com/kingjamesprogramming
- Krasnol 2y ago
  > The resulting interpreter will run very slowly because of the truth. It is so deep.
patcon 2y ago
This is what strikes me about the peter todd phenomenon -- that there are hidden glitch tokens within the LLM that seem to conjure some representation of pure hell, and some representation of pure good. https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petertodd-phenomenon https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...
timewizard 2y ago
[dead]
Trasmatta 2y ago
> the strange things that emerge are a wonderful form of glossolalia in my opinion that speak to some strange essence embedded in the collective space created by the sum of their corpi text. The Delphic oracle is real, and you can subscribe for a low fee of $20/month! I've had some surprisingly insightful tarot readings with the assistance of ChatGPT and Claude. I use tarot for introspection rather than divination, and it turns out LLMs are extremely good at providing a sounding board to mirror and understand those insights.

michaelsbradley 2y ago

I’ve been pretty impressed with ChatGPT’s promising capabilities as a research assistant/springboard for complex inquiries into the Bible and patristics. Just one example: Can you provide short excerpts from works in Latin and Greek written between 600 and 1300 that demonstrate the evolution over those centuries specifically of literary references to Jesus' miracle of the loaves and fishes? https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df78325 https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df783...

edflsafoiewq 2y ago
How certain are you that's correct? IME these "search problems" are the kind of thing almost always provokes hallucinations. For example, I looked up the quotation provided from Isidore of Seville's De fide catholica contra Iudaeos, Lib. II, cap. 19, using this copy on WikiSource, https://la.wikisource.org/wiki/De_fide_catholica_contra_Iudaeos https://la.wikisource.org/wiki/De_fide_catholica_contra_Iuda.... The quote certainly does not appear under LIBER SECUNDUS, CAPUT XIX. Nor could I find it in whole or in fragment anywhere in the document, nor indeed any mention of the miracle of loaves and fishes (granted, I could have missed one, I relied on Ctrl+F and my very rusty Latin). Perhaps the copy on WikiSource is incomplete, or perhaps there are differing manuscripts, but perhaps also the quote was a complete hallucination to begin with.
- FearNotDaniel 2y ago
  Exactly - it’s the same problem when using (current) LLMs for major programming tasks, generally useless if you don’t already have enough knowledge of the language/platform to spot and correct the mistakes, plus enough awareness of software design and architecture to recognise what is going to be secure, performant and maintainable in the long run.
FearNotDaniel 2y ago
I am by no means a professional in this area, but as a keen amateur I would worry about my inability to discern facts from hallucinations in such a scenario: while I could imagine such output provides a useful “springboard” set of references for someone already skilled in the right area, without being able to look up the original texts myself and make sense of the Latin/Greek I would not feel confident that such texts even really exist, let alone if they contain the actual words the LLM claims and if the translations are any good. And that’s before you get into questions of the “status” of any given work (was it considered accurate or apocryphal at the time of writing, for which audience was it intended using what kind of literary devices, what if any is the modern scholarly consensus on the value, truth or legitimacy of the text etc etc)

orionblastar 2y ago

There is this robot that reads the Bible: https://futurism.com/religious-robots-scripture-nursing-homes https://futurism.com/religious-robots-scripture-nursing-home...

GrumpyNl 2y ago
Isnt that just a modified mp3 player, according to the text they only just recite it.
- orionblastar 2y ago
  It is supposed to be an AI robot that explains the Bible and takes questions and answers them with Bible verses.

asim 2y ago

I had similar thoughts about using it for the Quran. I think this highlights you have to be very specific in your use cases especially when expecting an exact response on static text that shouldn't change. This is why I'm trying something a bit different. I've generated embeddings for the Quran and use chromem-go for this. So I'll ask the index the question first based on a similarity search and then feed the results in as context to an LLM. But in the response I'll still sight the references so I can see what they were. It's not perfect but a first step towards something. I think they call this RAG. What I'm working on https://reminder.dev https://reminder.dev

tokinonagare 2y ago
[flagged]
- danial 2y ago
  1. 4:144 This verse advises Muslims to prioritize loyalty within the community during a time of external threats. It is not a general prohibition but a caution in the context of potential betrayal. 2. 77:16 This refers to historical examples of past communities who faced consequences for rejecting divine guidance. It is a reminder of accountability, not a universal statement against non-believers. 3. 8:15 This verse gives instructions for battle, emphasizing courage and discipline during wartime. It applies to specific combat situations, not everyday relations with non-believers. 4. 5:41 This verse addresses the Prophet’s grief over those who rejected faith and distorted divine teachings. It critiques dishonesty and insincerity, not all members of specific groups. 5. 3:141 This verse speaks about trials that distinguish true believers and cleanse the community of wrongdoing. It emphasizes spiritual growth, not indiscriminate judgment of disbelievers. You seem to be making an accusation that Muslims widely practice "taqiyya" to deceive others. This is a baseless and Islamophobic trope. In mainstream Islam, lying is unequivocally condemned and considered an act of hypocrisy. While there is a narrow and rare historical exception permitting concealment of faith to protect one’s life under extreme duress, most Muslims have never encountered or practiced this concept. Ironically, those spreading this accusation often seem to know more about it than the Muslim communities they malign.

kittikitti 2y ago

I tried something similar with my favorite artist, Ariana Grande. Unfortunately, not even the most advanced AI could beat my knowledge of her lyrical work.

egeozcan 2y ago
As someone who usually listens to Anatolian Rock, even I know some Grande lyrics: > One taught me love, one taught me patience and one taught me pain. https://knowyourmeme.com/memes/thank-u-next https://knowyourmeme.com/memes/thank-u-next BTW, did someone already code an automated meme generator using LLMs? Only half-joking.
- hackernewds 2y ago
  a reaction to your comment https://knowyourmeme.com/memes/neil-degrasse-tyson-reaction https://knowyourmeme.com/memes/neil-degrasse-tyson-reaction

asimpleusecase 2y ago

This is nice work. The safest approach is using the look up - which his data shows to be very good - and combine that with a database of verses. That way textual accuracy can be retained and very useful lookup be carried out by LLM. This same approach can be used for other texts where accurate rendering of the text is critical. For example say you built a tool to cite federal regulations in an app. The text is public domain and likely in the training data of large LLMs but in most use cases hallucinating the text of a fed regulation could expose the user to significant liability. Better to have that canonical text in a database to insure accuracy.

sneak 2y ago

> While they can provide insightful discussions about faith, their tendency to hallucinate responses raises concerns when dealing with scripture I experience the exact same problem with human beings. > , which we regard as the inspired Word of God. QED

ddtaylor 2y ago

I'm heavily biased here because I don't find much value in the bible personally. Some of the stories are interesting and some interpretations seem useful, but as a whole I find it arbitrary. I never tell other people what to believe or how they should do that in any capacity. With that said I find the hallucination component here fascinating. From my perspective everyone who interprets various religious text does so differently and usually that involves varying levels of fabrication or something that looks a lot like it. I'm speaking about the "talking in tongues" and other methods here. I'm not trying to lump all religions into the same bag here, but I have seen that a lot have different ways of "receiving" communication or directive. To me this seems pretty consistent with the colloquial idea of a hallucination.

1123581321 2y ago
Most study and application tries to either source or fully work out from principles the meaning of the Bible. These can be wrong arguments but wouldn’t be hallucinations. Your experience sounds limited to Pentecostal-originated churches, which are 100-150 years old. In those churches, it’s acceptable to speak as if you’ve received a spontaneous understanding of the Bible and to not explain it. That does have a parallel to LLM hallucinations in face value output, I suppose, but the origination is completely different as the spontaneous human is making planned remarks passed off as spontaneous, trying to affect specific people in the room, or emotionally overwhelmed. None of those resemble why/how LLMs hallucinate.
- o11c 2y ago
  As quick as I am to criticize the bizarre versions of Christianity, I do think you're in error to assume Pentecostalism is all or even mostly about "planned remarks passed off as spontaneous". Improv is a thing, and can be trained as a skill even outside of comedy/entertainment. Though, outside of Charismatic sects, Christianity does see a more reasonable level of "I had prepared by thinking about (verse X), but suddenly now I'm thinking about (obscure verse Y)."
  - User23 2y ago

ks2048 2y ago

This is interesting. I'm curious about how much (and what) these LLMs memorize verbatim. Does anyone know any more thorough papers on this topic? For example, this could be tested on every verse in bible and lots of other text that is certainly in the training data: books in project gutenberg, wikipedia articles, etc. Edit: this (and its references) looks like a good place to start: https://arxiv.org/abs/2407.17817v1 https://arxiv.org/abs/2407.17817v1

int_19h 2y ago
For one anecdotal data point, GPT-4 knows the "navy SEAL copypasta" verbatim. It can reproduce it complete with all the original typos and misspellings, and it can recognize it from the first sentence.
deleted 2y ago
[deleted]