8 ms·
Bible Semantic Search
- _8j50 4y agoIs this site hijacking just my history or something (can't get back to hn after opening it all the history is spammed with their own domain). Poor quality site this behavior should be discouraged.
- saltdoo 4y agoNeed this for the Koran
- brink 4y agoVery cool! I could see myself really using this. I just wish it wasn't so dependent on SaaS - it tends to make things unreliable, and can disappear at any time.
- JasonFruit 4y agoThis is pretty good! I entered, "Every human thought is only evil," and got, among others, the verse I hoped for, Genesis 6:5: "And God saw that the wickedness of man was great in the earth, and that every imagination of the thoughts of his heart was only evil continually." "David cried" got me some unexpected results for "the Son of David", but also the verse about David and Jonathan I expected, and a couple others that were a very good fit. …and it's gone.
- wvlia5 4y agofunny story comes up first when you search "bears"
- blondin 4y agothis is cool. streamlit is super cool. totally missed when they made a sharing platform.
- blueberrychpstx 4y ago
- eatonphil 4y agoTried to use it and got: AttributeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app). Traceback: File "/home/appuser/venv/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 475, in _run_script exec(code, module.__dict__) File "/app/bible-semantic-search/app.py", line 11, in <module> index = PineconeIndex("qa-index") File "/app/bible-semantic-search/pinecone_index.py", line 16, in __init__ self.index = self.connect_to_index(index_name) File "/app/bible-semantic-search/pinecone_index.py", line 30, in connect_to_index index = pinecone.Index(index_name) File "/home/appuser/venv/lib/python3.8/site-packages/pinecone/index.py", line 34, in __init__ openapi_client_config.api_key = openapi_client_config.api_key or {}
- gk1 4y agoCredit goes to Chris Lee: https://www.chrisleeportfolio.com https://www.chrisleeportfolio.com Explanation when you click “Wat this?”: > This is a Streamlit app I prototyped for performing semantic search on the King James Bible. It conducts full text search as well as semantic search, which is useful for surfacing passages that are similar in meaning to the query, even if the passages don't explicitly contain the query keyword(s). Suppose you wanted to bring up all verses that reference the infamous snake that tempted Eve. In a traditional keyword search system, searching for 'snake' wouldn't yield any results because the KJV uses the term 'serpent'. A semantic search system would take that 'snake' query and retrieve the relevant verses that contain 'serpent' as well as similar verses like ones about reptiles. > Under the hood, I've generated vector embeddings of every verse in the Bible using SBERT (https://www.sbert.net/ https://www.sbert.net/), and stored those embeddings in a vector database called Pinecone (https://www.pinecone.io https://www.pinecone.io). Every time you submit a query, it's converted to its vector representation using SBERT. That query vector is then sent to Pinecone, which performs an Approximate Nearest Neighbor (https://www.pinecone.io/learn/what-is-similarity-search/ https://www.pinecone.io/learn/what-is-similarity-search/) search, retrieving the top n verses that are the most semantically similar to our query. The verses returned are ranked in order of most to least similar. (Full disclosure: I work for Pinecone, but I have no connection to this demo.)
- angrygoat 4y agoHow does pinecone compare to pgvector? https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- michaelmior 4y agoMilvus[0] would be another interesting comparison. [0] https://milvus.io/ https://milvus.io/
- gk1 4y agoSame general difference: Pinecone is SaaS that's ready to go with a few API calls, while Milvus an open-source tool that requires work to set up, scale, and maintain.
- lalaland1125 4y agoSBERT isn't trained on this type of archaic English and you can see it failing. It needs to be fine tuned or you should use a modern Bible translation. A clear example is the query: "homosexuality" This returns: > James 2:3 - And ye have respect to him that weareth the gay clothing, and say unto him, Sit thou here in a good place; and say to the poor, Stand thou there, or sit here under my footstool It's clearly seeing "gay" but is unaware that the meaning has changed. A classic issue when applying a ML model out of domain.
- stosto88 4y ago
- kadoban 4y agoCouldn't they search in a modern translation and just refer back to the ancient one? Seems like the way bible verses are split up should remove the ability of translation to move anything around much.
- chrislee973 4y agoAh, that's a good idea. I was considering just fully replacing the KJV with the NIV, but for people who want the KJV, having a mapping between the two versions would work. Do the verses between the two versions map each other exactly? Like they cover the same exact thing, just written in a different style?
- Yeroc 4y agoYes, for the majority of the English translations verses map to each other exactly with a few minor exceptions. NIV would map virtually exactly.
- chrismorgan 4y agoTo do this properly, you have to map versifications, because some systems use different conventions. As the most significant and pervasive example, some systems treat the title of psalms as verse one (French generally does this), while others treat it as separate (verse zero, effectively; English generally does this), so if you just reuse numbers across such transitions you’ll get consistent off-by-one errors. There are quite a few other similar off-by-one errors, and occasionally more, throughout. https://wiki.crosswire.org/Alternate_Versification https://wiki.crosswire.org/Alternate_Versification is a decent starting point for looking into the topic, with SWORD’s canon_.h files fairly tolerable for showing the number of verses in each chapter. Unfortunately, SWORD has never gone as far as doing proper mapping* between versifications for some reason—they have some basic mapping somewhere or other, but I can’t remember offhand where or what it is, as I only briefly looked into it five years or so ago. The most significant differences occur when you switch languages (there are quite a few differences if you switch from English to French or to many Indic languages), but there may be some differences between translations within a language too, e.g. SWORD’s NRSV versification has an extra verse in 3 John and Revelation 12 compared to its KJV versification.
- enw 4y ago> jesus being a gigachad > > Mark 1:25 - And Jesus rebuked him, saying, Hold thy peace, and come out of him.
- deleted 4y ago[deleted]
- slater 4y ago
- 1-6 4y ago> Joe Biden Matthew 22:8 - Then saith he to his servants, The wedding is ready, but they which were bidden were not worthy. Luke 14:24 - For I say unto you, That none of those men which were bidden shall taste of my supper. heh heh.
- irrational 4y ago> Republicans Isaiah 1:15 - And when ye spread forth your hands, I will hide mine eyes from you: yea, when ye make many prayers, I will not hear: your hands are full of blood.
- justinhj 4y ago6:7: “Do not be deceived, God is not mocked; for whatever a man sows, this he will also reap.”
- deleted 4y ago[deleted]
- ARandomerDude 4y agoThis seems to struggle with phrases. For example I expected Luke 9:13 as a result for the phrase "you feed them" or "you give them something to eat." The King James reads "give ye them to eat" but the app never found it.
- uean 4y agoHmm. Searching “baby” returns nothing but references to Babylon. Not sure why this was my first and only search, less clear why it is doing partial text matching.
- Yenrabbit 4y agoI'd guess Babylon is not in the models vocabulary and gets tokenized as <baby><lon> or similar
- pvg 4y agoIt's so that you can learn about the Tower of Babies and the great confusion it sowed among humankind.
- chrislee973 4y agoThe app performs full text search as well as semantic search. The full text search results are presented first, so if you scroll down to the bottom of the page you should see the semantically related results under the header "Semantic Search Results"
- yosito 4y agoWould have loved this back in my religious days, or in the brief period of time I was interested in criticizing religion. These days I'm not really interested at all in the Bible, but this is still a pretty cool idea. A little bit of an archaic choice to search the KJV though. Maybe it was chosen due to copyright issues.
- wincy 4y agoMust be for Mormons. They only read the King James Version because otherwise the Book of Mormon would sound super weird seeing as how it’s the worlds most popular Bible fanfic. Source: technically a Mormon.
- dbrueck 4y agoNot really. It's true that KJV is the default for English speakers, but less than half of the church's members are native English speakers. Even among those that speak English, many study the NIV and ESV, for example. It's not even that uncommon to hear non-KJV quotes in GenCon for that matter.
- wincy 4y agoHey today I learned, thanks! I catch maybe five minutes of General Conference, my wife’s more likely to watch it than I am. I was also very confused about what a tabletop gaming convention founded by Gary Gygax (of Dungeons and Dragons fame) had to do with Mormons at first hahah.
- eru 4y agoWell, I guess the Bible is also its own 'fanfic', as not every part was written at the same time and some later parts clearly reference earlier parts.
- hnal943 4y agoFanfic would imply that the work is non-canon, wouldn't it? The bible represents some of the earliest arguments about what is canon.
- hopsworks 4y agoIs the source code available for this?
- yosito 4y agoIs just two hours on HN enough to kill a Streamlit app? Or is this running into limits of a certain plan level? Or is the problem with the code itself?
- deleted 4y ago[deleted]
- sircastor 4y agoWhile serving a mission for my church, a fellow member bought me a copy of “Strongs Exhaustive Concordance of the Bible”. I found it really fascinating to flip through correlated terms and be able to draw conceptual lines between them. It was the first time I’d ever come across something like that. It strikes me as something that is relatively rare simply because what the Bible is, in terms of popularity and commonality across the western world.
- michaelsbradley 4y agohttps://www.blueletterbible.org/kjv/gen/1/1/ss1/s_1001 https://www.blueletterbible.org/kjv/gen/1/1/ss1/s_1001
- pbhjpbhj 4y agoBLB is great, their app works well too. I've been using it for what feels like ~2 decades.
- jmorse2 4y agoI've got a copy and still use it sometimes -- being able to immediately see nearby words (like "glad", "gladly", "gladness") is quite useful, which you'd miss just searching for "glad". Being able to see all uses on a hardcopy in a fixed position that doesn't scroll is helpful too, I find it harder to rationalise big lists when they're scrollable. (I also use it to explain type checking to friends, given the number of times I've looked up a Hebrew word in the Greek dictionary, or vice versa).
- elihu 4y agoI'd like to think it's called "Strong's Concordance" because that's what you'd be if you carried it around all the time.
- bayesian_horse 4y ago
- bibelo 4y agoWhy with a 400 year-old translation though?
- chrislee973 4y agoI figured it was the quintessential version of the Bible ¯\_(ツ)_/¯. I'm not religious so cut me some slack haha. Looking back, I should've used a version using modern English.
- elihu 4y ago...another reason is that KJV is not currently under copyright, unlike (as far as I know) all modern English translations.
- pbhjpbhj 4y agoI think there are a couple of free and open translations that are reasonable - the WEB (World English Bible) maybe. https://www.biblegateway.com/versions/World-English-Bible-WEB/ https://www.biblegateway.com/versions/World-English-Bible-WE...
- anjbe 4y agoFun fact: in the United Kingdom, the King James Version is still copyrighted—perpetually copyrighted, in fact—by (who else?) the British monarchy. Since the author of this app is based in the United States, he should be safe… we have a bit of precedent for ignoring royal prerogatives here :)
- elihu 4y agoInteresting.
- Amezarak 4y agoKJV is still wildly popular and some sects even regard it as an inspired translation. Never heard that about anything else. NKJV is more readable for people who grew up reading blog posts instead of old books and generally retains the poetry of the KJV, but it's under copyright. I think it would be allowed for a use like this though, because they explicitly allow (as do most translations) quoting up to a certain number of verses. EDIT: Maybe not, here's the rules: https://www.harpercollinschristian.com/sales-and-rights/permissions/#1 https://www.harpercollinschristian.com/sales-and-rights/perm... Seems like you'd fall afoul for the % of total text requirement.
- steve_gh 4y agoHmm. I entered "kingdom of heaven" - and it returned a ProtocolError Seems kind of appropriate. Obviously it is easier for a camel to thread the eye of a needle than to get a semantic search on the Kingdom of Heaven.
- beltsazar 4y agoA query "what is the purpose of life?" returns decent results with: - Precision@2 = 50% - Precision@5 = 20% - Precision@10 = 33% Relevant results: 2) Philippians 1:21 - For to me to live is Christ, and to die is gain. 6) Ecclesiastes 3:13 - And also that every man should eat and drink, and enjoy the good of all his labour, it is the gift of God. 10) Ecclesiastes 2:17 - Therefore I hated life; because the work that is wrought under the sun is grievous unto me: for all is vanity and vexation of spirit.
- shrimpx 4y agoIs the source available? Curious if the streamlit app makes requests to an external service or if it’s self-contained.
- punk_ihaq 4y agoIt is. Click the hamburger menu to the top-right, followed by 'View app source': https://github.com/chrislee973/bible-semantic-search/blob/main/app.py https://github.com/chrislee973/bible-semantic-search/blob/ma...
- shrimpx 4y agoThanks!
- punk_ihaq 4y agoIt makes a request to the Pinecone API to search through vector embeddings. [1][2] [1] https://github.com/chrislee973/bible-semantic-search/blob/ab5e532228c239fffc525c6e8fe865662bc06d92/pinecone_index.py#L20 https://github.com/chrislee973/bible-semantic-search/blob/ab... [2] https://www.pinecone.io/ https://www.pinecone.io/
- deleted 4y ago[deleted]
- phgn 4y agoFYI it's failing for me with: urllib3.exceptions.ProtocolError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app). Traceback: File "/home/appuser/venv/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 475, in _run_script exec(code, module.__dict__) File "/app/bible-semantic-search/app.py", line 50, in <module> query_results = controller.query( File "/app/bible-semantic-search/controller.py", line 127, in query results = self.index.query(query_emb, top_k, namespace) File "/app/bible-semantic-search/pinecone_index.py", line 63, in query return self.index.query(
- mfer 4y agoIt worked yesterday. Something broke in the meantime.
- giancarlostoro 4y agoI'm having the same issue. My searched phrase was two words very simple "Judge not" for reference. I'll check back later, curious how this looks and behaves overall.