5 ms·

This is one application of AI that I absolutely like. Imagine in future the AI will be able to ingest any video, blog, books, manuals, license and whole lot of

This is one application of AI that I absolutely like. Imagine in future the AI will be able to ingest any video, blog, books, manuals, license and whole lot of other things and we will just be able to ask questions to it or get a summary from it. I wonder what will happen to actual content then. Currently YouTube is showing info about most watched section of the clips. It saves so much time! Now imagine that happening to everything above.

PuppyTailWags 4y ago
Usually I find "most watched" ends up just being the most showy part of a clip, but I'm usually wanting the most informational stuff which isn't the same. E.g. I'm a rock climber and watch videos on technique and safety, but most watched is climbing a route with some pop song in the background. Useless.
- anamexis 4y ago
  One thing it's useful for is skipping past advertising segments.
mahathu 4y ago
How do you know what questions to ask if you haven't read the book
- mnkm 4y ago
  If you click the emoji there are some sample questions for each book.
- a1445c8b 4y ago
  I'd probably start with: "What is the main argument of the book?" if it's non-fiction.
cwkoss 4y ago
Perhaps even more exciting - you could write the documentation via a conversation with the chatbot, and it could ask further questions when it is unsure how to answer a user's query and even update the documentation when changes are noticed.
james-revisoai 4y ago
Hey! I'm working on building this vision at fragen.co.uk - which can take in Youtube videos (trascribed by whisper + postprocessed into useful chunks), PDFs (properly OCR'd - understands bullet points etc) and webpages (also OCR'd, experimental) - For the problem of data-supported AI search, the content really matters. Our edge is semantically chunking it, in other words we are splitting it up into key facts ready for recall. That makes this problem more solveable. ... and our tool "sees" the above text as ... For the problem of data-supported AI search, the content really matters. fragen.co.uk's edge is semantically chunking the content, in other words we are splitting the content up into key facts ready for recall. Splitting the content up into key facts ready for recall makes data-supported AI search solveable. (hope it's visible how an LLM like GPT able to use/quote the above can perform seriously better at those bothersome it/what/where questions and follow ups)
maegul 4y ago
Agree. I suspect that this is the short/medium term application for this tech. One basic but effective demonstration I’ve seen was summarising a 30 minute talk on YouTube into dot points.[1] I watched the video and read the summary afterwards and was almost completely satisfied with the summary. At scale, the flexible compression and expansion and navigation of information is potentially huge … like Google Maps for the internet. [1]: https://gist.github.com/simonw/9932c6f10e241cfa6b19a4e08b283ca9 https://gist.github.com/simonw/9932c6f10e241cfa6b19a4e08b283...
- a1445c8b 4y ago
  Google Maps for human knowledge! That's such an apt analogy(?) right there. Thanks for sharing!
fzliu 4y ago
If you sprinkle in a bit of infrastructure, I think we're already there. The ability to distill a variety of content into vectors and perform approximate nearest neighbor search (shameless plug: https://milvus.io https://milvus.io) across all of them can really help power a lot of these applications. With the retrieved vectors, you could match questions with answers or create a reverse index to the original content to perform summarization. With that being said, one of the main challenges ahead will be multimodal learning. We're sort-of there combining text with visual data, but there are many other modalities out there as well.
- williamcotton 4y ago
  How usable would the document embedding be for a nearest neighbor search if the dimensions were reduced to three?
  - hansvm 4y ago
    Nearly useless for most applications unless there's been a major improvement in the SOTA that I missed.
  - fzliu 4y ago
    Might be usable for low column count tabular data, but it would be pretty terrible for any other semantically dense modality e.g. video, molecules, geospatial, etc.
  - sdenton4 4y ago
    can i use the three dimensions to encode a space-filling curve over a 1000-dimensional embedding?
    - williamcotton 4y ago
      Not precisely, but if you had 50 documents in that 1000-dimensional embedding and you reduced the dimensions to three and still got at least the exact same nearest neighbor ordering then it would at least still function, right? I guess the problem is taking a new document (like a search term) in the higher dimensional embedding and reducing it to three dimensions for searching in that reduced space and expecting that to also maintain the same nearest neighbor ordering.
gremlinsinc 4y ago
Imagine listening to a book on tape, like even scifi/fantasy and being like wait, what just happened? Or it kinda acting like the Grandpa (minus humor) in Princess Bride, basically someone who's read the book and can stop/replay bits especially via audio. This would be a godsend for people w/ ADHD, or auditory processing disorders.
version_five 4y ago
I have a decent sized book collection, and what I'd like the most is a ctrl-f for books. Does that exist already in e-readers? Personally I think question answering is still very gimmicky, in particular because I can't clearly understand why the answer is what it is. Ctrl-f is completely explainable, I know why it works and why it fails, and is a much more useful tool to engage with books. The main problem seems that the full text of most books is not available unencumbered to process as one sees fit
- nutanc 4y ago
  Agreed. That's why the approach we are taking is to answer the question 'and' show the links/paragraphs which answer the question. Demo we have created for our website: https://speech-kws.ozonetel.com/ozosearch https://speech-kws.ozonetel.com/ozosearch
- joshspankit 4y ago
  I tend to download PDFs for an physical book I buy, but I’d like a hybrid approach. Ctrl-F is perfect for finding excerpts when you have a specific wording, but not great for finding something worded differently or something in a section you have not read yet. AI could step in for those and not only give specific references from the book (or it’s own referenced material) but could also give the text in full. “AI, what does this book say about integrity?” “(answer), based on paragraphs on page 143 and page 210” “Show me those paragraphs inline, starting with the paragraph where the author transitions in to the topic”
- rakejake 4y ago
  Not sure about e-readers. But any EPUB can be searched like any other text.
- arglebargle123 4y ago
  Yes but also no. If you have an exact string query that'll turn up the topic you want then, sure, you can find a topic easily. If your question is more tangential, say you kind of know what topic you're looking for but don't know what it's called in language X's standard library, you're pretty basically SOL unless an author happened to mention a phrase you can match. Using chat GPT to solve this problem seems pretty awesome, it would be super handy to query my (larger than it has any right to be) library of reference books using not-exactly-competent requests. This and translating technical material into 5th grade reading level explanations are probably the best two uses of chat GPT I've seen yet.
gala8y 4y ago
eg. https://huberman.rile.yt/ https://huberman.rile.yt/ based on popular The Huberman Lab podcast.
Giorgi 4y ago
imagine feeding it some person forum history, so you can talk to them even when they are dead, public figure or a family member.
d4rkp4ttern 4y ago
A related thing I’ve been wondering about — I have thousands of bookmarks and saved PDFs etc and I would love to have a way to scrape all the content from the URLs and PDFs and then ask questions and get answers along with links to relevant docs. Now that would be a killer app. EDIT — answered my own question: there is indeed an OpenAI Embeddings API : https://beta.openai.com/docs/guides/embeddings https://beta.openai.com/docs/guides/embeddings That plus a vector similarity engine (FAISS for example) is the key to these types of apps (Thanks to Simon Willison’s blog, which he pointed to elsewhere in this thread)

PuppyTailWags 4y ago

Usually I find "most watched" ends up just being the most showy part of a clip, but I'm usually wanting the most informational stuff which isn't the same. E.g. I'm a rock climber and watch videos on technique and safety, but most watched is climbing a route with some pop song in the background. Useless.

anamexis 4y ago
One thing it's useful for is skipping past advertising segments.

mahathu 4y ago

How do you know what questions to ask if you haven't read the book

mnkm 4y ago
If you click the emoji there are some sample questions for each book.
a1445c8b 4y ago
I'd probably start with: "What is the main argument of the book?" if it's non-fiction.

cwkoss 4y ago

Perhaps even more exciting - you could write the documentation via a conversation with the chatbot, and it could ask further questions when it is unsure how to answer a user's query and even update the documentation when changes are noticed.

james-revisoai 4y ago

Hey! I'm working on building this vision at fragen.co.uk - which can take in Youtube videos (trascribed by whisper + postprocessed into useful chunks), PDFs (properly OCR'd - understands bullet points etc) and webpages (also OCR'd, experimental) - For the problem of data-supported AI search, the content really matters. Our edge is semantically chunking it, in other words we are splitting it up into key facts ready for recall. That makes this problem more solveable. ... and our tool "sees" the above text as ... For the problem of data-supported AI search, the content really matters. fragen.co.uk's edge is semantically chunking the content, in other words we are splitting the content up into key facts ready for recall. Splitting the content up into key facts ready for recall makes data-supported AI search solveable. (hope it's visible how an LLM like GPT able to use/quote the above can perform seriously better at those bothersome it/what/where questions and follow ups)

maegul 4y ago

Agree. I suspect that this is the short/medium term application for this tech. One basic but effective demonstration I’ve seen was summarising a 30 minute talk on YouTube into dot points.[1] I watched the video and read the summary afterwards and was almost completely satisfied with the summary. At scale, the flexible compression and expansion and navigation of information is potentially huge … like Google Maps for the internet. [1]: https://gist.github.com/simonw/9932c6f10e241cfa6b19a4e08b283ca9 https://gist.github.com/simonw/9932c6f10e241cfa6b19a4e08b283...

a1445c8b 4y ago
Google Maps for human knowledge! That's such an apt analogy(?) right there. Thanks for sharing!

fzliu 4y ago

If you sprinkle in a bit of infrastructure, I think we're already there. The ability to distill a variety of content into vectors and perform approximate nearest neighbor search (shameless plug: https://milvus.io https://milvus.io) across all of them can really help power a lot of these applications. With the retrieved vectors, you could match questions with answers or create a reverse index to the original content to perform summarization. With that being said, one of the main challenges ahead will be multimodal learning. We're sort-of there combining text with visual data, but there are many other modalities out there as well.

williamcotton 4y ago
How usable would the document embedding be for a nearest neighbor search if the dimensions were reduced to three?
- hansvm 4y ago
  Nearly useless for most applications unless there's been a major improvement in the SOTA that I missed.
- fzliu 4y ago
  Might be usable for low column count tabular data, but it would be pretty terrible for any other semantically dense modality e.g. video, molecules, geospatial, etc.
- sdenton4 4y ago
  can i use the three dimensions to encode a space-filling curve over a 1000-dimensional embedding?
  - williamcotton 4y ago
    Not precisely, but if you had 50 documents in that 1000-dimensional embedding and you reduced the dimensions to three and still got at least the exact same nearest neighbor ordering then it would at least still function, right? I guess the problem is taking a new document (like a search term) in the higher dimensional embedding and reducing it to three dimensions for searching in that reduced space and expecting that to also maintain the same nearest neighbor ordering.

gremlinsinc 4y ago

Imagine listening to a book on tape, like even scifi/fantasy and being like wait, what just happened? Or it kinda acting like the Grandpa (minus humor) in Princess Bride, basically someone who's read the book and can stop/replay bits especially via audio. This would be a godsend for people w/ ADHD, or auditory processing disorders.

version_five 4y ago

I have a decent sized book collection, and what I'd like the most is a ctrl-f for books. Does that exist already in e-readers? Personally I think question answering is still very gimmicky, in particular because I can't clearly understand why the answer is what it is. Ctrl-f is completely explainable, I know why it works and why it fails, and is a much more useful tool to engage with books. The main problem seems that the full text of most books is not available unencumbered to process as one sees fit

nutanc 4y ago
Agreed. That's why the approach we are taking is to answer the question 'and' show the links/paragraphs which answer the question. Demo we have created for our website: https://speech-kws.ozonetel.com/ozosearch https://speech-kws.ozonetel.com/ozosearch
joshspankit 4y ago
I tend to download PDFs for an physical book I buy, but I’d like a hybrid approach. Ctrl-F is perfect for finding excerpts when you have a specific wording, but not great for finding something worded differently or something in a section you have not read yet. AI could step in for those and not only give specific references from the book (or it’s own referenced material) but could also give the text in full. “AI, what does this book say about integrity?” “(answer), based on paragraphs on page 143 and page 210” “Show me those paragraphs inline, starting with the paragraph where the author transitions in to the topic”
rakejake 4y ago
Not sure about e-readers. But any EPUB can be searched like any other text.
arglebargle123 4y ago
Yes but also no. If you have an exact string query that'll turn up the topic you want then, sure, you can find a topic easily. If your question is more tangential, say you kind of know what topic you're looking for but don't know what it's called in language X's standard library, you're pretty basically SOL unless an author happened to mention a phrase you can match. Using chat GPT to solve this problem seems pretty awesome, it would be super handy to query my (larger than it has any right to be) library of reference books using not-exactly-competent requests. This and translating technical material into 5th grade reading level explanations are probably the best two uses of chat GPT I've seen yet.

gala8y 4y ago

eg. https://huberman.rile.yt/ https://huberman.rile.yt/ based on popular The Huberman Lab podcast.

Giorgi 4y ago

imagine feeding it some person forum history, so you can talk to them even when they are dead, public figure or a family member.

d4rkp4ttern 4y ago

A related thing I’ve been wondering about — I have thousands of bookmarks and saved PDFs etc and I would love to have a way to scrape all the content from the URLs and PDFs and then ask questions and get answers along with links to relevant docs. Now that would be a killer app. EDIT — answered my own question: there is indeed an OpenAI Embeddings API : https://beta.openai.com/docs/guides/embeddings https://beta.openai.com/docs/guides/embeddings That plus a vector similarity engine (FAISS for example) is the key to these types of apps (Thanks to Simon Willison’s blog, which he pointed to elsewhere in this thread)