6 ms·
In my opinion a major obstacle right now is the token limit. This makes it very cumbersome to operate on large input documents, e.g. summarizing a large documen
by FooBarWidget 4y ago
In my opinion a major obstacle right now is the token limit. This makes it very cumbersome to operate on large input documents, e.g. summarizing a large document. It doesn't help that the limit includes the response as well.
Prompt engineering helps me better understand the limits and ways to work around them. But it's still painful.
Today I tried summarizing a long video (~1 hour) using Videohighlight but their AI quickly reached the token limit.
I also tried to use ChatGPT to (somewhat ironically) translate a new AI chat regulation proposal by the Chinese government, for which the government asks public feedback. http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm
I used prompt engineering to get a translation quality and readability better than what I can get through Google Translate. This isn't a short document but it's not very long either. I had to break it into three in order to get ChatGPT to translate it all.
In case anyone is curious about the final translation: https://gist.github.com/FooBarWidget/201ea5e0983d05d21f6719bacb46795e https://gist.github.com/FooBarWidget/201ea5e0983d05d21f6719b...
This is the prompt I used, incorporating prompt engineering lessons I've learned. I assigned a role to the LLM, provided appropriate context, and provided constraints for its output.
> You are a Chinese-to-English translator with decades of experience translating Chinese government policy documents to an audience that's not familiar with Chinese governance, nor with the way that the Chinese government writes. What follows is a part of a draft policy document on potentially regulating AI chat services. The Chinese government has asked the public for feedback.
> Begin introduction:
> ...
> End introduction.
> Begin part of policy document body:
> ...
> End part of policy document body.
> Translate the the policy document body part to English. Do not translate the introduction. Write in a manner that's easily readable for an audience that has no experience with Chinese policy, Chinese idioms or the way the Chinese government writes.
- summarity 4y agoProgressive summarization works pretty well - I'm using that for https://findsight.ai https://findsight.ai You can even get lesser LLMs to do the bulk reduction that have GPT clean it up on the way to even less content. Admittedly, that does take a lot of prompt engineering, chunk selection and reinforcement though (LLM supervising LLM).
- alchemist1e9 4y agoI understood everything except: > reinforcement though (LLM supervising LLM). Is there something I can read to understand what that looks like?
- summarity 4y agoI don't think this approach is formalized but I can give a few examples: A) Prompt leak prevention: chunk and embed LLM responses, than compare against original prompt to filter out chunks that leak the prompt B) Automatic prompt refinement: Prompt a cheap model, use an expensive model to judge the output and rewrite the prompt (this is in part how Vicuna[1] did eval for their LLaMa fine-tuning) Basically using LLMs in the feedback loop. [1] https://vicuna.lmsys.org https://vicuna.lmsys.org
- alehlopeh 4y agoIf telling Chat GPT that it has “decades of experience” leads to better results, why not say centuries of experience instead? If you don’t know how every token of input affects the output, is engineering an accurate way to describe what you’re doing?
- FooBarWidget 4y agoI don't think the training data has works of centuries-old translators. > If you don’t know how every token of input affects the output, is engineering an accurate way to describe what you’re doing? We in fact don't know exactly. Prompt engineering seems to be a bag of tricks that change the probabilities of outputs. But I didn't invent the term.
- simonw 4y agoI don't think this is a strong argument against calling it "prompt engineering". Engineers have built systems based around things they didn't fully understand for centuries. We have learned an enormous amount about materials engineering and metallurgy since building the Brooklyn Bridge for example. If you don't fully understand how a system you are building on top of works, the engineering approach is to methodically experiment. That's what prompt engineering is. (One argument that works for not calling prompt engineering "engineering" is to point out that in many disciplines engineering requires certifications and licensing - the same reason people sometimes argue against "software engineering" as an engineering discipline.)
- 4y ago
- ruxudev 4y agoVery interesting use of ChatGPT and prompt engineering. For your problem, summarizing a large document, splitting the document into smaller parts is indeed the way to go. I also had problems myself with operating on large documents. In my case, I had an insurance policy that I wanted to extract information from. My solution: use the OpenAI API to convert the document to OpenAI's embeddings and saving those embeddings to a vector database. Then, use similarity search on the database to find chunks of the document that might be related to my query and pass only those chunks to GPT for the information extraction prompt. I plan to create a guide on how to tackle these problems after I consolidate my findings.
- agomez314 4y agoI'm increasingly seeing these kinds of solutions for similar tasks. I wonder if we are seeing the discovery of new abstractions from using LLMs.
- a_bonobo 4y agoVery cool solution! My solution was to write a bit of code that writes a CSV, then I used a langchain-based CSV agent. Since that one calls on pandas it effectively has no token limit, but it also has no overview of the data, only what pandas tells it.
- Beefin 4y agothis is how we enable 10mb+ file ingestion and search /w http://mixpeek.com http://mixpeek.com
- monkeydust 4y agoAlso doing this, works really well. Check out James Briggs on YouTube. Excellent tutorials on how to achieve this.
- FooBarWidget 4y agoCan you refer to a specific video?
- v4dok 4y agoI've actually coded something like this. The trick is to split it in a smart way. What I did is the following: 1. Download the video 2. Split it in max length for whisper 3. Used whisper for transcription on the chunks 4. Used gpt for cleaning the transcription 5. Unioned the cleaned transcribed chunks 5. Used the langchain "refine" function of the summary chain It is quite slow and expensive because the refine summary is doing a lot of calls, but the result is amazing. And you don't need a vector DB for that because the summary is serial. For Q&A tho, an embeddings DB is a must
- freediver 4y agoTry Universal Summarizer. No token limit. https://kagi.com/summarizer https://kagi.com/summarizer
- highduc 4y agoAnd this is even worse on locally ran models. I get 2k tokens of context, max.