3 ms·

I tried something like that GPT trick and it is pretty hard to get it to not over-correct and over-index the vocabulary words. For instance if I tell it that "

I tried something like that GPT trick and it is pretty hard to get it to not over-correct and over-index the vocabulary words. For instance if I tell it that "OneTake AI" (my company) or "OneTake Chat" (one of our features) belong to the vocabulary words, it will twist correctly-transcribed sentences to make it appear in there. > Here is the biggest opportunity in artificial intelligence today if you're using Chat in your app (which is already grammatically correct!) would suddenly become > Here is the biggest opportunity in OneTake AI today if you're using OneTake Chat in your app This sycophantism of the LLM causes more trouble than it fixes. It's even worse if the vocabulary provided includes names, because then the model will randomly twist some sentences (mostly the benefit-driven ones, e.g. "so you can get faster results today") to add "... thanks to EXPERT_NAME!" It is also very difficult to get the model to edit structured text (like a JSON of the transcription) without breaking the schema or the IDs. I am still iterating to find a better method, and I think the "Functions" API can also help.

sebastiennight 3y ago
If this can be valuable to anybody else, here is an improved version of OP's prompt that will perform better than the one in the original article (it fixes every mistake plus punctuation and capitalization), with chatgpt-3.5-turbo-16k at a temperature of 0.7. I find that using the wording in the OP's article seems to create fewer sycophantic corrections than my previous approach, so that's a win. However the structure of my prompt seems to work better. So here's a combined result. System prompt: ``` The following English transcription is the output of a poor speech recognition system. Please correct its errors and reply with the corrected version without any preamble. Some of the uncommon words that the speech recognition system probably misspelled are: ["U2F", "CTAP", "FIDO", "CTAP2", "JSON", "FIDO2", "Web Authn", "bearer token", "hash", "phishing", "RP ID", "passkey", "caBLE"] If those words are present and misspelled, please fix their spelling. Please fix the spelling, grammar, any inconsistencies and transcription errors in each sentence. ``` Note: to improve performance, I also recommend: - providing the model with a before-and-after example, by using the USER and ASSISTANT roles to roleplay the whole thing (not putting the example in SYSTEM!) - providing the source text as a CSV or JSON, and asking the model to provide JSON as the output. - prefixing the Assistant response with an Assistant reminder of the instruction, like this: ASSISTANT ``` Return a corrected version with correct English grammar, punctuation and spelling. Correction (as JSON): ```

sebastiennight 3y ago

If this can be valuable to anybody else, here is an improved version of OP's prompt that will perform better than the one in the original article (it fixes every mistake plus punctuation and capitalization), with chatgpt-3.5-turbo-16k at a temperature of 0.7. I find that using the wording in the OP's article seems to create fewer sycophantic corrections than my previous approach, so that's a win. However the structure of my prompt seems to work better. So here's a combined result. System prompt: ``` The following English transcription is the output of a poor speech recognition system. Please correct its errors and reply with the corrected version without any preamble. Some of the uncommon words that the speech recognition system probably misspelled are: ["U2F", "CTAP", "FIDO", "CTAP2", "JSON", "FIDO2", "Web Authn", "bearer token", "hash", "phishing", "RP ID", "passkey", "caBLE"] If those words are present and misspelled, please fix their spelling. Please fix the spelling, grammar, any inconsistencies and transcription errors in each sentence. ``` Note: to improve performance, I also recommend: - providing the model with a before-and-after example, by using the USER and ASSISTANT roles to roleplay the whole thing (not putting the example in SYSTEM!) - providing the source text as a CSV or JSON, and asking the model to provide JSON as the output. - prefixing the Assistant response with an Assistant reminder of the instruction, like this: ASSISTANT ``` Return a corrected version with correct English grammar, punctuation and spelling. Correction (as JSON): ```