3 ms·
Listen to a video made by Karpathy about LLM, he explains why made up html tags work. It's to help the tokenizer
by p0w3n3d 1y ago
Listen to a video made by Karpathy about LLM, he explains why made up html tags work. It's to help the tokenizer
- victor106 1y agoCould not find it. Can you please provide a link?
- p0w3n3d 1y agohttps://youtu.be/7xTGNNLPyMI?si=eaqVjx8maPtl1STJ https://youtu.be/7xTGNNLPyMI?si=eaqVjx8maPtl1STJ He shows how the prompt is parsed etc. Very nice and eye opening. Also superstition dispelling
- dotancohen 1y agoI recall this even being in the Anthropic documentation.
- dotancohen 1y agoHere, found it: > Use XML tags to structure your prompts > There are no canonical “best” XML tags that Claude has been trained with in particular, although we recommend that your tag names make sense with the information they surround. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
- justsomehnguy 1y agoMy guess would be there is enough training materiel what a mere tagging sometging is enough to have a bigger SNR.