4 ms·
Show HN: Grammar Generator App for Llama.cpp
llama.cpp added context-free grammar guided generation functionality. It requires passing a file in a derivative of BNF notation, which gets messy very quickly for things like JSON.
To improve the experience, we built a small compiler from TypeScript interfaces to the grammar file format and have it hosted in a little browser app.
See more in discussion at https://github.com/ggerganov/llama.cpp/discussions/2494 https://github.com/ggerganov/llama.cpp/discussions/2494
- idkyall 3y agoSmall Nit: "This grammer can be used during LLMs generation." I didn't know Llama.cpp had this grammar feature, so this is interesting.
- codethief 3y agoIt's a relatively new feature, see https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773
- tarrekshaban 3y agoFixed! Thanks.
- dataf3l 3y agoCan you add examples on how to use this because I have honestly no idea how to use it
- aduffy 3y agoThanks for the feedback. There's an example in the linked GH discussion but let me provide a quick ETE example here, centered around parsing structured information from a hypothetical shipping company email. 1. In the app, write a TypeScript schema for this, one good example could be interface DeliveryInformation { /* Tracking number for the delivery */ tracking_number: string; /* Status of the delivery, one of "preparing", "out-for-delivery", or "delivered" */ status: string; /* Weight of the package, e.g. "2oz" or "3lb" */ weight: string; /* submission date time representation */ submitted_ts: string; } If you click Generate you'll see it generate a context-free grammar looking text, which is what llama.cpp reads. Click the download file to save it as grammar.gbnf 2. Grab a model for llama.cpp, e.g. this quantized Llama2 chat model: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/main/llama-2-7b-chat.ggmlv3.q8_0.bin https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma... 3. Grab the prompt in this gist (https://gist.github.com/a10y/d926039eee63cc2bcaf6345f9a419e3b https://gist.github.com/a10y/d926039eee63cc2bcaf6345f9a419e3...) and save as prompt.txt 4. Compile a recent commit of llama.cpp and run the following command ./main -m ./models/llama-2-13b-chat/llama-2-13b-chat.ggmlv3.q8_0.bin -f prompt.txt -c 4096 -n 1000 -t 1 --temp 0 --grammar-file ./grammar.gbnf EDIT: fixed some typos
- tkgally 3y agoYou might want to add an appropriate modifier before “grammar” on your top page to indicate your domain. Being a natural-language person, I thought your tool might be something that, say, generates rules for verb conjugations from a corpus of a human language.