4 ms·

As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is enc

As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). At which point, again I have to ask: what do you need the LLM for? You've already done all the hard work by hand and the LLM is only adding some extraneous natural language parsing on top. Plus, if you already have the grammar that can cover the anthropomorphic vegetable world it's only a bit more work to use it to parse such natural language requests, anyway. I think people forget that grammars were the staple for parsing natural language and stuffing it into structured form for a very long time before LLMs, and they still mostly are. The point is that if you have structure, someone has to hand-craft that structure. Frex, if you have a language with a compiler, someone has to write the compiler. Then, if you want to make some unstructured text conform to your hand-crafted structure, you can only do that to the extent that the unstructured text itself is made up of elements of the structured form. If you have a grammar for frogs and blueberries, and write a poem about the dawn and foxes, you can't use the former to structure the latter, no matter what you do, and LLMs won't make this happen magickally, either. Essentially, your grammar is a type and any unstructured text you want to convert to a structure with your grammar must be a value that you can cast to that type. >> I wasn't talking about deterministic Vs nondeterministic. Then what? What do you mean by "random string"?

creatonez 3y ago
> I think people forget that grammars were the staple for parsing natural language and stuffing it into structured form for a very long time before LLMs, and they still mostly are. This is a rewritten history of natural language processing tech. Years of fine-tuned theory-heavy grammar coding for parsing and generating human language got the field basically nowhere.
IanCal 3y ago
> As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). You wouldn't need to, that's the point here. You let the LLM work on generating semantically valid responses and use a tool like this to restrict it to syntactically correct ones. Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). Let the LLM deal with coming up with a name and backstory that work, making sure the description and type of the weapon make sense (gpt4 suggested a close range carrot dagger for example), and let this work as your type structure. { "type": "object", "title": "character", "properties": { "backstory": { "type": "string" }, "weapons": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "description": { "type": "string" }, "weapon_type": { "type": "string", "enum": ["ranged", "close", "magic"] }, "range": { "minimum": 0, "maximum": 150 }, "damage": { "type": "number" } }, "required": [ "name", "description", "range", "damage" ] } }, "name": { "type": "string" } }, "required": [ "backstory", "weapons", "name" ] } > Then what? What do you mean by "random string"? Nonsense. Like "Colorless green ideas sleep furiously" the famous sentence that's grammatically correct but utter nonsense. > Plus, if you already have the grammar that can cover the anthropomorphic vegetable world it's only a bit more work to use it to parse such natural language requests, anyway. I really do not think this is the case. Parsing and understanding arbitrary requests about something like this?
- YeGoblynQueenne 3y ago
  >> Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). That'd be nice, but it's not how this tool works. If you look at the repo, there's an example of following a json schema or pydantic model. It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: class Weapon(str, Enum): sword = "sword" axe = "axe" mace = "mace" spear = "spear" bow = "bow" crossbow = "crossbow" But perhaps I'm underestimating the tool's capabilities. If so, hopefully remilouf can correct me (and give an example of how the tool can be made to work as you want it). >> I really do not think this is the case. Parsing and understanding arbitrary requests about something like this? Not arbitrary. See my casting-to-type analogy. The point I'm trying really hard to get across is that generating free-form text is all nice and cool, but if you want to give it structure, you need to have the entire structure defined before-hand, otherwise the text that can't be made to conform to it simply won't. So if you haven't got anthropomorphic vegetables in your json schema, your LLM may generate them, they'll never end up in your json.
  - remilouf 3y ago
    You can definitely let the model improvise by defining `weapon` as `Union[Weapon, str]` if that's what you're asking.
  - IanCal 3y ago
    > It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: No, only if you want to explicitly limit it to a set of options. You can have freeform fields, just like the jsonschema I provided. If you look at the example there's a character name which has a constrained length but is not limited to a set of options: class Character(BaseModel): name: constr(max_length=10) age: int armor: Armor weapon: Weapon strength: int The name there can be anything you want. This tool is, unfortunately, outrageously slow so I put the json schema above with a few fixes into jsonformer and downloaded a small model and used it to convert the GPT4 description into valid json: { "backstory":"Born in the tranquil meadows of Veggie", "weapons":[ { "name":"Leek Lance", "description":"A long, green and white lance made from a leek", "weapon_type":"distance", "range":100.0, "damage":75.0 }, { "name":"Carrot Dagger", "description":"A short, pointed dagger. It's sharp", "weapon_type":"close", "range":10.0, "damage":50.0 } ], "name":"Sir Turnip Thistlebrook" } > Not arbitrary. Well exactly. If you want to support arbitrary requests while constraining the output, tools like this are an easy approach and I'm not sure what else comes close. An interactive character design flow would have something like the above as the defined output and you could just keep asking for alterations as a human would ("make it more whimsical" or "not a king, something lower class") and have useful structured output > See my casting-to-type analogy. The point I'm trying really hard to get across is that generating free-form text is all nice and cool, but if you want to give it structure, you need to have the entire structure defined before-hand, otherwise the text that can't be made to conform to it simply won't. The structure, sure. But the content can be extremely varied.

creatonez 3y ago

> I think people forget that grammars were the staple for parsing natural language and stuffing it into structured form for a very long time before LLMs, and they still mostly are. This is a rewritten history of natural language processing tech. Years of fine-tuned theory-heavy grammar coding for parsing and generating human language got the field basically nowhere.

IanCal 3y ago

> As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). You wouldn't need to, that's the point here. You let the LLM work on generating semantically valid responses and use a tool like this to restrict it to syntactically correct ones. Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). Let the LLM deal with coming up with a name and backstory that work, making sure the description and type of the weapon make sense (gpt4 suggested a close range carrot dagger for example), and let this work as your type structure. { "type": "object", "title": "character", "properties": { "backstory": { "type": "string" }, "weapons": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "description": { "type": "string" }, "weapon_type": { "type": "string", "enum": ["ranged", "close", "magic"] }, "range": { "minimum": 0, "maximum": 150 }, "damage": { "type": "number" } }, "required": [ "name", "description", "range", "damage" ] } }, "name": { "type": "string" } }, "required": [ "backstory", "weapons", "name" ] } > Then what? What do you mean by "random string"? Nonsense. Like "Colorless green ideas sleep furiously" the famous sentence that's grammatically correct but utter nonsense. > Plus, if you already have the grammar that can cover the anthropomorphic vegetable world it's only a bit more work to use it to parse such natural language requests, anyway. I really do not think this is the case. Parsing and understanding arbitrary requests about something like this?

YeGoblynQueenne 3y ago
>> Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). That'd be nice, but it's not how this tool works. If you look at the repo, there's an example of following a json schema or pydantic model. It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: class Weapon(str, Enum): sword = "sword" axe = "axe" mace = "mace" spear = "spear" bow = "bow" crossbow = "crossbow" But perhaps I'm underestimating the tool's capabilities. If so, hopefully remilouf can correct me (and give an example of how the tool can be made to work as you want it). >> I really do not think this is the case. Parsing and understanding arbitrary requests about something like this? Not arbitrary. See my casting-to-type analogy. The point I'm trying really hard to get across is that generating free-form text is all nice and cool, but if you want to give it structure, you need to have the entire structure defined before-hand, otherwise the text that can't be made to conform to it simply won't. So if you haven't got anthropomorphic vegetables in your json schema, your LLM may generate them, they'll never end up in your json.
- remilouf 3y ago
  You can definitely let the model improvise by defining `weapon` as `Union[Weapon, str]` if that's what you're asking.
- IanCal 3y ago
  > It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: No, only if you want to explicitly limit it to a set of options. You can have freeform fields, just like the jsonschema I provided. If you look at the example there's a character name which has a constrained length but is not limited to a set of options: class Character(BaseModel): name: constr(max_length=10) age: int armor: Armor weapon: Weapon strength: int The name there can be anything you want. This tool is, unfortunately, outrageously slow so I put the json schema above with a few fixes into jsonformer and downloaded a small model and used it to convert the GPT4 description into valid json: { "backstory":"Born in the tranquil meadows of Veggie", "weapons":[ { "name":"Leek Lance", "description":"A long, green and white lance made from a leek", "weapon_type":"distance", "range":100.0, "damage":75.0 }, { "name":"Carrot Dagger", "description":"A short, pointed dagger. It's sharp", "weapon_type":"close", "range":10.0, "damage":50.0 } ], "name":"Sir Turnip Thistlebrook" } > Not arbitrary. Well exactly. If you want to support arbitrary requests while constraining the output, tools like this are an easy approach and I'm not sure what else comes close. An interactive character design flow would have something like the above as the defined output and you could just keep asking for alterations as a human would ("make it more whimsical" or "not a king, something lower class") and have useful structured output > See my casting-to-type analogy. The point I'm trying really hard to get across is that generating free-form text is all nice and cool, but if you want to give it structure, you need to have the entire structure defined before-hand, otherwise the text that can't be made to conform to it simply won't. The structure, sure. But the content can be extremely varied.