2 ms·
Makes sense. In my experience, LLMs are not very good at maintaining deterministic state machines internally either. Our team did an experiment trying use an LL
by daxfohl 1y ago
Makes sense. In my experience, LLMs are not very good at maintaining deterministic state machines internally either. Our team did an experiment trying use an LLM to guide users through a four step signup flow, and gave it very explicit instructions what order to collect information, rules for validating it, under what conditions a step could be skipped, how many times each step could be retried. The rules were highly simplified, just as a proof of concept. Still, it would get lost half the time. So yeah, I have a hard time imagining it would do any better managing the full state of some RPG it made up
- duskwuff 1y agoI'm reminded of the infamous vending machine benchmark: https://arxiv.org/abs/2502.15840 https://arxiv.org/abs/2502.15840 in which some LLMs simply failed to manage the basic task of keeping a vending machine stocked with products to sell, and some went completely off the rails, e.g. by threatening their (simulated) supplier with "ABSOLUTE FINAL ULTIMATE TOTAL NUCLEAR LEGAL INTERVENTION".