4 ms·

And where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some

by 112233 2y ago

And where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some stereotypical character>". Absolutely no awareness about anything spatial - what is in which place, who is where, body positions. Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that. (edit) just asked GPT-4o to make a photo of hands tying two ropes (my favorite way to see how bad actually gen-ai is) - yup, it has an odd number of ropes coming out of the knot. That is state of art supposedly.

baq 2y ago
> And where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some stereotypical character>". > Absolutely no awareness about anything spatial - what is in which place, who is where, body positions. These are some things I'd like to at least attempt to quantify - LLMs are better at translation than thinking[0], so the idea was to translate the slop into something that can be then analyzed by a non-LLM tool. E.g. translate the story into a prolog fact database and generate a couple queries for each paragraph/chapter/combination of these. Rough idea, just something I haven't seen done. [0] don't @ me
thaumasiotes 2y ago
> Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that. I could see this working as comedy, though it'd be better on film than in print. Either way you'd have to establish a very specific tone in the surrounding material.
- 112233 2y ago
  This comedy is a good comedy while I am goofing around with "make a story". However, this wordpuker tech is positioned at solving strategic problems, writing software and (something equally serious). It has no mechanism to build a world model "in head" while it generinates. chess pieces teleport on board. people hug through doors. any exercise is a body horror. do you think this tech deals with invariants in software algorithms any better? I mean, haha but not funny.

by 112233 2y ago

baq 2y ago

> And where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some stereotypical character>". > Absolutely no awareness about anything spatial - what is in which place, who is where, body positions. These are some things I'd like to at least attempt to quantify - LLMs are better at translation than thinking[0], so the idea was to translate the slop into something that can be then analyzed by a non-LLM tool. E.g. translate the story into a prolog fact database and generate a couple queries for each paragraph/chapter/combination of these. Rough idea, just something I haven't seen done. [0] don't @ me

thaumasiotes 2y ago

> Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that. I could see this working as comedy, though it'd be better on film than in print. Either way you'd have to establish a very specific tone in the surrounding material.

112233 2y ago
This comedy is a good comedy while I am goofing around with "make a story". However, this wordpuker tech is positioned at solving strategic problems, writing software and (something equally serious). It has no mechanism to build a world model "in head" while it generinates. chess pieces teleport on board. people hug through doors. any exercise is a body horror. do you think this tech deals with invariants in software algorithms any better? I mean, haha but not funny.