3 ms·

I've noticed that with clojure(script) unless you specifically instruct them to keep nesting levels low, they can hit a point where they make a paren placement

I've noticed that with clojure(script) unless you specifically instruct them to keep nesting levels low, they can hit a point where they make a paren placement error and can't debug their way out of it. Although in my case while one model made the error then couldn't find what it had done, a second model that I switched to was then able to identify it and back it out. So I suspect this is a transient weakness in today's models, not something fundamental.

forlorn_mammoth 5mo ago
That's because you are holding it wrong. Just replace the ( with rs, like in strawberry.
isityettime 5mo ago
It's a bit of a pitiful way to fail. I wonder if diffusion models could handle parenthesis matching better. And I wonder if you could rig up tools for structural editing like with paredit.
- regularfry 5mo ago
  It's one of the drawbacks of having quite so little syntax. There's just less to grab hold of.
chamomeal 5mo ago
I don’t know, I think it might be fundamental. They think in tokens, and )))))))) might look equivalent to ))))))). Just like the strawberry problem. The calva backseat driver extension even includes a specific paren balancer for this reason, and it works quite well
- regularfry 5mo ago
  It's fundamental in that it's harder because there's less information per token. But we know it's not impossible because they can get nesting right at all, it's just a question of where the boundary is today. And if different models have different crapping-out points, then there's a gradient there and future models can do better. In token terms it's more like the fingers problem than the strawberry problem. ")" is a single token, but the model gets confused by several repeats of the same thing.
iLemming 5mo ago
> can't debug their way out of it Whoa, it seems you're using LLM to generate Clojure code like you'd do it with any other "static" PL. Give it a live REPL - it works wondrously.

forlorn_mammoth 5mo ago

That's because you are holding it wrong. Just replace the ( with rs, like in strawberry.

isityettime 5mo ago

It's a bit of a pitiful way to fail. I wonder if diffusion models could handle parenthesis matching better. And I wonder if you could rig up tools for structural editing like with paredit.

regularfry 5mo ago
It's one of the drawbacks of having quite so little syntax. There's just less to grab hold of.

chamomeal 5mo ago

I don’t know, I think it might be fundamental. They think in tokens, and )))))))) might look equivalent to ))))))). Just like the strawberry problem. The calva backseat driver extension even includes a specific paren balancer for this reason, and it works quite well

regularfry 5mo ago
It's fundamental in that it's harder because there's less information per token. But we know it's not impossible because they can get nesting right at all, it's just a question of where the boundary is today. And if different models have different crapping-out points, then there's a gradient there and future models can do better. In token terms it's more like the fingers problem than the strawberry problem. ")" is a single token, but the model gets confused by several repeats of the same thing.

iLemming 5mo ago

> can't debug their way out of it Whoa, it seems you're using LLM to generate Clojure code like you'd do it with any other "static" PL. Give it a live REPL - it works wondrously.