3 ms·

This seems like it has some potential, but is pretty much useless as it is. Shame there are no weights released - let alone the "compiler" tool they used to ac

This seems like it has some potential, but is pretty much useless as it is. Shame there are no weights released - let alone the "compiler" tool they used to actually synthesize computational primitives into model weights. It seems like a "small model" system that's amenable to low budget experiments, and I would love to see what this approach can be pushed towards. I disagree with the core premise, it's basically the old neurosymbolic garbage restated, but embedding predefined computational primitives into LLMs could have some uses nonetheless.

yorwba 7mo ago
If you want to experiment with hardcoding small programs into transformer weights, maybe try ALTA: https://arxiv.org/abs/2410.18077v2 https://arxiv.org/abs/2410.18077v2
- ACCount37 7mo ago
  I'm less interested in turning programs into transformers and more interested in turning programs into subnetworks within large language models. Which the blog post brings up as a research direction, but never actually elaborates upon. And the interface between the two is a hard problem. I'll check out the link though, thanks.
YeGoblynQueenne 7mo ago
What's "the old neurosymbolic garbage"?
- ACCount37 7mo ago
  Basically, a holdover from the days of symbolic AI, from back when neural network ML wasn't the dominant AI paradigm. Some people in the "symbolic AI" camp didn't take the loss well, so they pivoted towards "ML is not real AI and it needs a symbolic component to be a real AI", which is: the neurosymbolic garbage. This work isn't exactly that, and I do think it can amount to something useful, but the justification for it reeks of something similar.
  - YeGoblynQueenne 7mo ago
    Full disclosure: all my published work is on symbolic machine learning (a.k.a. Inductive Logic Programming) :O I think you're confusing various different things as "neurosymbolic AI". There is a NeSy symposium and I happen to have met many of the people there, and they are not GOFAI ideologues, rather they recognise the obvious limitations of neural nets (i.e. they're crap at deduction, though great at induction) and they look for ways to address them. Most of that crowd also has a predominantly statistical ML/ neural nets background, with symbolic AI as an afterthought. I don't think I've ever heard anyone say that "ML is not real AI" and I mainly move in symbolic AI circles. I would check my sources, if I were you. Anwyay, honestly, this is 2026, there is no sensible reason to be polarised about symbolic vs. statistical AI (or whatever distinction anyone wants to make). An analogy I like to make is as follows: a jetliner is a flying machine, a helicopter is a flying machine. We can use both for their advantages and disadvantages, but a flying machine is something too useful to give up on any one kind for ideological reasons. The practical benefits overwhelmingly make up for any ideological concerns (e.g. "jets bad" or "propellers bad"). And just to be clear, symbolic AI is still in rude health: automated theorem proving, planning and scheduling, program verification and model checking, constraint satisfaction, discrete optimisation, SAT solving, all those are fields where symbolic approaches are dominant, and where neural nets have not made significant inroads in many decades; nor are they likely to, not any more than symbolic approaches are likely to make any inroads in e.g. machine vision, or speech recognition. And that's just fine: lots of tools, lots of problems solved.
    - ACCount37 7mo ago
      I don't think symbolic approaches are completely useless. It's just that they're solving yesterday's problems 1.12% better. While ML is cracking open entirely new fields - and might go all the way to AGI, the way it's going now. One is near the end of its potential while another is only picking up steam. In many ways, the space ML dominates now is the space of "all the things symbolic approaches suck ass at". Which is a very wide space with many desirable things in it.
      - nextos 7mo ago

yorwba 7mo ago

If you want to experiment with hardcoding small programs into transformer weights, maybe try ALTA: https://arxiv.org/abs/2410.18077v2 https://arxiv.org/abs/2410.18077v2

ACCount37 7mo ago
I'm less interested in turning programs into transformers and more interested in turning programs into subnetworks within large language models. Which the blog post brings up as a research direction, but never actually elaborates upon. And the interface between the two is a hard problem. I'll check out the link though, thanks.

YeGoblynQueenne 7mo ago

What's "the old neurosymbolic garbage"?

ACCount37 7mo ago
Basically, a holdover from the days of symbolic AI, from back when neural network ML wasn't the dominant AI paradigm. Some people in the "symbolic AI" camp didn't take the loss well, so they pivoted towards "ML is not real AI and it needs a symbolic component to be a real AI", which is: the neurosymbolic garbage. This work isn't exactly that, and I do think it can amount to something useful, but the justification for it reeks of something similar.
- YeGoblynQueenne 7mo ago
  Full disclosure: all my published work is on symbolic machine learning (a.k.a. Inductive Logic Programming) :O I think you're confusing various different things as "neurosymbolic AI". There is a NeSy symposium and I happen to have met many of the people there, and they are not GOFAI ideologues, rather they recognise the obvious limitations of neural nets (i.e. they're crap at deduction, though great at induction) and they look for ways to address them. Most of that crowd also has a predominantly statistical ML/ neural nets background, with symbolic AI as an afterthought. I don't think I've ever heard anyone say that "ML is not real AI" and I mainly move in symbolic AI circles. I would check my sources, if I were you. Anwyay, honestly, this is 2026, there is no sensible reason to be polarised about symbolic vs. statistical AI (or whatever distinction anyone wants to make). An analogy I like to make is as follows: a jetliner is a flying machine, a helicopter is a flying machine. We can use both for their advantages and disadvantages, but a flying machine is something too useful to give up on any one kind for ideological reasons. The practical benefits overwhelmingly make up for any ideological concerns (e.g. "jets bad" or "propellers bad"). And just to be clear, symbolic AI is still in rude health: automated theorem proving, planning and scheduling, program verification and model checking, constraint satisfaction, discrete optimisation, SAT solving, all those are fields where symbolic approaches are dominant, and where neural nets have not made significant inroads in many decades; nor are they likely to, not any more than symbolic approaches are likely to make any inroads in e.g. machine vision, or speech recognition. And that's just fine: lots of tools, lots of problems solved.
  - ACCount37 7mo ago
    I don't think symbolic approaches are completely useless. It's just that they're solving yesterday's problems 1.12% better. While ML is cracking open entirely new fields - and might go all the way to AGI, the way it's going now. One is near the end of its potential while another is only picking up steam. In many ways, the space ML dominates now is the space of "all the things symbolic approaches suck ass at". Which is a very wide space with many desirable things in it.
    - nextos 7mo ago