Yeah, i think that neural networks learn because they can represent if/else relation (via step function (activation)) and that makes them learn good heuristics

Yeah, i think that neural networks learn because they can represent if/else relation (via step function (activation)) and that makes them learn good heuristics at least. I wonder if someone can prove that NN can generate arbitrary programs because as we know 2 layer NNs can approxomate any calculation. Given some state and how easy is to create turing complete machine i think they already might have it. I speculate that the trick is that these programs sometimes dont evaluate in time when response is short. That’s why LLMs should be given way to adjust output and something like stop signal. Adjusting output can be done using DiffusER approach but not sure how to handle stop signal. I guess we could try to add fake input tokens for silience/thinking.