That's true, but my argument is that we can basically guarantee that a simple neural net won't become self-aware, in the same way that we know that a linear reg

That's true, but my argument is that we can basically guarantee that a simple neural net won't become self-aware, in the same way that we know that a linear regression library won't become self-aware. The complexity is bounded, just like it is in the experiments described in this paper.