3 ms·

A neural network with a hidden layer can approximate F=GMm/r^2 if given appropriate training input. I'm not clear on what you're saying. Is it that LLMs specif

A neural network with a hidden layer can approximate F=GMm/r^2 if given appropriate training input. I'm not clear on what you're saying. Is it that LLMs specifically don't have this same property of being able to approximate such functions? Is it that a neural network wouldn't learn that model if you gave it real world measurements (because I think it would, but such a thing should be fairly easily testable)?

mjburgess 3y ago
most formulas, including that one, are neural networks (such is the absurdity of the term) -- so it is trivially learnable The issue is that to prepare the dataset from which that formula is learnt requires already knowing it. This is the triviality of applications of universal function approximators to science -- empirical data modelling isnt new, and neural networks are just one example of it; not all that special. All observational data on the solar system, at any point in time, would not yield this formula via empirical function approximation. there really isnt "observational data" to collect, in this sense This is what I mean about rigging -- there is no 'bare dataset' which tells you what the world is like. to construct experiments which yield data that 'presents' scientific laws as if statistical patterns requires millenia of theory-building science science uncovers the necessary causal relations between objects and their properties, as determined by extremely controlled experiments which take millenia of engineering and theory-building to even conceive, let alone execute stats is the dumb 'accounting' of this data -- by the time you actually have it all the science (and indeed, all the intelligence) is done what can be automated at this point is 'stamp collecting' as rutherford said
- kybernetikos 3y ago
  > The issue is that to prepare the dataset from which that formula is learnt requires already knowing it. Why is that? Are you saying that universal function approximators can't invent higher level models? Isn't that what happens in the hidden layers when an neural network is trying to predict things that require those higher level models?
  - mjburgess 3y ago
    No, consider how you would collect a dataset to show F=GMm/r^2 you'd need to measure the force between planets, the distance between them, etc. no such direct measurements have taken place to my knowledge, certainly not at newton's time All "universal fn approximator" means is that given any dataset which is sampled from a function, you can recover that function with enough data points. It does not mean, eg., that given a 2D dataset you can derive a 3D function -- you cannot, this is impossible. So you need to be sampling from F=GMm/r^2 to find that function. So you need to know the answer before you begin. Fn approximators are only useful for empirical refinements to existing knowledge. In order to construct such datasets you need to do science; hence experiments, etc.

mjburgess 3y ago

most formulas, including that one, are neural networks (such is the absurdity of the term) -- so it is trivially learnable The issue is that to prepare the dataset from which that formula is learnt requires already knowing it. This is the triviality of applications of universal function approximators to science -- empirical data modelling isnt new, and neural networks are just one example of it; not all that special. All observational data on the solar system, at any point in time, would not yield this formula via empirical function approximation. there really isnt "observational data" to collect, in this sense This is what I mean about rigging -- there is no 'bare dataset' which tells you what the world is like. to construct experiments which yield data that 'presents' scientific laws as if statistical patterns requires millenia of theory-building science science uncovers the necessary causal relations between objects and their properties, as determined by extremely controlled experiments which take millenia of engineering and theory-building to even conceive, let alone execute stats is the dumb 'accounting' of this data -- by the time you actually have it all the science (and indeed, all the intelligence) is done what can be automated at this point is 'stamp collecting' as rutherford said

kybernetikos 3y ago
> The issue is that to prepare the dataset from which that formula is learnt requires already knowing it. Why is that? Are you saying that universal function approximators can't invent higher level models? Isn't that what happens in the hidden layers when an neural network is trying to predict things that require those higher level models?
- mjburgess 3y ago
  No, consider how you would collect a dataset to show F=GMm/r^2 you'd need to measure the force between planets, the distance between them, etc. no such direct measurements have taken place to my knowledge, certainly not at newton's time All "universal fn approximator" means is that given any dataset which is sampled from a function, you can recover that function with enough data points. It does not mean, eg., that given a 2D dataset you can derive a 3D function -- you cannot, this is impossible. So you need to be sampling from F=GMm/r^2 to find that function. So you need to know the answer before you begin. Fn approximators are only useful for empirical refinements to existing knowledge. In order to construct such datasets you need to do science; hence experiments, etc.