4 ms·

No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficie

No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficiently large training corpus will naturally have this sort of regularity in it. My point is that a general learning algorithm trained on a sufficiently large and representative corpus will capture causal regularity without the sort of fine-tuning of the algorithm or the training data you are suggesting. You haven't given any reason to think otherwise.

mjburgess 6y ago
identifying causes cannot be done statistically, as a fact of statistics events A then B then C do not imply C is caused by B is caused by A This problem is worse the more data you have, as with my example above of giving the machine /every/ event within a 1km radius of a pot boiling. Identifying a cause is a dynamic experimental process. A machine can only accept a highly prepared dataset which has already been chosen because the associations are known to be causal.
- hackinthebochs 6y ago
  But this problem isn't specific to machines. The fact that my hand hurts after touching an open flame stove doesn't entail the flame caused my pain. No amount of first-hand experience with flames can entail that flames cause pain. All we can do is increase the likelihood of this model. Experimental processes are in the same boat, except that the statistical power is greater. But our lived experiences with flames and pain and all other stimuli is a sort of low power ongoing experiment. But with enough of these poor experimental runs we can converge on an approximately true model. The same goes for a statistical learning algorithm and a large training corpus.
  - mjburgess 6y ago
    Yes but the body solves the problem of concluding that the stove caused the pain. The body is a mechanism of relevance. The more data you feed a machine, the worse it performs. The body isnt necessarily right about causation (though it often is) but it provides a minimal mechanism of relevance to enable inference. Data itself contains no causal information.
    - plutonorm 6y ago
      You are just pushing the solution to the problem of relevance (which isn't actually a real problem) down the stack, into the body. How does the body magically create this new kind of relation called cause? You are mistaking the image in your mind for reality. The image you see in your mind is the product of an extraordinarily large neural network that has done all the statistical analysis for you.

mjburgess 6y ago

identifying causes cannot be done statistically, as a fact of statistics events A then B then C do not imply C is caused by B is caused by A This problem is worse the more data you have, as with my example above of giving the machine /every/ event within a 1km radius of a pot boiling. Identifying a cause is a dynamic experimental process. A machine can only accept a highly prepared dataset which has already been chosen because the associations are known to be causal.

hackinthebochs 6y ago
But this problem isn't specific to machines. The fact that my hand hurts after touching an open flame stove doesn't entail the flame caused my pain. No amount of first-hand experience with flames can entail that flames cause pain. All we can do is increase the likelihood of this model. Experimental processes are in the same boat, except that the statistical power is greater. But our lived experiences with flames and pain and all other stimuli is a sort of low power ongoing experiment. But with enough of these poor experimental runs we can converge on an approximately true model. The same goes for a statistical learning algorithm and a large training corpus.
- mjburgess 6y ago
  Yes but the body solves the problem of concluding that the stove caused the pain. The body is a mechanism of relevance. The more data you feed a machine, the worse it performs. The body isnt necessarily right about causation (though it often is) but it provides a minimal mechanism of relevance to enable inference. Data itself contains no causal information.
  - plutonorm 6y ago
    You are just pushing the solution to the problem of relevance (which isn't actually a real problem) down the stack, into the body. How does the body magically create this new kind of relation called cause? You are mistaking the image in your mind for reality. The image you see in your mind is the product of an extraordinarily large neural network that has done all the statistical analysis for you.