4 ms·

Causal. Imagine a video. Each frame is plotted in space, so instead of a sequence of 2d images, you have a cube. The z axis of the cube shows each frame of the

Causal. Imagine a video. Each frame is plotted in space, so instead of a sequence of 2d images, you have a cube. The z axis of the cube shows each frame of the video in turn. This is an equivalent representation to the representation that we are used to. We have just switched the time dimension for a physical dimension. What you are reverently calling causality, is only correlation along the z spacial dimension. In the same way an image can be compressed due to the similarities across it's 2 dimensions, so can a 3d representation of video be compressed along all 3 of it's axis. An appreciation of causality is only induction and induction is only a correlation, if a then probably b. Finding correlations in one of the dimensions you are more familiar with, like the pixels of an image is no different to finding it in time. Within language the elements of time are encoded, just as they are within the video represented as a cube. "The boy threw the ball, the ball landed and then rolled." Just because the time dimension is represented to the neural network within the input parameters of a single iteration of the neural network, does not make it any less able to understand correlations across time.

mjburgess 6y ago
So this is to repeat hume and mistake causal analysis as a kind of induction or inference. It isnt. Our bodies are the primary site of "causal analysis". Eg., when I touch the hot surface of a stove I do not infer that temperature is a cause of my hand being hot. Such an inference, as the basis for our models of the world, would be -- as you/hume/etc. say -- deeply insufficient. The operation of the world on our bodies is already laden with causal information. A hand striking a face does not just create a "painful sense impression"... rather the body encodes it as caused by the object you also saw. It is the action of the world upon the body that is the bedrock of our causal model building. Scientific/inferential processes sit on top of that to disambiguate between possible distal causes. A machine, as a person, given "mere data" cannot hope to do much.
- hackinthebochs 6y ago
  None of this explains how modelling time sequences of events does not sufficiently approximate the kind of causal knowledge you mention.
  - mjburgess 6y ago
    Suppose I place a pot on a stove and the water boils. Now I feed into the machine what data? Here, it gets this: all gravitational, electomagnetic, etc data within 1km; all geometric information about all objects within 1km (, and all of their properties, etc.) Now, machine, what caused the pot to boil? It has no clue. There are an infinite number of antecedent temporal events. The problem isn't the mathematics of causal inference. The problem is relevance. That isn't an inferential problem.
    - hackinthebochs 6y ago
      >It has no clue. There are an infinite number of antecedent temporal events. Right, determining the cause of the pot boiling given only local information about this one event is impossible. But that's not a good representation of learning from a real-world data set. A real training corpus might also have an example of a pot boiling in a campfire, a water heater heating up water using fire, someone touching an open flame and going "ouch!", fire burning down a house, etc. All these examples taken together can reasonably lead one to infer that fire causes things to become hot. This is the kinds of regularity found in real world datasets and a good learning algorithm will extract this regularity in the course of predicting the dataset.
      - mjburgess 6y ago
        Who prepares the dataset? You are just shifting relevance problem to the human to solve. That's my point. The "regularity" isn't very hard to find when the data which exposes it is already selected.
        hackinthebochs 6y ago
        No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficiently large training corpus will naturally have this sort of regularity in it. My point is that a general learning algorithm trained on a sufficiently large and representative corpus will capture causal regularity without the sort of fine-tuning of the algorithm or the training data you are suggesting. You haven't given any reason to think otherwise.

mjburgess 6y ago

So this is to repeat hume and mistake causal analysis as a kind of induction or inference. It isnt. Our bodies are the primary site of "causal analysis". Eg., when I touch the hot surface of a stove I do not infer that temperature is a cause of my hand being hot. Such an inference, as the basis for our models of the world, would be -- as you/hume/etc. say -- deeply insufficient. The operation of the world on our bodies is already laden with causal information. A hand striking a face does not just create a "painful sense impression"... rather the body encodes it as caused by the object you also saw. It is the action of the world upon the body that is the bedrock of our causal model building. Scientific/inferential processes sit on top of that to disambiguate between possible distal causes. A machine, as a person, given "mere data" cannot hope to do much.

hackinthebochs 6y ago
None of this explains how modelling time sequences of events does not sufficiently approximate the kind of causal knowledge you mention.
- mjburgess 6y ago
  Suppose I place a pot on a stove and the water boils. Now I feed into the machine what data? Here, it gets this: all gravitational, electomagnetic, etc data within 1km; all geometric information about all objects within 1km (, and all of their properties, etc.) Now, machine, what caused the pot to boil? It has no clue. There are an infinite number of antecedent temporal events. The problem isn't the mathematics of causal inference. The problem is relevance. That isn't an inferential problem.
  - hackinthebochs 6y ago
    >It has no clue. There are an infinite number of antecedent temporal events. Right, determining the cause of the pot boiling given only local information about this one event is impossible. But that's not a good representation of learning from a real-world data set. A real training corpus might also have an example of a pot boiling in a campfire, a water heater heating up water using fire, someone touching an open flame and going "ouch!", fire burning down a house, etc. All these examples taken together can reasonably lead one to infer that fire causes things to become hot. This is the kinds of regularity found in real world datasets and a good learning algorithm will extract this regularity in the course of predicting the dataset.
    - mjburgess 6y ago
      Who prepares the dataset? You are just shifting relevance problem to the human to solve. That's my point. The "regularity" isn't very hard to find when the data which exposes it is already selected.
      - hackinthebochs 6y ago
        No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficiently large training corpus will naturally have this sort of regularity in it. My point is that a general learning algorithm trained on a sufficiently large and representative corpus will capture causal regularity without the sort of fine-tuning of the algorithm or the training data you are suggesting. You haven't given any reason to think otherwise.