4 ms·

The author doesn't know what causation is. Heck not even correlation establishes causation so OF course a counterfactual can't establish causation. Not only ar

The author doesn't know what causation is. Heck not even correlation establishes causation so OF course a counterfactual can't establish causation. Not only are counterfactuals infinite in number but ALL hypothesis are infinite in number. You can literally say ANYTHING about why something didn't build. I can say someone didn't purge some files or I can say someone purged files or I can say aliens destroyed the server with a laser. There are an infinite number of statements that can be made here and whether it's a "counterfactual" is irrelevant. What the author doesn't know is that NOTHING can establish causation except for a very specific type of experiment. Basically there is nothing that can establish causality UNLESS you have control over the causal switch. You flick the switch randomly and if the randomness of the switch flicking correlates with the consequence you have then have established causation. BUT you have only established it to a certain correlative degree. It is fundamentally impossible in this reality to establish causation to be 100%. True causation can never be fully verified and at best the experiment I described is the best fuzzy verification. If you truly study how science works you will find that this flaw extends to all of science and therefore reality. In reality as we know it, nothing can be proven. Proof is fundamentally impossible. If you understand this concept then you will truly understand what science is. Anyway, Going to the authors Kubernetes example, his "counterfactuals" are basically null hypothesis and can be helpful in establishing causation but by themselves are pointless. It's simple... If you hypothesize that something didn't build because the admins did not "configure file purging" then to establish causation is SIMPLE. Configure the file purging and see if your system builds. Then remove the configuration and see if the system stops building. Do this enough times and you have established causation or zero causation to a certain degree. That's it. This is science and probability and statistics. So crazy to see someone mention causality without even knowing what it really is.

zwaps 5y ago
You hint at it, but causation is a fuzzy concept even with a RCT. Causation is "established" in the usual sense by having a causal model of a data generating process. Such model may or may not identify causal parameters given the data. That is, it is not the "randomized control trial" itself that establishes causation, rather it is a more general concept of statistical identification. Case in point, there are instances where a RCT fails. For example, even if you can assign treatments, you might find that treatment and control groups influence each other or whatever else can go wrong in a RCT. By contrast, there are instances where we do not control the causal switch, but we can observe "natural experiments" nonetheless. Usually, causation is fuzzy, since there are at least some assumptions which need to be made for us to believe the analysis to be causal. However, really, the same is true for an RCT. For example, if you observe an engineer sometimes purging the cache, and sometimes not purging the cache, you do not actually control the assignment of treatment. However, it is easy to think of a mental model where observations of this engineer, and the state of the cluster, gives you high confidence to proclaim: "Purging the cache makes the cluster start!" So, this is just to expand on what you wrote: Causation is not a precise concept, because it depends on our model of the DGP and the data we actually have. Identification of causal parameters depend on these factors, and thereby on assumptions - this is true for RCTs where I control the treatment, just as much for observational studies where I do not. Even if I assign treatments, I have to make the assumption that my assignment is truly random, and that there is no non-random factors influencing my observations (etc). In quite a broad sense, causation as a scientific concept depends on assumptions I need to make. If I get to control the treatment variable (or variable of interest), it is typically easier to create "exogenous" variation that identifies causal parameters... in English: Allows me to establish causation. However, the whole concept does not depend on controlling or not controlling treatment. RCTs can fail, or there are instances in which I can establish and measure causation even without controlling the treatment.
- deltaonefour 5y ago
  Yeah you're right. It may be that I'm coming from a more philosophical perspective. What is randomness? If randomness involves free will then only an entity with free will can create that random variable aka only another person or yourself can create randomness. Pedant-ism aside, can you think of a "truly random" data generative process that doesn't involve someone with their hand on the switch? I guess this is the assumption I am making in my explanation: Only people with this magical thing called "free will" can generate random data, nothing else can be trusted to be truly random so you or another person has to have their hand on the switch in order to establish causality. Note that even your example stayed within the limits of this assumption. You were possibly unable to choose another example that didn't involve someone with free will. Either I have my hand on a switch, or I'm observing another engineer who has his hand on the switch. Typically to illustrate a point one would usually pick an outlandish example that strays very far from a human having his hand on a switch and show that the example is STILL true. But you didn't do this and that may mean something.
  - zwaps 5y ago
    I cannot really engage with your question. First, because I am not an expert in philosophy (a Philistine, if you will). Second, because it is not axial in the identification of causality, such as we normally think about it. You see, in the pragmatist view of a statistician (and in the math that goes with it), randomness - for example in the form of statistical independence - is not required to identify causality (or causal parameters of a model). Instead, we need the "switch" variable to be uncorrelated with things we do not observe, or can not adequately model. This requirement holds for a random experiment as much as for an observational study. However, you will quickly convince yourself that this is an assumption. It depends on the data and how we got it, it depends on what we assume about the factors at play, and it depends also on the model and how we take into account factors that influence the outcome. Specifically, the identifying assumptions for a causal model are untestable. Let me emphasize this again: Every causal analysis is fuzzy, because they are only causal based on the assumptions. As such, the question of true randomness is somewhat ancillary: Statisticians know that the idea of causality is by default not a philosophically "true" concept. This, of course, leads to a pragmatist view. One last point: Between free will and flipping a switch, and a causal analysis, stand many assumptions about the data, the measurement, and so forth. In that sense, statisticians probably do not feel that a random flip switching is less assumptive than any other causal analysis. Related to this, let me offer you the perspective of social scientists. They feel that human behavior IS influenced by other factors, just as a machine or physical system would be (broadly speaking). That is, they would reject the notion that free will is indeed (or at least always) random. In such studies, for example, choices from individuals would also be subject to intense scrutiny with respect to unobserved influences or other factors that might endanger the exogeneity of the observed choices. Causal identification has been likened to a Zoo, with many kinds of animals in them. I am sure the philosophical idea of causation is part of the Zoo. On the other hand, statisticians speaking of causal analysis find themselves unable to "prove" any sort of causality, even weak forms as indicated above, without underlying assumptions. Case in point, a randomized controlled trial is chock-full of identification and exclusion assumptions. They are just standardized and accepted. Sorry again that I have to rely on the statistical view - it is the only one I know a little bit about.

zwaps 5y ago

You hint at it, but causation is a fuzzy concept even with a RCT. Causation is "established" in the usual sense by having a causal model of a data generating process. Such model may or may not identify causal parameters given the data. That is, it is not the "randomized control trial" itself that establishes causation, rather it is a more general concept of statistical identification. Case in point, there are instances where a RCT fails. For example, even if you can assign treatments, you might find that treatment and control groups influence each other or whatever else can go wrong in a RCT. By contrast, there are instances where we do not control the causal switch, but we can observe "natural experiments" nonetheless. Usually, causation is fuzzy, since there are at least some assumptions which need to be made for us to believe the analysis to be causal. However, really, the same is true for an RCT. For example, if you observe an engineer sometimes purging the cache, and sometimes not purging the cache, you do not actually control the assignment of treatment. However, it is easy to think of a mental model where observations of this engineer, and the state of the cluster, gives you high confidence to proclaim: "Purging the cache makes the cluster start!" So, this is just to expand on what you wrote: Causation is not a precise concept, because it depends on our model of the DGP and the data we actually have. Identification of causal parameters depend on these factors, and thereby on assumptions - this is true for RCTs where I control the treatment, just as much for observational studies where I do not. Even if I assign treatments, I have to make the assumption that my assignment is truly random, and that there is no non-random factors influencing my observations (etc). In quite a broad sense, causation as a scientific concept depends on assumptions I need to make. If I get to control the treatment variable (or variable of interest), it is typically easier to create "exogenous" variation that identifies causal parameters... in English: Allows me to establish causation. However, the whole concept does not depend on controlling or not controlling treatment. RCTs can fail, or there are instances in which I can establish and measure causation even without controlling the treatment.

deltaonefour 5y ago
Yeah you're right. It may be that I'm coming from a more philosophical perspective. What is randomness? If randomness involves free will then only an entity with free will can create that random variable aka only another person or yourself can create randomness. Pedant-ism aside, can you think of a "truly random" data generative process that doesn't involve someone with their hand on the switch? I guess this is the assumption I am making in my explanation: Only people with this magical thing called "free will" can generate random data, nothing else can be trusted to be truly random so you or another person has to have their hand on the switch in order to establish causality. Note that even your example stayed within the limits of this assumption. You were possibly unable to choose another example that didn't involve someone with free will. Either I have my hand on a switch, or I'm observing another engineer who has his hand on the switch. Typically to illustrate a point one would usually pick an outlandish example that strays very far from a human having his hand on a switch and show that the example is STILL true. But you didn't do this and that may mean something.
- zwaps 5y ago
  I cannot really engage with your question. First, because I am not an expert in philosophy (a Philistine, if you will). Second, because it is not axial in the identification of causality, such as we normally think about it. You see, in the pragmatist view of a statistician (and in the math that goes with it), randomness - for example in the form of statistical independence - is not required to identify causality (or causal parameters of a model). Instead, we need the "switch" variable to be uncorrelated with things we do not observe, or can not adequately model. This requirement holds for a random experiment as much as for an observational study. However, you will quickly convince yourself that this is an assumption. It depends on the data and how we got it, it depends on what we assume about the factors at play, and it depends also on the model and how we take into account factors that influence the outcome. Specifically, the identifying assumptions for a causal model are untestable. Let me emphasize this again: Every causal analysis is fuzzy, because they are only causal based on the assumptions. As such, the question of true randomness is somewhat ancillary: Statisticians know that the idea of causality is by default not a philosophically "true" concept. This, of course, leads to a pragmatist view. One last point: Between free will and flipping a switch, and a causal analysis, stand many assumptions about the data, the measurement, and so forth. In that sense, statisticians probably do not feel that a random flip switching is less assumptive than any other causal analysis. Related to this, let me offer you the perspective of social scientists. They feel that human behavior IS influenced by other factors, just as a machine or physical system would be (broadly speaking). That is, they would reject the notion that free will is indeed (or at least always) random. In such studies, for example, choices from individuals would also be subject to intense scrutiny with respect to unobserved influences or other factors that might endanger the exogeneity of the observed choices. Causal identification has been likened to a Zoo, with many kinds of animals in them. I am sure the philosophical idea of causation is part of the Zoo. On the other hand, statisticians speaking of causal analysis find themselves unable to "prove" any sort of causality, even weak forms as indicated above, without underlying assumptions. Case in point, a randomized controlled trial is chock-full of identification and exclusion assumptions. They are just standardized and accepted. Sorry again that I have to rely on the statistical view - it is the only one I know a little bit about.