There is a long tradition of representing causal relationships by directed acyclic graphs (Wright 1934). Spirtes (1992) and Spirtes, Glymour and Scheines (1993) using some ideas in Pearl and Verma (1991) describe procedures for inferring the presence or absence of causal arrows in the graph even if there might be unobserved confounding variables, and/or an unknown time order, and that under weak conditions, for certain combinations of DAGs and probability distributions, are asymptotically (in sample size) consistent. These results are surprising since they seem to contradict the standard statistical wisdom that consistent estimates of causal effects do not exist for non-randomized studies if there are potentially unobserved confounding variables.
We resolve the apparent incompatibility of these views by closely examining the asymptotic properties of these causal inference procedures. We show that the asymptotically consistent procedures are "pointwise consistent" but "uniformly consistent" tests do not exist. Thus, no finite sample size can ever be guaranteed to approximate the asymptotic results. We also show the non-existence of valid, consistent confidence intervals for causal effects and the non-existence of uniformly consistent point estimates. Our results make no assumptions about the form of the tests or estimates. In particular, the tests could be classical independence tests, they could be Bayes tests or they could be tests based on scoring methods such as BIC or AIC. The implications of our results for observational studies are controversial and are discussed briefly in the last section of the paper.
The results hinge on the following fact: it is possible to find, for each sample size n, distributions P and Q such that: P and Q are empirically indistinguishable and yet P and Q correspond to different causal effects.