When we found that BI was statistically inconsistent and the cause was integrating over branch lengths, we knew this result would be controversial, so we wanted to be sure the experiments were truly airtight.
The insistence of integration over branch lengths bothered me since I first read this paper. Of course, I was not the only sceptic out there. Joe Felsenstein, who is moderately anti-Bayesian, voiced his concerns at dechronization. Surely enough, always vigilant Mike Steel found a very serious mathematical mistake in Kolaczkowski and Thornton analysis. The authors wanted to streamline their computations by integrating out branch lengths from the posterior, but completely messed up the integral. Here is the correction posted by Thornton on PLoS ONE: Correction: BI is biased but not asymptotically inconsistent on resolved trees.
Although Thornton tries to downplay the mistake in his comment on PLoS ONE, this is a huge blow to the authors' argument. Let me address bias and inconsistency separately here.
Bayesian inconsistency. This was the sensational claim, but it did not hold the water. The authors admit that they've got nothing on Bayesian inconsistency.
Bayesian bias. The authors claim that Bayesian analysis is biased in the case of finite sample size. Big deal, so is the maximum likelihood. OK, the authors claim that Bayesian estimates are more biased than the mles. I doubt this is true in general. Simulation studies are not capable of examining all possible parameter values and choices of priors. I have a bigger problem with this claim though. What is phylogenetic bias? Bias is defined as the difference between the expected value of the estimator and the true parameter value. I rephrase my question. What is the mean of a phylogenetic tree? Even if we agree to ignore branch lengths and concentrate on the branching order or topology, we still have a problem of defining the mean of a categorial random variable...
Some final thoughts:
- I predict that this article will have zero impact on the field, contrary to Thornton's hopes: "We think this paper has profound implications for phylogenetic practice and theory." With Bayesian inconsistency debunked, the rest of paper's results are merely expected if not trivial.
- There is nothing wrong with Kolaczkowski and Thornton messing up on integration. Anyone can make a mistake. What bothers is me is the obvious desire of the authors to publish a controversial paper. They seem to be so eager to prove everyone wrong that they become unnecessarily defensive: "Our experience with the review process in phylogenetic methods, unfortunately, is that many reviewers evaluate manuscripts based on whether or not the results confirm their world-view. This is a legacy of decades of internecine warfare in the field between the adherents of different methodological camps. We write papers in other fields, and while peer-review always has its ups and downs, our experience in phylogenetics is unusual in that solid papers are often rejected for philosophical reasons rather than for reasons of scientific validity and quality." I presume Thornton is saying that criticizing Bayesian inference is mauvais ton in phylogenetic circles. Anyone familiar with the literature knows better than that.
- Biologists are more eager to engage in high risk/high reward behavior than statisticians or machine learning researchers. Kolaczkowski and Thornton's claim of Bayesian inconsistency is equivalent to a statistician questioning properties of Lebesgue integration. For that, Kolaczkowski and Thornton deserve to be alpha dogs of the week on The Colbert Report.
- Despite the spectacular gaffe by Kolaczkowski and Thornton, practitioners should not stop questioning conventional statistical wisdom. Bayesian inference can indeed be inconsistent, maximum likelihood is not bullet proof either.
