I have recently found an article written by James O. Berger and Donald A. Berry, and its title is “Statistical Analysis and the Illusion of Objectivity“. It was published in American Scientist, Mar-Apr 1988 issue, Vol. 76 Issue 2, p. 159-165. I think I discovered this article in a blog entry written by Andrew Gelman (or in one of the comments to that entry).
I include short fragments from this critical article because I had my share of P-values both as a consumer of scientific articles and producer of them so I believe it is very important to question the fundamental methods of making inferences from data, and find better ways of doing that if possible at all. That is finding better ways to do science.
“Acknowledging the subjectivity of statistical analysis would be healthy for science as a whole for at least two reasons. The first is that the straightforward methods of subjective analysis, called Bayesian analysis, yield answers which are much easier to understand than standard statistical answers, and hence much less likely to be misinterpreted. This will be dramatically illustrated in our first example.
The second reason is that even standard statistical methods turn out to be based on subjective input — input of a type that science should seek to avoid. In particular, standard methods depend on the intentions of the investigator, including intentions about data that might have been obtained but were not. This kind of subjectivity is doubly dangerous. First, it is hidden; few researchers realize how subjective standard methods really are. Second, the subjective input arises from the producer rather than the consumer of the data — from the investigator rather than the individual scientist who reads or is told the results of the experiment.
…
Many nonstatisticians mistakenly think that a P-value is the probability of the null hypothesis or, equivalently the probability that one is making an error in rejecting the hypothesis. The example of the vitamin C experiment provides a dramatic demonstration that this is not the case.
…
Where does this initial probability come from? The answer is simple: it must be subjectively chosen by the person interpreting the data. A person who doubts the hypothesis initially might choose a probability of 0.1; by contrast, someone who believes in it might choose 0.9. We would argue that a consideration of such initial probabilities is unavoidable in reaching a conclusion about the truth of H. The investigator, however, need not be concerned with the initial probability chosen by possible consumer of data; it suffices for the investigator to show how the data will change this initial probability into a final probability. The mechanism for doing this is called the Bayes factor, which is essentially the odds against the hypothesis provided by the data.
…
The reason P-values can be very deceptive is that they involve probabilities of the unobserved data that are more extreme than the observed data. The actual data for the vitamin C experiment were 13, yet the P-value in effect pretends that the set of data shown in color in Figure 1 is that data. The logic behind this step is weak, and we have now seen that the conclusion can be misleading.
To be fair, we should point out that standard statisticians constantly reiterate that a P-value is not to be interpreted as a final probability. But consumers of data want a final probability; they want to know how probable it is that the hypothesis is true in the light of the data. Since standard statistics cannot answer this question, and indeed gives no guidance in translating P-values into an answer to this question, it is difficult to blame consumers for taking the number provided — the P-value — and interpreting as an answer.
…
The advantages of acknowledging the role of subjectivity and adopting Bayesian methods are substantial. Bayesian probabilities can be calculated as the experiment proceeds and reported to others at any time. … Finally, in many problems arising in fields such as medicine, business, or engineering, it can be vitally important to involve the subjective information possessed by the decision-maker, who is often an expert in the area; Bayesian analysis is ideally suited for this task.
…
Statistical analysis plays a central role in scientific inquiry. The adoption of today’s statistical methods has led to enormous improvements in the understanding of experimental evidence. But common usage of statistics seems to have become fossilized, mainly because of the view that standard statistics is the objective way to analyze data. Discarding this notion, and indeed embracing the need for subjectivity through Bayesian analysis, can lead to more flexible, powerful, and understandable analysis of data.”
FZ Blogs 