Discussion on the meeting on ‘Signs and sizes:understanding and replicating statistical findings’

Two‐sided tests are often used by ‘experts’ as well as ‘non‐experts’ to verify a certain hypothesis. I congratulate Rice, Bonnett and Krakauer for rejuvenating this topic by giving an alternative decision theoretic approach by viewing the signs of an underlying parameter and making a decision based on whether the parameter is positive or negative. The whole procedure looks very complicated and it seems that plenty of subjectivity is involved, which is a little difficult to interpret. For example, how do we set an optimal value of a which minimizes the risk in testing the significance? Killeen (2006) proposed a decision‐theory‐based approach for hypothesis testing which calculates the expected utility of an effect on the basis of the probability of replicating it. However, it seems like it was only a theoretical contribution.

Rice and his colleagues have not given any examples to show how the results may be different from conventional statistical tests or a p‐value approach. Conventional statistical tests look more transparent and can lead to further analysis like confidence intervals, which are lacking in this approach.