Statistical Significance and p-values

• Stephen Senn (Statistical Consultant, Edinburgh, Scotland, United Kingdom)
• Ron Kenett (KPA Group and Samuel Neaman Institute, Technion, Israel)
• Bernard Francq (GSK)
• Daniël Lakens (Eindhoven University of Technology, Netherlands)

This meeting is organized to present and discuss the issues listed in the title. It consists of 3 shorts presentations and discussions of recognized experts. The objective is to both, provide an introduction and a review of a topic with current significant impact of the role of statistics in healthcare and beyond. **Trends towards significance** Stephen Senn There are many valid criticisms of P-values but the criticism that they are largely responsible for the reproducibility crisis has been accepted rather lightly in some quarters. Whatever the inferential statistic that is used, it is quite illogical to assume that as the sample size increases it will tend to show more evidence against the null hypothesis. This applies to Bayesian posterior probabilities as much as it does to P-values. In the context of P-values it can be referred to as the trend towards significance fallacy but more generally, for reasons I shall explain, it could be referred to as the anticipated evidence fallacy. The anticipated evidence fallacy is itself an example of the overstated evidence fallacy. I shall also discuss this fallacy and other relevant matters affecting reproducible science including the problem of false negatives. **p-value, s-value, B-value, D-value, … what else? Tolerance intervals: Beyond the t-test and p-values** Bernard G Francq, Ron Kenett The statistical significance is often based on confidence intervals (or credible intervals in Bayesian analysis) and p-values, the reporting of which is requested by most top-level medical journals. However, in recent years there have been ongoing debates on their usefulness, leading to a ‘significance crisis’ in science. In this talk, some alternative solutions proposed in the literature like the s-value, D-value or B-value (namely, the probability that a patient under treatment A ends up with a better clinical outcome compared to another patient under treatment B) will be reviewed. We'll show that, in medical research, the treatment successes on the patient level can be elaborated using the concept of individual success probability (ISP) which generalizes the B-value. The ISP allows a more pragmatic interpretation under both, frequentist and Bayesian, paradigms. The relationships between p-value, ISP and tolerance intervals will be discussed and illustrated with toy examples (1 sample t-test, paired t-test, cross-over trials, 2 samples t-test) and real-world data sets.