Beyond ‘significance’: Principles and practice of the analysis of credibility

Robert A.J. Matthews*

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

The inferential inadequacies of statistical significance testing are now widely recognized. There is, however, no consensus on how to move research into a ‘post p < 0.05’ era. We present a potential route forward via the Analysis of Credibility, a novel methodology that allows researchers to go beyond the simplistic dichotomy of significance testing and extract more insight from new findings. Using standard summary statistics, AnCred assesses the credibility of significant and non-significant findings on the basis of their evidential weight, and in the context of existing knowledge. The outcome is expressed in quantitative terms of direct relevance to the substantive research question, providing greater protection against misinterpretation. Worked examples are given to illustrate how AnCred extracts additional insight from the outcome of typical research study designs. Its ability to cast light on the use of p-values, the interpretation of non-significant findings and the so-called ‘replication crisis’ is also discussed.

Original languageEnglish
Article number171047
JournalRoyal Society Open Science
Volume5
Issue number1
Early online date17 Jan 2018
DOIs
Publication statusPublished - 17 Jan 2018

Fingerprint

credibility
statistical significance
statistics
interpretation
methodology
ability
knowledge
Values

Bibliographical note

© 2018 The Authors. Published by the Royal Society under the terms of the Creative Commons
Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted
use, provided the original author and source are credited

Keywords

  • Bayesian methods
  • Credibility
  • Replication crisis
  • Significance testing
  • Statistical inference

Cite this

Matthews, Robert A.J. / Beyond ‘significance’ : Principles and practice of the analysis of credibility. In: Royal Society Open Science. 2018 ; Vol. 5, No. 1.
@article{818d2ccd9c1745899e8f350867626137,
title = "Beyond ‘significance’: Principles and practice of the analysis of credibility",
abstract = "The inferential inadequacies of statistical significance testing are now widely recognized. There is, however, no consensus on how to move research into a ‘post p < 0.05’ era. We present a potential route forward via the Analysis of Credibility, a novel methodology that allows researchers to go beyond the simplistic dichotomy of significance testing and extract more insight from new findings. Using standard summary statistics, AnCred assesses the credibility of significant and non-significant findings on the basis of their evidential weight, and in the context of existing knowledge. The outcome is expressed in quantitative terms of direct relevance to the substantive research question, providing greater protection against misinterpretation. Worked examples are given to illustrate how AnCred extracts additional insight from the outcome of typical research study designs. Its ability to cast light on the use of p-values, the interpretation of non-significant findings and the so-called ‘replication crisis’ is also discussed.",
keywords = "Bayesian methods, Credibility, Replication crisis, Significance testing, Statistical inference",
author = "Matthews, {Robert A.J.}",
note = "{\circledC} 2018 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, provided the original author and source are credited",
year = "2018",
month = "1",
day = "17",
doi = "10.1098/rsos.171047",
language = "English",
volume = "5",
number = "1",

}

Beyond ‘significance’ : Principles and practice of the analysis of credibility. / Matthews, Robert A.J.

In: Royal Society Open Science, Vol. 5, No. 1, 171047, 17.01.2018.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Beyond ‘significance’

T2 - Principles and practice of the analysis of credibility

AU - Matthews, Robert A.J.

N1 - © 2018 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, provided the original author and source are credited

PY - 2018/1/17

Y1 - 2018/1/17

N2 - The inferential inadequacies of statistical significance testing are now widely recognized. There is, however, no consensus on how to move research into a ‘post p < 0.05’ era. We present a potential route forward via the Analysis of Credibility, a novel methodology that allows researchers to go beyond the simplistic dichotomy of significance testing and extract more insight from new findings. Using standard summary statistics, AnCred assesses the credibility of significant and non-significant findings on the basis of their evidential weight, and in the context of existing knowledge. The outcome is expressed in quantitative terms of direct relevance to the substantive research question, providing greater protection against misinterpretation. Worked examples are given to illustrate how AnCred extracts additional insight from the outcome of typical research study designs. Its ability to cast light on the use of p-values, the interpretation of non-significant findings and the so-called ‘replication crisis’ is also discussed.

AB - The inferential inadequacies of statistical significance testing are now widely recognized. There is, however, no consensus on how to move research into a ‘post p < 0.05’ era. We present a potential route forward via the Analysis of Credibility, a novel methodology that allows researchers to go beyond the simplistic dichotomy of significance testing and extract more insight from new findings. Using standard summary statistics, AnCred assesses the credibility of significant and non-significant findings on the basis of their evidential weight, and in the context of existing knowledge. The outcome is expressed in quantitative terms of direct relevance to the substantive research question, providing greater protection against misinterpretation. Worked examples are given to illustrate how AnCred extracts additional insight from the outcome of typical research study designs. Its ability to cast light on the use of p-values, the interpretation of non-significant findings and the so-called ‘replication crisis’ is also discussed.

KW - Bayesian methods

KW - Credibility

KW - Replication crisis

KW - Significance testing

KW - Statistical inference

UR - http://www.scopus.com/inward/record.url?scp=85040946992&partnerID=8YFLogxK

U2 - 10.1098/rsos.171047

DO - 10.1098/rsos.171047

M3 - Article

AN - SCOPUS:85040946992

VL - 5

IS - 1

M1 - 171047

ER -