Ressort Presse

• 02.04.2026

An international study, published in the journal "Nature", shows that different, methodologically sound analyses of the same dataset can lead to different results. The study* highlights why transparency regarding methodological decisions is a key component of empirical research. The project involved 457 researchers, including researchers from the University of Duisburg-Essen and the Centre for Advanced Internet Studies (CAIS).

For the study Investigating the Analytic Robustness of the Social and Behavioural Sciences, an international team of 457 independent researchers re-analysed data from 100 previously published studies in the social and behavioural sciences. Different teams were given the same dataset and the same research question. However, how they analysed the data remained open – for example, regarding the choice of statistical models, the definition of variables or data preparation.

The majority of the reanalyses confirmed the key findings of the original studies: in 74% of cases, the analysts reached the same conclusion. At the same time, the analyses often differed in the magnitude of the effect found. Furthermore, the average effect size of the reanalyses was smaller than that of the original studies.

Such differences occur particularly frequently in research using survey data, as is often the case in the social sciences. This data is often complex and requires individual analytical decisions within comparatively wide margins of discretion during evaluation.

“The study does not prove that scientific results are unreliable. Rather, it highlights how scientific findings are generated and validated. Individual studies are rarely the final word on a research question – reliable knowledge usually emerges from many studies that cross-check and complement one another,” says Prof. Dr. Conrad Ziller (photo, right), a political scientist at the University of Duisburg-Essen (UDE).

Prof. Dr. Johannes Breuer (photo left), Head of the Research Data & Methods Department at the Center for Advanced Internet Studies (CAIS) and Professor of Digital Social Sciences at UDE, adds: “Above all, the new study shows how important transparency regarding analytical decisions is, so that results can be better contextualised. Empirical research often allows for several methodologically valid analytical approaches. Researchers are inevitably required to make various decisions during the analysis process, for example regarding data cleaning, model selection or the interpretation of statistical results. This scope for decision-making can influence the magnitude of measured effects or statistical uncertainty. Therefore, precise documentation and justification of methodological decisions, as well as an assessment of their robustness against possible alternatives, are crucial for reliable research findings.”

* The large-scale collaboration was coordinated by Hungarian professors Balázs Aczél and Barnabás Szászi (Eötvös Loránd University and Corvinus University) as part of the ‘Systematizing Confidence in Open Research and Evidence’ (SCORE) programme. The aim of the project is to develop new approaches to make the reliability of scientific results more transparent.

Dr Teresa Hummler and Dr Paul Vierus from the UDE were also involved in the study. Like Professors Breuer and Ziller, they were co-authors and contributed reanalyses of datasets.

Publication: https://www.nature.com/articles/s41586-025-09844-9