DOING JUSTICE TO UNCERTAINTY IN RESEARCH
Does playing violent games make you more aggressive? Does cognitive behavioural therapy work against depression? Socially relevant questions that scientists like to find clear answers to. After research, crystal clear and unequivocal conclusions often follow: no, you do not become more aggressive from violent games; yes, cognitive behavioural therapy helps against depression. Yet such conclusions can be undermined by a subsequent study where results are not replicated. In other words, the methodological approach does not always do justice to diffuse reality.
For twenty years, the Psychological Methods Lab (PML) has been committed to promoting robust and transparent statistics that take more account of uncertainty in psychological research. The research group aims to break the status quo where one researcher reports the results of one specific analysis, conducted with only one model and often using outdated methodology. Their critical research shows that the idea that there is only one correct interpretation after analysing research data is a fallacy. To illustrate, the team asked four different research groups to analyse a simple dataset. This produced four completely different results, all of which differed from those of the original authors. The message: diverse models lead to diverse conclusions, and for reliable science it is essential to embrace and explicitly name this diversity in the social sciences.
The idea that there is only one correct interpretation of research data is a fallacy
The research group actively contributes to increasing diversity in research methods. For example, they are developing new network approaches that allow psychological constructs, such as IQ or mental disorders, to be understood as a complex of interacting variables. In traditional psychology, problems in mental disorders are often seen as isolated symptoms that can be explained from a single ‘cause’. For example, if someone sleeps poorly, broods, and becomes increasingly insecure, these may be considered signs of possible depression. In the network approach, however, all symptoms are put into a broader perspective to see how they interact. After all, if you sleep badly, you get tired and make more mistakes, after which you start feeling insecure and fretting and you may sleep even worse. In short, the network approach shows how these kinds of symptoms can cause and influence each other. PML’s innovative models make it possible to map this complex interplay and make more realistic predictions, for example about the course of mental illness.
The team’s next goal is to develop innovative methods that help prevent hindsight bias. Suppose a researcher tests a hypothesis suggesting a link between coffee drinking and the likelihood of health problems. After collecting data and conducting statistical analyses, it turns out that there is no significant relationship between coffee consumption and health risks. Hindsight bias occurs when the researcher subsequently decides to adjust the analyses to bring the results more in line with the original hypothesis. For instance, by analysing specific subgroups, using different statistical methods, or removing certain outliers, a significant association may still be found. Although common in scientific statistics, the PML team stresses the need to avoid such adjustments to ensure the integrity of psychological research.
Diverse models lead to diverse conclusions
Another special feature is that the team continuously integrates their state-of-the-art statistical modelling into JASP which is an open-source software package which they developed, and which is now used by more than 292 universities in 67 countries worldwide. Open science is therefore a guiding principle in the research group’s work. The new methods to be developed to prevent hindsight bias will also be made available to researchers worldwide through JASP.
All the different lines of research share a common goal: to develop accurate statistical models that enable researchers to extract more reliable information from (psychological) datasets. This goal is both ambitious and bold, given the tendency of most scientists to stick to existing methods. Typical of this research group is therefore their enormous determination: even if they meet resistance, they continue to try. If a method they develop is successful, it can significantly improve the reliability of a significant amount of empirical research.
