Cassie Ann Short is a postdoctoral researcher in the Department of Psychology at Carl von Ossietzky Universität Oldenburg. Her methodological work focuses on robustness and multiverse analyses, replicability, transparent research workflows, and the psychometric properties of psychophysiological estimates. Cassie has previous experience of working in large collaborative replicability projects, and she is active in open science initiatives and in developing practical tools for documenting and evaluating analytical decisions.
Nate Breznau is a principal investigator in the Department of Organization and Program Planning at the DIE, German Institute for Adult Education - Leibniz Institute for Lifelong Learning, Bonn. His work involves public opinion, social policy, income inequality, and immigration. He has a long track record of open science and metascience work investigating the reproducibility of the social and behavioral sciences. He has taught several courses on open science and replication and authored a paper on teaching replications.
They will teach the Spring Seminar course Replication in Practice: From Study Design to Publication, held over several weeks in a hybrid format in Cologne and online: Day 1 (Thursday, 18 Feb) and Day 5 (Friday, 02 Apr) are held online only; Days 2, 3, and 4 are held in hybrid mode (GESIS Cologne | Online).
How did you become interested in your subject?
Nate: My interest in replication, metascience and reproducible workflows comes directly from my participation in the Open Science Movement. In my graduate studies from 2005 through 2009, there was no attention paid to reproducibility and robustness in empirical analyses. Almost none of the published work was computationally reproducible – no code and data sharing – and little of it was replicable – the findings were mostly hacked rather than robust. The culture I learned social science in was driven by p-hacking – keep running different models until you find the result you want – and HARKing – if you find results you did not expect, pretend that you hypothesized them all along. This way of doing science is totally untrustworthy. It is garbage. It presents wild, outlier findings as if they are reliable. And because of this I made it my mission to do things better, and to teach others to do things better.
Cassie: My interest in metascience and open science really developed during my first postdoc after moving to Germany, when I joined the CoScience project. That work gave me first-hand experience of just how much analytic uncertainty can arise in EEG individual differences research. Seemingly reasonable alternative analytical choices can lead to quite different results and, in turn, quite different scientific conclusions. I found that both challenging and fascinating, because it raised a fundamental question for me as a scientist: how can we make our conclusions more transparent and robust, and communicate the uncertainty that is actually present in the research process? I have since worked on methodological development for multiverse analysis, mostly with application to substantive neuropsychological individual differences questions.
What lessons can participants draw from your GESIS course?
Cassie & Nate: We hope participants will leave the course with a much more practical understanding of what it actually takes to replicate published research. By carrying out their own replication from start to finish, they will learn how to turn a published study into a transparent and reproducible workflow, how to deal with ambiguities and unexpected decisions along the way, and how to interpret replication results carefully. An important lesson is that replication is not simply about deciding whether an original finding was “right” or “wrong”; it can teach us a great deal about methods, evidence, and the research process itself. Ideally, participants will also take these practices back into their own research.
What do you enjoy most about being a social scientist?
Nate: Starting to work on an important question when you only have a foggy glimpse of what an answer might look like – and then seeing an answer come into focus, whether gradually or all at once.
What do you think is the most exciting recent development in your field?
Cassie: What I find most exciting about metascience is the increasing ability to make analytic uncertainty visible and to study it systematically, rather than treating a single analytical pathway as if it were the only reasonable one. I especially like that this does not have to be about finding fault; it can be a very constructive way of learning more about our methods and practices. I am particularly excited about the statistical development of multiverse analysis. Working across many defensible analytical choices creates a complex statistical environment. Developing statistical approaches to summarise, model, and interpret that structure is essential if multiverse analysis can move beyond describing analytic variability to drawing more informative conclusions about analytic uncertainty. This ongoing, creative statistical development is the area I am most excited by.
Nate: The new metascience movement which takes careful stock of previous research to expose serious flaws in the way we are doing things in social science, especially sociology, I find exciting. I think that social sciences, particularly sociology, have a lot of work to do to gain back the trust of the public and funding institutions. Social interaction is more complicated than quantum physics, and we need better and more reliable social science to enable us to understand ourselves and our social worlds.
We thank Cassie & Nate for their insights and look forward to their upcoming Spring Seminar course.
