Text, techné and tenure: what remains out of scope of research evaluation in Humanities disciplines and how to change it for the better?

Text, techné and tenure: what remains out of scope of research evaluation in Humanities disciplines and how to change it for the better? (Slides presented at the OAI12 conference: https://oai.events/) Peer review is central scholarly practice that carries fundamental paradoxes from its inception. On the one hand, it is very difficult to open up peer review for the sake of empirical analysis, as it usually happens in closed black boxes of publishing and other gatekeeping workflows that are embedded in a myriad of disciplinary cultures, each of which comes very different, and usually competing notions of excellence. On the other hand, it is a practice that carries an enormous weight in terms of gatekeeping; shaping disciplines, publication patterns and power relations within academia. This central role of peer review alone explains why it is crucial to study to better understand situated evaluation practices, and to continually rethink them to strive for their best, and least imperfect (or reasonably imperfect) instances. How the notion of excellence and other peer review proxies are constructed and (re)negotiated in everyday practices across the SSH disciplines; who are involved in the processes and who remain out; what are the boundaries of peer review in terms of inclusiveness with content types; and how the processes are aligned or misaligned to research realities? What are the underlying reasons behind the persistence of certain proxies in the system and what are emerging trends and future innovations? To gain an in-depth understanding of these questions, as part of the H2020 project OPERAS-P, our task force collected and analysed 32 in-depth interviews with scholars about their motivations, challenges and experiences with novel practices in scholarly writing and in peer-review. The presentation will showcase the results of this study. Focus will be on the conflict between the richness of contemporary scholarship and the prestige economy that defines our current academic evaluation culture. The encoded and pseudonymized interview transcripts that form the basis of our analysis will be shared as open data in a certified data repository together with a rich documentation of the process so that our interpretations, conclusions and the resulting recommendations are clearly delineable from the rich input we had been working with and which are thus openly reusable for other purposes.