This article explores the affordances and functionalities of the Dutch CLARIAH research infrastructure – and the integrated video annotation tool – for doing media historical research with digitised audiovisual sources from television archives. The growing importance of digital research infrastructures, archives and tools, has enticed media historians to rethink their research practices more and more in terms of methodological transparency, tool criticism and reflection. Moreover, also questions related to the heuristics and hermeneutics of our scholarly work need to be reconsidered. The article hence sketches the role of digital research infrastructures for the humanities (in the Netherlands), and the use of video annotation in media studies and other research domains. By doing so, the authors reflect on their own specific engagements with the CLARIAH infrastructure and its tools, both as media historians and co-developers. This dual position greatly determines the possibilities and constraints for the various modes of digital scholarship relevant to media history. To exemplify this, two short case studies – based on a pilot project ‘Me and Myself. Tracing First Person in Documentary History in AV-Collections’ (M&M) – show how the authors deployed video annotation to segment interpretative units of interest, rather than opting for units of analysis common in statistical analysis. The deliberate choice to abandon formal modes of moving image annotation and analysis ensued from a delicate interplay between the desired interpretative research goals, and the integration of tool criticism and reflection in the research design. The authors found that due to the formal and stylistic complexity of documentaries, also alternative, hermeneutic research strategies ought to be supported by digital infrastructures and its tools.