Three axes form the basic structure for the research program:
- Visual Computing: including novel transdisciplinary applications of computer vision and image processing, computer graphics, human-computer interaction and data visualization, etc.
- Textual Computing: including natural language processing, textual-statistical historiography of art history (trends, terminologies, schools, translations), the relationship between word and image, new forms of digital publishing for current and historic
- Spatiotemporal Computing (also known as 4D analysis): including space- and place-syntax analysis, video and movement analysis, geospatial modeling, geo-economic simulation, computationally-enriched historical mapping.
The first research focus is set on computer vision’s potential contribution to the history of art and architecture. As the photographic collection of the BHMPI is currently being digitized a a fast rate, and a massive number of art-related research data is being pooled by several partnering institutions (see 3.3), new methodologies will be developed to read, order, and represent art historical visual material. Supervised and semi-supervised machine learning will be possible thanks to the high-quality metadata of existing visual records, automatically predicting metadata for the new unlabeled images – whilst unsupervised methods will also be useful to organize and search the image-datasets through elements not present in the metadata (style, iconography, gesture). These will combine a number of existing techniques, applied to different material (face recognition, style recognition, gesture recognition, sketch-based image retrieval) with new problems in computer vision (architectural pattern recognition, artistic attribute detection). Further annotations will be collected during users’ and scholars’ employment of these prototype systems, improving the quality of both the underlying data and – through online machine learning – future models: such a mechanism necessitates the rapid technology transfer of subsequent computer vision prototypes to the BHMPI’s online photo archive interface. As well as defining new problems in computer vision, the annotations created in such a process can be offered to the computer vision and pattern recognition community, providing new machine learning databases and baselines for relevant computer science research communities, such as ECCV-VISART and ACM Expressive.
Examples for possible research topics include:
The second focal point uses and develops natural language processing techniques within the methodology of Distant Reading. As the digitization initiative of the BHMPI library will then have added special collections (see 3.1.1), the doctoral students will be able to develop new methods and technologies for recognizing linguistic patterns in the textual data. The main goal of this axis of research is the development of a computational historiography that allows researchers to recon-struct the history and geography of artistic and art-historical concepts and ideas, by being able to trace art historical terminology and argumentation contextually. Indeed, scholarly experience shows that studying the history of Art History becomes increasingly challenging after 1900 due to the large quantity of textual material. On the other hand, since the 1990s, textual scholarship itself has become digital. In this light, digital research methods become not only necessary (due to the scale and complexity of the material) but contextually appropriate (with increasingly born-digital scientific publications) to map understand the development of the intellectual history and geography of the discipline in the last century.
This shift towards a corpus historiography echoes previous similar shifts in corpus linguistics: such a historiography, however, requires different computational methods to those developed for linguistic and literary phenomena. Novel computational problems include multilingualism (the interrelation between language-groups and schools of scientific thought), contextualization (particularly in relation to intertextual and intervisual references), and polysemy, which however has well progressed over the last two decades thanks to the Getty Research Institute and others, including the BHMPI. New textual search engines will change the virtual structure of the BHMPI library and its scholarly use significantly, whilst the semantic analysis of text can lead towards a recommendation engine for secondary references for links made in the BHMPI’s knowledge-graph. Specific tools for identifying handwriting will be necessary for the identification of the digitized photo material. The developed technologies also be of particular interest to the MPIWG and other MPIs of the Section.
Examples for possible research topics driven by the new infrastructures and tools:
The spatiotemporal domain not only allows visual data to be linked with the textual data (interconnecting, for instance, the BHMPI’s library and photographic collection), but also enables new forms of organizing and representing these data through historical space and time. The rapidly-expanding field of the Digital Spatial Humanities will be particularly relevant to this research axis, using a cross-scalar approach: from micro-resolution 3D scanning of artistic artefacts and architectural details, to urban- and continent-scale geo-mapping. Recent advances in differentiable rendering systems, and a more general convergence between computer graphics and vision methods, al-low for 3D data to be not only reproduced but analyzed. Methods for the simulation and analysis of three-dimensional artefacts and architectural spaces will be developed, building on current work in acoustic simulation, light simulation and space-syntax analysis. Measures of centrality and accessibility analysis can add invaluable insights about the how the spatial material conditions of a rich and cumulative urban tissue (such as Rome) influences the material practices through access, expo-sure and co-presence of resources, of artists, of galleries, of publics, of institutions, etc. As well as adapting current methods from architecture, computational urban studies, spatial econometrics and
acoustic modelling, novel techniques must be developed which take account of the important temporal dimension of historic data (so-called 4D modelling), as well as the unique uncertainties in simulation and modelling that arise from philological and archeological evidence.
Examples of possible art historical research topics driven by the new infrastructures and tools: