Research

Three axes form the basic structure for the research program:

- Visual Computing: including novel transdisciplinary applications of computer vision and image processing, computer graphics, human-computer interaction and data visualization, etc.

- Textual Computing: including natural language processing, textual-statistical historiography of art history (trends, terminologies, schools, translations), the relationship between word and image, new forms of digital publishing for current and historic

- Spatiotemporal Computing (also known as 4D analysis): including space- and place-syntax analysis, video and movement analysis, geospatial modeling, geo-economic simulation, computationally-enriched historical mapping.

The first research focus is set on computer vision’s potential contribution to the history of art and architecture. As the photographic collection of the BHMPI is currently being digitized a a fast rate, and a massive number of art-related research data is being pooled by several partnering institutions (see 3.3), new methodologies will be developed to read, order, and represent art historical visual material. Supervised and semi-supervised machine learning will be possible thanks to the high-quality metadata of existing visual records, automatically predicting metadata for the new unlabeled images – whilst unsupervised methods will also be useful to organize and search the image-datasets through elements not present in the metadata (style, iconography, gesture). These will combine a number of existing techniques, applied to different material (face recognition, style recognition, gesture recognition, sketch-based image retrieval) with new problems in computer vision (architectural pattern recognition, artistic attribute detection). Further annotations will be collected during users’ and scholars’ employment of these prototype systems, improving the quality of both the underlying data and – through online machine learning – future models: such a mechanism necessitates the rapid technology transfer of subsequent computer vision prototypes to the BHMPI’s online photo archive interface. As well as defining new problems in computer vision, the annotations created in such a process can be offered to the computer vision and pattern recognition community, providing new machine learning databases and baselines for relevant computer science research communities, such as ECCV-VISART and ACM Expressive.

Examples for possible research topics include:

• face recognition and social network analysis of people’s representation in photographs or other visual media;
• automatic cataloging and ordering of photographic collections based on individual hand-written metadata as to reconstruct the history of collections;
• developing machine learning for visual similarity as to test theories and study practices of visual comparison in scholarly publications;
• automatic movement categorization and correlation for performing art, video art and cinema as to extract and structure information from moving images, enabling new types of comparative analysis;
• embedding of textual, audio and visual information in digital representations, and the study of cross-correlations between textual and visual content;
• automatic semantic classification and /annotation of 2D and 3D elements using machine learning and textual resources;
• spatialization of textual and visual information within 2D and 3D representations;
• digital reconstructions of light, color, acoustics and other sensory phenomena, and computational analysis of these reconstructions
• critique of digital approaches for the classification and operationalization of visual information.

The second focal point uses and develops natural language processing techniques within the methodology of Distant Reading. As the digitization initiative of the BHMPI library will then have added special collections (see 3.1.1), the doctoral students will be able to develop new methods and technologies for recognizing linguistic patterns in the textual data. The main goal of this axis of research is the development of a computational historiography that allows researchers to recon-struct the history and geography of artistic and art-historical concepts and ideas, by being able to trace art historical terminology and argumentation contextually. Indeed, scholarly experience shows that studying the history of Art History becomes increasingly challenging after 1900 due to the large quantity of textual material. On the other hand, since the 1990s, textual scholarship itself has become digital. In this light, digital research methods become not only necessary (due to the scale and complexity of the material) but contextually appropriate (with increasingly born-digital scientific publications) to map understand the development of the intellectual history and geography of the discipline in the last century.

This shift towards a corpus historiography echoes previous similar shifts in corpus linguistics: such a historiography, however, requires different computational methods to those developed for linguistic and literary phenomena. Novel computational problems include multilingualism (the interrelation between language-groups and schools of scientific thought), contextualization (particularly in relation to intertextual and intervisual references), and polysemy, which however has well progressed over the last two decades thanks to the Getty Research Institute and others, including the BHMPI. New textual search engines will change the virtual structure of the BHMPI library and its scholarly use significantly, whilst the semantic analysis of text can lead towards a recommendation engine for secondary references for links made in the BHMPI’s knowledge-graph. Specific tools for identifying handwriting will be necessary for the identification of the digitized photo material. The developed technologies also be of particular interest to the MPIWG and other MPIs of the Section.

Examples for possible research topics driven by the new infrastructures and tools:

• study of the semantic drift of artistic vocabulary in time and space;
• visualizing the diffusion of art historical terms and trends in social networks;
• exploring intertextual references across publications;
• automatic transcription, comparison and categorization of hand-written and early printed documents;
• context-aware intelligent ‘diff’ tools for intratextual comparisons, i.e. between different editions, translations or transcriptions of the same text;
• analysis of aesthetic terms in texts denoting color, space, forms etc. and topological representation of their interrelations and values in comparison to the object they describe;
• stylometric analysis of art historical writings and their developments and affiliations;
• machine learning-driven and thesaurus-based multilingual text search as to reach out more easily to foreign-language publications and sources;
• research-trend analysis and forecast in relation to times, places, terms, actors and networks as to be able to study the contemporary history of Art History;
• statistical and archival research into the provenance, the value development and mobility of works of art;
• use of digital tools for exploring and correlating visual images in texts, furthering recent technical developments in image-text relations, e.g. in automatic image caption generation;
• Cross-lingual text mining and joint text and image mining from art historical publications.

The spatiotemporal domain not only allows visual data to be linked with the textual data (interconnecting, for instance, the BHMPI’s library and photographic collection), but also enables new forms of organizing and representing these data through historical space and time. The rapidly-expanding field of the Digital Spatial Humanities will be particularly relevant to this research axis, using a cross-scalar approach: from micro-resolution 3D scanning of artistic artefacts and architectural details, to urban- and continent-scale geo-mapping. Recent advances in differentiable rendering systems, and a more general convergence between computer graphics and vision methods, al-low for 3D data to be not only reproduced but analyzed. Methods for the simulation and analysis of three-dimensional artefacts and architectural spaces will be developed, building on current work in acoustic simulation, light simulation and space-syntax analysis. Measures of centrality and accessibility analysis can add invaluable insights about the how the spatial material conditions of a rich and cumulative urban tissue (such as Rome) influences the material practices through access, expo-sure and co-presence of resources, of artists, of galleries, of publics, of institutions, etc. As well as adapting current methods from architecture, computational urban studies, spatial econometrics and
acoustic modelling, novel techniques must be developed which take account of the important temporal dimension of historic data (so-called 4D modelling), as well as the unique uncertainties in simulation and modelling that arise from philological and archeological evidence.
Examples of possible art historical research topics driven by the new infrastructures and tools:

• analysis of urban and architectural fields of visibility and their relationship with visual art;
• 3D analysis and reconstructions of cultural artifacts (object, disposition, interactions);
• social and historical geographies of gesture and body-language through analysis of gesture, pose, movement and interaction in representational art;
• automatic or semi-automatic integration of historical data (2D/3D) in a digital reconstruction;
• visualization of 2D/3D reconstruction within other media using digital or semi-digital technologies;
• analysis of text-image patterns in publications;
• analysis of the temporality of the constitution of works of art, both representational and non-representational, based on metadata and the detection of traces of movement;
• 4D exhibition reconstruction from archival records, including the encoding of uncertainty.