Evaluation of deep pose detectors for automatic analysis of film style

Description

Identifying human characters and how they are portrayed on-screen is inherently linked to how we perceive and interpretthe story and artistic value of visual media. Building computational models sensible towards story will thus require a formalrepresentation of the character. Yet this kind of data is complex and tedious to annotate on a large scale. Human pose estimation(HPE) can facilitate this task, to identify features such as position, size, and movement that can be transformed into input tomachine learning models, and enable higher artistic and storytelling interpretation. However, current HPE methods operatemainly on non-professional image content, with no comprehensive evaluation of their performance on artistic film.Our goal in this paper is thus to evaluate the performance of HPE methods on artistic film content. We first propose a formalrepresentation of the character based on cinematography theory, then sample and annotate 2700 images from three datasetswith this representation, one of which we introduce to the community. An in-depth analysis is then conducted to measure thegeneral performance of two recent HPE methods on metrics of precision and recall for character detection , and to examinethe impact of cinematographic style. From these findings, we highlight the advantages of HPE for automated film analysis, andpropose future directions to improve their performance on artistic film content.

Abstract

International audience

Additional details