Thin Structures in Image Based Rendering

Description

We propose a novel method to handle thin structures in Image-Based Rendering (IBR), and specifically structures supportedby simple geometric shapes such as planes, cylinders, etc. These structures, e.g. railings, fences, oven grills etc, are present inmany man-made environments and are extremely challenging for multi-view 3D reconstruction, representing a major limitationof existing IBR methods. Our key insight is to exploit multi-view information. After a handful of user clicks to specify thesupporting geometry, we compute multi-view and multi-layer alpha mattes to extract the thin structures. We use two multi-viewterms in a graph-cut segmentation, the first based on multi-view foreground color prediction and the second ensuring multiviewconsistency of labels. Occlusion of the background can challenge reprojection error calculation and we use multiviewmedian images and variance, with multiple layers of thin structures. Our end-to-end solution uses the multi-layer segmentationto create per-view mattes and the median colors and variance to create a clean background. We introduce a new multi-pass IBRalgorithm based on depth-peeling to allow free-viewpoint navigation of multi-layer semi-transparent thin structures. Our resultsshow significant improvement in rendering quality for thin structures compared to previous image-based rendering solutions.

Abstract

International audience

Additional details