Image-Based Reconstruction and Synthesis of Dense Foliage

This project presents a finer scale technique to demonstrate for the first time the processing of densely leafed foliage – computation of 3D structure, plus extraction of statistics for leaf shape and the configuration of neighboring leaves.

Flora is an element in many computer-generated scenes. But trees, bushes, and plants have complex geometry and appearance and are difficult to model manually. One way to address this problem is to capture models directly from the real world. Existing techniques have focused on extracting macro structure such as the branching structure of trees, or the structure of broad-leaved plants with a relatively small number of surfaces. This project presents a finer scale technique to demonstrate for the first time the processing of densely leafed foliage – computation of 3D structure, plus extraction of statistics for leaf shape and the configuration of neighboring leaves.

Our method starts with a mesh of a single exemplar leaf of the target foliage. Using a small number of images, point cloud data is obtained from multi-view stereo, and the exemplar leaf mesh is fitted non-rigidly to the point cloud over several iterations. Initialization of the fitting is done using RANSAC, making it robust to outliers in the stereo reconstruction and suitable for the chaotic point cloud obtained from foliage. In addition, our method learns a statistical model of leaf shape and appearance during the reconstruction phase, and a model of the transformations between neighboring leaves. This information can subsequently be used to generate a variety of plausible leaves for that plant species in plausible configurations, and is useful in two ways – to augment and increase leaf density in reconstructions of captured foliage and to synthesize new foliage that conforms to a user-specified layout and density. The result of our technique is a dense set of captured leaves with a realistic appearance and a method for leaf synthesis. Our approach excels at reconstructing plants and bushes that are primarily defined by dense leaves and is demonstrated with multiple examples.

The first stage of this project has been successfully completed. The results are expected to be integrated with related projects ’3D Reconstruction of Natural Environments using Scanning Robots’ and ’3D Reconstruction of Trees’.