ANISE: Assembly-based Neural Implicit Surface rEconstruction

1University of Massachusetts Amherst
2Adobe Research

We introduce ANISE -- a shape representation for 3D reconstruction through a set assembly of neural implicits (left). Our approach results in state-of-the-art part-aware reconstruction quality. It can also perform reconstruction constrained on a database of reference parts or shapes (middle) – the model here uses only shapes provided by the user. Finally, it allows part-level editing of outputs (right).

Method overview

An overview of our approach. It consists of three modules. The first module predicts a coarse part arrangement in the form of part transformations (structure prediction). Then, conditioned on these part transformations, the next module predicts an implicit function representing the actual geometry (geometry prediction). The last module transforms each part implicit functions according to the predicted transformations and combines them into a single output implicit function (assembly).

Part assembly

Summary of assembly module. We assemble complete shapes by combining the implicit function representation of the individual parts in a differentiable manner. For SDF, the geometric union of implicit functions can be expressed using a minimum operation, after transforming the part according to the decoded transformation parameters.

Qualitative Results

ANISE vs part-aware single view reconstruction methods. We compare with Joint Retrieval and Deformation (JLRD) and PQ-Net on PartNet dataset. Compared to JLRD our model better models shape topology. Compared to PQ-Net our model better reconstructs small geometric details. Click the image for greater details.

Part-constrained shape assembly. Our model is capable of reconstructing sparse point clouds by assembling parts from a user-provided reference database of shapes or parts. In this example, the user selects 5 shapes (left) pre-segmented into their parts. Our model reconstructs shapes that closely approximate the input point clouds while only using parts from the reference shapes. Differently from traditional shape retrieval- based methods our approach can mix parts from different shapes to better approximate the target. Best viewed in color.

Part editing. Given a query shape, users can replace or interpolate a particular part to produce a full shape with modified parts. Left shapes: original reconstruction. Right shapes: edited reconstruction.

Quantitative evaluation

Part-aware single-view reconstruction. Our method significantly outperforms both part-based (PQ-Net) and retrieval and deformation based (JLRD) approaches on chosen PartNet categories. PR&A stands for part retrieval and assembly version of ANISE: reconstruction is based on part retrieval from dataset of reference shapes (same database that was used by JLRD authors). PQ-Net-regr stands for regression model trained to regress PQ-Net representations from input image embeddings. IoU is multiplied by 102 . Chamfer distance (CD) is multiplied by 103.


First author (Dmitry Petrov) was partially supported by Adobe Research internship during this work. Authors thank Marios Loizou for providing style templates for this website.


ANISE: Assembly-based Neural Implicit Surface rEconstruction

Dmitry Petrov, Matheus Gadelha, Radomír Měch, Evangelos Kalogerakis

picture_as_pdf PDF
format_quote BibTeX