ANISE: Assembly-based Neural Implicit Surface rEconstruction

An overview of our approach. It consists of three modules. The first module predicts a coarse part arrangement in the form of part transformations (structure prediction). Then, conditioned on these part transformations, the next module predicts an implicit function representing the actual geometry (geometry prediction). The last module transforms each part implicit functions according to the predicted transformations and combines them into a single output implicit function (assembly).

Summary of assembly module. We assemble complete shapes by combining the implicit function representation of the individual parts in a differentiable manner. For SDF, the geometric union of implicit functions can be expressed using a minimum operation, after transforming the part according to the decoded transformation parameters.

ANISE vs part-aware single view reconstruction methods. We compare with Joint Retrieval and Deformation (JLRD) and PQ-Net on PartNet dataset. Compared to JLRD our model better models shape topology. Compared to PQ-Net our model better reconstructs small geometric details. Click the image for greater details.

Part-constrained shape assembly. Our model is capable of reconstructing sparse point clouds by assembling parts from a user-provided reference database of shapes or parts. In this example, the user selects 5 shapes (left) pre-segmented into their parts. Our model reconstructs shapes that closely approximate the input point clouds while only using parts from the reference shapes. Differently from traditional shape retrieval- based methods our approach can mix parts from different shapes to better approximate the target. Best viewed in color.

Part editing. Given a query shape, users can replace or interpolate a particular part to produce a full shape with modified parts. Left shapes: original reconstruction. Right shapes: edited reconstruction.

Part-aware single-view reconstruction. Our method significantly outperforms both part-based (PQ-Net) and retrieval and deformation based (JLRD) approaches on chosen PartNet categories. PR&A stands for part retrieval and assembly version of ANISE: reconstruction is based on part retrieval from dataset of reference shapes (same database that was used by JLRD authors). PQ-Net-regr stands for regression model trained to regress PQ-Net representations from input image embeddings. IoU is multiplied by 10² . Chamfer distance (CD) is multiplied by 10³.