Abstract

Due to the increased popularity of augmented and virtual reality experiences, the interest in capturing high-resolution real-world point clouds has never been higher. Loss of details and irregularities in point cloud geometry can occur during the capturing, processing, and compression pipeline. It is essential to address these challenges by being able to upsample a low Level-of-Detail (LoD) point cloud into a high LoD point cloud. Current upsampling methods suffer from several weaknesses in handling point cloud upsampling, especially in dense real-world photo-realistic point clouds. In this paper, we present a novel geometry upsampling technique, PU-Dense, which can process a diverse set of point clouds including synthetic mesh-based point clouds, real-world high-resolution point clouds, real-world indoor LiDAR scanned objects, as well as outdoor dynamically acquired LiDAR-based point clouds. PU-Dense employs a 3D multiscale architecture using sparse convolutional networks that hierarchically reconstruct an upsampled point cloud geometry via progressive rescaling and multiscale feature extraction. The framework employs a UNet type architecture that downscales the point cloud to a bottleneck and then upscales it to a higher level-of-detail (LoD) point cloud. PU-Dense introduces a novel Feature Extraction Unit that incorporates multiscale spatial learning by employing filters at multiple sampling rates and field of view. The architecture is memory efficient and is driven by a binary voxel occupancy classification loss that allows it to process high-resolution dense point clouds with millions of points during inference time. Qualitative and quantitative experimental results show that our method significantly outperforms the state-of-the-art approaches by a large margin while having much lower inference time complexity. We further test our dataset on high-resolution photo-realistic datasets. In addition, our method can handle noisy data well. We further show that our approach is memory efficient compared to the state-of-the-art methods.

Overview

Surface Reconstruction Results

We show the visualized results of 3D surface reconstruction using the ball-pivoting algorithm for the ground truth, input point cloud, and the upsampled point clouds from 3PU, PU-GAN, PU-GCN, Dis-PU, and PU-Dense on ShapeNet dataset. In the 3D mesh reconstruction task, the result is greatly influenced by the density as well as the quality of the upsampled point cloud. We can see the effectiveness of the proposed upsampling method by surface reconstruction in the figure below. For both 4x and 8x upsampling, surfaces reconstructed from PU-Dense upsampled point clouds are a lot more structured and exhibit richer geometry details. PU-Dense can recover more details and better preserve the smoothness of smooth regions as well as preserve the sharp edges without creating outliers.

Upsampling Results on High-Resolution Point Clouds

We show the visual results of our experiments for 4x and 8x upsampling in the figures below using a point cloud frame from sequence RedandBlack and Longdress respectively from 8iVFB dataset. This is a real-world captured dense photo-realistic dynamic point cloud dataset for immersive communication used in MPEG and JPEG point cloud compression standardization. We also plot the error map based on the point-to-point (P2point) D1 distance between the point cloud and its ground truth to visualize the error distribution. We can see that our method generates limited outliers while populating points very close to the surface of the ground truth point cloud. This results in a high-resolution point cloud with considerably better quality than the other state-of-the-arts. We also show the zoomed-in point cloud geometry views. The results of our proposed method are more precise, sharper, and cleaner, especially around key positions, such as corners and edges.

4x upsampling visual results on sequence RedandBlack.

8x upsampling visual results on sequence Longdress

Results on Real Scanned Object Dataset

We also examined the performance of the proposed method on real-world scanned object dataset with 2048 points from ScanObjectNN dataset. We show the comparative visual results of upsampling a point cloud from ScanObjectNN dataset in the figure below. ScanObjectNN dataset is captured through LiDAR and consists of sparse point clouds. We visualize the upsampled point cloud as well as the reconstructed mesh surface using the ball-pivoting algorithm for the input point cloud and each of the upsampled point clouds. The proposed method (PU-Dense) works well on even sparse object point cloud datasets. In the upsampled point cloud generated by PU-Dense, the points generated are a lot more structured. We can notice that the holes in the point cloud get impainted by the other methods, whereas, PU-Dense keeps the original shape of the point cloud. The reconstructed mesh surface quality for PU-dense is also better compared with 3PU and PU-GAN.

Visual comparison for 4x upsampling on ScanObjectNN dataset

Results on Dynamically Acquired Outdoor Dataset

We examine the performance of PU-Dense on dynamically-acquired outdoor LiDAR dataset from KITTI dataset used for autonomous driving. We show the visual results from four street point clouds in the figure below. Even for sparse and non-uniform point clouds from real-world LiDAR dataset, our proposed method can significantly improve the quality by upsampling these point clouds. In our experiments, we trained PU-Dense on regularly sampled and relatively dense ShapeNet dataset. Therefore, it is a considerable achievement for PU-Dense to be able to generalize to a sparse non-uniform dataset.

Visual results for outdoor LiDAR dataset from KITTI for 4x upsampling.

Robustness against Noise

In this section, we evaluate the robustness of different methods to noise. The plot below quantitatively compares the methods on different levels of noise using ShapeNet dataset for 4x upsampling. We plot D1 MSE PSNR for each method under 7 levels of Gaussian noise. It can be seen that the performance of all methods decreases as the noise level increase. Nevertheless, the proposed method consistently achieves the best performance under each noise level. For PU-Dense, we notice the sharpest decrease when the noise is initially added and then the quality consistently decreases with an increase in noise. The robustness against noise can be further increased by augmenting noise during training of PU-Dense, something that was not employed during our training. We further show visual upsampling results for the PU-Dense on various noisy point clouds in the figure below. We visualize both 4x and 8x upsampling with 0%, 1%, and 2% Gaussian noise. We observe that even after adding noise, the upsampled point clouds visually look similar to the noise-free upsampled point cloud, demonstrating the robustness of the proposed method against noise.

Quantitative comparisons on data with various levels of noise for 4x upsampling using ShapeNet dataset.

Visual results for PU-Dense on noisy data for 4x and 8x upsampling. Left: (a1), (b1), and (c1) are the sparse inputs with 0%, 1%, and 2% Gaussian noise, respectively. Right: (a2), (b2), and (c2) are the upsampled results from (a1), (b1), and (c1) respectively.

Citation

Paper accepted in IEEE Transactions on Image Processing. Citation available soon.
If PU-Dense is useful for your research, please consider citing.