🌦WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting

Chenghao Qian1*, Yuhu Guo2*, Wenjing Li1, Gustav Markkula1
1University of Leeds, 2Carnegie Mellon University
*Indicates Equal Contribution
arXiv Code Dataset (Coming soon)

Can 3D reconstruction models work in adverse weather conditions?

WeatherGS Animation

Abstract

3D Gaussian Splatting (3DGS) has gained significant attention for 3D scene reconstruction, but still suffers from complex outdoor environments, especially under adverse weather. This is because 3DGS treats the artifacts caused by adverse weather as part of the scene and will directly reconstruct them, largely reducing the clarity of the reconstructed scene. To address this challenge, we propose WeatherGS, a 3DGS-based framework for reconstructing clear scenes from multi-view images under different weather conditions. Specifically, we explicitly categorize the multi-weather artifacts into the dense particles and lens occlusions that have very different characters, in which the former are caused by snowflakes and raindrops in the air, and the latter are raised by the precipitation on the camera lens. In light of this, we propose a dense-to-sparse preprocess strategy, which sequentially removes the dense particles by an Atmospheric Effect Filter (AEF) and then extracts the relatively sparse occlusion masks with a Lens Effect Detector (LED). Finally, we train a set of 3D Gaussians by the processed images and generated masks for excluding occluded areas, and accurately recover the underlying clear scene by Gaussian splatting. We conduct a diverse and challenging benchmark to facilitate the evaluation of 3D reconstruction under complex weather scenarios. Extensive experiments on this benchmark demonstrate that our WeatherGS consistently produces high-quality, clean scenes across various weather scenarios, outperforming existing state-of-the-art methods.

1.Problems

MY ALT TEXT

Reconstructed scene in snowy weather using NeRF and 3DGS.

MY ALT TEXT

Rendering examples in adverse weather conditions using NeRF, 3DGS, and WeatherGS.

2.Dataset Construction

MY ALT TEXT

3.Method Pipeline

MY ALT TEXT

The overview of WeatherGS. The WeatherGS preprocesses multi-view images by removing weather-related visual artifacts. An atmospheric filter removes dense particles like raindrops and snowflakes, while a lens effect detector identifies and masks occlusions caused by precipitation on the camera lens. To ensure high-quality 3D scene reconstruction, we train the preprocessed images with 3D Gaussian splatting excluding the occlusion area to model the clear scene's geometry and appearance.

4.Final Results

3DGS reconstructs scenes with dense weather particles and lens occlusions, which significantly obscure visibility. The proposed WeatherGS method effectively eliminates these artifacts, resulting in clean 3D scene reconstructions and artifact-free image rendering.

BibTeX


@misc{weathergs2024qian,
      title={WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting}, 
      author={Chenghao Qian and Yuhu Guo and Wenjing Li and Gustav Markkula},
      year={2024},
      eprint={2412.18862},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.18862}, 
}