Neuralux: Improving the decomposition of single-illumination multiview outdoor scenes

Abstract

We present a novel approach that combines intrinsic decomposition of outdoor scenes with real-time rendering of new views under unknown illumination. Building on top of the state of the art, our method tackles the challenges of limited information in single-illumination scenarios by introducing pixel-level regularization terms aligning inferred material segmentation labels with albedo consistency estimators. For outdoor illumination, we adopt a physically-based sky model which increases the intrinsic decomposition robustness by relying on a reduced set of expressive parameters. Our approach enables partial retraining of 2DGS/3DGS models to render de-illuminated scenes in real time, with seamless integration into rendering engines for enhanced scene lighting, achieving better decomposition results than the state of the art. We show several experiments, including ablation studies and material segmentation source comparisons, proving our method's advantages over previous work, despite remaining challenges in handling fine shadow details and view-dependent effects due to the limitations of the Lambertian shading model.