This paper introduces a Gaussian Splatting (GS) approach to relighting and reconstructing 3D scenes containing real in-camera image-based lighting (RIC-IBL) sources. This is applied Virtual Studio (VS) film production, where LED walls project live virtual backgrounds that produce RIC-IBL effects. The goal of VS reconstruction and relighting (VSR) is to provide the ability to change the RIC-IBL background texture and propagate the lighting changes to a 3-D scene. As no prior RIC-IBL research exists, this paper establishes a robust platform for future work by introducing a novel pipeline, a handful of baselines for RIC-IBL texture sampling, and several datasets including real VS scenes. The main method is a geometry-independent lighting model that explicitly represents the IBL texture coordinates and lighting intensity as view-dependent GS parameters. This avoids computing physically-based events linked to inverse rendering (IR). Hence, our method does not rely on depth/normal priors, is capable of reconstructing diverse scenes including transparent objects, and can be implemented without custom CUDA or RTX code. In practice, our method removes the need for costly VS-specific hardware and offers numerous possibilities for revolutionizing the VS production process to benefit post production artistry.
We propose a new VP workflow/pipeline. The output is a fully controllable photorealistic IBL light simulation and 3-D Gaussian Splatting scene, reconstructed from multi-view footage of a VP stage under varying IBL conditions. We attempt to precisely match the simulated IBL exposure setting to the real VP LED stage, meaning that what you see in simulation should be a replica of realisty.
We test this on a variety of LCD and LED set-ups. Whether its a full VP stage, a minature set or a minor IBL source (TV, monitort etc.), our method attempts to simulate the 3-D scene and light field in post without inheriting the capture artifacts related to photographing luminous screens with video camera settings (low ISO, shallow color bandwith etc.). This means no more overly expensive capture equiptment...
By simulating the VP set using our Gaussian Splatting pipeline users can design, edit and modify the background textures in real-time and without even visiting the VP set. This example was streamed from Photoshop.
By capturing the real IBL exposure setting (with lambda) we can modify this parameter to control the intesity of the IBL screen on the remainder of the scene. Future applications could extend this to localized exposure modifications based on material properties to enhance artistic feel. Such effects include vignetting, contrast adjustments and saturation adjustments and hue-based exposure suppression (e.g. enhancing red values while desaturating blue colors).
Our VSR pipeline produces a 3-D Gaussian Splatting scene so novel views can be tested for lighting design quality. Future work on dynamic VSR could use this to capture cinematic shots in simulation. Compared to capturing the final shot during production, this would trivialize capturing complex shots while being equiptment-free (low cost).
...
@article{...}