VSR: Virtual studio 3-D reconstruction and relighting with real image-based relighting

Anonymous Submission
Showcase Paper Info Datasets Appendix Code
Abstract:

Virtual production (VP) has emerged as an effective means of film production. A VP studio combines the capture of real foreground material with virtual backgrounds displayed on LED walls. While supporting image-based lighting (IBL) and in-camera visual effects (VFX), VP constrains lighting at capture-time, making downstream VFX non-trivial. Moreover, stage lighting is coupled to LED wall pose and display settings, limiting pre-visualization and leading to costly re-shoots when creative intent is not met. We address these challenges by introducing a VP 3D scene reconstruction and relighting (VSR) pipeline. This enables pre-visualization and post-capture modification of LED wall content and settings while faithfully propagating the corresponding lighting effects to foreground objects. We propose an unconstrained, geometry-independent Gaussian Splatting (GS) lighting model that encodes IBL texture sample coordinates and lighting intensities as deformable view-dependent Gaussian parameters. By avoiding ray transmission functions, we eliminate the need for depth or normal priors, naturally support transparency and reflections, and do not require custom CUDA/RTX code. Our representation is compact and efficient, requiring under 5 GB of RAM and VRAM to train 1080p scenes in under 2 hours. Objective and perceptual evaluations demonstrate state-of-the-art reconstruction and relighting capabilities, and shows that mip-map based IBL texture sampling outperforms other baselines by up to 3 dB in PSNR and 0.04 in SSIM. We show that in practice our approach reduces the reliance on VP-specific hardware and enables greater creative flexibility in both pre- and post- production.



~Missing Image~

This primary focus is to tackle relighting scenes that contain real in-camera IBL (RIC-IBL) sources by:

(1) Introducing a new 3-D reconstruction and relighting pipeline - using Gaussian Splatting
(2) Producing a several datasets to support future work



This application area is Virtual Production (VP) as:

(1) VP relies on RIC-IBL sources for in-camera lighting effects
(2) VP is extremely costly and so the proposed model works towards more accessible filmmaking
(3) VP technology is primitive with a high reliance on pre-production and background and lighting design. So, production pipelines do not yet consider the variety of tasks that could be automated. This includes photorealistic footage relighting.



As we tie together Graphics, VS filmmaking and Gaussian Splatting, it is important to outline what this paper is and is not.

This paper is:

(1) An attempt to provide lighting editability and customisability in post-production
(2) An novel attempt at VSR with a focus on producing arbitrary output variables (AOVs)
(3) A geometry-independent approach to relighting real static 3-D scenes using Gaussian Splatting
(4) Attempting to provide additional baselines to support objective studies and future work


This paper is not:

(1) An approach to 3-D inverse rendering - we do not model material properties like albedo/roughness
(2) A synthetic lighting pipeline for GS - we do not rely on synthetic light sources for relighting
(3) An active solution to VSR. This is new research - a couple of steps exist between here and actual application.

BibTeX

@article{...}