VSR: Virtual studio 3-D reconstruction and relighting with real image-based relighting

Anonymous Submission
Showcase Paper Info Datasets Appendix Code
Abstract:

This paper introduces a Gaussian Splatting (GS) approach to relighting and reconstructing 3-D scenes containing real in-camera image-based lighting (RIC-IBL) sources. This is applied Virtual Studio (VS) film production, where LED walls project live virtual backgrounds that produce RIC-IBL effects. The goal of VS reconstruction and relighting (VSR) is to provide the ability to change the RIC-IBL background texture and propagate the lighting changes to a 3-D scene. As no prior RIC-IBL research exists, this paper establishes a robust platform for future work by introducing a novel pipeline, a handful of baselines for RIC-IBL texture sampling, and several datasets including real VS scenes. The main method is a geometry-independent lighting model that explicitly represents the IBL texture coordinates and lighting intensity as view-dependent GS parameters. This avoids computing physically-based events linked to inverse rendering (IR). Hence, our method does not rely on depth/normal priors, is capable of reconstructing diverse scenes including transparent objects, and can be implemented without custom CUDA or RTX code. The final model only uses less than 5GB of RAM and VRAM for training 1080p scenes. In practice, our method removes the need for costly VS-specific hardware and offers numerous possibilities for revolutionizing the VS production process to benefit post production artistry.



~Missing Image~

This primary focus is to tackle relighting scenes that contain real in-camera IBL (RIC-IBL) sources by:

(1) Introducing a new 3-D reconstruction and relighting pipeline - using Gaussian Splatting
(2) Producing a several datasets to support future work



This application area is Virtual Studio (VS) film production as:

(1) VS production relies on RIC-IBL sources for in-camera lighting effects
(2) VS production is extremely costly and so the proposed model works towards more accessible filmmaking
(3) VS technology is primitive with a high reliance on preproduction stage and background design. So, production pipelines do not yet consider the variety of tasks that could be automated. This includes the work investigated in this paper



As we tie together Graphics, VS filmmaking and Gaussian Splatting, it is important to outline what this paper is and is not.

This paper is:

(1) An attempt to provide lighting editability and customisability in post production
(2) An novel attempt at VSR with a focus on producing arbitrary output variables (AOVs)
(3) A geometry-independent approach to relighting static 3-D scenes using Gaussian Splatting
(4) Attempting to provide additional baselines to support objective studies and future work


This paper is not:

(1) An approach to 3-D inverse rendering - we do not model material properties like albedo/roughness
(2) An active solution to VSR. This is new research - a couple of steps exist between here and actual application.

BibTeX

@article{...}