Gaussian splatting virtual production

Luma AI’s Dream Machine and its earlier 3D capture tools use NeRF (Neural Radiance Fields) and Gaussian splatting (3DGS) to turn a set of ordinary photos or video frames into a dense, explorable 3D scene that can be viewed, navigated, and — with the right pipeline — imported into real-time engines like Unreal Engine. For virtual production studios working with LED volumes and in-camera VFX, this matters because it offers a faster, cheaper path to photoreal digital environments than traditional photogrammetry or manual 3D modelling. At Sinfull Studios in Regina, Saskatchewan, we see this as one of the more practical recent developments in the capture-to-production pipeline.

What Is Gaussian Splatting and How Does It Differ from NeRF?

Both NeRF and 3DGS reconstruct a 3D scene from 2D images, but they do it differently. A NeRF trains a neural network to predict colour and density at any point in space, which produces high-quality results but is computationally heavy — both to train and to render. Gaussian splatting instead represents the scene as a large collection of small, semi-transparent 3D ellipsoids (“splats”), each with its own colour, opacity, and orientation. Rendering those splats is far faster because it maps better to GPU rasterisation pipelines, which means you can navigate a 3DGS scene in real time at reasonable resolutions without waiting for ray-march convergence. For production work, that interactivity matters: you can actually move a camera through the space and judge framing before you commit to a shoot.

What Does Luma AI Actually Do?

Luma AI built its initial reputation on NeRF-based capture — you shoot a subject or location with your phone, upload the footage, and Luma’s cloud processing returns an interactive 3D scene you can embed or share. More recently the platform has moved toward Gaussian splatting as the underlying representation for at least some of its capture workflows, because of the real-time rendering advantages. Dream Machine is Luma’s generative AI video model, which is a separate product — it synthesises video from text or image prompts rather than capturing real-world geometry. It is worth keeping those two things distinct: the capture tools and the generative model are not the same pipeline, and they solve different problems for production.

Where Does 3DGS Shine for Virtual Production?

  • Speed of capture to usable asset: A location shoot with a phone or mirrorless camera can produce a navigable 3D environment in hours, not weeks. That is a meaningful compression of the traditional photogrammetry-to-mesh-to-texture workflow.
  • Photorealism at capture: Because the scene is derived directly from real photographs, the lighting, surface detail, and atmospheric character of the actual place are baked in. This is hard to fake with procedural assets.
  • Previs and camera planning: Even before any pixel hits an LED wall, the 3DGS scene lets a director and DP plan shots inside the real captured space.
  • Reference for environment artists: A splat capture of a Saskatchewan location gives the Unreal Engine environment team an accurate spatial and visual reference that goes far beyond a photo library.

What Are the Current Limits You Need to Know?

3DGS is not a replacement for a clean, editable mesh, and that gap matters in production. Relighting a Gaussian splat scene is genuinely hard — the colour of each splat encodes the lighting from the original capture, so changing the light source means re-synthesising or overriding data rather than just moving a light in the engine. Editability is similarly constrained: you cannot easily extract a clean polygon mesh from a splat cloud to retopologise, UV-unwrap, or texture-paint. Transparency and fine structures like hair, foliage edges, or chain-link fences are also areas where current 3DGS reconstructions can produce artefacts. If you need a set piece that will be animated, collided with, or composited against live action with dynamic relighting, you are likely still combining 3DGS reference with conventional modelling rather than using the splat directly as a final asset.

How Does This Feed an Unreal Engine Virtual Production Pipeline?

Unreal Engine has been adding native and plugin-based support for Gaussian splat rendering, which means a captured scene can be loaded directly into UE and used as a background element on an LED volume. The practical workflow at a studio like Sinfull Studios might run like this: capture a Saskatchewan exterior location with video footage, process it through Luma AI or an equivalent tool into a 3DGS asset, bring that into Unreal Engine alongside nDisplay for the LED wall, and use it as the photoreal background plate for an interior scene shot on stage. The splat handles the distant environment and the character or vehicle lives in the foreground under real-stage lighting. You get the authenticity of the actual place without the logistics of shooting on location.

What Does Capturing Saskatchewan Locations Actually Look Like?

Saskatchewan has a specific visual character — big sky, flat horizon, grain elevators, weathered industrial structures, river valleys, boreal edge — that is genuinely hard to replicate with generic asset packs built for international markets. Capturing those locations as 3DGS assets means a Regina-based production has access to a library of real, local environments that can be dressed and scaled on an LED volume without pulling a full crew out of the city. The capture process is approachable: overlapping video passes around a subject or along a location, processed in the cloud. The main investment is in the processing pipeline and in the Unreal Engine workflow that makes those assets production-ready.

Is This Ready for Production Use Today?

For background environments, previs, and reference, yes — 3DGS capture is production-viable right now. For hero set pieces that need full relighting and geometry editing, the technology is still catching up. The honest position is that 3DGS fits best as one layer in a hybrid workflow: use it for the photoreal environmental context where the baked lighting is acceptable, and lean on conventional Unreal Engine assets where you need full control. The tooling is improving quickly — relightable Gaussian splatting is an active research area — so the balance of what you can rely on natively versus what needs supplementing will shift over the next few production cycles.

Explore Virtual Production with Unreal Engine at Sinfull Studios for more.

Frequently Asked Questions

Can Luma AI captures be used directly in Unreal Engine for LED volume virtual production?

Yes, with caveats. Gaussian splatting assets from tools like Luma AI can be imported into Unreal Engine via native or plugin-based 3DGS support and used as background environments on an LED volume. They work best for distant environments where the baked lighting is acceptable. For elements that require dynamic relighting or clean mesh geometry, you will typically combine the splat capture with conventional Unreal Engine assets.

What is the difference between NeRF and Gaussian splatting (3DGS)?

Both reconstruct a 3D scene from 2D photographs, but they use different representations. A NeRF trains a neural network to predict colour and density at arbitrary points in space, which is high-quality but slow to render. Gaussian splatting represents the scene as a cloud of small 3D ellipsoids that can be rasterised rapidly on a GPU, enabling real-time navigation of the captured scene. For interactive previs and LED volume use, the real-time rendering advantage of 3DGS is significant.

What are the main limitations of Gaussian splatting for film and virtual production today?

The three main limits are: relighting (splat colour encodes the original capture lighting, making dynamic relighting difficult), editability (it is not straightforward to extract a clean polygon mesh for retopology or animation), and handling of fine transparent structures like foliage edges or hair. For backgrounds and environment reference these limits are often acceptable; for hero assets that need full production control they typically require supplementing with conventional 3D modelling.

Related reading from Sinfull Studios

Planning a virtual production, Unreal Engine, or VFX project in Regina or anywhere in Saskatchewan? Request a quote from Sinfull Studios.