In our last technical deep-dive, we walked through our Mo-Sys StarTracker Max setup — the optical tracking system that gives us sub-pixel-accurate 6DoF camera position data. But tracking data on its own doesn’t make a virtual production shot. It’s what happens after the tracking data leaves the StarTracker that turns a green screen stage into a living, breathing virtual world.
This post is about the other half of the equation: how we take that stream of positional data and turn it into a real-time composite inside Unreal Engine, with a virtual camera that moves in perfect sync with the physical one on our stage floor.
The Signal Chain
Understanding our pipeline means understanding the journey of a single frame. It starts with the StarTracker reading the retroreflective dot pattern on our ceiling and calculating the camera’s exact position and orientation in 3D space. That positional data — X, Y, Z coordinates plus pan, tilt, and roll — travels over our 10Gb local network link to the render node running Unreal Engine. The entire hop takes under two frames of latency.
On the render node, Mo-Sys VP Pro receives the tracking stream and maps it onto a virtual camera inside the Unreal scene. Every physical movement of our Blackmagic Pyxis 6K on set is mirrored by the virtual camera in the engine — pan left on the dolly, and the CG background pans with you. Push in on the subject, and the parallax in the virtual environment shifts exactly as it would if you were shooting on a real location.
This is where the green screen compositing approach diverges from an LED volume workflow. On an LED stage, the rendered output goes to the wall panels and the camera simply photographs it. In our setup, the rendered output is composited with the live camera feed in real time — the green is keyed out and replaced with the virtual environment. The result is essentially the same on the monitor, but the path to get there is different, and that difference has practical implications we’ll get into.
VP Pro: The Engine Inside the Engine
Mo-Sys VP Pro is the plugin that ties everything together. It sits inside Unreal Engine as a native plugin, which means we’re not bolting on external compositing software or routing signals through additional hardware. The compositing happens within the same application that’s rendering the virtual world.
VP Pro handles several things simultaneously. It ingests the live video feed from our camera via SDI capture, receives the tracking data stream, drives the virtual camera, runs an internal chroma keyer, and outputs the final composite — all within the Unreal render loop. Because it’s native to the engine, we have full access to Unreal’s rendering features: ray tracing, Lumen global illumination, Nanite geometry, post-processing volumes, and everything else the engine offers.
The practical benefit of this integration is that what you see on the monitor during the shoot is extremely close to the final output. The director, the DP, and the client can make creative decisions about the virtual environment in real time — adjusting lighting in the CG scene, swapping background elements, or shifting the time of day — and see the result immediately composited with the live action.
Lens Metadata and Why It Matters
Camera tracking gives you position and rotation, but there’s another critical data stream that often gets overlooked: lens metadata. Our StarTracker system also captures focal length, focus distance, and aperture data from the lens. This information is fed into Unreal Engine alongside the positional data, and it’s what allows the virtual camera to match the physical camera’s field of view and depth of field in real time.
Without accurate lens data, you’d get a composite where the perspective of the CG background doesn’t quite match the live-action foreground. It’s one of those things that audiences can’t necessarily articulate, but they can feel — something just looks “off.” With proper lens metadata flowing into the engine, the virtual camera’s focal characteristics update frame by frame, keeping the composite photorealistic even when the camera operator racks focus or zooms during a take.
Getting this right requires careful calibration at the start of each shoot day. We run a lens calibration process that maps the physical lens characteristics to the virtual camera model inside Unreal. It takes a bit of time upfront, but it saves hours of potential headaches in post.
The Compositing Layer Stack
A real-time composite isn’t just “foreground over background.” Our typical setup uses a three-layer stack within VP Pro: the CG background (everything behind the talent), the keyed live-action plate (the talent and any physical set pieces), and a CG foreground layer (elements that sit in front of the talent, like virtual set dressing, particles, or atmospheric effects).
This layered approach is what allows us to create convincing depth in our shots. A character can walk behind a virtual pillar, or rain can fall in front of them, all composited in real time. The keyer separates the green screen cleanly, and the foreground and background layers are rendered with proper depth sorting based on the tracked camera position.
We also use this layering to handle interactive lighting. If the virtual environment has a warm sunset, we can feed colour information from the CG scene back to our physical lighting rig on set, matching the ambient light on the talent to the virtual world they’re standing in. This is an area where green screen compositing has caught up significantly — it used to be a major selling point of LED volumes that the light from the wall naturally spilled onto the actors, but with smart lighting integration, we achieve a similar result with more control over each element independently.
Network Architecture and Keeping Latency Low
Latency is the enemy of real-time compositing. Any delay between a camera movement and the corresponding update in the virtual environment creates a visible disconnect — the background slides against the foreground, and the illusion breaks. Our 10Gb network link between the tracking PC and the render node is a deliberate choice to keep that chain as short and fast as possible.
We run a dedicated local network for tracking data, separate from any studio Wi-Fi or internet traffic. There’s no switch congestion, no competing traffic, no variable latency. The tracking data has a clear, predictable path from StarTracker to VP Pro, and we consistently measure under two frames of total system latency — from photon hitting the tracking camera sensor to updated pixel on the output monitor.
Two frames at 24fps is roughly 83 milliseconds. At normal camera movement speeds, that’s effectively imperceptible. It’s only if you whip-pan aggressively that you’d start to notice, and even then, the motion blur in the live-action plate helps mask it. For the vast majority of shots — dialogue scenes, product reveals, slow dolly moves, music video performances — the latency is invisible.
What This Means for Productions Booking With Us
If you’re considering a virtual production shoot at Vedri North Wales, the pipeline we’ve described here is running in the background of every session. You don’t need to understand the signal chain to benefit from it — that’s our job. But knowing that it exists hopefully gives you confidence that when you see a composite on our monitor, it’s not a rough preview that’ll need months of post-production to look right. It’s very close to your finished shot.
The practical upshot is faster turnarounds, more creative control on the day, and fewer surprises in the edit. If you’ve been burned by green screen shoots where the composite never quite came together in post, our real-time pipeline is designed to eliminate that uncertainty. You see what you’re getting before you wrap.
In our next post, we’ll look at how we prepare and optimise Unreal Engine scenes for real-time performance — because the best pipeline in the world won’t help if your virtual environment is running at five frames per second.
