What is a LED Virtual Production Studio? | In-depth Technical Analysis
In 2020, the rise of XR extension technology has brought about new changes in the film and television industry. LED virtual production based on LED background walls has become a hot topic in the industry. The combination of XR (Extend Reality) technology and LED display has built a bridge between virtual and reality, and has made great achievements in the field of virtual film and television production.
This article analyzes and summarizes the LED virtual production technology and solutions based on the research and development and testing of dozens of LED studio solutions provided by LEADING LED TECH™. This article will focus on an in-depth analysis of the overall production process, integrated equipment application, core advantages, etc., to provide a reference for the successful implementation of more film and television virtualization production projects.
What is LED Virtual Production Studio?
LED Virtual Production Studio is a comprehensive solution, tool, and approach. We define LED virtual production as “real-time digital production”. In actual use, LED virtual production can be divided into two types of applications: “VP studio” and “XR extended studio”.
What is VP Studio？
VP studio is a new type of film and television shooting method, which is more used for filming and TV series. It enables film and television producers to replace green screens with LED screens and display real-time backgrounds and visual effects directly on the set.
What are the advantages of VP studio shooting?
1. The photography space is liberalized, and the shooting of various scenes can be completed in the indoor studio. Whether it is a forest, grassland, or snow-capped mountains, it can be created in real-time using the rendering engine, which greatly reduces the cost of framing and shooting.
2. The whole production process is simplified, “what you see is what you get”, during the shooting process, the producer can view the desired shooting picture on the screen in time, and the scene content and narrative space can be modified and adjusted in real-time, which greatly improves the performance of the film. The efficiency of scene and scene change.
3. The performance space is immersed. The actors on the set stand on the XR stage where the scene is generated in real-time, and can perform in the immersive space and experience it directly. The performance of the actors is more realistic and natural. At the same time, the light source of the LED display provides real light and shadow effects and delicate color performance lighting for the scene, and the shooting effect is more realistic and perfect, which greatly improves the overall quality of the film.
4. Shorten the return on the investment cycle. Compared with the traditional time-consuming and labor-intensive film shooting process, virtual filming and production can realize the release of the film faster because of its high efficiency and greatly reduce the cycle, saving actors’ remuneration and staff expenses. cost, greatly reducing the cost of shooting.
This kind of virtual film production based on LED background walls is considered a huge development in film production, providing a new impetus for the future of the film and television industry.
What is XR Extended Studio？
XR extended studio refers to the use of visual interaction technology to combine the real and virtual through the production server, and use the LED display to create a virtual environment that can be interacted with humans and machines, bringing the audience a seamless connection between the virtual world and the real world. The “immersion” of the conversion.
Why use XR studio and what are the advantages?
The XR Extended Studio can be used for live webcasts, live TV broadcasts, virtual concerts, virtual evening parties, and commercial shooting. XR extended studio shooting can expand virtual content beyond the LED stage, superimpose virtual and reality in real-time, enhance the audience’s sense of visual impact and artistic creativity, and allow content creators to create unlimited possibilities in a limited space, and pursue endless possibilities. Visual experience.
What is the Workflow of LED Virtual Production Studio?
In the LED virtual production studio, the entire shooting process of “VP studio” and “XR extended studio” is roughly the same, which is divided into four parts: pre-preparation, pre-production, on-site production, and post-production.
What is the VP studio shooting process?
The biggest difference between VP film and television production and traditional film production methods lies in the changes in the process, and the most important feature is “post-preparation”. VP film and television production move the 3D asset production and other links in traditional visual effects films to the actual filming of the film.
In the past, the virtual content produced in the pre-production can be directly used for the on-site visual effects shooting in the camera, while the post-processing links such as rendering and synthesis are moved to the shooting site, and the composite picture is completed in real-time, which greatly reduces the workload of post-production and improves the production efficiency. In the early stages of video shooting, VFX artists use real-time rendering engines and virtual production systems to create 3D digital assets. Next, use the seamless splicing LED display with high display performance as the background wall (usually presented as a ring screen or a spherical screen) to build an LED stage in the studio, and load the pre-production 3D rendering scene through the XR virtual server. The LED background wall creates an immersive virtual scene with high picture quality. Then, an accurate camera tracking system is used, and the object position tracking and positioning technology is used to track and shoot the object. After the final shooting is completed, the captured material is sent back to the XR virtual server through a specific protocol (Free-D) for viewing and editing.
What is the XR Extended Studio Shooting Process?
The steps for an XR stretch shot are roughly the same as for a VP studio shot, but in a VP studio shot the entire shot is usually captured in-camera without the need for stretch.
In the XR extension studio, because of the particularity of the extension of the picture, there are more links to expand the “background” picture in the post-production. After the shot material is sent back to the XR virtual server, it is necessary to extend the scene to the outer cone and screenless area through the method of an image overlay and integrate the real scene with the virtual position. Achieve more realistic and immersive background effects. Then, through image processing technologies such as color calibration, positioning correction, color matching and edge overshoot, the harmony and unity of the images inside and outside the screen are realized, and the expanded overall image is finally output. In the background of the director system, you can see and output the completed virtual scene. On the basis of extended reality, XR extended shooting can also add motion capture sensors to achieve the interactive effect of AR tracking, and performers can interact with virtual elements in three-dimensional space instantly and unlimitedly on the stage.
What equipment are used in LED virtual production?
The LED virtual production studio is a fusion of technologies. The required hardware and software equipment includes a virtual production server, rendering engine, camera tracking system, lighting system, LED display, etc.
Only through the perfect integration of these hardware and software systems, can fantastic and cool visual effects be captured and the final effect be achieved. Although the LED display of the XR extension studio has a small construction area, it needs low-latency features to support live broadcasts, requires greater data transmission and real-time interaction, and requires a system with stronger performance to support real-time image processing. VP studio LED construction area is large, but because there is no need for screen expansion, the system requirements are relatively low, but high-quality image shooting is required, and the configuration of other equipment such as virtual engines and cameras must reach professional level.
Virtual production server:
It is the infrastructure that connects the physical stage with the virtual scene, highly integrated LED display hardware, control system, content rendering engine and camera tracking.
The XR virtual production server is the core of the virtual shooting workflow. It is responsible for accessing the camera tracking system + virtual production content + real-time images captured by cameras, outputting virtual content to the LED wall, and outputting the synthesized XR video images to the director station for live broadcast and storage. The most common virtual production systems are: Disguise, Hecoos.
Rendering Engine:
The rendering engine of video production is the performer of various latest graphics technologies. The rendering engine applied to video production is like the engine of a car, which determines the realism, fluency and attractiveness of the program screen. The pictures, scenes, color effects, etc. seen by the audience are directly controlled by the engine. The realization of these effects includes many rendering techniques: ray tracing – image pixels are calculated by particles of light; path tracing – rays are reflected back to the viewport Calculations; Photon Mapping – Light source emits “photons” calculations; Radiosity – Lighting paths reflected from scattering surfaces into the camera.
The most common rendering engines are: Unreal Engine, Unity3D, Notch, Maya, 3D MAX.
Camera Tracking System:
It consists of a complete set of equipment such as a positioning machine, rocker arm, camera stabilizer, pan/tilt, encoder, and binocular stereo vision machine.
Camera position tracking refers to the process of tracking the position and pose of a camera in real space. The captured spatial information is transmitted to the rendering engine in real-time, providing the virtual camera in the engine with the position information of the real camera in the three-dimensional space, as well as lens parameter information, including focal length, focus, and aperture. Relying on real-time rendering and real-time keying and synthesis technology, the movement and composition of the virtual camera and the real camera are synchronized, so that the real-time picture and virtual elements can be synthesized in real-time. Camera real-time dynamic tracking technology is divided into three types: grid recognition – a tracking method based on the slight movement of the camera in the two-dimensional X, and Y axis plane to perform real-time calculations, without the need to modify the camera tracking method; mechanical sensor tracking – installed on the base On a stand or on a tripod, provides a very accurate high-resolution solution; IR Tracking – calculates camera position via IR and transmitter.
Lighting system:
Under the premise that the existing natural light source of the scene is insufficient, the lighting system must artificially enhance the lighting in the scene to enhance the sensitivity of the film and improve the brightness of the picture, which plays an important role in the establishment of space and the creation of atmosphere.
Basic Lighting – In the case of the basic lighting provided by the LED background wall, it is still necessary to use lighting equipment to supplement the lighting to make up for the problems of the background wall lighting with narrow spectrum, low color rendering, no directionality, and no hard light; Supplementary Lighting-LED Lighting Matrix+ Soft light makes up for the problems of narrow spectrum, low color rendering, no directionality, and no hard light in background wall lighting; special effects lighting – lights that are temporarily arranged according to the needs of the scene + plot in a specific shooting scene; lighting control system – CNC lighting It can be directly driven and controlled by the virtual engine and XR fusion system through DMX/Artnet and other protocols.
Commonly used lighting systems are: ARRI.
LED display:
LED virtual production studio is a new scenario for large-screen display applications, a new market derived from the continuous development of the LED small-pitch market and the continuous improvement of the technical level of LED display equipment. Compared with the traditional LED screen application, the virtual LED display screen has more accurate color reproduction, dynamic high refresh, dynamic high brightness, dynamic high contrast, wide viewing angle without color shift, high-quality picture display, etc. for stringent requirements.
What special requirements does the LED virtual studio have for the specifications of the LED display?
After hundreds of tests and actual construction experience by our R&D department, the display screen used in the LED virtual studio needs to meet at least the following conditions: can suppress moiré, low reflection, wide color gamut and high CRI, wide viewing angle without a color cast, Low scan and high refresh.
Can suppress moiré：
Moiré is the high-frequency interference formed when the LED screen is imaged on the photosensitive element of the camera device, which causes colored stripes to appear in the picture. The Mini 4-in-1 technology can make the LED display achieve a luminous aperture ratio of more than 80%. Mini LED is matched with special types of lens, depth of field and aperture, which can obtain wider adjustment freedom, lower degree of defocus, cleverly avoid the situation that the display screen produces moiré in the finished film, and finally realize the realistic immersion of the shooting angle. formula effect.
Low reflection:
The smaller the light reflectivity, the smaller the impact on the screen surface, and a more realistic background picture can be achieved. On the surface of the Mini small-pitch screen, there is a unique light-absorbing groove design between the lamp groups to reduce the influence of light; and the surface adopts a matte matte process to control the light reflectivity below 5%, and the shooting angle can be close to 180 degrees . Easily cope with various light environments on the shooting site, the picture is cleaner, and the shooting is not easy to wear.
Wide color gamut and high CRI:
The application of color gamut matching and conversion technology can truly and accurately restore natural colors, and the color gamut of Alto LED display can reach 110% NTSC color gamut after correction. High CRI ensures good color uniformity, optimizes the continuous spectrum of each color and each band, and improves the color rendering index. At the same time, the luminous efficiency is improved, and under the premise of ensuring the luminous efficiency, part of the blue light energy is suppressed, and the red (R9) spectrum is compensated.
Wide viewing angle without color cast:
Mini LED products can provide a wider shooting angle of nearly 180° to ensure no color cast; 24bit processing depth can better present the level and details of the image, and can show good visual effects from any angle of observation or shooting.
Low scan and high refresh:
With a 1/6 scan rate and a refresh rate of 7680Hz, the Mini LED small-pitch screen has a stable shooting image under a high refresh rate, no ripple black screen, clear image edges, and accurately restores the real information of the image.
