There’s been a lot of talk about virtual production recently, which combines real-time CG assets delivered by a game engine such as Unity or Unreal Engine, with physical actors and sets. Most famously, The Mandalorian on Disney+ has used the technique to great effect, shooting scenes with a 270-degree LED wall dubbed The Volume.
But what do you need to do to get started with virtual production, and should you even bother?
That’s the first question that Martin Taylor, Director and Co-founder at Prox & Reverie usually asks his clients. Do you actually need virtual production?
“It’s become a buzz phrase, but it’s not necessarily a silver bullet,” Taylor says. “We’ve done a few of these projects now and it’s all about looking at your project and asking very basic questions such as… Is the project for screen? Is there a true benefit from creating 3D real-time assets for the production?
“As an example, we’re doing a true hybrid production at the moment. We’re making a short promotional film, but all of the assets that are produced for the film are going to get drawn down into an interactive simulation experience,” he continues. “So that’s a really strong case for virtual production. But, if it’s just a scene in a regular place like in a house, for example, there’s less reason to use these kinds of techniques.”
Solving problems in pre-production
“Virtual production has enabled an earlier and often more successful collaboration between the VFX dept, director and other key crew such as the director of photography (DoP) and production designer,” says Christian Manz, creative director for film at Framestore.
“It is now easier to solve creative and logistical problems during pre-production that could otherwise have led to issues during the shoot and in post-production. Real-time workflows, digital extensions and environments can be explored and refined with the filmmakers, just as a practical set build would be. They can be visualised in various ways during the shoot itself, which can lead to a much more cohesive end result. Creatures can be conceived and put through their paces at the same time as the actors that will be sharing the screen with them.
- · Watch the webinar replay: Virtual production – a creative revolution
“Tools such as motion capture that have been around for a while have now been joined by innovations such as LED volumes, which have shown that, under certain conditions, real-time, in-camera VFX are coming a step closer every day,” continues Manz. “In short, virtual production means that a key decision or creative discovery at the start of pre-production can make it through to the final rendered pixel.”
Working in a controlled environment
There are many benefits to virtual production, especially when visual effects are involved. Instead of shooting actors on a green-screen stage and compositing the visual environment in later, virtual production can play the rendered content on background screens on stage.
“I think it helps take the edge off for talent,” says Steve Griffith, executive producer for virtual production, DNEG. “It provides better eyelines and gives filmmakers a better idea of what they are making, along with providing shots for editorial to use right away versus green-screen shots.”
Martin Taylor says that his rule of thumb is: “If you need to shoot in a controlled environment for an impossible location, VP might be the answer.” That was the case with the virtual production work on the multilingual period mystery television series 1899 at Studio Babelsberg in Germany, where production company DARK BAY built a dedicated virtual production stage, complete with a turntable and rain rig for water SFX. Scenes in the series required a 19th-century sailing ship, not an easy task.”
- · Read more: Go behind the scenes on Netflix drama 1899
“In a case like that ship, it’s a lot cheaper to do it in a studio using virtual production than you can deliver by going out on the water; it’s similar if you’re doing something in space, or a shoot that we did recently was set in India,” reveals Taylor.
“We’re not currently going to be able to fly all our crew out to India and shoot in a village. So, we built a partial interior, but we then had to figure out how many windows and doors we could put into this set so you can see the location beyond. So, really the new big thing is all about leveraging the technology to speed up production and delivering mostly exterior shots, shot inside a studio, which is amazing because then you’ve got godlike powers to control the weather and the time of day.”
What does virtual production offer a film or TV series?
“Virtual production covers a lot of services, and so each aspect provides something unique,” says Steve Griffith. “There’s real-time (RT) pre-vis, which gives you your movie before you shoot it, and helps everyone to know what needs to be done. Every film should have pre-vis - RT pre-vis just gets you there faster and is cheaper to do.”
Of course, there are a host of other visualisation stages, such as tech-vis and post-vis, that make use of virtual production. “These provide similar insight to what you need to do before you start production, and end up saving you more, by preventing mistakes,” adds Griffith. “SimulCam, a fusion of a real and virtual camera first used on Avatar] is great if you still go with a [non-LED] greenscreen route, but want to get post-vis immediately, see line-ups of the CG components of your scene, and make sure you are framing your shots correctly.”
Then there’s in-camera final pixel, what most people who have seen The Mandalorian think of as virtual production, where the technology combines real actors and physical sets with photorealistic virtual backdrops displayed on LED screens. “This gives your talent reference and a better experience, your director something to shoot, and producers the final VFX immediately on a compressed schedule,” says Griffith.
Ask the right questions
“Do your research,” says Taylor. “Figure out if your project is really in need of virtual production, and which part of virtual production is needed? And if it involves anything to do with an LED wall, find a good vendor partner, because it’s very, very expensive to get into. So don’t try and do it yourself when you first production, if it matters.”
It’s sound advice and is being followed. The DARK Bay facility worked closely with technology companies. The LED volume used ROE Visual LED walls. Tracking systems that made sure the imagery in the volume matched the camera view were from Vicon and Trackmen, while ARRI SkyPanels were in an adjustable ceiling to provide the interactive lighting for the physical set.
According to Framestore’s Christian Kaestner, overall VFX Supervisor on 1899 for Netflix, the technology is ever-evolving. “The evolution of virtual production and real-time technology will have a big impact,” he says. “However, I don’t think it will revolutionise our industry overnight. It will possibly shift and influence how episodic VFX is approached and executed. Most technical achievements do not replace the creative vision or visual result, they do, however, often make it easier or less time consuming to achieve a certain result.
“Who knows, maybe one day we will be doing several ‘visual effects’ takes live with the director and DOP on the day.”
Keeping any eye on the money
Just how much do you need to budget for when using virtual production compared to a normal studio shoot?
“You need to be prepared to make decisions ahead of time, and plan more in advance,” says Steve Griffith. “Virtual production offers great flexibility, but it does mean that other decisions need to be made upfront. All the background environments are chosen and created before shooting, whereas with a traditional green screen process, you have time to figure that out after you’ve shot.”
“I’d never even seen an LED volume, but now I’ve done quite a few in the last two or three months,” says Martin Taylor. “The budgets work differently. One thing you have to consider is what is real versus digital in the mix of the in-camera shoot. You still do need to build sets. You still need to build foreground props. You then have to factor in artists’ time and it’s a different makeup of crew to location shoots as well.”
Yes, crew. There are VP specialists out there, but they’re unsurprisingly in huge demand. For example, there’s the ‘Brain Bar’, the array of workstations on set that actually drive the virtual production, and the operators who run it.
“As with any emerging technology, finding talent with experience is tricky. It’s all so new,” says Griffith. “So, getting the right team who know how to use the tech is critical. From Unreal artists to LED technicians and engineers, it’s all about the operators. We see a lot of LED stages popping up, and major investments into the infrastructure. But we aren’t seeing the right people behind the gear.”
That gear will involve a game engine for rendering the content, but just as important is the tracking system. “Generally, we use just about every motion tracking system out there and Unreal Engine for 3D content, but a variety of other systems for 2D or 2.5D. For LED panels we use ROE BP2 v2 for in-camera VFX and CB5 for lighting. But can use, and have used others, depending on the project. We build our own custom UE workstations, servers, and engineering rooms [too].”
There are several partnerships in this field such as the one formed by Mo-Sys with its StarTracker camera/lens tracking system and LED content server solution VP Pro XR, Bendac Group’s InfiLED LED displays and LED video processing technology from Brompton Technology. Virtual production is becoming very popular, so productions can get kitted out quickly.
Steve Griffith’s advice for choosing a studio is to check the credits list. “Not of the studio itself, but of the team they have hired, and who they are partnered with. Often vendor relationships are key,” he explains.
Producers also need to be aware of costs and have realistic expectations. Griffith says the latter is not currently common. “Producers still think it’s a button-push technology to some degree, and that they can just convert from one methodology to another. A lot of what we do during the budgeting and pre-production phases is educational. We are seeing more producers that have done some of the research and come knowing what questions to ask.”
They also need to build in extra time and investment.
“Pre-production needs to start earlier, and pre-production costs may go up,” says Griffith. “Engaging Art Departments, Production Design, the Director, and others early on is critical. But producers have to pull everything together sooner than normal. The whole mantra is now ‘fix it in PRE-production’. Virtual production does make for a better product, and those that get on board with the ideology end up appreciating the process - it just takes some getting used to.”
Virtual production is growing in popularity. How fast? For extra insight into this area of the industry, read: Virtual production: Ten years of growth achieved in one?
No comments yet