The current crop of films featuring digitally de-aged characters has brought new life to older actors, and now a pioneering example of age regression for a TV series shows just how much the technology is evolving.
The quest for the fountain of youth is over, at least in Hollywood, where a new clutch of films features leading actors playing much younger versions of themselves.
In Ang Lee’s Gemini Man, Will Smith plays both 50-year old assassin Henry Brogan and 23-year old ‘Junior’, who is sent to kill him, while in Martin Scorsese’s The Irishman, Robert De Niro and Al Pacino are digitally de-aged to portray younger versions of hitman Frank Sheeran and Teamsters boss Jimmy Hoffa, respectively.
Disney first explored the idea of getting an actor to play against himself for Gemini Man almost two decades ago, but the technology was nowhere near advanced enough.
- Read more: Behind the scenes: Gemini Man
Fast forward to 2016, and the project was jolted back to life with the hiring of Ang Lee as director, who got fully behind the concept of a man facing a younger version of himself in an action thriller. But, according to producer Jerry Bruckheimer, “the creation of Junior is not de-aging. This is a one hundred percent digital human character as portrayed by Will Smith.”
Weta Digital, behind such CG character creations as Gollum and many other digital creatures of recent blockbusters, was brought onboard to create Junior, using facial motion capture and performance capture techniques.
In production notes for Gemini Man, Weta’s VFX supervisor Guy Williams revealed: “We had to scour every single piece of footage we could find, and every photograph of Will at the age of 23 so that we could start building Junior. We were also working with Will where we put him in front of eight calibrated cameras capturing all the positions of his face.”
According to the film’s VFX supervisor Bill Westenhofer, ‘an army’ of artists at Weta captured and recreated Smith in minute detail, even down to the enamel on his teeth. “All of these details are replicated, so that when we create the digital version of Junior based on Will’s performance, we can add those details,” says Westenhofer.
When shooting a sequence in which Junior but not Henry was in the scene, Weta had Smith perform on set wearing a helmet rig to track his facial animation. This allowed him to perform the role with the other actors in frame. “There’s a stereo pair of cameras hanging in front that are looking back at Will, with little tracking dots on his face,” Williams explains. “Those cameras allow us to understand what expressions he uses so we can make sure that we’re faithful to his performance in post-production.”
During filming, specially designed and placed cameras allowed Smith to see himself projected on a screen as Henry, so that when he was acting ‘opposite’ Junior, he could play against his own lines. “We wanted to make sure that the timing of the scenes with Henry and Junior happened naturally, so we had his real self with his real dialogue,” says Westenhofer. “Then we re-did the scene in motion capture for Junior.”
Ready for the closeup
Like Gemini Man, The Irishman had spent years in development hell, but with the promise of new technology from ILM, Scorsese was able to make his return to historical gangster drama, with financial backing from Netflix.
ILM had to create a digital facelift for De Niro (75), Pacino (79) and Joe Pesci (76), to be deployed as younger versions of the characters in various scenes spanning the latter half of the 20th Century. Precise details of the workflow used by VFX supervisor Pablo Helman was still under wraps at the time of writing, but by all accounts, it was responsible for a large part of the $160 million budget on the project. Reportedly, with De Niro against any form of helmet camera, facial markers or retakes for VFX, ILM developed a proprietary system of facial capture and used this motion and geometry data to directly drive CG versions of the younger characters. Cinematographer Rodrigo Pietro shot the actors with a special rig consisting of two digital cameras positioned on either side of the director’s camera to capture the data in the same lighting on set as the main performance.
Marvellous de-aging medicine
The Marvel Cinematic Universe (MCU) is the franchise to beat when it comes to de-aging (or, as in some films, digital ageing) characters. Last year we saw Samuel L. Jackson and Clark Gregg, respectively playing their recurring MCU characters Nick Fury and Phil Coulson, get de-aged by 25 years by Lola Visual Effects for Captain Marvel.
Driven by the performance of the actor and based on visual references from Jackson’s 1990s films and research on how we change with age, a digital Fury was used in almost the whole film. With around 60 artists on the project, the facial de-aging, wardrobe and posture adjustment was carried out in painstaking 2D compositing rather than 3D modelling replacement, using a combination of Autodesk Flame and The Foundry’s Nuke.
- Read more: Behind the scenes: Captain Marvel
Christopher Townsend, the award-winning visual effects supervisor on that film - and many others in this genre - has been involved with de-aging characters for over a decade. “We first used the 2D techniques on Captain America: The First Avenger with our work with ‘Skinny Steve’,” says Townsend. “Using various compositing tools, smoothing, patching, painting and warping, we have manipulated actors so that they appear younger, smaller, stronger or thinner.
“Using various compositing tools, smoothing, patching, painting and warping, we have manipulated actors so that they appear younger, smaller, stronger or thinner.” Christopher Townsend, VFX supervisor
“Because we are always basing the work on the actual photography of the actor, we have to be incredibly cognizant of the actor’s intent, charisma and performance nuances,” he adds. “Stepping on any of that changes the moment; reshaping the curl of a lip in the wrong way can change the performance from a smirk to a smile, which will alter the emotion of a scene.
“On top of the performance intent, photorealistic reality is absolutely key; if we don’t capture all the subtleties of the physical performance, we slip into the uncanny valley,” he continues.
The uncanny valley is when a simulation of a human looks not quite artificial enough and yet not lifelike enough to the viewer.
Townsend says compositing techniques and toolsets have developed over the years, and now definitely speed up the processes. “Ultimately we’re relying on the skills and the eye of the artists using those tools; their understanding of physiology, of the way the light falls across smoother rather than rougher skin, their attention to the tiniest detail, this all ultimately dictates the success of an individual shot,” he says.
The impact on production varies too. “Depending on the specific actor, and the age range we’re trying to hit, sometimes we need to use a body double to guide us as a reference of how a younger person would look in a particular shot; sometimes we need to digitally skin graft some parts of the younger face onto the older,” he explains. “This can impact production as we need to shoot each scene twice, with the actor, then with the double mimicking the original performance. However, in some instances, such as Nick Fury in Captain Marvel, we were able to use just Samuel L. Jackson’s performance and manipulate the photography to recreate the younger Fury; that sped up the shooting schedule considerably.”
Townsend says the VFX team involved in the de-aging avoid impacting on what the director or cinematographer want to shoot; they’re more likely to adapt and try to follow what happens on set.
“We’re involved from an early stage, so that we know what other reference we’ll need to capture, but it’s a matter of us adjusting our approaches, rather than dictating what the shots need to be,” he says. “Camera angles and lighting affect everything; people look very different through different lenses, so trying to maintain a consistency throughout a film is one of the biggest challenges.”
Small screen shapeshifting
De-aging is no longer confined to the big screen. For an episode in the HBO series The Righteous Gemstones, Gradient Effects used an AI-assisted workflow and its proprietary Shapeshifter tool to de-age actor John Goodman for an entire episode.
“Shapeshifter is the first of its kind – an entirely new way to de-age people,” says Olcun Tan, owner and visual effects supervisor at Gradient Effects.
Tan says the technology was developed to help to capture motion information from the actual plates, without the need for motion capture equipment. Using the source footage, Shapeshifter continuously captures the face, including all of its essential details, as its guide. With the data being constantly logged, artists can extract movement information from anywhere on the face.
To turn the clock back to 1989 on Righteous Gemstones, Shapeshifter analysed the underlying shape of Goodman’s face. It then extracted important anatomical characteristics, like skin details, stretching and muscle movements. With the extracted elements saved out as layers to be reapplied at the end of the process, artists could start reshaping his face without breaking the original performance or footage. Artists could also tweak additional frames in 3D as required, but with such a process hardly called upon, the de-aging process was nearly all automated.
In comparison to the months required for film de-ageing work, on Righteous Gemstones, Gradient had 35 people working for six weeks with Shapeshifter.
One sequence in the episode shows stage crew walking in front of the de-aged Goodman.
“In the past, a studio would have recommended a full CGI replacement for Goodman’s character because it would be too hard or take too much time to maintain consistency across the shot,” says Tan. “With Shapeshifter, we can just reshape one frame and the work is done.”
As it is based on the actual footage, the process carries no impact for the performance, on set lighting or cinematography.
“Most productions are limited by time or money…we can turn around award-quality VFX on a TV schedule, opening up new possibilities for shows and films.” Olcun Tan, Gradient Effects
“It actually brings everyone closer in the process,” says Tan. “It will further collaboration between makeup departments and the VFX process. While most productions are limited by time or money, we can turn around award-quality VFX on a TV schedule, opening up new possibilities for shows and films.”
Longevity of legends
Projects like The Irishman and Righteous Gemstones are bringing a concept long accepted in sci-fi and fantasy blockbusters, where CGI is the norm, to a new drama and TV comedy audience. It’s an indication of the artistry and technology involved that this audience seems to completely accept the artifice and just enjoys the story.
As a result, some older ‘serious’ actors now see de-aging as a way to extend their careers – and directors are sure to welcome such experience made newly available.
Speaking before The Irishman premiere at the BFI London Film Festival, Martin Scorsese called CGI an ‘evolution of make-up’. “You accept certain norms in make-up; you know he’s not that old, she’s not that young, you accept the illusion,” he said.
No comments yet