From Avatar to Dawn of the Planet of the Apes, motion capture has enabled animation to become ever more lifelike and photo realistic. Now Disney Research is advancing this science with emotion capture, and systems that offer filmmakers continuous control of facial performance in live action motion pictures.

“People can be surprised that a certain character in a film is digital – a prime example being the Winklevoss twins in The Social Network – but with a trained eye you can discern the nuances,” says Disney Research scientist Dr. Derek Bradley, who told IBC2016 all about it in a special conference session. “The biggest issue is when motion starts. You can make a pretty realistic render of a face, but this can break down if you want real subtlety, especially when rendering a known person.”

Source: YouTube

With a trained eye you can spot if a character is digital

The Canadian studied computer science in Ottawa and gained a Ph.D. from the University of British Columbia, before being recruited by Disney Research in 2010.

“I had studied the 3D reconstruction of objects from cameras, in particular objects like clothing and faces that can deform over time,” he explains. “I was driven by a love of movies and VFX and hoped one day to apply these techniques to films. I was quite fortunate that when Disney was starting up a new office in Zurich, [Lab Director] Markus Gross happened to be in Vancouver, came by for a visit, saw my research and invited me to come over.”

It was the right call. Bradley’s efforts have helped result in the Medusa Performance Capture System, a mobile rig of cameras and lights coupled with proprietary software that can reconstruct actors’ faces in full motion, without using traditional motion-capture dots.

Medusa performance capture system

Medusa performance capture system

Source: Disney Research

“We know where every point in an expression moves on the face, which is a key ingredient for VFX artists building a facial animation rig for an actor.” So key that Medusa has been used on such high-profile features as Star Wars: The Force Awakens, Maleficent, and The Jungle Book.

While these films are pushing towards the ultimate merger of production with post production – virtual production – Bradley thinks there’s a way to go yet.

Improvements ahead

“Real-time facial performance capture is not at the quality or resolution of offline processes. Every stage of the pipeline can still be improved. You can achieve a good representation of a face in real time, but it won’t exactly model their expressions.”

Among Bradley’s other projects is FaceDirector, a prototype intended to reduce the number of takes required for performance capture during principal photography.

“FaceDirector blends together different takes of an actor’s performance,” he explains. “You could have a single take where the actor is angry, another take where they are sad, then choose the moments of transition in post.”

Source: Disney Research

Of Zurich, Bradley says there’s a growing tech scene with IBM, Google and several spin-offs located nearby. “One of the reasons Disney is based here was because it had the smart idea to partner with universities which were already strong in the areas it wanted to research. It chose to base next to the campus of the Swiss Federal Institute of Technology because it’s one of the world’s leading centres for computer graphics.”

“We know where every point in an expression moves on the face”

The ultimate goal is to create a fully realistic digital human, but to get there, filmmakers must overcome the ‘uncanny valley’, the curious disconnect between actual humans and digitally constructed ones.

“The trouble is no-one knows exactly what it is or how to fix it,” he says. “Most people believe it’s a problem around the eyes and mouth. We have focused some research into that area.”

If anyone can crack it, then you can bet it will be Bradley and the team of world-class scientists at Disney Research.

PROFILE

Derek bradley

Derek bradley

Derek Bradley

Research Scientist, Disney Research

Region: Switzerland

Dr. Derek Bradley is a Research Scientist at Disney Research Zurich. He completed his Bachelor of Computer Science in 2003 and Master of Computer Science in 2005, both at Carleton University in Canada. In 2010, Derek obtained a PhD from the University of British Columbia in Canada, and then started with Disney Research in September 2010.

Derek’s main research interest is in creating realistic digital humans for visual effects in film. This includes new technology in both computer graphics and computer vision, designed for 3D reconstruction of human faces, performance capture, facial animation and data-driven simulation.

 This interview was first conducted for IBC2016.