Volumetric capture’s ability to achieve engaging and immersive experiences means it is finding a receptive audience throughout the M&E world, writes David Davies.

In the explosion of new technologies and techniques to have hit the film and broadcast production industry in recent years, it’s possible the term ‘volumetric video’ hasn’t yet achieved the general profile or understanding of – for example – VR or AR. But with an increasing flow of volumetric projects since 2019, and some flagship productions expected to launch in 2023, it surely won’t be the case for much longer.

Dimension - Sony Pictures - I Wanna Dance With Somebody 3

Dimension: I wanna Dance With Sombody

Source: Sony Pictures

What is volumetric video?

So first things first: what is volumetric video? Developed to bring depth and new creative possibilities to video production, the technique involves the use of multiple cameras to capture a three-dimensional space. Typically, the type of content being filmed in this way might include a person, an object or a complete performance. Subsequently, the video can then be replayed with a smart device, a 2D browser on a computer, VR glasses or an AR display – thereby allowing the viewer to see the subject from any angle.

Piotr Uzarowicz

Piotr Uzarowicz, Arcturus

The significant processing and general hardware requirements of volumetric video have, to date, meant that its reach has been somewhat limited. But like every visual technology before it, the costs associated with capture and delivery are gradually falling as processing power increases and the surrounding product ecosystem becomes more comprehensive. Indeed, it is possible that a tipping point for this technique could be on the cards as soon as next year.

‘A post tool was the missing piece’

One of a handful of leading innovators currently focused on volumetric video solutions, Arcturus was established by a creative and executive team with stints at companies including YouTube, Google, Dreamworks and Pixar to their collective credit. The starting point for the business that became Arcturus in 2016 was: “the realisation that there was one missing piece, and that was a post-production and streaming tool for volumetric video. So that is what they set out to build,” explained Piotr Uzarowicz, head of partnerships and marketing at Arcturus.

Read more Accelerator - Volumetric video for broadcasting, metaverse production and RT3D assets

The result of their labours is HoloSuite, a capture-agnostic SaaS post-production platform that offers tools for editing and distributing volumetric video. “It comprises two pieces: HoloEdit, which contains tools for editing and compressing footage so it can be outputted to [immersive creation tools] Unreal or WebXR, and HoloStream, which is a streaming solution for mobile, web, desktop and immersive headsets.”

ARCTURUS Crescent_Stills_Overlay

Arcturus: HoloEdit

Source: Crescent

Arcturus has also consulted and undertaken production work on a number of high-profile projects, including Hulu’s Emmy-nominated ‘Light as a Feather’ Magic Leap activation and Madonna’s volumetric holograms at the 2019 Billboard Music Awards. The latter project – billed as the first-ever live broadcast of a volumetric performance – has proven to be especially significant in terms of conveying volumetric’s long-term potential to a wider audience, although it’s clear that we should expect live volumetric productions to occur more frequently from now onwards.

To this end, Uzarowicz indicates that – rather than inhibiting its development – the pandemic may have provided an opportunity “for more people to take the time to find out about [the technique]. Another interesting aspect is that during the production of volumetric video the artist is alone in that space. You can really separate out the components, so you might have the capture crew in one space, the actor in another room, and the director joining remotely on Zoom. [Volumetric video] made it easy to accommodate the [Covid] restrictions.”

Dimension - Universal Music Group - Coldplay x BTS My Universe Music Video

Dimension: Coldplay x BTS - My Universe Music Video

Source: Universal Music Group

Meanwhile, the HoloSuite tools continue to evolve, with a particular emphasis placed on streamlining the capture and distribution process. For example, an update in late 2021 added full support for Scatter’s Depthkit Studio 3D capture software, enabling content creators to capture, edit and distribute their volumetric projects using one pipeline, thereby making it easier to create holograms for AR/VR, frame media and 360-degree spatial videos.

‘Volumetric and virtual production are intersecting’

The Madonna/Billboard Awards performance was also a milestone for Dimension Studio, for it was at their London facility that the singer’s creative director, Jamie King, and Reflector Entertainment captured four of her ‘personas’ – namely, a musician, cha-cha instructor, secret agent and bride. The volumetrically captured models and effects were played back live through Reflector’s custom-built volumetric capture player, and high-quality playback took place within Unreal Engine 4. The resulting AR performance – which also incorporated special effects such as clouds, rain and butterflies – was presented alongside live components, including Latin singer Maluma and more than a dozen dancers.

Dimension - Sky Sports The Open 2022 MORIKAWApng

Dimension: The Open 2022

Source: Sky Sports

Surveying the current production landscape, Dimension Studio co-founder & co-CEO Simon Windsor highlighted the tendency for “work in virtual production and volumetric video to be intersecting more these days”. It will also come as no surprise to discover that sports are proving to be an early adopter, with the all-encompassing possibilities of volumetric capture lending themselves to detailed analysis of player performance. For example, Dimension’s Polymotion Stage was used during the production of Sky Sports’ free-viewpoint swing analysis segment at the 148th Open Championship, which allowed a full 360-degree inspection of a golfer’s swing. Once golfers had been captured at the studio, their holograms were being used on air only 48 hours later.

While sports and entertainment are arguably leading the field, Windsor says that there are strong signs of “3D techniques [being adopted] across the board for everything from concept visualisation to production and post-production.” He suggests a significant increase in live volumetric capture is around the corner, whilst one of Dimension’s current areas of R&D focus is the use of volumetric video for “digital crowd creation, which leverages the experience and systems we have developed over the last few years. If you are able to film [a crowd] volumetrically, it gives you a lot more flexibility in terms of changing aspects such as clothing, hairstyles and so on.”

‘Volumetric has to fit in with the rest of post’

4Dviews is another company for whom the scope of volumetric video has become increasingly expansive. Established in 2007, the company’s product offer is now led by the HOLOSYS volumetric video capture system, which comprises capture equipment (cameras, lighting and electronics), a software suite with a lifetime product license, installation and operator-customer training. As well as including plug-ins for Unity3D and Unreal Engine, the export option of the Alembic format (.ABC) enables users to natively import all volumetric video in Cinema4D, Autodesk Maya, Houdini and Blender3D.

Simon Windsor Dimension

Simon Windsor, Dimension

The starting point for any creative to begin using volumetric video is “to adapt to the fact that video now has three dimensions, which can initially seem a bit weird, I guess!” said CEO Richard Broadbridge. Nonetheless, 4Dviews’ philosophy has always been to make the entire process as seamless as possible: “Our approach is that it has to fit in with the rest of the post-production process, be that web stuff, 2D presentations or music videos.”

Indeed, music promos have been a strong contributor to the client base from early on, along with projects in the fashion and art worlds. And as the market has grown, so have the expectations around image quality: “The main focus since time immemorial has been on image quality, and that is never-ending really.”

Like the other contributors to this feature, Broadbridge indicates that the Covid-era lockdowns have resulted in more intensive developmental work across the industry. “All of a sudden people were able to focus more on R&D than shooting activities,” he recalls, adding that in 4Dviews’ own case there was a chance for “our engineers to try and leap ahead as far as possible. [In terms of projects] we have had a really good this year and I think the market as a whole is feeling more reassured about moving ahead; there is certainly a lot of experimentation taking place.”

ARCTURUS TetaVi_Stills_Overlay

Arcturus: Holoedit

Source: TetaVi

In fact, a recent forecast by Research and Markets would suggest that the sky is the limit for volumetric video, with the global market expected to grow from USD 1.5 billion in 2021 to USD 4.9 billion by 2026 – representing a CAGR of 26.9%. As well as volumetric’s ability to deliver enhanced viewing experiences for sports and entertainment content, Research and Markets cites factors such as burgeoning demand for AR in retail and e-commerce, and the overall rising level of investment in the AR/VR ecosystem.

In short, it appears that we are still in the relatively early chapters of what is set to become a huge global success story.

Read more Artificial intelligence in broadcasting