Unprecedented times call for new solutions, or at least the deployment of a range of existing solutions in an innovative way. This is certainly the case with independent post facility Halo, which has responded to the Covid-19 crisis by offering a virtual version of all its normal post production processes.
London-based post facility Halo is now providing offline, online, audio or grade up to UHD ‘from home’ as workers tackle the coronavirus lockdown.
According to chief executive John Rogerson, the virtual solution is already running 90% of Halo’s client workload remotely, and “can honour all existing obligations from offline through to finishing post and delivery.”
Halo had actually been testing and using remote solutions for post for several years before the current pandemic, beginning with an original remote offline edit offering called Halo Connect. This made use of PC over IP (PCoIP), a technology which uses display compression to provide end users with on-premises or cloud-based virtual machines.
“This initial trial led us into using PCoIP and CWDM [multiplexing over fibre] technologies more widely throughout the facilities,” says James Cregeen, head of operations at Halo.
“These were primarily introduced to offer the flexibility to ‘spin up’ different suites from offline to online and grade without the need for extensive local infrastructure changes. They have since become ingrained in our ecosystem, with the bulk of our tech support and finishing suites operating with these technologies as standard. Through secure VPN access, remote solutions were already in use for some of our services prior to recent events.”
“The Covid-19 pandemic has greatly restricted access to the Halo facilities in London to both our clients and our staff so we have had to react quickly to expand on those remote solutions already in place, as well as adapting our culture of working collaboratively using these technologies,” says commercial director Will Garbutt. “Right now, nearly all of our services from data management, offline edit, grade, online/VFX, audio, QC and delivery have a remote solution we can offer.”
First past the post
The coronavirus had an early impact on Halo. “We were working on a series for CCTV and as China was the first country to be affected, crews were unable to film and so we immediately felt the shockwaves,” says Garbutt. “At that point, we knew huge restrictions on filming would follow globally, so we started to formulate our remote strategy at that point.”
Initially, the challenges were largely logistical. “As London started to lock down, fear and hysteria for clients and our staff were issues to begin with, particularly around travel in out of central London – people were afraid to use the Tube and buses,” Garbutt says. “[Personnel] with even the mildest of symptoms had to self-isolate, so it became challenging to staff with short notice absences.”
There followed what Garbutt describes as “a post version of panic buying”, with some clients suddenly desperate to get their media out of the facility. “We were tasked with mass copying of assets to standalone drives as well as trying to migrate our staff and resources out of Soho,” he adds.
“As clients were desperate to get out, some were desperate to get in. This all happened at the end of the financial year, which is always a busy time for post, so we were also under pressure to deliver projects before the end of March in order for clients to fulfil their contractual obligations. It was pretty crazy!”
Stemming the tide
Now with lockdown in place, Halo has a skeleton crew of support staff handling physical media and carrying out hardware support in its MCRs. The rest has become virtual. However, with the bulk of Halo’s work falling into the ‘non-scripted’ category, multi-suite, collaborative working with lots of footage has traditionally been the norm.
“Our workflow for ‘editorial’ hasn’t really changed much,” says Cregeen. “We currently receive data from the client either by transport drives or using the Aspera upload accounts we have set up. From there the tech team back up and transcode the rushes as normal to our Avid storage and the media populates the bins from there.
“The main difference is that the editors are using remote systems to log into Avid Media Composer machines at the facility and that’s where the grunt work is happening,” he adds. “The tech teams can then access all projects and perform sound syncing, exports/uploads in exactly the same way as they are used to.”
The edit itself takes places over secure VPN and uses Z Central Remote Boost, a remote graphics solution for HP workstations.
“The editor is able to share his screen with the edit producer via HP Z Central Remote Boost, allowing real-time collaboration as if they were in the suite together,” says Cregeen.
“They can also record guide VO and carry out any archive ingest from data, transferred by a virtual machine [VM] PC. The editor can download and transfer data locally. They don’t have to wait for our tech team to do this on their behalf.
“Once we move into final post, our creatives have access to the conformed media and can crack on with respective grades, onlines and mixes using kit we have either migrated to their homes or they can remote into [the network] using secure PCoIP access, supplemented by Sohonet ClearView Flex and StreamBox Chroma for their external monitoring and collaborative suite outputs with our clients. Thankfully, nearly all of our sound editors and dubbing mixers have either stereo or 5.1 environments in place at home in which to edit and mix, with Zoom and Source View being used for in session review.
“Our QC Ops have the equipment needed to work from home so they can pick up the master file via Aspera and QC as normal on local hardened machines [VM images made more resilient to cyber-attacks]. Deliverables are again all achieved by remoting into our various delivery hardware such as Colorfront Transkoder, and Vidchecker,” adds Cregeen.
Developing the virtual service was aided by the direction Halo was already travelling in. “Our buildings are all connected via dark fibre and a significant proportion of the finishing suites were already making use of PCoIP technology to extend from our primary MCR.
”Making these systems remote and using their existing tech was pretty straightforward,” says Cregeen. “The use of ClearView Flex and StreamBox had been in discussion for a while so it was again an easy choice to invest in these products and implement that into the setup – which has been turned around very quickly thanks to our suppliers.
“Our support partners have been a big part of rolling out services so quickly,” agrees Garbutt. “Jigsaw24 assisted with both StreamBox implementation and getting our PCoIP connection broker working remotely.
Sohonet assisted with ClearView Flex installation which was turned around quickly and was pre-configured to operate with our firewall which they manage, meaning implementation was seamless. Object Matrix is working with us to roll out Vision to our wider client base, and Salon our rental kit provider has been reliable and quick to respond to the situation as ever, helping us provide kit to remote sites as well as offering a free tech support/advice line to anyone needing support.”
“As we are heavily ingrained in HP Z workstation hardware across our offline, edit support and finishing suites, Z Central Remote Boost was a tried and tested solution we were able to spin up quickly with no change to the technology utilised in the suites,” says Cregeen. “With staff already allowed to work remotely via VPN, providing secure access to all these services was relatively quick to facilitate.”
In terms of further development, the Halo team are working on a number of products currently in beta to improve both flexibility and greater access to remote services, as well as looking at implementing products and services in previously unused ways.
Garbutt says its vital to keep the collaborative element of post as free flowing as possible. “Naturally we are having to compensate for the capabilities of the end user such as bandwidth, available hardware like their own home computer and of course their own technical knowledge,” he says. “Unlike onsite support, we cannot simply walk into a suite and diagnose and solve a problem, but there are many ways we can either remote in or offer phone support to perhaps an edit producer who is struggling with passwords and installing software.”
The support team have adapted their skills to include a call centre-style technical support for those who need it. “They are a polite and patient bunch as well as being very knowledgeable, so pretty good at talking a client through an installation process or technical issues,” says Garbutt.
“From a solutions standpoint the main challenge has been the unique nature of each home environment,” says Cregeen. “The technical solutions were available to roll out fairly immediately. Once these had been signed off from a security perspective by studios and broadcasters, the major obstacles have often been related to domestic broadband setups and, latency issues associated with those environments.
There’s also the greater demands for support that our clients have needed in order to get them set up working remotely, whether that’s on their own kit, on Halo’s or through our rental kit support partners, Salon.
“There was also a degree of education required for staff as much as clients in how the new remote setup would operate, and the additional steps or lack thereof in the remote workflows,” adds Cregeen.
“In many cases, the workflow hasn’t changed as much as perhaps may be expected. Gradually though, as each individual has resolved any issues with their home setup and as documentation is now in place and widely circulated to help explain new workflows, things are beginning to settle down.”
The response has been encouraging, according to the team. “We have had incredible feedback from editors, directors, PMs and execs,” says Garbutt.
“It’s pretty overwhelming, all thanks to the Halo team who got their act together the minute we made the decision to put the remote plan into action. What has been really impressive is the willingness from productions to adapt and improvise with new ways of operating.
”It has certainly been a joint effort from production and post to quickly transition projects across to remote working. As a result, we have enabled our clients to continue to work in a very short space of time.
“Although there is much going on in the background to keep it all flowing, we have adapted to the new way of working and are finding new ways to improve the service and overcome any outstanding technical challenges,” he continues.
Moving on from the current situation, it is foreseeable that a solution like this can help combat the climate emergency.
“Absolutely,” agrees Garbutt. “Our biggest footprint is our need for premises in central London, particularly for editorial (offline). The cost to keep these buildings rented, heated and staffed is huge.
”If after this pandemic the culture of remote editing can stick, we may not need so much real estate, which will greatly reduce our carbon footprint. Not to mention the environmental impact of travelling to and from London.”