CBC/Radio-Canada outlines how automation put the finishing touches to its new IP-based production centre in Montréal.
IP has been one of the core themes across the broadcast industry this year, with the Covid-19 pandemic forcing broadcasters to reconsider how they can be more flexible across workflows.
CBC/Radio-Canada is putting the finishing touches to its new IP-based production hub in Montréal. During the design and early deployment of this major facility, it became clear that we would need to automate and manage the staging and configuration of the thousands of media devices in a fashion similar to an IT data centre.
Speaking during a SMPTE2020 session, CBC/Radio Canada director of media transport architecture Felix Poulin explained the facility will become the headquarters of its French language network in Montreal.
Joined by his colleague, project leader – core systems – Alexandre Dugas, Poulin said the new 420,000 square foot space operates 10 studio floors and four automated control rooms. There are also two TV studios and two multifunctional studios, with two control rooms.
It will be home to a continuous news channel, 13 radio stations and 60 post-production units. All of CBC’s French speaking arms will be based there, with 20 TV channels and 80 radio channels operating out of the centre across both traditional broadcast and IP.
To facilitate this, CBC has engaged in a full IT refresh across all of Canada. This includes its first major project based around state-of-the-art IP-based technology.
The architecture is based on JT-NM TR-1001-1 and uses SMPTE ST 2110 (-20, -30 and -40) for transport and a dual spine-leaf COTS network with SMPTE ST 2022-7 redundancy, said Poulin.
IEEE 1588 Precision Time Protocol operating with the SMPTE ST 2059 profile maintains accurate timing while the Canadian broadcaster is also using AMWA NMOS, Ember+ and proprietary API protocols for control, he said.
“With this quantity of project to deliver in the coming years, we believe that investing in automating the deployment is the only way to achieve this huge task,” he said.
This setup allows CBC to connect 1000s of devices and overcome significant challenges, such as the need for regular updates and setup of more parameters per device.
“It is not sustainable to maintain a manual facility when it is as large as this one,” he explained.
Casting an eye on IT
To address the challenges, CBC looked across IT industries and how they have implemented technology and automation, ultimate focussing on data centres. They applied concepts from continuous integration and continuous deployment, such as scripted configurations, sanity checks and monitoring.
“This is a major switch for a broadcast operation,” he claimed.
Key to the plans outlined by Poulin is automating deployment. Firstly, network deployment is vital, he explained, as it eliminates the need for separate staging stations and the risk of errors from human manipulation. This means configuring network switches and ports that allow for installation at the place they were used, cutting out staging stations that need human interaction.
The second stage centres on programming the minimum set of endpoint device parameters to make it possible to reach the devices’ management interface. These can be multicast addresses of audio, video and data streams; frame sync-enabling, output timing alignments, or audio mapping. These are “often subject to change for different production needs,” he added.
Finally, device configuration, entailing “software programming all of the media-specific parameters to make the endpoint usable and ready to be provisioned in the broadcast control room,” he said.
- Read more: Right content, right place, right time
The solution itself is architected from several key components, according to Poulin’s colleague Alexandre Dugas who joined him at the SMPTE session. It is made up from a continuous delivery pipeline, a dynamic host configuration protocol, server which is used to automatically assign IP addresses; a firmware repository which boots from a file server; a daily sanity check pipeline; and a network infrastructure physical path.
“The role of the continuous delivery pipeline is to organize sets of tasks into what the broadcaster calls workflows—which are involved in the initial staging process, can play a role in network configuration and “perform function calls in an organized manner with proper metadata that are fetched to the IT automation tool,” said Dugas. “Continuous delivery pipeline tools provide a level of abstraction to the end user that is not necessarily familiar with scripting language.”
The architecture has three key parts, which CBC is calling “sources of truth”, to ensure an error-free automated configuration—a schematic database which provides a clear understanding of the physical network, a source version control tool that regularly saves data on the state of the IT automation tool and the IPAM, which provides “a source of truth for all of the logical network components,” said Dugas.
“The role of the continuous delivery pipeline is to organize sets of tasks into what the broadcaster calls workflows—which are involved in the initial staging process, can play a role in network configuration and “perform function calls in an organized manner with proper metadata that are fetched to the IT automation tool,” explained Dugas. “Continuous delivery pipeline tools provide a level of abstraction to the end user that is not necessarily familiar with scripting language.”
Dugas also outlined an agentless IT automation tool that will help synchronise the components outlined in CBC’s architecture by translating staging parameters into comments, offering up-to-date information on the system’s endpoints and architecture.
We can say a task is idempotent if it provides the exact same results regardless of the initial stage of the IT automation tool and also regardless of the number of times it was executed,” he added, describing the criteria that underpins the system.
One of the core challenges for CBC came in testing and configuring media-over-IP devices, which, according to Poulin, can take “five to 10 times as long” as equivalent SDI equipment. This is why automation is so important, he added.
Closing out the SMPTE session, Poulin offered advice for broadcasters who may be looking to deploy a similar automated IP system, saying that engineers should focus on building deployment tools ahead of building the actual system.
He explained: “That way you can recover your investment by automating the initial deployment and avoid having to reconcile the state of your system with the configuration scripts [which would be necessary] if you waited to automate at a later stage of the project.”
Beyond this, broadcasters need to “bring everybody on board so they can implement efficiently the new methodologies,” Poulin said. “Using a continuity of deployment methodology is a major change of practice.”
“This is a major shift for a broadcast media organization,” he said. “But it is a change that is necessary to adopt the IT technology at the core of our media infrastructure and get the expected benefits of flexibility and agility promised by the complex transition.”