Subjecting 32 vendors to 8,000 individual Tests: George Jarrett talks to the EBU and BBC R&D about the JT-NM Tested Program that lit up IBC with green and red.
The giveaway that something important had occurred was the sighting of green and red plaques all around IBC2019. Some 32 vendors had put up over 70 products for precise testing, to answer industry concerns over the maturity of our live IP standards and specifications.
Crucially it saw the real coming of NMOS, and the inclusion in tests of adherence to IS-08 - the audio channel mapping specification.
A first test in March (for NAB) had focussed on SMPTE ST 2110 as data plane, and this was repeated in August with some editorial cures for ambiguities. The new elements were ST 2059 and JT-NM TR-1001, the umbrella standard for AMWA IS-04 and IS-05 – the discovery and registration spec, and the connection management spec. IS-08 is not part of TR-1001. Other areas inspected were cybersecurity and UHDTV.
A ‘snapshot in time’, this program is gold dust for users, vendors and integrators working with engineered broadcast facilities based on IP.
The Joint Task Force on Networked Media (JT-NM) group of experts who created the test plan featured key inputs from Andrew Bonney of BBC R&D, Arne Bönninghoff of the test host company Riedel, Bill McLaughlin of EEG, Jean Lapierre of Matrox, John Mailhot of Imagine Communications, Leigh Whitcomb of Imagine Communications, Mike Overton of Tektronix, Pedro Ferreira from Bisect, Peter Brightwell of BBC R&D, Serge Grondin form Grass Valley, Thomas Kernen of Mellanox Technologies, and Willem Vermost from the EBU.
Start controlling it
Vermost, Bonney plus Levgen Kostiukevych, senior IP media technology architect at the EBU, gathered to explain what they had wanted to achieve.
Kostiukevych started by saying: “The Test Plan can live its life outside of the catalogue. We have done the user job of writing down how to test and which actual environment you should use.”
Bonney explained the full weight of TR 1001. “It is a technical recommendation document and the idea is that it brings together a number of standards and specs. A standard in its own right, it aims to minimise the amount of manual configuration that needs to happen when you bring an IP device into the IP network, and then you want to start controlling it.”
The motivation for the second, fuller JT-NM tested program is an insight into how vendor kit aligns with open media standards.
“The first JT-NM Tested program proved that SMPTE ST 2110 is a go-to standard for media over IP transport. However, as described in the EBU Pyramid (EBU Tech 3371/Minimum User Requirements To Build And Manage An IP-Based Media Facility), having just a media transport, without an open control plane, is not enough. Therefore, we decided that the next iteration of the program had to address that,” said Kostiukevych.
- Read more: Interview with Noel Curran, EBU
Were the cybersecurity IS-08 and UHD elements ‘extra goodies’? And were the NMOS factors important given that many vendors have been slow to adopt NMOS?
“The goal of the program is indeed to encourage the industry to address all the layers of the EBU Pyramid. NMOS is certainly a piece of the puzzle, and the EBU has recently published a position statement document that calls on industry to adopt NMOS faster,” said Kostiukevych. “The JT-NM TR-1001-1 mandates the use of NMOS IS-04 and IS-05 and adds even more standard IT protocols to enable end-users to manage their ST 2110 installations efficiently.
“While IS-08 and cybersecurity are not addressed by the TR-1001-1, they are essential parts of any IP-based installation and are required by the EBU Pyramid,” he added. “The UHD testing was as extensive as HD testing, however and unfortunately there were not a lot of devices submitted for UHD testing.
Certified or approved
What precisely merited a green plaque, and what merited a red plaque? AJA, Grass Valley, Imagine, Phabrix, Net Insight, EVS, Harmonic, and Matrox, plus over 20 other companies displayed them.
Bonney said they call them Badges. The Green one is the SMPTE ST 2110 JT-NM Tested badge and the red one is AMWA NMOS/JT-NM TR-1001-1 JT-NM Tested Badge. Bonney added: “Vendors were able to display them and use them for marketing a device if the test results of that device were published in one of the open catalogues, but the Badge itself did not certify that a product has a ‘pass’ or that it is certified or approved.”
If you saw a badge at IBC, the test results of this device are openly published in the JT-NM Tested catalogue. In simple terms how did the sets of tests break down?
The official sets of tests were: the data plane (media transport) – the ST 2110 test plan and catalogue, the control plane – the NMOS/TR-1001 test plan and catalogue. The control plane testing was led by Bonney.
Kostiukevych added: “The cybersecurity assessment was not an official part of the JT-NM Tested this time, therefore there is no test plan and catalogue yet.
“Indeed, EBU R-148 was used as a scope for cybersecurity assessment. This was done by the EBU’s Media Cyber Security Group and was led by Gerben Dierick and Álvaro Martin Santos,” he added.
“The data plane ST 2110 test plan was expanded to include UHD testing, more sophisticated ST 2022-7 (multiport redundancy) testing, and some ambiguities were resolved as well.”
Discussing the potential revision of JT-NM TR-1001
Looking at what was a very deep ‘snapshot in time’ what were the discoveries that most impressed the expert group, and what were the small disappointments?
“Overall the success rate was much better than last time. However, some challenges are still there,” said Kostiukevych. “Vendors recognised the need and value of mandatory self-pre-testing, and it makes a big difference!
“Not all participating products were submitted also to the NMOS/TR-1001 testing,” he added. “At this point in time, we were expecting much higher levels of adoption. The PTP and network stability was excellent. It is critically important to feed the results and findings back into SMPTE for ambiguities resolution.”
Fully featured implementations were quick to test proving the principles of JT-NM TR-1001-1. Would a third snapshot in time have to wait for certain market maturities to happen? The audio channel mapping specification (IS-08) should maybe be within TR-1001?
“The results of the Tested events are supposed to drive the industry forward. We stay in close contact and collaboration with standardisation bodies and of course we expect that certain progress will be made,” said Kostiukevych. “We are currently discussing the potential revision of JT-NM TR-1001 so it may include IS-08. The EBU Pyramid should also be revised to adapt to new challenges, as more and more broadcasters decide to go full IP. There is definitely the need for publication of best practices and technical recommendations, and the EBU will be working very closely with JT-NM in this field.
Struggles with RTP time stamping
Vermost summarised the key stats from the program.
“The ST 2110 Badge saw a 100% pass rate from 31 out of 71 devices, and all but 22 devices achieved 90% pass rate across the board,” he said. “The worst test results were ST 2022-7 related.”
Some 26% (nine in total) failed to deliver an SDP file, or delivered a faulty SDP file.
“The biggest fail rate was in ST 2022-7 testing. A disturbing fail rate appeared in ST 2110-21 and ST 2110-10, which are fundamental,” said Vermost. “A lot of products are still struggling with RTP time stamping – which can be linked to the ambiguity in the standard – and there is still not a lot of UHD-capable products.”
Bonney, who led the NMOS/TR-1001 aspects, added his results summary.
“We were testing the main fundamentals of this document. With the NMOS/TR-1001 badge, the 100% pass rate concerned six out of 34 devices. And the global 100% pass rate concerned 20 out of 34 devices,” he said.
With NMOS IS-04 and IS-05 all but four devices achieved a 90% pass rate across the board, but “Link Layer Discovery Protocol (LLDP) had the worst results,” he added. For management interfaces, there was a 39% pass rate.
Dynamic Host Configuration Protocol (DHCP) was better with 92% support on management interfaces and 78% support for media interfaces.
Almost 8000 individual tests were carried out, with around 90% carried out using automated means, the automated NMOS tool being an example.
“There were very few common failures across devices, beyond items, which simply hadn’t been implemented,” said Bonney.
“There were some issues with the IS-04 Universally Unique Identifier (UUID) consistency. IS-05 changes taking effect in streams see the use of IGMPv3 (Internet Group Management Protocol) source filters,” he added. “We observed DHCP configuration confusion across Nodes with multiple network interfaces.”
Kostiukevych added: “How do you use our tool? It is free and open source for everyone.”
No comments yet