The Fast Lane: How F1 TV Tackled Live Streaming at Scale

When you consider the stats, it’s clear F1 TV has a substantial challenge to tackle. F1 events are streamed 13 million times a month, with each event broadcast live in six languages, as well as managing 24 concurrent streams for various different camera views, including the angles from within the actual cars.

Customers pay £19.99 a year or £2.29 a month, and gain access to streams over smartphone and tablet apps, a web player and TV apps for Apple TV, Google TV, Fire TV and Roku. It’s a lot of eyeballs, distributed across a wide array of platforms.

Watch Now Formula 1: Racing Through Video

Working at the speed of Formula 1

When representing such a fast-paced sport that sets social media alight, there is little room for technical issues to hold back the show. The F1 streaming team experienced its fair share of problems getting F1 TV to the level seen today.

“When we launched F1 TV, we focused on quality and we focused on scale. And it was delivered on a really strict deadline too, and that meant keeping the scope tight,” said James Bradshaw, Formula 1 Head of Digital Technology. “So a single streaming format of course, was preferential.”

Clean and simple was the idea, but trouble-shooting led the team to use a much more convoluted method to ensure a great, consistent user experience.

“This can’t just be a problem that affects us,” said Tim Edmonds, Accenture Architecture Lead, who was also a key figure on the project.

“This must be a wider problem where people are spending inordinate amounts of time on video complexity in a way that maybe they don’t need to.”

The causes of complexity

This complexity arose when F1 TV’s ‘clean’ plan was met with the realities of streaming high profile events live.

“When we started out, we decided that we were going to, race by race, rollout each of our different platforms. We started with web, moved on to mobile and then the big screen so that we could kind of concentrate on each platform [and] fix issues as we started rolling it out,” explained Edmonds.

However, as is so often the case, beta testing ahead of a real launch was only going to reveal a majority (or perhaps even a fraction) of the potential problems ahead.

“Our strategy of a single streaming format appeared to be working really well. We thoroughly tested the platform. We performance tested it, functionally tested it to within an inch of our sanity, but no amounts of testing or instrumentation and monitoring will capture all scenarios,” said Formula 1’s Bradshaw.

“It wasn’t really until round four and the Portugal [race] that we thought we were in quite a good place but all of a sudden, we started seeing buffering issues, the worst kind of issues. Now the buffering wasn’t terrible, but it definitely was notable,” said Edmonds.

“In this instance, this was isolated to web users. And the beauty of web versus native apps is that the rollout and, crucially, the rollback is much more straightforward and quicker… a CDN wizard from Akamai was able to apply a fix, and our new instrumentation was able to prove that we actually solved the problem. The buffering issue had gone away.”

This first problem revealed the weakness of Formula 1’s use of a single content delivery network (CDN) to power F1 TV.

“We always intended to be multi CDN, but the timescales, the relaunch didn’t really allow for that. It would have brought us substantial benefits,” said Bradshaw. The team switched to a multi-CDN approach for greater resilience.

Lap 2: DRM

Digital rights management (DRM) improvements, used to control who can and cannot access F1 TV content, further saw a further layer of complication added to the pipeline.

“It just didn’t work,” said Edmonds of their original simple plans to apply DRM, thanks to limitations of the new CDN partners. “That led us to going with an HLS and the Dash stream with the various flavours of DRM, which was slightly sad because again, it got us into a situation where we had even more streams to operate and run. And of course, with all of those streams you get a whole myriad of errors, much of which are very opaque, difficult to debug, often.”

Confounding factors like the abundance of platforms and hardware used to access streams like those of F1 TV makes achieving a streamlined back-end almost impossible, it seems.

Lap 3: The problems you can’t solve

Perhaps the most frustrating issue of them all related to parameters completely out of the F1 team’s control.

“Device and OS releases also present their own set of challenges. So just keeping a platform stable when everything around you is changing takes effort and time,” said Bradshaw. “And unlike issues that we occasionally introduce ourselves through our development of our products, you can’t control the timelines of Apple or any other software giant. Their release schedule happens on their release schedule, and you have to just work around it.”

This exact issue popped up with an iPhone software release that appeared to almost completely break F1 TV’s streaming tech.

“It became clear that on this new operating system, our streams just didn’t start. But we didn’t think anything of it because actually, when you’re testing live streams, when there isn’t actually live racing, you do get the sort of transitory kind of issues,” said Edmonds.

“But as we got closer to the public launch of that OS, and as we got closer, more importantly, to the first race where people would have actually started to install that new OS, it became clear that there really was a problem,” said Edmonds.

This is a nightmare scenario. “What was really difficult is it didn’t matter what change we made, it didn’t seem to fix it,” said Edmonds. F1 fans with iPhones would be met with completely non-functional streams.

The solution they found was one that, again, dramatically increased the complexity of the back-end processes.

“What we found was if we embedded the audio into the video, the problem went away. But obviously [that] isn’t really a fix. It was kind of a hack,” said Edmonds.

“We weren’t in a good place because all of a sudden we’ve gone from one stream to many, many, many hundreds of streams, which is a big overhead operationally. It cost more, it reduced the efficiency on the CDN through the same number of users more streams spread across them.” F1 TV operates in six languages, with 24 concurrent camera streams.

Apple did eventually release an update that solved the problem, but it gave the team an important lesson. “The important takeaway in all of this is when there’s a problem, which is not inside your gift to solve, and you don’t know how long for somebody else, fix it, all you can do is mitigate”, said Bradshaw.

Another learning point here is that technical issues are unavoidable regardless of how much talent is present on an software engineering team. And at times, design aims have to be shifted to deal with them.

“It became even more clear to us that even with the might of Formula One and Accenture there was one thing that we couldn’t change. And that was really the passage of time. And it was actually a time related problem. So as time was passing our streams were going from working to not working working to not working,” Edmonds explained.

Formula 1’s Head of Digital Technology James Bradshaw and Accenture Architecture lead Tim Edmonds led a talk on the challenges of creating F1 TV at IBC2023.

Read more IBC Changemakers Podcast: F1 Cars, a Yellow Bear and Supporting Young People