The Value of Benchmark Testing for Video Codecs

If you’ve ever developed or created something, you likely realized it is rarely sufficient to only test it by yourself. While “in your lab” everything works as you imagined it, when applied to the real world you may quickly find that you missed something—whether it’s because others use your product in a different way than you intended, or that your product was tested on a fast computer and now must perform on a lesser-powered system. For this reason, it is extremely valuable to have reliable partners to assist with independent testing and validation.

Who performs benchmark testing for video codecs?

Moscow State University (MSU) is the preeminent resource for codec testing. MainConcept® is fortunate to have worked with them for over a decade as they provide independent verification of our codec technology and test it against other available solutions. Their tests have motivated our engineering teams to improve, rework and optimize our video encoders so they remain at the forefront of the codec world.

How do you test a video codec?
MSU develops and runs a set of standardized tests that provide an objective assessment of the qualities that need to be measured, including speed, quality and speed-quality trade-offs. Since a video codec can be looked at in many different ways and given the vast array of parameters that can be used, one codec may excel at low resolutions while another may flex its muscles at very high bitrates. One codec may offer a plethora of settings for the user to tweak, while others may appear to be a simple black box.

For anyone who wants to test a codec, the very first step is to “make sense” of all the variables and parameters, followed by defining a few dedicated use cases and elaborating on them very clearly. For example, the most recent iteration of MSU’s HEVC comparison focused on:

One resolution
One chroma-sampling type
Two bit-depths
Two use-cases

With these defined scenarios, the testing challenge became manageable, with all codecs playing by the same rules. Now, the tester could simply just take a codec, run the application that controls it and list the results. But this is not MSU’s approach. Prior to starting the benchmark testing, MSU’s experts work with the codec vendors to find and identify the optimal settings for the given use cases. They spend countless hours analyzing the tests, discussing variables, tuning parameters and listening to vendors’ feedback on how to achieve the best results.

What do the results of codec testing reveal?
MSU’s results represent not just looking at a “5-minute black-box” test, but instead the result of performance tuning for each benchmark candidate to get the best possible result. But MSU continued to go above and beyond. In the report, they share their findings and the optimal tuning for each codec, effectively enabling anyone to reproduce and validate their results.

When reading through the MSU report, you not only get a “who’s who” in the codec industry and a “who’s best,” but the details of the report provide a definitive guide to setting up a codec for a well-defined use case.

The process leading up to the report helps us as technologists to incorporate all of MSU’s feedback into future versions of our codec, so that our customers benefit from the collaborative effort. And it is not just MainConcept’s direct customers who benefit, but the users of our customers’ products benefit as well since our technology is deployed in industry-leading products such as Adobe Premiere Pro, MAGIX Video Deluxe and VEGAS Pro, AVID Media Composer and Media Central, and Wowza Streaming Engine.

How do MainConcept HEVC codecs compare to other products?
In the most recent HEVC video codec benchmark comparisons by MSU, MainConcept products continued to lead the industry. In the 10-bit tests, which used x265 as a reference, the MainConcept HEVC codec produced the best quality. Improving on the previous study, it showed up to 30% higher bitrate encoding efficiency (when compared with open source).

In the 8-bit encoding tests, MainConcept led the list for best speed-to-quality trade-off, an important factor for streaming as well as other use cases that rely on fast and reliable content delivery.

The best is yet to come
As we look towards the future of codec technology, I believe the best is yet to come and that MainConcept, established over a quarter of a century ago, will continue at the forefront of technical innovation. If you have questions or need assistance as you plan for the future of digital video post-production, broadcast or delivery, we are always here to help. Just email us or apply for a free trial of our software.