Published in News

AnandTech alleges MediaTek cheating on benchmarks

by on09 April 2020


Helio P95 outperforming Dimensity 1000L

AnandTech claims to have caught MediaTek fudging its benchmarks.

In a long and rambling article where the writer spends rather too much time talking about himself, Andrei Frumusanu said that he smelt a rat when he had first received Oppo’s new Reno3 Pro – the European version with MediaTek’s Helio P95 chipset.

“The phone surprised me quite a bit at first, as in systems benchmarks such as PCMark it was punching quite above its weight and what I had expected out of a Cortex-A75 class SoC. Things got weirder when I received a Chinese Reno3 with the MediaTek Dimensity 1000L – a much more powerful and recent chip, but which for some reason performed worse than its P95 sibling. It’s when you see such odd results that alarm bells go off as there’s something that is quite amiss.”

It appeared that the chip was engineered to recognise a benchmark test and give an excellent result to match the test.

Frumusanu alleged that the cheating mechanism had seemingly been sitting in plain sight to users for several years:

In the device’s firmware files, there’s a power_whitelist_cfg.xml file, most found in the phone’s /vendor/etc folders. Inspect the file, and there is a list of popular applications with various power management tweaks applied to them and a list of multiple benchmarks. We find the APK ID for PCMark, and we see that some power management hints are being configured for it, one common one is called a “Sports Mode”.

"The benchmark list here isn’t exhaustive. But it contains the most popular benchmarks in the industry today – GeekBench, AnTuTu and 3DBench, PCMark, and some older ones like Quadrant or popular Chinese benchmark 鲁大师 / Master Lu. There’s also a storage benchmark like AndroBench2 which is a bit odd – more details on that later."

“Sports Mode” fixes some DVFS characteristics of the SoC such as running the memory controller at the maximum frequency all the time. The scheduler is being set up to be a lot more aggressive in its load tracking – meaning it’s easier for workloads to have the CPU cores ramp up in frequency faster and stay there for a more extended period, applying a few common boosting mechanisms.

After ruling out phone makers, it appears that the mechanism isn’t stemming from the individual vendors, but originates from MediaTek and is integrated into the SoC’s BSP (Board Support Package).

MediaTek told AnandTech that it follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of its chipsets.

“We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.”
Media Tech said that in real-world scenarios, there is a multitude of factors that will determine how chipsets perform.

“MediaTek’s chipsets are designed to optimise power and performance to provide the best user experience possible while maximising battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for great user experience. Additionally, some brands have different types of modes turned on in different regions, so device performance can vary based on regional market requirements.”

 

Last modified on 09 April 2020
Rate this item
(0 votes)

Read more about: