Published in PC Hardware

Intel may have commissioned some misleading benchmarks

by on10 October 2018

But the press is still reporting them

On Monday, Intel unveiled the 9th Gen Core i9-9900K, which will rival AMD's Ryzen 2700X when it goes on sale in two weeks but may be over egging its performance pudding with the tech press.

Intel claims it is the "world's best gaming processor" to see how exactly it fares against its AMD counterpart but is doing its best to make sure that none of the tech press go off that message.

Reviewers test the new CPU and comply with an NDA/embargo (non-disclosure agreement) with Intel, which requires them to not share performance data of Intel's new CPU for another few days.

But stories starting to come out this week claim that "Intel's Core i9 9900K is up to 50 per cent faster than AMD's Ryzen 7 2700X in games".

However the reports cite data from an Intel-commissioned report penned by third-party firm Principle Technologies to make the claim.

Basically, Intel has published its own "testing" done suspiciously through a third party ten days before reviews, which reviewers can’t contradict because they would be refuting the claims due to the NDA.

Instead they will be contractually obliged to gush like PCGamesN.

“Intel must be able to hold out the 9900K as hands down the best gaming processor compared with the AMD competition, and in that it seems to have excelled. On some games, such as Civ 6 and PUBG, the performance delta isn't necessarily that great, but for the most part you're looking at between 30 and 50 per cent higher frame rates from the 9900K versus the 2700X."

But reviewers who have not signed the NDA but have some experience of benchmarking smell a rat and called PCGamesN out on it.

Over at the Principled Technologies website you can find the full report which states how they tested and the hardware used. Timings are important and they used Corsair Vengeance memory without loading the extreme memory profile or XMP setting, instead they just set the memory frequency to 2933 and left the default memory timings in place. These loose timings ensure compatibility so systems will boot up, but after that point you need to enable the memory profile.

Techspot has pointed out that  it's misleading to conduct benchmarks without executing this crucial step.

Techspot noted that it would have been fair if they had done the same for Intel, but they didn’t. For all Intel platforms they first set the memory to XMP and then adjusted the frequency manually, handling Intel a significant performance advantage, particularly for games.

Another manipulation of the results was to only test at 1080p with a GTX 1080 Ti using quality presets that were a step or two down from the maximum level. In many cases this simulates the kind of performance we see when testing at 720p using ultra quality presets.

PCGamesN has done the decent thing and published a story which contradicts its earlier one

Now it is saying the test platforms used by Principled Technologies are similar, but not identical. The motherboard and CPU have been changed per platform, but it's the 16GB DDR4 memory (at either 2,666MHz or 2,933MHz) that has been the subject of scrutiny.

Specifically, it appears that the XMP-equivalent technology for AMD motherboards wasn’t enabled, lending to worse than expected performance on AMD’s top Ryzen chip.

Last modified on 10 October 2018
Rate this item
(0 votes)

Read more about: