A new report from Ars Technica claims Samsung might again be exploiting benchmark scores to make its devices look better. Earlier this year, it was suggested that Samsung's Galaxy S4 was manipulated to perform better in industry benchmark tests, and that appears to be the case with the company's new Galaxy Note 3. In response to previous accusations, Samsung unsurprisingly said it wasn't up to any shenanigans; this new report suggests Samsung knows exactly what it's doing.
"We can confidently say Samsung appears to be artificially boosting the U.S. Note 3's benchmarking scores with a special, high-power CPU mode that kicks in when the device runs a large number of popular benchmarking apps," Ars Technica wrote.
Ars Technica first noticed something was up after comparing the Note 3 with the LG G2, which has the same Snapdragon 800 chip clocked at 2.3GHz. Apparently, the Note 3 was scoring "really, really well in benchmark tests—puzzlingly well." So well, in fact, that it scored much higher than the G2, which is a beastly device in its own right. When Samsung was caught manipulating benchmark scores with its S4, the company used a special booster tool so the device would treat benchmarking apps differently, and that seems to be the case here.
After some sleuthing, Ars Technica was able to actually disable Samsung's "special CPU mode," giving them a better idea of how the artificial optimizations affect the overall scores. Basically, Samsung purposely forces the Note 3 CPU to lock into 2.3GHz mode—none of the cores can shut off—when a popular benchmarking app is loaded, which then leads to the inflated scores. During normal use, when a Note 3 is idling, only one core runs at 300MHz to conserve power; benchmarking apps are designed to measure normal usage, "and a device should never treat a benchmark app differently than a normal app," Ars Technica said.
Ars Technica was actually able to run more honest tests, and trick the Note 3 into running normally; this proves that Samsung does indeed manipulate its devices to treat benchmarking apps differently. When running Geekbench, the Note 3 locked into that aforementioned 2.3GHz mode. Running a repackaged (changing the name) Stealthbench test, which was identical to Geekbench, the device was able to idle normally, and thus present the Note 3's real scores.
"In Geekbench's multicore test, the Note 3's benchmark mode gives the devices a 20 percent boost over its 'natural score,'" Ars Technica said. When ran without Samsung's trickery, the device is more on par with LG's G2 (actually slightly better), as anticipated.
Ars Technica's report is fascinating, and collects plenty of evidence to suggest Samsung is playing a little unfairly. In today's quad-core world, benchmarking scores mean about as much as giving water to a fish, but folks still take them seriously. Samsung, it seems, takes them a little too seriously, even when the inflated scores don't contribute to the real-world experience.