HTC 10 cheats battery life benchmarks

by Ashley Gullen | 18th, March 2017

The HTC 10 with Nougat (Android 7.0) update appears to be specifically designed to cheat battery life benchmarks. Here's what I've found.

We make Construct 2, a game creator. A user reported a bug involving choppy animations (often called "jank"). I tried out their demo on my phone — a HTC 10 — and sure enough it was pretty choppy. According to an FPS measurement, it was running at 50 FPS instead of syncing to the display refresh rate at 60 FPS.

Initially I suspected a Chrome bug, so I reported it. However upon investigation, Google engineers identified the fundamental cause as the Android OS throttling the v-sync signal. This means the HTC 10's version of Android is pretending the display is running at 50 Hz (with 20ms intervals) instead of the correct 60 Hz (with ~16ms intervals). Since it wasn't a Chrome bug the issue was closed, and one of the Chrome engineers left a comment saying they suspected it was done for benchmark purposes.

It also emerged that the device does actually render at a smooth 60 FPS, until it is left idle for two or three seconds — then it drops down to 50 FPS. This means when you interact with the device, like scrolling, it appears smooth. Then when you leave it idle for a few seconds, it starts throttling. In my opinion this is an attempt at hiding the throttling from users: so long as you keep tapping it stays smooth, but if you wait and watch something like a game just for a moment, it suddenly drops in to a horrible juddering. (It looks bad because 50 does not divide evenly in to 60, so the frame update pattern is irregular, which many people find uncomfortable to look at.)

Our engine uses WebGL for games in the browser, and this made me think of battery life benchmarks. For example in Ars Technica's original HTC 10 review, they specifically use a WebGL battery life test on page 2, which is one of their standard tests. Assuming nobody was interacting with the device during the test (which appears to run for hours), the HTC 10 will be running at 50 FPS, while other devices it is compared to are running at 60 FPS, therefore giving the HTC 10 an unfair advantage. Not only is 50 FPS simply less work per second, but every millisecond counts for saving power, and a few extra milliseconds idle per frame allows the hardware to remain in a low-power state for longer. This all makes the HTC 10 look like it has better power characteristics than it really has.

HTC may try to claim this is some kind of power saving feature, but it doesn't appear it can be switched off anywhere despite obnoxiously affecting performance, and it still happens when the phone is charging. Alternatively they may claim it's a bug, but it appears to be carefully calibrated to avoid it being noticed — it won't happen while scrolling, it's not a massive change which might be too noticable, and it reliably switches between smooth and choppy mode depending on when you last touched the screen. Therefore I would conclude this is deliberately designed to cheat battery life benchmarks. If you run benchmarks you might want to look out for this — it's possible other devices are affected too.

You can check for this yourself by visiting this test on a HTC 10. It runs a very simple animation and shows an FPS counter in the corner. On a HTC 10 the FPS clearly switches between 49-50 and 58-60 depending on when you interact with the device. I should point out the device originally came with Marshmallow (Android 6.0), but I only noticed this when the device eventually updated to Nougat (Android 7.0) — I don't know if Marshmallow was also affected.

This is incredibly frustrating both as a developer and a customer. As a developer I work hard to make sure users get a smooth 60 FPS experience, and Google has made enormous efforts to do the same for Chrome and Android, and then HTC ruin it to cheat a benchmark. And as a customer, I want this expensive high-end phone to run smoothly in all situations, not just when it thinks I'm not running a benchmark.

Guess I should have bought a Pixel after all.

Now follow us and share this

Comments

5
skymen 2,705 rep

Yes you should have bought a Pixel x).
Anyway, thank you for this post. We should try to share this in order to make other developers aware of this.

Saturday, March 18, 2017 at 8:42:55 AM
0
uboel 9,420 rep

Hi. It's perhaps an Android 7 mechanism. I have got the same kind of behavior on my HTC M9 running Android 7.

Saturday, March 18, 2017 at 8:47:12 AM
1
uboel 9,420 rep

Layer today, I'll try on my son's HTC M8s which is still on Android 6. I'll keep you posted...

Saturday, March 18, 2017 at 8:49:14 AM
4
Rhindon 14.3k rep

It'll be a while before I make another attempt at a mobile game, but I'm glad to learn about this. As a naive novice, I don't think I would have ever thought about this sort of thing.

But what frustrates me even more is that such a high-end cell phone maker would sacrifice quality like this. I shouldn't be surprised, and I'm not really. It's just upsetting when it happens.

I, too, own a Google Pixel, and while I am not attuned to the nuances of the sort of things this blog is about, I've never noticed (yet) any obvious issues. Everything is smooth and always crisp in terms of animations and image quality.

Saturday, March 18, 2017 at 9:53:47 AM
1
Lordshiva1948 38.3k rep

Thanks Ashley I have not use HTC 10, I will try out soon as I get my hands on

Saturday, March 18, 2017 at 10:10:53 AM
1
MangoPolo 924 rep

I'm on Nougat with my HTC Desire 816. How to check whether or not I get the same issues?

Saturday, March 18, 2017 at 10:20:03 AM
1
Anonynoise 4,079 rep

Wow, wait... My Galaxy s5 running custom Android 6.0.1 ROM (Resurrection Remix)
Also shows the same result from the demo! Locked 60fps while spamming touch, drops to between 48 and 56 fps after 2-3 seconds of idle o.O.

Saturday, March 18, 2017 at 2:14:35 PM
1
Bleenx 6,562 rep

Great investigatory work! This makes me wary of other devices now.

Saturday, March 18, 2017 at 3:23:31 PM
1
KENYONB 1,970 rep

HTC sounds like the Volkswagen of phones ;)

Saturday, March 18, 2017 at 5:19:01 PM
1
TheDoctor 4,052 rep

Very interesting read. We'll find out Monday if your blog post kills their stock price! I hate when companies mislead their customers!

Personally, I wanted the Samsung Galaxy Note, I just didn't want to burn a hole in my pocket (one way or the other), so I settled for an LG Stylo. I mean really, $900 bucks for a phone in pocket? One I could break or lose? The insurance on it was a total joke. Turns out I loved the LG Stylo! The stylus worked great, just like writing with a pen! Now I have the LG Stylo 2 Plus and it has even more features! You can pick one up for like $150-$200. Nice size screen, thin, and like I said great stylus! I also know I dropped the first Stylo like 5 or 6 times thinking "Oh crap, I must have at least broke the screen", but not a scratch.

Sunday, March 19, 2017 at 2:19:47 AM
2
uboel 9,420 rep

Hello,

I made some tests on different hardware I have home and checked for performances throttling using Ashley's little test application.

Here are the results:

- HTC M8s (Android 6) : fps remains stable around 60
- HTC M9 (Android 7) : fps going to 50 after a few seconds if the phone is left alone...and back to 60 if you touch the screen. Just note that even at 50fps, the animation remains perfectly fuid.
- nVidia shield Tablet (Android 7) : fps remains stable around 60

It really seems that HTC tweaked their Android 7 releases (at least on M9 an M10) with performance throttling when the phone is not touched.

Sunday, March 19, 2017 at 2:51:51 PM
1
Kisakuku 159 rep

50 fps lock was present on MM, known about since mid-2016 ([Hyperlink removed - users with less than 500 rep cannot post links]) and removed by various PnP mods (Freak07's and ivicask's).

Sunday, March 19, 2017 at 2:56:54 PM
1
jarnMod 309 rep

Even this is true, HTC10 fans aren't going to be happy. They may try to deny it, saying it isn't true or does not matter, according to DABDA grief model.

Monday, March 20, 2017 at 2:48:25 AM
3
Haldi 167 rep

Wanna know something else?
The HTC10 screen does NOT run at 60fps! It's 57 or 58Hz.
Why? Nobody knows. Not all devices are affected. Some do run at 60hz, some don't. It's not related to which type of screen you have (Sharp or Tianma) or what ROM you're using.
Fps2d is a good app to test this.

Monday, March 20, 2017 at 3:42:28 AM
1
yassercv 505 rep

Excelent! :D

Monday, March 20, 2017 at 11:20:15 AM

Leave a comment

Everyone is welcome to leave their thoughts! Register a new account or login.