The Great HTML5 Gaming Performance Test: 2014 edition

by Ashley | 9th, May 2014

We've done a big performance test in the past, but that last test is nearly two years old now (the references to the Playbook make it look particularly dated!) and things are moving quickly in the world of web technology. This time around we've tested a range of desktop and mobile devices, browsers and OSs covering a total of 30 combinations of device and software. The overall state of HTML5 performance is considerably ahead of where it was in 2012. Browser technology, Javascript engines, device hardware, drivers and our own HTML5 game engine have seen countless improvements. The future of HTML5 gaming is very promising - but there are still a few lingering issues. Read on to learn more.

The test

Our standard test is the same as before: sbperftest, an automated version of our stock demo game Space Blaster. It's based on a real-world game using our engine, it doesn't need input so runs the same every time, and it automatically measures the framerate producing a final test score which is the average framerate throughout the test. It actually gets pretty intense with the sprite counts and blend modes on the lasers and explosions, and we've found it's a pretty good indicator of a platform's performance - if sbperftest runs well, most other HTML5 games should run decently as well.

Screenshot of the Space Blaster performance test running.

Note the CPU utilisation measurement is a slight misnomer - it's measured from Javascript and is based on timers so it should be viewed with some skepticism, especially since it's only measuring the main thread and many browsers like Chrome are parallel in key components like the renderer. A better name would be "percentage of time available to run Javascript that is used up", but that's a bit of a mouthful! It's not taken in to account for the test result anyway, it's just interesting to observe sometimes.

The devices

For testing and development, we've amassed a fair few devices in the Scirra office. This puts us in a good position to test a wide range of devices and OSs. For brevity in the results table we've given them nicknames (in bold below), but the full specs are listed here if you're curious. Note mobile device specs can be hard to come by so some details below are sourced from Wikipedia.

Desktop devices:

  • Windows desktop: Windows 8.1 x64, Intel Core i5-2500 @ 3.3 GHz, 8 GB RAM, nVidia GeForce GTX 660
  • Mac Mini: Mac OS X 10.9.2, Intel Core i5 @ 2.5 GHz, 2 GB RAM, Intel HD Graphics 3000
  • Ubuntu desktop: Ubuntu 14.04 x64, Intel Core i7-3820 @ 3.6 GHz, 16 GM RAM, AMD Radeon HD 6570

Mobile devices:

  • Nexus 5: Android 4.4.2, Krait 400 @ 2.3 GHz, 2 GB RAM, Adreno 330
  • Nexus 7 (first generation, 2012 model): Android 4.4.2, nVidia Tegra 3 T30L @ 1.2 GHz, 1 GB RAM
  • SGS3 (Samsung Galaxy S3 i9300): CyanogenMod 10.2.0 (Android 4.3.1), Cortex-A9 @ 1.4 GHz, 1 GB RAM, Mali-400 MP4
  • HTC Sensation: Android 4.0.3, Qualcomm Scorpion @ 1.2 GHz, 768 MB RAM, Adreno 220
  • iPad 2: iOS 7.1, Cortex-A9 @ 1 GHz, 512 MB RAM, PowerVR SGX543MP2
  • iPad 3: iOS 7.1, Cortex-A9 @ 1 GHz, 1 GB RAM, PowerVR SGX543MP4
  • iPhone 4S: iOS 7.1, Cortex-A9 @ 800 MHz, 512 MB RAM, PowerVR SGX543MP2
  • iPhone 5: iOS 7.1, A6 @ 1.3 GHz, 1 GB RAM, PowerVR SGX543MP3
  • Lumia 520: Windows Phone 8.1 (preview), Krait @ 1 GHz, 512 MB RAM, Adreno 305
  • Kindle Fire HD 7" (first generation, 2012): Fire OS (based on Android), 1.2 GHz, 1 GB RAM, PowerVR SGX540
  • Blackberry Z10: Blackberry OS 10.2, Krait 1.5 GHz, 2 GB RAM, Adreno 225
  • Tizen: Tizen OS 2.2, developer device with unknown specs but looks/feels like a Galaxy S4

The results

Without further ado, here are our measurements!


Device Software Notes Score
Windows desktop Chrome 34 webgl
60
Windows desktop Firefox 28 webgl
60
Windows desktop IE 11 webgl
60
Mac Mini Chrome 34 webgl
60
Mac Mini Firefox 28 webgl
60
Mac Mini Safari 7 canvas2d
60
Ubuntu desktop Chromium 34 canvas2d, blacklisted gpu
60
Ubuntu desktop Firefox 29 webgl
55
Nexus 5 Chrome 34 webgl
59
Nexus 5 Firefox 28 webgl
58
Nexus 7 Chrome 34 webgl
59
Nexus 7 Firefox 28 webgl
51
SGS3 Chrome 34 canvas2d, blacklisted webgl
58
SGS3 Firefox 28 webgl
58
SGS3 Stock browser webgl (enabled by CyanogenMod)
52
HTC Chrome 34 canvas2d, blacklisted gpu
7
HTC Firefox 28 webgl
48
HTC Stock browser canvas2d, no gpu support
18
iPad 2 Safari 7.1 canvas2d
46
iPad 2 Ejecta webgl, no JIT
55
iPad 3 Safari 7.1 canvas2d
42
iPad 3 Ejecta webgl, no JIT
52
iPhone 4S Safari 7.1 canvas2d
40
iPhone 4S Ejecta webgl, no JIT
51
iPhone 5 Safari 7.1 canvas2d
60
Lumia 520 IE 11 webgl
47
Kindle Silk canvas2d
44
Z10 Browser webgl
54
Tizen Browser webgl
57

Analysis

The results are far stronger than our previous test in 2012. Performance is good across a wide range of devices and OSs. Thanks to optimisations to OSs, browsers, and our own engine, many scores were improved. Mobile browsers have benefited particularly well, most of which can now easily keep the framerate above 30 FPS at all times.

The main problem device was the HTC sensation. This ageing Android device never got updated beyond Android 4.0.3, and Chrome has blacklisted its GPU presumably due to stability issues. (Graphics driver issues are back to haunt us!) Stuck with software rendering on a weak mobile CPU, it produces an unplayable framerate averaging at just 7 FPS. The stock browser is not much better: it never supported GPU acceleration in the first place, so similarly gets a lousy average of 18 FPS due to software rendering. Interestingly Firefox for Android makes full use of the GPU even running with WebGL support, and scores a respectable and playable average of 48 FPS. This is better than some of the fully-up-to-date iOS devices. Chrome can actually also score an average 48 FPS on the HTC Sensation if you head in to chrome://flags and enable 'Override software rendering list', so it ignores the GPU blacklist. This illustrates that even on a crusty old Android phone on its last legs, the hardware is still perfectly capable of handling the game. The only problem is that sometimes the browser will not use the GPU.

WebGL support has come on in strides since our last test. Previously WebGL support on mobile was practically unheard of, and now Apple platforms are the only major platforms without support. We've already seen how WebGL support significantly improves performance, so a future version of iOS adding WebGL support could see even the older iOS devices getting their results topped up to 60. Apart from Apple, there are a few spots where it still fell back to canvas2d: Chrome on Ubuntu, where Chrome didn't trust the bundled open-source graphics driver, but the mighty i7 CPU was fast enough to run software rendering at 60 FPS; Chrome on the Galaxy S3 where the GPU was blacklisted for WebGL but could still be used for canvas2d, and the hardware was good enough to still get a great result; the HTC Sensation which suffered badly from blacklisting; and the Kindle Fire HD which appears to lack WebGL support but could get by with canvas2d.

The newly-supported Ejecta wrapper for iOS also performs excellently, even outperforming Safari despite Apple's restrictions that native apps cannot JIT-compile Javascript for top performance. That's the kind of difference WebGL support can make, and also allows you to make use of WebGL shaders on iOS.

Note we didn't test Crosswalk on Android - there is very little point, since it is essentially the same as Chrome for Android. It can be assumed that Crosswalk will perform identically to Chrome.

Conclusion

Desktop devices are generally all sorted. Despite being plagued by performance issues in 2011, with a now-solid selection of browsers, OSs and drivers you can be pretty confident that desktops will be able to blow through HTML5 games just fine.

Mobile devices have for a long time now had sufficient hardware to run HTML5 games just fine. Even the HTC Sensation, now a junk Android phone from 2011, can manage a decent framerate providing the browser makes full use of the GPU. However many mobile manufacturers have the somewhat galling attitude of abandoning their devices a year or two after releasing them. This is frustrating as a developer and offensive as a consumer: it's analogous to buying a brand new Windows 7 desktop, then finding 18 months later it's officially impossible to update it to Windows 8 or newer. Google's line of excellent Nexus devices shows it doesn't have to be this way, but for devices like the Sensation it ends up stuck with an old OS with broken drivers that crash regularly, forcing browser developers to blacklist them so you don't have to keep pulling the battery out of your phone because it locked up while browsing. When you hear about HTML5 performance complaints, it is almost always actually a device with perfectly sufficient hardware that has been blacklisted by a browser maker, so it gets software rendering. (The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.) Ironically the stability problems are caused by the native technology in the drivers, not the web technology.

The situation on mobile is very similar to the situation on desktop in 2011. The problem is effectively already solved: the latest crop of devices have great performance and support, but there are some legacy systems still hanging around being left behind by their manufacturers. These can be problematic, but the good news is there can be workarounds (such as switching to Firefox on Android, or skipping the blacklist), and the old broken devices are steadily dying off. Even new budget devices like the Lumia 520 can pull off decent performance even with limited specs thanks to good software. The wait for the devices running abandoned software die out is awkward, but the hardware has been good enough for years, and the software today is already good enough. This would appear to guarantee that the future of mobile HTML5 gaming is high performance everywhere.

Now follow us and share this

Tags:

Comments

7
Mishfire 2,018 rep

Great post really "informative"

Friday, May 09, 2014 at 12:20:52 PM
2
GEEKEV 283 rep

awesomeeeeeeeeeeeeeeeeeeeeeeee

Friday, May 09, 2014 at 12:32:01 PM
2
Aleq 873 rep

Good to see that construct 2 has works on improoving the performance of the framework. We would love to see real "native" app converters for mobile devices in this framework in the future. Keep-up the great work you have done so far

Friday, May 09, 2014 at 12:34:32 PM
4
oceldot 1,366 rep

Maybe it's just me, but in my experience, previewing a game on my Nexus 5 in the browser and running the game when it is installed as an app (built with Intel XDK with the Android for Crosswalk option), will not have identical performance. I'm really struggling with using physics on mobile, even having only one object with physics the FPS drops like a mother.

Friday, May 09, 2014 at 12:40:04 PM
2
rethitokiri 760 rep

Great information. Thanks for the update.

Friday, May 09, 2014 at 12:45:21 PM
3
TGeorgeMihai 5,937 rep

Awesome ! Performance has increased since last year (also now we have more powerful smartphones/tablets in the market)

Friday, May 09, 2014 at 12:45:58 PM
3
Unprofessional 3,965 rep

Great times ahead for HTML5 games!

Friday, May 09, 2014 at 1:07:18 PM
1
iGamersBox 4,266 rep

Just 2 hours back i was test my first game(WIP) which has been exported for android using cocoonjs and uploaded to ludei. I compiled the zip file using canvas and when i tested there was stage where i have lots of sprite stacked on screen more than 50.

At this point the fps drop down to 18 and then picked up after few seconds.

while exporting to mobile is canvas a best option or will webview give a better performance. or the reson for my fps drop is something else.

I am new to game development. so probably not aware of technical things.

Friday, May 09, 2014 at 1:13:08 PM
3
Ashesh 6,293 rep

I think Scirra will lead world of game developing very soon!

Friday, May 09, 2014 at 1:17:20 PM
1
iceangel 26.3k rep

Thanks Scirra.

Friday, May 09, 2014 at 1:18:28 PM
1
isasaurio 1,596 rep

Genial
Muchas gracias
Saludos desde Chile!!!

Friday, May 09, 2014 at 1:19:55 PM
1
gamepopper 3,015 rep

Very detailed report. It would be great if there were not just proper native app converters for mobile, but also desktop platforms. Node-Webkit is great, but being pretty much a Chrome Browser in a desktop window means that it faces whatever bugs and limitations the Chromium engine version it has.

Friday, May 09, 2014 at 1:28:43 PM
1
Lordshiva1948 23.2k rep

Scirra is one of best and I know from my experience. I started with Vic 20 from then on I have went through so much in terms of learning new lingo, and year ago found C2. With C2 it.s so easy that my nearly 6 year old grandson Kai Stoneman started learning too. I think Kai be best if he carry on like he is. Ashley my boy thanks again for C2. Keep up the good work.

Friday, May 09, 2014 at 2:02:13 PM
1
Tedg 7,019 rep

Great info thanks for the update.

Friday, May 09, 2014 at 2:05:50 PM
2
Fimbul 6,764 rep

The bad thing about this test is that it's capped at 60fps, so we cannot know what browsers/devices are better than others. Maybe it runs "great" with sbperftest, but many games are far more complex. For instance, I cannot know from this test if Windows Desktop runs better than the iPhone 5 (it obviously does, but this test doesn't say it). We also can't compare chrome/firefox/IE11 performance.

I'd like to see a repeat of this test with framerates uncapped or a heavier test that makes no one able to reach 60 FPS.

Friday, May 09, 2014 at 2:06:16 PM

Leave a comment

Everyone is welcome to leave their thoughts! Register a new account or login.