Scirra cog

About Us

We're a London based startup that develops Construct 2, software that lets you make your own computer games!

Archives

Browse all our blog posts

Latest Blog Entries

We love brains!

Join us! Joiiinnn ussss! Mooooree brains!

The Great HTML5 Mobile Gaming Performance Comparison

by Ashley | 23rd, May 2012

Update 11th June 2012: added score for Playbook OS 2.1. It's literally 3x faster than OS 2.0, so the conclusion for Playbook has changed from "unusable" to "great"!

Much has been said about the performance of HTML5 games on mobile, often around the fact it's too slow to be practical. This week we measured the performance of 23 combinations of browser and device with a real-world game test made using Construct 2, our HTML5 canvas based game creator. We've spent some time optimising our engine for best performance, and results are promising: despite the naysayers, it looks like HTML5 mobile gaming is a genuinely viable option already - and it is likely to improve further.

The test is an automated version of Space Blaster, our stock demo game for Construct 2. It's a simple arcade space shooter that, despite occasionally having a lot of on-screen objects, should not cause any problems for any desktop or mobile device using hardware acceleration. Of course, not all platforms use hardware acceleration or they don't implement it efficiently, which is the cause of the well-known performance problems.

Screenshot of the Space Blaster performance test running.

To run the test just visit the performance test URL and watch it go. The test scores by taking an average framerate over a period of about a minute. Browsers often spend the first few seconds optimising Javascript, so to prevent JIT compiles and other startup processes affecting the result the first five seconds are not counted. Further, the test has no sound since some platforms don't play audio (like iOS's Safari browser), and devices which can sometimes take a performance hit from having to also process audio, so for fairness sound is excluded completely. Since the test has been adapted from a real game using a real engine, this is probably the best real-world test we can come up with. Hopefully its results provide a good indication of the practical performance you can expect from each platform.

The devices listed below are desktop (Windows 7 Core i5-2500 quad-core @ 3.3 GHz; 8 GB RAM; AMD Radeon HD 6570 graphics), iPad 2, iPad 3, iPhone 4S (all on iOS 5.1.1), Android 4 (Ice Cream Sandwich) (on a HTC Sensation Z710e), Windows Phone 7.5 (Mango) (on a Nokia Lumia 800), and a BlackBerry Playbook. Browsers tested include various desktop and mobile variants of Chrome, Firefox, Internet Explorer, Safari, and Opera, as well as the Android stock browser ("Internet") and the Playbook stock browser ("Browser"). We also tested the non-browser HTML5 game wrappers directCanvas by appMobi (currently only available for iOS, but coming soon to Android) and CocoonJS by Ludei (for both iOS and Android), both designed to boost the performance of HTML5 games and supported by Construct 2. Obviously the total number of combinations of device, platform and browser out there is mind-boggling (which some people cite as part of the problem), but I think we've covered a fairly good cross-section. It's also important to note the test is running identically on all 23 tested platforms, with fluid scaling depending on the device screen size including support for Retina displays, all with the same code base - the only variant is the performance. This is why we don't believe there is a problem with having lots of platforms. Take this as a comparison: Windows runs on a mind-boggling array of hardware, and most Windows apps run just fine everywhere. You can make a Windows app and just trust that it will run on any PC with any hardware running Windows. Likewise we believe while testing is prudent since HTML5 is newer technology, we hope one day you can similarly design a HTML5 game and simply trust it will run everywhere. This test is a good working example of that already beginning to happen.

Another thing essential for HTML5 mobile gaming is multitouch support. Imagine a jump and run game: single touch means you can't jump while holding down to run - game over! All platforms support multitouch, with the exception of Android's stock browser in 2.x (support was added in 4.0) and Windows Phone 7, which shockingly only gives you a "click" event. Frustratingly both actually support multitouch in their hardware, but are held back by archaic software support. Windows Phone 7 is nearly totally useless for HTML5 gaming in its browser because it does not even have proper single-touch support! Luckily the PhoneGap team came to the rescue and added support for single-touch via mouse events (make sure you enable 'use mouse input' in Construct 2's Touch plugin), but as mentioned single-touch still rules out many games. I don't know why anyone would want to develop for a device with a vanishingly small market share and severe device limitations. Still, if you have a game that works with single-touch, you might want to give it a shot with PhoneGap Build. Android 2.x is rescued by directCanvas and CocoonJS - both add support for multitouch even if the stock browser does not support it.

So now on to the results. Remember the average framerate was measured, so higher is better. Much below 20 means it is questionable if the game will be playable; 30 FPS is perfectly playable; 60 FPS is the target. Notes: on iOS the results for Safari are after 'Add to home screen', so the game is rendered as close to fullscreen as possible. Also for technical reasons some technologies don't use the "retina" high-resolution display - they will render at low-resolution and just stretch the game up to fill the screen. These technologies are noted "using retina" or "not using retina". Using retina looks better with crisper graphics but should be expected to be slower since it has more pixels to fill. The iPhone 4S and iPad 3 have hi-res retina displays, but the iPad 2 does not. Finally, PhoneGap for the Playbook was not tested because I couldn't find an obvious way to run the output of PhoneGap Build on it without making a full submission to the Blackberry App World.

The Test Results

Device Technology Notes Score
Desktop Chrome 19
60
Desktop Firefox 13
64
Desktop IE9
60
Desktop Safari 5.1
26
Desktop Opera 12 beta
27
iPhone 4S Safari From home screen, using retina.
34
iPhone 4S PhoneGap Using retina.
27
iPhone 4S directCanvas Not using retina.
52
iPad 2 Safari From home screen.
42
iPad 2 PhoneGap
 14
iPad 2 directCanvas
53
iPad 3 Safari From home screen, using retina.
30
iPad 3 PhoneGap Using retina.
 3
iPad 3 directCanvas Not using retina.
54
Android 4 Stock browser Fail! Severe display glitches. Multi-touch only Android 4+.
 11
Android 4 PhoneGap Fail! Severe display glitches. Multi-touch only Android 4+.
 10
Android 4 Chrome 18 beta
24
Android 4 Firefox 14 beta
 7
Android 4 Opera 12 mobile
 10
Android 4 CocoonJS
34
Windows Phone 7 IE Mobile "Click" event only.
 14
Windows Phone 7 PhoneGap Single-touch only.
 10
Playbook Browser Playbook OS 2.0
 9
Playbook Browser Playbook OS 2.1 (beta)
30


Review

Performance is extremely varied: ranging from 3-60 FPS, a difference of a factor of 20!

Desktop performance is generally no longer a problem. The top three browsers barely broke a sweat thanks to their hardware acceleration. Desktop users still need to make sure they have up-to-date graphics card drivers else the browser may switch off hardware acceleration; however even with software rendering (like Safari and Opera) desktop performance is usually in the playable range. Our WebGL renderer also helps Chrome and Firefox run about twice as fast as Canvas 2D, although we currently have disabled WebGL rendering on mobile devices since it has always been slower than Canvas 2D in our tests.

PhoneGap unfortunately appears on the whole to be unsuitable for publishing HTML5 games. This is mainly because it is stuck with the system default browser which is usually too slow. On iOS the system browser is Safari which actually performs very well, but when you wrap it up in PhoneGap it drops three times slower on iPad 2, ten times slower (!!) on iPad 3 (using the retina display), and bizarrely for reasons I really struggle to comprehend, is only marginally slower on the iPhone 4S. So the only utility for PhoneGap appears to be for targeting the iPhone. Windows Phone 7 also seemed to take a performance hit from using PhoneGap, so perhaps there are difficult technical issues around putting a browser in a native app.

Safari on iOS is consistently good - the best performing mobile browser by a clear margin, even when rendering to a high-resolution retina display. If you 'Add to home screen' like was done for this test, the game also gets an icon which makes it look like a native app. Combine that with the fact Construct 2 games still work offline, and no 30% fee if you charge for your game, and adding to the home screen may be a viable way to distribute HTML5 games. Unfortunately you will run in to problems trying to get Safari to play audio - you might have to stick to just a music track. More information in How to make an iOS Web App.

Across the board directCanvas and CocoonJS out-perform all the browsers. This is not surprising since they are optimised to maximise performance and always use hardware-acceleration. They also add multitouch for Android 2.x. These options are definitely the best ways to package up your HTML5 game as a native app for mobile. Both are works-in-progress with Construct 2 integration for the time being though, but they should both be robust and ready for production in the near future. However they prove good performance is definitely possible with HTML5 games - it's only the browser software that is holding anything back.

Android has a few browsers available, but Chrome for Android beta is the only actual browser that provides playable performance. It's a separate download and can't be used with PhoneGap, but it's GPU-accelerated fast and supports multi-touch. Hopefully one day Chrome will become the default browser on Android and that will improve the situation considerably, but it seems that is years off becoming reality given the prevalence of Android 2.x devices. Until then you're stuck with a very slow stock browser that in this test started glitching so badly you couldn't tell what was going on. Use directCanvas or CocoonJS for Android.

The Playbook comes in with a suprise result: OS 2.1 is great! OS 2.0 is too slow to be usable, but OS 2.1 fixes it and is literally a whole 3x faster. Its result is competitive with the Android results, but with the advantage of having good support in the native browser (with multi-touch, too). It's also worth noting the Playbook is considerably cheaper than most other devices - given how much less than an iPad it costs, it puts out a very good framerate indeed. I did notice a few jumps probably due to garbage collection pauses, so hopefully those can be ironed out eventually too. OS 2.1 is still a beta, so let's hope it gets pushed out to all the Playbook owners soon.

Windows Phone 7 comes in a sore last. With poor touch support and an unusable performance result, I cannot see any HTML5 developers seriously targeting this platform. The Playbook stomps all over it with multitouch support, far better performance, and a low one-off cost.

tl;dr: Use directCanvas or CocoonJS to get your games on mobile. The Playbook is surprisingly good, so consider it. Windows Phone 7 is useless. And if you can live with limited audio, consider an iOS web app as well.

Conclusion

Mobile browsers are still largely a big mess. Safari on iOS is good, except when PhoneGap'd. The Blackberry Playbook also has an excellent browser, but the realities of market share make it a small target compared to iOS. However, if your game just works on it, it's definitely worth considering putting in the time to publish to the App World.

For making Android and iOS native apps, directCanvas and CocoonJS are two much-needed technologies that help make HTML5 games really fast. Doubtless mobile browsers will continue to improve in the future. In the long term, I am confident both browser and hardware improvements will make HTML5 games in the browser practical for all platforms. Chrome for Android beta and the Playbook browser are glimmers of this distant hope, as well as the idea that, maybe one day, using PhoneGap won't make Safari totally useless. In the short term, we can work around with these other wrappers which really perform very nicely indeed.

I barely mentioned audio at all, because I could basically write another article this long on the subject. However it should suffice to say that directCanvas and CocoonJS both support polyphonic audio on all platforms. So HTML5 audio on mobile is not the problem it used to be either, if you use those wrappers.

The "write once run anywhere" slogan has been thrown around for decades, but consider this: the game is a single Construct 2 project, and it directly exports and runs successfully on 21 of the 23 platforms tested. And it's early days for HTML5.

Now follow us and share this

Tags:

Comments

0
Animate 11.7k rep

Ashley, so based on your observations. PhoneGap is not a viable option to distritute an HTML5 game right now?

Wednesday, May 23, 2012 at 9:05:57 PM
0
Velojet 21.0k rep

Fascinating and extremely helpful for mobile platform targeting. Some interesting surprises e.g. Safari on the iPad (3) using Retina is actually faster than Safari on your desktop! Thanks for all the effort put into testing, analysing and publishing!

Wednesday, May 23, 2012 at 10:34:17 PM
1
sqiddster 32.7k rep

This looks great! Why are there no tests for cocoonjs on iOS devices, however...

Wednesday, May 23, 2012 at 11:50:26 PM
0
Wink 10.0k rep

Thanks for doing all this research testing, and giving us all the facts. I feel a little disappointed but at the same time relieved to know the direction we need to pursue. Now I can quit wasting my time with the many disappointments, and failures Iv had with PhoneGap, and just wait for appMobi, and Cocoonjs to iron out there bugs and life will be good :D

Thursday, May 24, 2012 at 1:13:39 AM
0
Joannesalfa 28.4k rep

Thanks for researching about performance, also there a something bothers me that Blackberry Playbook, you should to add "Playbook 1" to avoid the confusion. Besides Playbook 2 supports HTML5 and WebGL.

Thursday, May 24, 2012 at 9:15:11 AM
0
Ashley 193.8k rep

@relswick, @Joannesalfa - it was Playbook OS 2. I don't think there's a "Playbook 2" in the sense there's an "iPad 2" which is a different hardware device, it's just a software update. I tried the WebGL renderer and like all other mobile devices, rendering with WebGL is slower than rendering with canvas 2D.

Thursday, May 24, 2012 at 1:07:19 PM
0
balaporte 3,009 rep

I really enjoyed your "Optimisation: don't waste your time" post, but here you say you spent time optimising your engine for best performance. Are you referring here to something like leaving out audio, or are there are other optimisations you performed that might help others get better framerates with their projects?

Also, I'd personally love to see a post about audio...

Thursday, May 24, 2012 at 4:18:20 PM
0
Ashley 193.8k rep

@balaporte - I meant in general while developing Construct 2 we've done lots of work to optimise the Construct 2 runtime engine - nothing specific for this project except the things mentioned in the 'Performance Tips' manual entry. BTW we have 4 blog posts about audio if you go through the archives!

Thursday, May 24, 2012 at 4:29:21 PM
0
ludei 3,616 rep

@ sqiddster - Thanks for the question! CocoonJS supports both Android and iOS. We have our own "launcher" where you can test out your HTML5 game directly on the device itself. Although our launcher is still pending with Apple, we provide an xarchive file.
Here is the tutorial if anyone is interested:
http://blog.ludei.com/installing-the-cocoonjs-launcher-in-ios/

Also, we do have a video of SpaceBlaster deployed on various iOS and Android devices. You can see how it runs here: https://vimeo.com/41737282

Thursday, May 24, 2012 at 6:10:19 PM
0
sqiddster 32.7k rep

@ ludei - thanks for your response! I am very interested in your technology indeed. I was merely wondering if Ashley had forgotten to test your software on iOS devices, but I suppose it is because your iOS integration is so young.
Keep working on C2 support, there are still a few bugs but I am confident that you can get it working very well indeed with our more advanced C2 games.

Thursday, May 24, 2012 at 11:03:02 PM
0
Ashley 193.8k rep

@ludei,@sqiddster - we wanted everyone to be able to reproduce our results, and and I thought the technical knowledge and hardware barrier (needing a mac) to set up CocoonJS for iOS would place it out of reach of most of our users. I was waiting on the iOS launcher app but it's a shame that Apple are being slow about it. When it's released, I'll update the results with it and then other people can easily try it for themselves too.

Thursday, May 24, 2012 at 11:29:12 PM
0
relswick 2,211 rep

@Ashley thanks, that was going to be my next question, was WebGL enabled and it sounded like it was. Yes, OS ver 2 for the Playbook, which I tried it out and the results were similar. Have to hit up someone at RIM and find out why it isn't running well, as it should have been. Thanks!

Friday, May 25, 2012 at 2:48:14 AM
0
balaporte 3,009 rep

@Ashley Oh, I thought there was some magic you did with the game engine itself. I think I've read the other audio posts, it's just something I still find a struggle (especially delays between an event occurring and the sound effect starting) so I look forward to any more info on the subject. Thanks for the comparison testing, that's the kind of data that helps users direct their energies in the right place.

Friday, May 25, 2012 at 5:50:05 PM
1
jimand 2,705 rep

What a difference a driver can make! I ran the performance test on my laptop (an Lenovo Ideapad with Windows 7), using the Google Chrome browser, and got a score of 33. It was OK, but seemed low for this kind of machine. So, I poked around on the Lenovo web site, found an AMD video driver for this model, downloaded and installed it. Turns out, the driver was never on my computer at all. After the driver update, I ran the performance test again, and this time got a score of 60. So now I am a believer in checking your video driver!

Saturday, May 26, 2012 at 2:14:07 PM
0
Ashley 193.8k rep

@jimand, good to hear! Your story is not unusual at all, which is why we try to mention it regularly!

Saturday, May 26, 2012 at 5:29:49 PM

Leave a comment

Everyone is welcome to leave their thoughts! Register a new account or login.