(CC vs) C2 performance (community?) benchmark #1.

Discussion and feedback on Construct 2

Post » Sun Feb 09, 2014 3:58 pm

Hello everyone, recently i met some guys developing a game in C2 for desktop platforms and we had a conversation, in which we discussed if CC would be a viable option instead of C2 and how the performance of the two compares.

Now, i looked around the forum and the blog in hopes of finding a benchmark of the two, and while i did find one that did compare them (link), the blog post was made in 2011. Now, you'll probably agree that this is a long time and much could have changed in it, so i set out to make a new benchmark.


The idea.
The idea i had is to create benchmarks for both CC and C2, which could provide us with some information which is more performant and how the older implementation of different things (e.g. physics, collision detection) has aged, as well as to see how well different machines perform in general. Of course, you can't always make a 1:1 copy of the implementation of something, but i guess that is part of the challenge!


What i came up with so far.
For the initial testing i decided to try comparing something simple - the performance of drawing, rotating and resizing sprites. I made 3 benchmarks, testing each of the features in that order, both for CC and C2. Below you can compare the event sheet views as well as download an archive of both the .cap and .capx files.

Download of the editor files:
LINK ("every 5 ticks" version)
LINK ("every tick" version)

bench1 - drawing test
The simplest of all tests, this benchmark draws a new 256x256 32bit .png file with transparent areas on screen every 5 ticks.
C2:
CC:

bench2 - rotation test
A bit more intensive benchmark, this one does the same as above, as well as rotates the sprites by 5 degrees clocwise every tick.
C2:
CC:

bench3 - scaling test
The last benchmark i made for this thread, it changes sprite sizes every tick, until they either reach a width of 256 or 128, then the process is inverted.
C2:
CC:


Differences between the benchmarks.
There's a couple of things that you'll instantly notice, that are different. First, i used the spritefont object in C2 as opposed to the default font one, as there have been plenty of complaints on the forums about the performance issues with it. Secondly, there is no tick counter built-in inside of C2, therefore i just opted for using a global variable, which should perform almost as good, if not better. Lastly, in CC, there is no fps counter (unless you use the one in the caption, but it is lost on export), therefore i made one of my own, however the accuracy is rough at best (+/- 5 fps).

Furthermore, i adjusted the settings for the best possible performance at the expense of graphical quality, because this is a benchmark for exactly that, not a comparison between the eye-candy options available or roundness of the pixels.
CC (r2) settings: 800x600 window size, point sampling, fixed framerate, 60fps, aliased text, no multisampling
C2 (r156) settings: 800x600 window size, pixel rounding off, point sampling, low quality scaling, no high dpi display, do not clear background, webgl enabled


How to participate.
Of course, none of this is important, unless you participate!
The goal i'd like to reach is to have as many people as possible download the samples and run them, write down the results and post them here, so more people know what to expect performance-wise on different desktop configurations.

Because of this, i tried to make the process as simple as possible, all you have to do is:
1.) download the archive and extract the executables (you can build them yourself, but then you need to select webkit as the export platform in c2, to test the executable exporter).
--LINK(~75MB, .zip archive, "every 5 ticks" version)--
--LINK(75MB, .zip archive, "every tick" version)--
2.) download the spreadsheet i made for easily comparing the results, depending on which office software suite you have.
--LINK(.xls for Microsoft Excel, converted from .ods)--
--LINK(.ods for OpenOffice / LibreOffice etc.)--
3.) run the executables, keep a close eye on the fps values and write down the object counts at which fps reaches the values in the tables, also, if you can write down some basic information about your hardware in the indicated cells, that would be nice. furthermore, you really should check what renderer is being used by c2 and put it in the table (webgl or canvas2d)
4.) take a screenshot of the resulting charts / spreadsheet, host it somewhere (imgur is usually good for this) and post it here!


My results.
Here are the results i got on my outdated laptop i made the benchmark on, but please keep in mind that WebGL is off because my hardware doesn't support it (yay, legacy, right?), therefore the performance is pretty bad there. Another thing, you'll notice that there are "x" in a couple of cells - this means that C2 didn't achieve the indicated framerate at all (shouldn't be a problem on most modern machines, though).

Also, if you'd rather view this in .pdf format, here are the results: LINK.


Also, i should probably mention that higher means better in the charts... :/
This is pretty much just a test post to see what kind of feedback i get, if there is demand for stuff like this, i could compare things like the efficiency of physics, sounds, particle effects etc.
Good luck, everyone!Stiivais2014-02-09 18:33:18
B
10
S
1
Posts: 163
Reputation: 1,436

Post » Sun Feb 09, 2014 4:22 pm

@Stiivais, I definitely like graphs and benchmark tests. These look great!

I haven't tried CC before, but by being around the forum quite a bit, I've gathered that CC was all about performance and the developers had lots of little tricks/hacks in the game engine. But, unfortunately, these hacks caused stability issues so there were a lot of annoying crashes. The CC engine would just have been too unreliable for long term support, thus the decision was made to start over from scratch with all the knowledge learned to make a system that was vastly more resilient and error-proof.

Another deciding factors to change to C2: Which is better, a game that runs at 60 fps but crashes often, or a game that runs at 40fps and always work?

Though, I must admit, I didn't realize that the difference in performance was this much! This has been very eye-opening, thank you!
B
20
S
4
G
1
Posts: 185
Reputation: 2,563

Post » Sun Feb 09, 2014 4:33 pm

Well, imo, the stability issues were mainly encountered in the editor, for example, large event sheets took minutes to open, editing was not very responsive and so on - the actual rendering was pretty good and stable. I do agree that C2 is a better program, however the performance leaves a lot to be desired, but i guess that you can't have both performance and multi platform support along with the ability to run in the browser...

Either way, as i said, the vast difference is likely to be caused because my hardware doesn't support WebGL (or because the drivers are no longer developed) and therefore runs with the slower renderer. I'm curious to see what results others post, though!
B
10
S
1
Posts: 163
Reputation: 1,436

Post » Sun Feb 09, 2014 5:20 pm

For C2 to have a fair chance you really have to test with WebGL available and pixel rounding off. Without WebGL you'll see a big hit from the browser engine overhead. I know not all systems support WebGL, but it's increasing all the time and represents the best possible performance with C2, which is probably what you really want to test. Also with pixel rounding on you make C2 do extra work on each draw call to round the position where in Classic the option was effectively off (it was not supported) so it wouldn't be doing that work for each object.

It's also vitally important to use exactly the same window size at runtime for the test to be fair. If you run a CC test in an 800x600 window, you must also ensure the viewport in the browser window is also 800x600 when testing C2 - otherwise the amount of work for the GPU is different and will skew the test. It's not enough to just set the window size property.

We have in the past found one case where C2 outperforms Classic with WebGL rendering on some systems: https://www.scirra.com/blog/102/html5-games-faster-than-native

C2 also has some algorithmic improvements over Classic in some cases:
- in C2 pathfinding is asynchronous (runs in a separate thread via a web worker) whereas in CC it was synchronous and run in the main thread. This meant even with native code long pathfinding searches would make the game jank, whereas this does not happen in C2.
- C2 recently got the collision cells optimisation, making very large layout collision-test intensive games far faster. CC never did this optimisation and brute forced everything all the time, so C2 probably actually scales better for large games.
- some areas of the event engine and rendering engine do less work via a lazy loading scheme whereas Classic brute forced things again.

C2 is also considerably more stable, including at runtime - Classic had some weird crashes and glitches due to the gone-too-far optimisations we implemented.Ashley2014-02-09 17:21:47
Scirra Founder
B
399
S
236
G
89
Posts: 24,539
Reputation: 195,422

Post » Sun Feb 09, 2014 5:32 pm

@Ashley First off, it's nice to have someone such as yourself explain these technicalities, but now i feel really bad about myself - who could have known that more precision actually is good for performance in this case?
I'll look into updating the benchmarks for this to be fair now...

But about the resolution - what you are saying is, that it's insufficient just to have a specific window size set with the resizing/scaling parameter set to "off", is that right? What else could be possibly done? Also, is this still a problem if the file is exported as an executable?

Also, if you're already here, i have another question i'd like for you to answer. Does the minification and/or the image compression parameters have any noticeable effect on the performance?Stiivais2014-02-09 17:34:32
B
10
S
1
Posts: 163
Reputation: 1,436

Post » Sun Feb 09, 2014 5:59 pm

Ive started with testing, but it takes too much time, it would use 30 mins per test to actually come into the 30fps area (estimated).
But if theres one thing i can say already now, then it is the fact, that C2 uses far less GPU Usage.
Btw, how does it come that the FPS gets over 60?
When im using C2, the fps never gets over 60.
(Note: my GPU gets hotter then using furmark)SgtConti2014-02-09 18:01:46
"It's done when it's done"

Shadows of War
Buy on Steam ;)
B
24
S
10
G
7
Posts: 253
Reputation: 4,931

Post » Sun Feb 09, 2014 6:11 pm

I don't use CC, but the performance comparaison on optimals parameters for both CC and C2 can be somewhat interresting, however I think it'll continue to evolve with time for C2, with each update, so you might want to precise which versions of both software were tested

Also in both cases I think we should need more complete tests (some little games for exemple) to really compare at the end.
Game design is all about decomposing the core of your game so it becomes simple instructions.
B
53
S
22
G
18
Posts: 2,122
Reputation: 17,123

Post » Sun Feb 09, 2014 6:13 pm

I updated the benchmarks now, pixel rounding is now off (although i didn't get very much improvement after re-running the tests).

@SgtConti If i'm not mistaken, then the framerate of C2 is locked to 60fps (or perhaps that's just the refresh rate of your monitor?) and while it should be the same in CC, i had to figure out a way of manually calculating the current fps, which probably isn't too precise (the project has it set at a fixed rate of 60)...

As for the time testing takes, it is so because new sprites are created every 5 ticks, as opposed to every single tick. If you want it to be faster, you should download the .cap(x) files and edit it to say "every tick", i can update the .cap(x) files for you, if you want this (although giving you pre-made .exe files will take a couple of minutes, as my upload speeds are not the best).

Update: I made a version, which should work 5 times faster and thus be more useful for modern computers. Just get the "every tick" version...

@Aphrodite I used the latest version of CC (r2) and a somewhat recent one of C2 (r15X), i'll check and update the main post. Also, it's a bit too early to speak of playable games that can be used to compare the performance, as there still are a couple of problems that need to be solved, for example, the speed at which the applications run (what SgtConti had) and the optimal parameters for both CC and C2, so everything is fair...Stiivais2014-02-09 18:36:08
B
10
S
1
Posts: 163
Reputation: 1,436

Post » Sun Feb 09, 2014 8:14 pm

@Stiivais - if you're using a non-fullscreen mode that should be fair, yes. Also exporting can actually help it run faster - the export-time optimisations can help and some preview-mode bug checks are removed. Obviously that's a bit more hassle, but if you want it to be totally fair, export as a HTML5 website and run it from a web page (e.g. Dropbox)! (That also makes it easy for others to run your tests)

I think Classic would beat C2 in several tests, but hopefully not by as much as you might think, and I reckon C2 would beat CC on some other tests as well. I'd be curious to see your results with WebGL support.
Scirra Founder
B
399
S
236
G
89
Posts: 24,539
Reputation: 195,422

Post » Sun Feb 09, 2014 8:39 pm



for what it's worth I ran the 'every tick' binaries are here are my results - roughly



Hope it helps...
B
9
S
3
G
1
Posts: 112
Reputation: 1,827

Next

Return to Construct 2 General

Who is online

Users browsing this forum: No registered users and 6 guests