Get color at pixel

For developers using the Construct 2 Javascript SDK

Post » Tue Mar 29, 2016 2:34 am

I'm looking to create a color utility plugin, to provide a variety of color related expressions and conditions.
The first expressions I want to add would form a color picker that will get a color at a specified pixel coord on C2's game canvas.
[edit] (By "canvas", I mean C2's game window area, not the 3rd party "Canvas" plugin.)

It would look something like this:
pxR( x , y )
pxG( x , y )
pxB( x , y )


Ideally the picker expressions would get the color as it appears on the screen. (i.e. After all webGL effects have been applied.)
Does anyone know how best to do this from within the SDK?
Thanks in advance for any advice or suggestions. :)


Possibly related R0J0hound example code
I saw in another post that @R0J0hound posted some javascript that could be executed from within a C2 event sheet to retrieve color at a given pixel, and this looks like what I want to do.
Granted, I'm not sure if this is the best way to do it from within the SDK, since I might be able to get access to the canvas directly, rather than looking it up in the DOM by element id.
span-class-posthilit-color-span-of-a-span-class-posthilit-pixel-spa_p946044?#p946044
Code: Select all
// This is a paraphrase of R0J0hound's original code found at the link above.
var canvas = document.getElementById( 'c2canvas' );
var ctx = canvas.getContext( '2d' );
var pixel = ctx.getImageData( x , y , 1 , 1 );
pixel.data[ 0 ];")
(That said, I wasn't able to get this code to work from within C2. It seemed to work up until assigning the canvas context into "ctx", which as near as I can tell always assigned null/undefined, as I could never get any data out of it, or .toString() the object. So I'm probably doing something wrong.)
Last edited by fisholith on Tue Mar 29, 2016 4:24 am, edited 1 time in total.
B
27
S
18
G
8
Posts: 331
Reputation: 6,164

Post » Tue Mar 29, 2016 2:41 am

Reading color of pixel in canvas plugin will return the original color in this canvas, not the final result after shader of weblg, imo.
B
109
S
27
G
276
Posts: 4,479
Reputation: 154,418

Post » Tue Mar 29, 2016 4:28 am

@fisholith
That should work with webgl off i'm pretty sure. Even with webgl on I think it should work. For reference you could look at the snapshot canvas action, which basically uses the same approach i think. i can''t test webgl currently.

From a plugin you should be able to access the canvas from the runtime variable i think. There is a way to grab a pixel with just webgl which I tried to use with the paster plugin when webgl was working for me but it proved unreliable.
B
94
S
33
G
113
Posts: 5,354
Reputation: 73,269

Post » Tue Mar 29, 2016 4:32 am

Thanks for the reply @rexrainbow, :)

I'm familiar with that aspect of the third party Canvas plugin. I meant the HTML5 canvas that C2 renders the game into. Apologies, I should have been more specific in my first post. I think (though I may be wrong) R0J0hound's code I linked to in the first post samples C2's main HTML5 canvas directly.

I was also thinking that since C2 can store a screencap of the canvas that includes applied shaders, would it be better to try to use an approach related to that screencap mechanism to get the post shader colors?

I've been trying to get a simple method for picking screen colors for a while and all the event-and-object approaches I've tried end up being kind of fiddly, especially when it comes to capturing post shader colors.

[edit] Oh, hey @R0J0hound, I think you just posted right before I did. :D
I have to head out briefly, but I'll be able to reply shortly. Thanks for the info. :)

[edit 2] Thanks for the info, and the suggestion @R0J0hound, I'll take a look at the screencap action code.
B
27
S
18
G
8
Posts: 331
Reputation: 6,164

Post » Tue Mar 29, 2016 11:41 am

Okay, I've done some digging through the C2 js files, but I'm having trouble finding the point at which any image data is actually encountered.


C2 snapshot code
The Action in system.js calls "doCanvasSnapshot()"
[system.js , line 1913]
Code: Select all
SysActs.prototype.SnapshotCanvas = function (format_, quality_)
{
   this.runtime.doCanvasSnapshot(format_ === 0 ? "image/png" : "image/jpeg", quality_ / 100);
};


"doCanvasSnapshot()" is called in system.js, and seems to be declared in preview.js ... maybe?
I'm not especially familiar with the setup of the HTML5 canvas in C2, or in general. So I suspect there are probably some conventions I'm not recognizing.
[preview.js , line 5043]
Code: Select all
Runtime.prototype.doCanvasSnapshot = function (format_, quality_)
{
   this.snapshotCanvas = [format_, quality_];
   this.redraw = true;      // force redraw so snapshot is always taken
};


I found a spot in the tick code that seems to be involved with the snapshot process, and uses "this.snapshotCanvas" , but it seems like it's arranging for the snapshot data to be available to some other system somewhere, rather than handling image data itself.
[preview.js , line 2346]
Code: Select all
// Snapshot the canvas if enabled
if (this.snapshotCanvas)
{
   if (this.canvas && this.canvas.toDataURL)
   {
      this.snapshotData = this.canvas.toDataURL(this.snapshotCanvas[0], this.snapshotCanvas[1]);
      
      if (window["cr_onSnapshot"])
         window["cr_onSnapshot"](this.snapshotData);
      
      this.trigger(cr.system_object.prototype.cnds.OnCanvasSnapshot, null);
   }
      
   this.snapshotCanvas = null;
}


The only other place I'm even seeing the word snapshot come up is in an assignment to an array named "window".
I have no idea what's going on here. Is this some kind of variable function callback thing?
[preview.js , line 6061]
Code: Select all
window["cr_getSnapshot"] = function (format_, quality_)
{
   var runtime = window["cr_getC2Runtime"]();
   
   if (runtime)
      runtime.doCanvasSnapshot(format_, quality_);
}



My thoughts so far
So I'm starting to suspect that the image data saved by the snapshot action is setup inside code much closer to the actual drawing of the main canvas, and the snapshot action just flags it to be saved to a variable for later use. I could be completely wrong though, as I'm not quite sure what I'm looking at.

All that said, I'm not sure if I really need to get at the snapshot code in the first place. I was going to take a look at it just to get an idea of how to pick a color from the final rendered scene, though I don't know if that's the direction I should be going, or if there's a better way to get at the color data.

Any suggestions are still welcome. I'm still not quite clear on how to access the canvas from within the SDK.
B
27
S
18
G
8
Posts: 331
Reputation: 6,164

Post » Tue Mar 29, 2016 7:40 pm

I found a pc to test webgl on, and it seems that you can't get both a webgl and 2d context on the same canvas. That explains why my code doesn't work.

You can still save the entire canvas. Ex:

var canvas = document.getElementById( 'c2canvas' );
canvas.toDataURL("image/png")
or
canvas.toDataURL("image/jpeg", 0.75)

That is all the runtime is doing, except it waits till after everything is drawn for that frame.

From a plugin you can get the canvas with:
this.runtime.canvas
instead of
document.getElementById( 'c2canvas' );

So the process to get the color of a pixel that also works with webgl would be
Code: Select all
var img = new Image();
var self = this;

img.onload = function ()
{
   var canvas = document.createElement('canvas');
   canvas.width = img.width;
   canvas.height = img.height;
   var ctx = canvas.getContext('2d');
   ctx.drawImage(img,0,0);
   var pixel = ctx.getImageData( x , y , 1 , 1 );
   
   // save the value in the plugin and call a trigger to use the color in your program
   self.redvalue=pixel.data[ 0 ];
   self.runtime.trigger(cr.plugins_.Sprite.prototype.cnds.OnURLLoaded, self);
}


img["crossOrigin"] = "anonymous";
img.src = this.runtime.canvas.toDataURL("image/png")

It's referenced from the loadUrl sprite action.

Basically load toDataUrl to a image. This is asynchronous so we need a callback when it finishes. When it's done loading we can then create a canvas, draw the image to it and retrieve a pixel with a 2d context. Then we save the pixel to a variable we can access from an expression and call a event sheet trigger.

Whew, that's not exactly straightforward. When webgl is off a pixel can be retrieved in place with the code in your op.

Now to go the webgl route instead there's a nifty readpixels function so supposedly it's as simple as this:
https://developer.mozilla.org/en-US/doc ... readPixels
Code: Select all
var canvas=document.getElementById('c2canvas');
var gl=canvas.c2runtime.gl;
var pixel = new Uint8Array(4);
gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);
pixel[0];

which works only on frame 1 it seems. I don't know why, and my google fu hasn't been helped me yet. But that could be a start.

Edit:
Here's the proper solution:
https://asalga.wordpress.com/2011/08/01 ... ingbuffer/
But that requires modifying the runtime in preview.js.
B
94
S
33
G
113
Posts: 5,354
Reputation: 73,269

Post » Wed Mar 30, 2016 2:31 am

Wow, thanks @R0J0hound, that's super helpful. :)


Sprite loadURL code
I just looked up the Sprite's loadURL code.
I see what look like 4 parts to the code.

1. Declare some variables, including the image "img". [3 lines]
2. Assign an onload() function. [a bunch of lines]
3. An if-statement that checks stuff about the "url_". [2 lines]
4. Finally "img.src = url_;". [1 line]


So does that final assignment in part 4 "img.src = url_;" trigger the onload() function assigned in part 2? Or rather, does the completion of loading that newly assigned URL trigger it?

Because otherwise I'm not sure where the loading of the image is happening.

If that's the case, I may understand what's going on in your code. I think the assignment of that onload() function just threw me a bit, because it looked like nothing explicitly called it. So I'm assuming that again the assignment to img.src sets some behind-the-scenes stuff in motion that ultimately triggers the onload() function.


Possibly realized my in-event color picking troubles
As an added bonus, when looking through the Sprite's loadURL code, I may have figured out why my previous in-event attempts to pick colors acted so finicky. I think I was waiting till the screenshot was taken, via the triggered event, but I wasn't waiting until after I was done loading it into a sprite, via the sprite's triggered event.

I've been trying to pick the post-webgl colors out of the 3rd party Canvas plugin by saving a screenshot with the System action, loading the screenshot into a Sprite, and then pasting that screenshot Sprite into a Canvas plugin, so I can pick it's colors. I keep forgetting that every image loading step is going to be asynchronous, not just the screenshot.


Some questions about your code example
Will the "canvas.getContext('2d');" in your first code example work if webgl is enabled?
Or does that not matter because, once inside the onload() function, we're dealing with our own custom canvas for the screencap, rather than C2's game canvas?


Thoughts on readpixels
The "working only on frame one" thing might be because of how buffer clearing works when using webgl with preserveDrawingBuffer set to false, which I assume it is for C2 for performance reasons. Not sure though.

At the second link you posted one of the comments seems to indicate that it might be possible to get data from the buffer after it's drawn but before it's handed to the compositor and cleared. Granted that would require inserting code into C2's runtime I gather, which is probably not the direction I should be going. Of course I could be totally wrong about all of the above, I'm just trying to follow along. :)



[update]
So I set up the plugin and implemented the example code, but I'm always getting [ 0 , 0 , 0 , 255 ] as the pixel values regardless of the coordinates I use. I'm using an alert() to print the pixel directly out of the onload() function, right after the call to ctx.getImageData().

Still working on figuring out what's going on, but I'm open to any suggestions.

Do I maybe need to wait for some aspect of the C2 canvas to load before passing it to "img.src"?
B
27
S
18
G
8
Posts: 331
Reputation: 6,164

Post » Wed Mar 30, 2016 3:41 pm

In js as soon as you give a URL to img.src the image will be loaded asycronously and when it finishes it calls the function at img.onload. It's an html thing.

Yeah you'd probably want to want to wait till the image load trigger to draw it to the canvas.

It should work fine getting a 2d context because it's using the new canvas created right above it.

The preserveDrawingBuffer setting is slower when it's on which is why it typically is off. The fact readpixels worked once with that flag off probably has to do with undefined behavior since it's not supposed to work when the flag is false. As far as I can tell, you can only set that flag when first getting the context, aka when the runtime creates it. But ya it's not really acceptable for a user to have to make that change when using such a plugin. :p

If the image URL idea isn't working paste the code for that function and the trigger it calls. The events should look like this:
On click
--- get pixel at X,y

On pixel got
--- set redvar to pixel.red
B
94
S
33
G
113
Posts: 5,354
Reputation: 73,269

Post » Thu Mar 31, 2016 5:19 am

At the moment I have an action that captures a pixel to a plugin instance variable.
It sets the onload() callback, and sets the img.src.
When the onload() callback runs it grabs a pixel out of the image data at the coords passed to the action function.

Speaking of which, now that I think of it, I should try printing the coords from within the onload() function to see if they are the correct values or undefined, as the action function scope is gone when onload() runs. Maybe that's the issue, and so I need to store the coords in a plugin instance variable.
That's the point at which everything I print out is (0,0,0,255).


So within the onload() I save the pixel data to a plugin instance variable, and then trigger a C2 plugin condition to indicate that the load is complete, and is ready for use.

From there I can use an expression to get the R, G, or B values. Or at least I could if they weren't all 0.

I'll post my code for the action condition and expression in a little bit, in an update.
Thanks again for all the extra info and clarification. :)
B
27
S
18
G
8
Posts: 331
Reputation: 6,164

Post » Thu Mar 31, 2016 8:08 am

One thing I didn't cover was in the action right before the callback you see the line:
var self=this;
Then inside the callback self is used instead of this. The reason for that is "this" is the object instance inside the action, but when the onload function is called "this" is instead the global variable space because the function is called without a object. You can find more information by googling "JavaScript this".

Anyways that just means to just use self instead of this everywhere inside the onload function. If you used "this" instead it would explain why the instance variables aren't being changed.
B
94
S
33
G
113
Posts: 5,354
Reputation: 73,269

Next

Return to Javascript SDK

Who is online

Users browsing this forum: No registered users and 1 guest