Get color at pixel

For developers using the Construct 2 Javascript SDK

Post » Thu Mar 31, 2016 11:31 am

Fortunately I've run into the disembodied "this" issue before, so I think I set that up correctly, but it's good to know I wasn't being paranoid. That said, I did set it up wrong the very first time. :)

img.src always blank
Within the onload() function, I set up an alert() to print out the image URL that was being assigned to img.src. The src was always a base64 image string of a blank canvas with a size matching the game's window dimensions. For some reason, I'm just never getting anything but a blank image from:
this.runtime.canvas.toDataURL("image/png")
I was able to use this base64 image decoder to see what exactly was stored in the base64 image text.
http://codebeautify.org/base64-to-image-converter


Workaround
So with some experimenting I may have found a workaround that allows me to get legitimate pixel color values.
Rather than trying to capture the canvas inside my Action function, I'm instead passing one extra parameter to the action, an Image URL, so I can simply pass C2's System CanvasSnapshot URL variable to my Action.

This guarantees that the URL contains an actual image. And it seems to work, so far. I'm actually getting correct color data. Ultimately I would like to be able to capture the game canvas entirely from within my plugin, but I don't know if there's a way to ensure that the capture is timed and sequenced correctly so that I actually get an image.


Capturing from within my plugin?
Do you have any ideas on what might be going wrong with capturing the canvas from within my plugin? You mentioned waiting for the image load trigger. Are you talking about waiting for a trigger related to the game canvas itself? It looks like when this.runtime.canvas.toDataURL("image/png") is used from within my plugin, the URL returned is a text representation of a large blank image matching the game window size. If there is a way to wait for it to contain image data that would be cool.


My current code

Instance vars.
Code: Select all
instanceProto.onCreate = function()
{
   // Captured pixel.
   this.pixelCap = []; // Will be filled by a getImageData() call.
   this.pixelCapX = 0; // Coord of captured pixel.
   this.pixelCapY = 0; // Coord of captured pixel.
};


Action.
Code: Select all
Acts.prototype.pixelCap = function ( posX , posY , dataURL )
{
   var img = new Image();
   var self = this;

   img.onload = function ()
   {
      // Make temp canvas, and draw image data into it.
      var canvas = document.createElement('canvas');
      canvas.width = img.width;
      canvas.height = img.height;
      var ctx = canvas.getContext('2d');
      ctx.drawImage(img,0,0); // Paste image starting at coord (0,0).

      // Save pixel data.
      self.pixelCap = ctx.getImageData( posX , posY , 1 , 1 ); // Save to instance var.
      self.pixelCapX = posX;
      self.pixelCapY = posY;

      // Trigger notice of completion.
      self.runtime.trigger(cr.plugins_.fi_color.prototype.cnds.onCapturedPixel, self);
   }

   img["crossOrigin"] = "anonymous";
   img.src = dataURL;
};


Condition triggered from the onload in the Action code above.
Code: Select all
Cnds.prototype.onCapturedPixel = function () // REMEMBER: These params are C2 event sheet params.
{
   return   true;
};


The Expression that retrieves the Red value of the pixel.
Code: Select all
Exps.prototype.cpR = function ( ret )
{
   ret.set_float(   this.pixelCap.data[ 0 ]   );
};
B
27
S
18
G
8
Posts: 328
Reputation: 6,158

Post » Thu Mar 31, 2016 6:27 pm

I think the answer is in c2's runtime with it's capture action. It doesn't grab the canvas right there, instead it grabs it later (I think it's right after the next redraw).

I was fiddling with that with the paster plugin before and you have some places in the plugin that's run at different times.

Two that come to mind are the tick() and tick2() functions. I forget if there are others. One is run before the event sheet and one is done after. You do have to tell the runtime to call those functions with I think a tickme() call? Searching for tick2 probably would give a post with more info as I recall, as well as any I'm fogetting here. Anyways my guess is in those places might be better times to grab the canvas, but it's kind of test and see.

Basically the process would be to set a Boolean with your action and in the tick function it would check if the Boolean is set and then do what your action does currently.

The idea can probably be more definite, you just need to break down what the runtime is doing. Off hand I think it looks something like this? I'd have to verify where the capture is done.

Tick()
Events
Tick2()
Capture canvas
Draw
Repeat
B
92
S
32
G
109
Posts: 5,290
Reputation: 70,991

Post » Fri Apr 01, 2016 6:41 am

The tick2() suggestion sounds like it could be pretty cool, but I actually just started looking through the "preview.js" file, and I'm not sure if tick2() will work. Though I could be misinterpreting what I found.

In preview.js I regex searched for
tick\w*\(
to find all the different tick types.

I found the various tick calls inside the logic() function declaration:
Code: Select all
// Process application logic
Runtime.prototype.logic = function (cur_time)

pretick() "// Tick objects-to-pre-tick"
tick() "// Tick behaviors"
posttick() "// Call posttick on behaviors"
tick() "// Tick objects-to-tick"
run() "// If the running layout has an event sheet, run it"
tick2() "// Post-event ticking (tick2)" [Looks to tick behaviors]
tick2() "// Tick objects-to-tick2"
(The comments are taken from preview.js and precede the respective tick calls.)


The logic() function is called from within a different runtime tick() function, not to be confused with the object tick() function above.
Code: Select all
Runtime.prototype.tick = function (background_wake, timestamp, debug_step)
(Note this is a function also named "tick" but it belongs to the runtime, where all the tick() functions called inside the logic() function belong to objects or behaviors.)

Within this runtime tick() function, logic() is called, and afterwards draw() and drawGL() are called, and after that the snapshot canvas code runs.

So I think the canvas may not get redrawn until after all of the various object tick calls are completed, and the logic() function exits. Would this matter? I'm not sure if there's a reason I wouldn't just get the canvas result from the previous drawGL call, unless it's been erased by the time the tick code runs.

[update]
I had a weird and possibly terrible idea. Could I force the canvas to redraw from within the plugin by calling draw() or drawGL()? That seems like it could work, but also be really bad if put inside a loop in the event sheet. :)
B
27
S
18
G
8
Posts: 328
Reputation: 6,158

Post » Fri Apr 01, 2016 5:36 pm

Looks like the closest you can get to the runtime snapshot is to use pretick.

Basically run this when the plugin is created:
this.runtime.pretickMe(this);

Then when you run the action to capture the pixel, you set a boolean in your plugin and tell the runtime to redraw.
this.runtime.redraw=true;

Then in pretick you'd grab it.


I wouldn't call drawgl directly. I mean you could but it would be very slow. Getting pixels from the canvas is slow anyway. What we're doing is grabbing the entire canvas and loading it into an image, which can take several frames. If you called drawgl directly you'd just be making the runtime draw something it would draw anyway.
B
92
S
32
G
109
Posts: 5,290
Reputation: 70,991

Post » Sat Apr 02, 2016 7:32 am

Yeah, I figured the drawGL() thing would be hilariously awful.

I'm not quite sure if I follow how pretick is the closest to the snapshot, unless you mean, it's the closest to a snapshot taken on the prior tick, in which case I think I understand.

At the moment I'm triggering a condition (belonging to my plugin) when the img.onload() function runs. This means that I get to act as soon as the color info is available, but outside of that triggered condition, I can't guarantee that the image has been loaded yet.

So, is the boolean you're talking about part of a system that would make the color info accessible from regular non-triggered events? That is, as soon as the boolean is true, then the data would be stored during the very next pretick and, by the time the event sheet code runs, you know you can trust the image data?

Finally I'm also not sure if I understand the reason for setting the runtime redraw to true. Would that indicate to C2 that the scene need to be redrawn during the next render phase?

All that said, with your help so far, I've gotten a system that does allow me to pick any color on the screen as it appears after shaders are applied, which is really handy, so thanks. :)
My main plan was to use it as part of a color picking UI. Fortunately a color picking UI doesn't require thousands of picks per frame-draw or anything, so it's probably okay that it's not super efficient at the moment.

I also just noticed that because my current version of the picking function takes a dataURL as an image parameter, it also works with Pode's base64 image extractor and injector. Though again, that extracts and injects underneath the shading pass, but it's a cool discovery none the less. :)
B
27
S
18
G
8
Posts: 328
Reputation: 6,158

Post » Sat Apr 02, 2016 6:06 pm

Pretick is the very next spot we can run code after the game draws, and since C2's snapshot runs right after a draw, this will get you pretty close.

Best I can tell telling the runtime to redraw before using imageUrl is needed. The snapshot function does this and I recall it not working if we don't.


By boolean I mean something you'd add to your code. Something like this:
action()
{
this.grabcanvas = True;
}

pretick()
{
if (this.grabcanvas)
{
this.grabcanvas=false;
...
}
}
B
92
S
32
G
109
Posts: 5,290
Reputation: 70,991

Post » Mon Apr 04, 2016 2:26 pm

Wow okay, I think I can actually stay logged in for real this time.

Thanks for the information.

Sorry for the delayed response. I kept getting taken to the login page after every button click or page change, and couldn't get to any part of the forum interface that would let me post without asking me to log in again. This is the 3rd time I've typed this message up. :)
I checked the "Keep me logged in" box last time, so here's hoping it posts this time.

[update]
Okay it worked. :D
Maybe something is set up weird in my browser because if "Keep me logged in" is unchecked on this site's login page, it goes into "log me out the instant I do anything" mode.
B
27
S
18
G
8
Posts: 328
Reputation: 6,158

Previous

Return to Javascript SDK

Who is online

Users browsing this forum: No registered users and 0 guests