Note: this document has been replaced with a new proposal at https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html. This document remains as a discussion of a prior proposal.

This document proposes some additional ImageData methods to help asynchronously convert ImageData to and from Blob and Image.

This is a first draft to propose these features for the public-webapps mailing list.

Introduction

Note: this document has been replaced with a new proposal at https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html. This document remains as a discussion of a prior proposal.

ImageData represents the size and pixel data of an image, and was originally specified for use with the canvas [[2dcontext]]. Modern web apps need asynchronous functions to process image data without "janking" the browser UI, but currently all means of creating or converting ImageData objects involve synchronous use of a canvas. Two new asynchronous methods are proposed to create and convert ImageData to and from Blob. Converting an Image to ImageData also requires synchronous use of a canvas, so an additional method is added for this case.

Proposed WebIDL

Promise<Blob> toBlob (optional DOMString type, optional any... encoderOptions)
Asynchronously encode the image pixel data in to a blob with the given compressed format. This is intended to work identically to the HTMLCanvasElement toBlob() method, as if the ImageData had first been written to a canvas with putImageData and then toBlob called on the canvas with the same arguments, except that it processes fully asynchronously.
optional DOMString type
If provided, controls the type of the image to be returned (e.g. PNG or JPEG). The default is image/png; that type is also used if the given type isn't supported.
optional any... encoderOptions
Depends on the type. If image/png is used, this parameter is ignored. If image/jpeg is used and this parameter provides a number in the range 0.0 to 1.0 inclusive, then it is treated as the desired quality level.
static Promise<ImageData> fromBlob (Blob blob)
Asynchronously decode the compressed image data in the blob and return an ImageData object representing the decompressed image pixel data. This is intended to work identically to using URL.createObjectURL(blob), setting an image's src attribute to the blob URL, drawing the resulting image to a canvas, and then using getImageData, except that it processes asynchronously. This method should therefore be capable of decoding any compressed image format that can be read by image elements. The promise rejects under the same conditions that loading an image from the blob URL fires onerror.
Blob blob
Specifies the blob to asynchronously decompress to ImageData.
static Promise<ImageData> fromImage (Image image)
Asynchronously retrieve or decode the image data for the given Image object and return an ImageData object representing the decompressed image pixel data. This is intended to work identically to drawing the image to a canvas and then using getImageData, except that it processes asynchronously. In other words it is the same as fromBlob once the image has been obtained from the blob.
Note: This must obey the same restrictions that apply when using getImageData on a canvas which has been tainted by drawing a cross-origin Image.
Image image
Specifies the image to asynchronously decompress to ImageData.

Note: while toImage would be the logical counterpart to fromImage, it is omitted because it raises the question of what to set for the resulting Image object's src property. To avoid having to deal with this problem, it is left to developers to use the toBlob method instead, from which they can create a URL to the resulting blob to pass to an Image object. This is still fully asynchronous.

Examples

Example 1 demonstrates synchronously decompressing a Blob to ImageData. Note that some browsers optimise this path such that decoding the image data is not done until the getImageData call, so while it may appear to load the image asynchronously, the actual image decode to an ImageData is still synchronous.

// note: looks asynchronous because it necessarily returns a promise
// due to loading the blob as an Image, but the important parts are
// still synchronous.
function BlobToImageData(blob)
{
	let blobUrl = URL.createObjectURL(blob);
	
	return new Promise((resolve, reject) =>
	{
		// asynchronous part, which may not actually decode any image data
		let img = new Image();
		img.onload = () => resolve(img);
		img.onerror = err => reject(err);
		img.src = blobUrl;
	}).then(img =>
	{
		URL.revokeObjectURL(blobUrl);
		
		let w = img.width;
		let h = img.height;
		
		let canvas = document.createElement("canvas");
		canvas.width = w;
		canvas.height = h;
		let ctx = canvas.getContext("2d");
		ctx.drawImage(img, 0, 0);
		return ctx.getImageData(0, 0, w, h);	// some browsers synchronously decode image here
	});
};

With the new method this can be simplified to:

function BlobToImageData(blob)
{
	return ImageData.fromBlob(blob);
};

Example 3 demonstrates synchronously compressing an ImageData to a Blob. Note the canvas element's toBlob method is asynchronous, in theory allowing the compression to happen asynchronously, but this still includes the overhead of creating a canvas 2D context followed by putImageData.

function ImageDataToBlob(imageData)
{
	let w = imageData.width;
	let h = imageData.height;
	let canvas = document.createElement("canvas");
	canvas.width = w;
	canvas.height = h;
	let ctx = canvas.getContext("2d");
	ctx.putImageData(0, 0, w, h);	// synchronous
	
	return new Promise((resolve, reject) =>
	{
		// note Blink does not support toBlob, so in Chrome this would have to be another
		// synchronous call to toDataURL(), making the entire function synchronous.
		canvas.toBlob(resolve);	// implied image/png format
	});
};

With the new method this can be simplified to:

function ImageDataToBlob(imageData)
{
	return imageData.toBlob();	// implied image/png format
};

In some cases, it may be desirable to convert an existing Image object in to an ImageData. For example the Image may have been obtained over the network, and it would be inconvenient to first convert it to a blob (which would also require synchronous use of a canvas again). Example 5 demonstrates synchronously converting an Image to ImageData.

function ImageToImageData(img)
{
	let w = img.width;
	let h = img.height;
	
	let canvas = document.createElement("canvas");
	canvas.width = w;
	canvas.height = h;
	let ctx = canvas.getContext("2d");
	ctx.drawImage(img, 0, 0);
	return ctx.getImageData(0, 0, w, h);	// synchronous
};

With the new method this can be simplified to:

function ImageToImageData(img)
{
	return ImageData.fromImage(img);
};

Note: fromImage is expected to obey identical cross-origin security restrictions that apply to getImageData from the previous example.

Potential alternative solutions

Asynchronous getImageData

One alternative solution would be to add an asynchronous getImageData method to the 2d context. However most modern implementations use GPU-accelerated 2d contexts and such a method could require readback from the GPU, which has costly implications for performance. To avoid race conditions with subsequent drawing during the processing of an asynchronous getImageData implementations would likely first have to synchronously make a copy of the canvas contents, which adds more overhead than would be necessary with the proposed ImageData methods.

Further, the logical counterpart to an asynchronous getImageData method is an asynchronous putImageData method. This method would be necessary for converting an ImageData back to a Blob or Image. It is hard to see how this could be implemented without creating race conditions if more synchronous drawing happens during the processing. Should drawing done while it is processing be queued up to happen after, or should the drawing be overwritten when putImageData completes? Neither solution seems great, and the proposed ImageData methods circumvent the canvas entirely, side-stepping this problem.

Canvas-in-workers

Another possible solution is to run the canvas in a worker. Then the synchronous use of getImageData and putImageData will be processed in parallel to the main thread. However to implement this efficiently requires Image, ImageData and Blob to all be transferrable. It is not known how feasible this is, and it is assumed to be unlikely to be possible considering that Image is a DOM element and workers have no DOM. Blobs and ImageDatas can be converted to transferrable ArrayBuffers, but likely involves making copies to transfer to the worker since the objects are not themselves transferrable, and it does not support conversions involving the non-transferrable Image. It could also be argued this is an unnecessarily complicated solution and that the web platform ought to provide this type of feature as a built-in primitive to make it as easy as possible for web developers to build effective asynchronous web apps.