node.js - How to convert a Uint8Array to a CanvasImageSource?

Question

Welcome To Ask or Share your Answers For Others

node.js - How to convert a Uint8Array to a CanvasImageSource?

asked Jan 27, 2021 in Technique[技术] by 深蓝 (71.8m points)

node.js - How to convert a Uint8Array to a CanvasImageSource?

In TypeScript (NodeJS), I am struggling to convert a Uint8Array with bitmap image data into a CanvasImageSource type.

More Context

I am working on a typescript library that will be used in the browser as well as NodeJS environments. The library does image operations with WebGL, so in NodeJS environments, I am attempting to leverage headless-gl. My library includes a function (getCanvasImageSource) that returns a CanvasImageSource for clients to use.

A bunch of code was removed for the purposes of asking the question. A WebGL shader will create the desired image on the gl context, which the client can retrieve through the CanvasImageSource. This works as intended in a browser client.

/**
 * Browser version of the library.
 */
export class MyLibrary {
    protected gl!: WebGLRenderingContext;
    protected width: number;
    protected height: number;

    public getCanvasImageSource(): CanvasImageSource {
        return this.gl.canvas;
    }

    /**
     * the GL context can only be created in a browser.
     */
    protected makeGL(): WebGLRenderingContext {
        const canvas = document.createElement('canvas');
        canvas.width = this.width;
        canvas.height = this.height;
        const glContext = canvas.getContext('webgl');
        if (!glContext) {
            throw new Error("Unable to initialize WebGL. This browser or device may not support it.");
        }
        return glContext;
    }
}

import gl from 'gl';

/**
 * A subclass of MyLibrary that overrides the browser-specific functionality.
 */
export class MyHeadlessLibrary extends MyLibrary {
    public getCanvasImageSource(): CanvasImageSource {
        // The canvas is `undefined` from headless-gl.
        // this.gl.canvas === undefined;

        // But, I can read the pixel data as a bitmap.
        const format = this.gl.RGBA;
        const type = this.gl.UNSIGNED_BYTE;
        const bitmapData = new Uint8Array(this.width * this.height * 4);
        this.gl.readPixels(0, 0, this.width, this.height, format, type, bitmapData);

        // This is where I am struggling...
        // Is there a way to convert my `bitmapData` into a `CanvasImageSource`?
    }

    /**
     * Overrides the browser's WebGL context with the headless-gl implementation.
     */
    protected makeGL(): WebGLRenderingContext {
        const glContext = gl(this.width, this.height);
        return glContext;
    }
}

However, I am struggling to find a way to successfully convert the Uint8Array data read from the headless-gl context into a CanvasImageSource object.

Here are some things that I have tried:

1. Return `this.gl.canvas`

This ends up being undefined in the case of headless-gl.

2. Use JSDOM

JSDOM's canvas does not support the WebGL render context.

3. Use JSDOM to create an HTMLImageElement and set the src to a base64 URL

I haven't quite figured out why, but the promise here does not ever resolve or reject. This results in my library timing out. So perhaps this strategy will work, but there is an issue with my implementation.

This strategy has been used in other areas of the library, but none that involve headless-gl or even WebGL. Just the 2D canvas context.

import gl from 'gl';
import { JSDOM } from 'jsdom';

export class MyHeadlessLibrary extends MyLibrary {
    /**
     * In this attempt, I changed the return type to Promise<CanvasImageSource> here, in MyLibrary, and in the client code.
     */
    public getCanvasImageSource(): Promise<CanvasImageSource> {
        // The canvas is `undefined` from headless-gl.
        // this.gl.canvas === undefined;

        // But, I can read the pixel data as a bitmap.
        const format = this.gl.RGBA;
        const type = this.gl.UNSIGNED_BYTE;
        const bitmapData = new Uint8Array(this.width * this.height * 4);
        this.gl.readPixels(0, 0, this.width, this.height, format, type, bitmapData);

        // Create a DOM and HTMLImageElement.
        const html: string = `<!DOCTYPE html><html><head><meta charset="utf-8" /><title>DOM</title></head><body></body></html>`;
        const dom = new JSDOM(html);
        const img = dom.window.document.createElement('img');

        // Create a base64 data URL
        const buffer = Buffer.from(bitmapData);
        const dataurl = `data:image/bmp;base64,${buffer.toString('base64')}`;

        // Set the image source and wrap the result in a promise
        return new Promise((resolve, reject) => {
            img.onerror = reject;
            img.onload = () => resolve(img);
            img.src = dataurl;
        });
    }
}

Please let me know if something in my code jumps out as a problem, or point me to a potential solution to this problem!

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…

1 Answer

深蓝 · Answer 1 · 2021-01-27T04:41:29+0000

According to the spec a CanvasImageSource is

typedef (HTMLOrSVGImageElement or
         HTMLVideoElement or
         HTMLCanvasElement or
         ImageBitmap or
         OffscreenCanvas) CanvasImageSource;

So it depends on what your needs are. If you don't need any alpha then one of those is HTMLCanvasElement so given pixels you could do

function pixelsToCanvas(pixels, width, height) {
  const canvas = document.createElement('canvas');
  canvas.width = width;
  canvas.height = height;
  const ctx = canvas.getContext('2d');
  const imgData = ctx.createImageData(width, height);
  imgData.data.set(pixels);
  ctx.putImageData(imgData, 0, 0);

  // flip the image
  ctx.scale(1, -1);
  ctx.globalCompositeOperation = 'copy';
  ctx.drawImage(canvas, 0, -height, width, height);

  return canvas;
}

This should work as long as either you have no alpha or you don't care about lossy alpha. Note: the code assumes you're providing unpremliplied alpha

The issue is pixels could have a pixel like this 255, 192, 128, 0. But because alpha is zero, when you pass that through the function above you'll get 0, 0, 0, 0 in the canvas that comes out because canvases always use premultiplied alpha. That may not be an issue because for most use cases 255, 192, 128, 0 appears as 0,0,0,0 anyway but if you have a special use case then this solution won't work.

Note: You'll need the canvas package

As for your Image from dataURL example this code makes no sense

// Create a base64 data URL
const buffer = Buffer.from(bitmapData);
const dataurl = `data:image/bmp;base64,${buffer.toString('base64')}`;

First off, whether image/bmp is supported or not is browser dependent so it's possible JSDOM doesn't support image/bmp but further a bmp file has a header which the code is not supplying. Without that header there is no way for any API to know what's in the data. If you gave it 256 bytes is that an 8x8 4byte per pixel image? A 16x4 4byte per pixel image? A black and white 32x64 1 bit per pixel image? etc.. You need the header.

Maybe writing the header will make that code work?

function convertPixelsToBMP(pixels, width, height) {
  const BYTES_PER_PIXEL = 4;
  const FILE_HEADER_SIZE = 14;
  const INFO_HEADER_SIZE = 40;

  const dst = new Uint8Array(FILE_HEADER_SIZE + INFO_HEADER_SIZE + width * height * 4);
  
  {
    const data = new DataView(dst.buffer);
    const fileSize = FILE_HEADER_SIZE + INFO_HEADER_SIZE + (width * height * 4);

    data.setUint8 ( 0, 0x42); // 'B'
    data.setUint8 ( 1, 0x4D); // 'M'
    data.setUint32( 2, fileSize, true)
    data.setUint8 (10, FILE_HEADER_SIZE + INFO_HEADER_SIZE);

    data.setUint32(14, INFO_HEADER_SIZE, true);
    data.setUint32(18, width, true);
    data.setUint32(22, height, true);
    data.setUint16(26, 1, true);
    data.setUint16(28, BYTES_PER_PIXEL * 8, true);
  }

  // bmp expects colors in BGRA format
  const pdst = new Uint8Array(dst.buffer, FILE_HEADER_SIZE + INFO_HEADER_SIZE);
  for (let i = 0; i < pixels.length; i += 4) {
    pdst[i    ] = pixels[i + 2];
    pdst[i + 1] = pixels[i + 1];
    pdst[i + 2] = pixels[i + 0];
    pdst[i + 3] = pixels[i + 3];
  }
  return dst;
}

note: this code also assumes you're providing unpremultiplied alpha.

Categories

node.js - How to convert a Uint8Array to a CanvasImageSource?

node.js - How to convert a Uint8Array to a CanvasImageSource?

More Context

1. Return `this.gl.canvas`

2. Use JSDOM

3. Use JSDOM to create an HTMLImageElement and set the src to a base64 URL

Please log in or register to add a comment.

Please log in or register to answer this question.

1 Answer

Please log in or register to add a comment.

Just Browsing Browsing

Most popular tags

Categories

node.js - How to convert a Uint8Array to a CanvasImageSource?

node.js - How to convert a Uint8Array to a CanvasImageSource?

More Context

1. Return this.gl.canvas

2. Use JSDOM

3. Use JSDOM to create an HTMLImageElement and set the src to a base64 URL

Please log in or register to add a comment.

Please log in or register to answer this question.

1 Answer

Please log in or register to add a comment.

Just Browsing Browsing

Most popular tags

1. Return `this.gl.canvas`