Posts Tagged ‘computer graphics’

Pushing computation to the front: thumbnail generation

Frontend possibilities

As the APIs brought forward by HTML5 about a decade ago have matured and the devices running web browsers have continued to improve in computational power, looking at what’s possible on the frontend and the ability to bring backend computations to the frontend has been increasingly interesting to me. Such architectures would see each user’s browsers as a worker for certain tasks and could simply backend systems, as those tasks are pushed forward to the client. Using Canvas for image processing tasks is one area that interesting and that I’ve had success with.

For Mural, I did the following Medium-esque image preload effect, the basis of which is generating a tiny (16×16) thumbnail which is loaded with the page. That thumbnail is blurred via CSS filter, and transitions to the full-resolution image once it’s loaded. The thumbnail itself is generated entirely on the frontend when a card is created and saved alongside the card data.

In this post, I’ll run though generating and handling that 16×16 thumbnail. This is fairly straightforward use of the Canvas API, but it does highlight how frontend clients can be utilized for operations typically relegated to server-side systems.

The image processing code presented is encapsulated in the canvas-image-transformer library.

<img> → <canvas>

A precursor for any sort of image processing is getting the image data into a <canvas>. The <img> element and corresponding HTMLImageElement interface don’t provide any sort of pixel-level read/write functionality, whereas the <canvas> element and corresponding HTMLCanvasElement interface does. This transformation is pretty straightforward:

The code is as follows (an interesting thing to note here is that this can all be done without injecting anything into the DOM or rendering anything onto the screen):

const img = new Image(); img.onload = function() { const canvas = document.createElement('canvas'); canvas.width = img.width; canvas.height = img.height; const canvasCtx = canvas.getContext('2d'); canvasCtx.drawImage(img, 0, 0, img.width, img.width); // the image has now been rendered onto canvas } img.src = "https://some-image-url";

Resizing an image

Resizing is trivial, as it can be handled directly via arguments to CanvasRenderingContext2D.drawImage(). Adding in a bit of math to do proportional scaling (i.e. preserve aspect ratio), we can wrap the transformation logic into the following method:

/** * * @param {HTMLImageElement} img * @param {Number} newWidth * @param {Number} newHeight * @param {Boolean} proportionalScale * @returns {Canvas} */ imageToCanvas: function(img, newWidth, newHeight, proportionalScale) { if(proportionalScale) { if(img.width > img.height) { newHeight = newHeight * (img.height / img.width); } else if(img.height > img.width) { newWidth = newWidth * (img.width / img.height); } else {} } var canvas = document.createElement('canvas'); canvas.width = newWidth; canvas.height = newHeight; var canvasCtx = canvas.getContext('2d'); canvasCtx.drawImage(img, 0, 0, newWidth, newHeight); return canvas; }

Getting the transformed image from the canvas

My goto method for getting the data off a canvas and into a more interoperable form is to use the HTMLCanvasElement.toDataURL() method, which allows easily getting the image as a PNG or JPEG. I do have mixed feeling about data-URIs; they’re great for the web, because so much of the web is textually based, but they’re also horribly bloated and inefficient. In any case, I think interoperability and ease-of-use usually wins out (esp. here where we’re dealing with a 16×16 thumbnail and the data-uri is relatively lightweight) and getting a data-uri is generally the best solution.

Using CanvasRenderingContext2D.getImageData() to get the raw pixel from a canvas is also an option but, for a lot of use-cases, you’d likely need to compress and/or package the data in some way to make use of it.

Save the transformed image

With a data-uri, saving the image is pretty straightforward. Send it to the server via some HTTP method (POST, PUT, etc.) and save it. For a 16×16 PNG the data-uri textual representation is small enough that we can put it directly in a relational database and not worry about a conversion to binary.

Alternatives & limitations

The status quo alternative is having this sort of image manipulation logic encapsulated within some backend component (method, microservice, etc.) and, to be fair, such systems work well. There’s also some very concrete benefits:

  • You are aware of and have control over the environment in which the image processing is done, so you’re isolated from browser quirks or issues stemming from a user’s computing environment.
  • You have an easier path for any sort of backfill (e.g. how do you generate thumbnails for images previously uploaded?) or migration needs (e.g. how can you move to a different sized thumbnail?); you can’t just run though rows in a database and make a call to get what you need.

However, something worth looking at is that backend systems and server-side environments are typically not optimized for any sort of graphics workload, as processing is centered around CPU cores. In contrast, the majority of frontend environments have access to a GPU, even fairly cheap phone have some sort of GPU that is better suited for “embarassing parallel”-esque graphics operations, the performance benefits of which you get for free with the Canvas API in all modern browsers.

In Chrome, see the output of chrome://gpu:

chrome settings, canvas hardware acceleration

Scale, complexity and cost also come into play. Thinking of frontend clients as computational nodes can change the architecture of systems. The need for server-side resources (hardware, VMs, containers, etc.) is eliminated. Scaling concerns are also, to a large extent, eliminated or radically changed as operations are pushed forward to the client.

Future work

What’s presented here is just scratching the surface of what’s possible with Canvas. WebGL also presents as a ton of possibilities and abstraction layers like gpu.js are really interesting. Overall, it’s exciting to see the web frontend evolve beyond a mechanism for user input and into a layer in which substantive computation can be done.

Using feColorMatrix to dynamically recolor icons (part 2, two-color icons)

Previously, I looked at how to use feColorMatrix to dynamically change the color of single-color icons. In this post, we’ll look at using feColorMatrix, feBlend, and feComposite to implement an algorithm that allow for dynamically changing the the colors of an icon with 2 colors.

The Algorithm

With a single color, we could think of the process as being a single step: applying the color transformation matrix with the desired R, G, B values. For two colors, there are multiple steps and matrices involved, so it’s worth having a high-level overview and conceptual understanding of the algorithm before delving into the details.

  • The input icon will have 2 color, black and white; black areas will be changed to colorA and white areas will be changed to colorB
  • Add colorA to the source image (black areas will take on the new color, white areas will remain white), the result is imageA
  • Invert the source image, then add colorB to it (black areas will become white and remain white, white areas will become black and take on the new color), the result is imageB
  • Combine imageA and imageB, such that the alpha component from the source image is preserved, output the result
Dynamically recolor two-color icon

Note that from the above, we see the key operations that are needed:

  • Add
  • Invert
  • Combine

Another look at the color transformation matrix for applying a single-color

Note that the transformation matrix used previously for the single-color case, only preserves the alpha component from the input. The R, G, and B components are thrown away:

[[0, 0, 0, 0, R], [0, 0, 0, 0, G], [0, 0, 0, 0, B], [0, 0, 0, 1, 0]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(R), (G), (B), (A_(src))]

It doesn’t matter what color the input pixel is, applying the transformation matrix will result in those Rsrc Gsrc Bsrc input values being multiplied by zero, before the new/output R, G, B values are added in.

While this is fine for the single-color case, for the two-color algorithm to work, the distinction between the black areas and the white areas need to be kept intact, so we have to work with the Rsrc Gsrc Bsrc values from the input vector and preserve the distinction.

Making Rsrc Gsrc Bsrc part of the transformation

Modifying the transformation matrix to allow Rsrc Gsrc Bsrc to be part of the calculations requires the first 3 diagonal elements of the matrix to be non-zero.

The simplest case of this is the identity matrix:

[[1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0], [0, 0, 0, 1, 0]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(R_(src)), (G_(src)), (B_(src)), (A_(src))]

Let’s look at a few matrices that define transformations needed for the algorithm.

The add colorK matrix:

[[1, 0, 0, 0, R_(k)], [0, 1, 0, 0, G_(k)], [0, 0, 1, 0, B_(k)], [0, 0, 0, 1, 0]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(R_(src) + R_(k)), (G_(src) + G_(k)), (B_(src) + B_(k)), (A_(src))]

The invert matrix:

[[-1, 0, 0, 0, 1], [0, -1, 0, 0, 1], [0, 0, -1, 0, 1], [0, 0, 0, 1, 0]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(-R_(src) + 1), (-G_(src) + 1), (-B_(src) + 1), (A_(src))]

The above can be combined into a single transformation matrix, the invert & add colorK matrix:

[[-1, 0, 0, 0, 1 + R_(k)], [0, -1, 0, 0, 1 + G_(k)], [0, 0, -1, 0, 1 + B_(k)], [0, 0, 0, 1, 0]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(-R_(src) + 1 + R_(k)), (-G_(src) + 1 + G_(k)), (-B_(src) + 1 + B_(k)), (A_(src))]

Putting aside and referencing intermediate results

Note that the algorithm requires us to create 2 independent images (imageA and imageB) and then subsequently combine them.

This can be accomplished by utilizing the result attribute available on SVG filter primitive elements. By specifying a result identifier we are able to apply a transformation and put aside the resultant image.

Combining the intermediate results

Combining intermediate results/images can’t be done with feColorMatrix, it’s simply not an operation that can be constructed as a transformation. To handle this, SVG provides the feBlend element with a number of predefined operations available via the mode attribute. Based on the color transformations done to create imageA and imageB (mainly that the areas that are not of concern, the background, are set to white [1,1,1]), the multiply operation will work to combine the images (not perfectly, there’s one big problem with the alpha channel, but we’ll deal with that in a bit).

image multiply
<feBlend 
    color-interpolation-filters="linearRGB"
    in="imageA" 
    in2="imageB" 
    mode="multiply" 
    result="output" />        

The alpha channel problem

While feBlend seem to accomplish what’s needed, for anything other than a white background you’ll notice a white-ist outline around elements in the image, as you can see below.

alpha channel problems with feBlend

The problem is that for what we’re trying to accomplish we just want to simply preserve the alpha channel, not blend it in any way. However, all feBlend operations will perform a blend of the alpha channel. From the SVG spec:

image multiply

So there’s no way we can really work with feBlend to get a solution for the alpha channel, but we do have an idea of what the solution is: copy the alpha channel from the source image.

Fixing the alpha channel

Fixing the alpha channel will involve 2 steps:

  • For the image outputted by feBlend, get the alpha value to be 1 for every pixel
    (the reason this step will become apparent once we look at how feComposite has to be used)
  • Use feComposite to construct an image with the alpha values from the source image and the R,G,B values from the image outputted by feBlend

The first step is simple. We just need a slight modification to the 2 color transformation matrices used, such that one or both set the alpha channel to 1 for every pixel (note from the alpha blending equation, this will effectively set the alpha value to 1 for every pixel in the output). The modified matrices are shown below and it’s a good point to start showing some code.

The full-alpha add colorK matrix:

[[1, 0, 0, 0, R_(k)], [0, 1, 0, 0, G_(k)], [0, 0, 1, 0, B_(k)], [0, 0, 0, 0, 1]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(R_(src) + R_(k)), (G_(src) + G_(k)), (B_(src) + B_(k)), (1) ]

<feColorMatrix in="SourceGraphic" type="matrix" result="imageA"
    values="1 0 0 0 0
            0 1 0 0 0.68235294117
            0 0 1 0 0.93725490196
            0 0 0 0 1" /> 

The full-alpha invert & add colorK matrix:

[[-1, 0, 0, 0, 1 + R_(k)], [0, -1, 0, 0, 1 + G_(k)], [0, 0, -1, 0, 1 + B_(k)], [0, 0, 0, 0, 1]][(R_(src)), (G_(src)), (B_(src)), (A_(src))] = [(-R_(src) + 1 + R_(k)), (-G_(src) + 1 + G_(k)), (-B_(src) + 1 + B_(k)), (1)]

<feColorMatrix in="SourceGraphic" type="matrix" result="imageA"
    values="-1 0 0 0 1.0784313725
            0 -1 0 0 1.7058823529
            0 0 -1 0 1.2431372549
            0 0 0 0 1" />

For the next and final step, we need to take a look at feComposite.

The feComposite filter primitive allows for more fine-grained operations on pixels using the supported “arithmetic” operation:

feComposite arithmetic operation

As this operation is done on every channel, including the alpha channel, we have way to control what happens to the alpha pixels in a composition of 2 images.

We’re going to make use of this by:

  • Using an feColorMatrix, where the input in the source image, and transforming such that R,G,B is white [1,1,1] and the alpha remains unchanged
  • Using feComposite to do a simple arithmetic multiply (k1=1, k2=0, k3=0, k4=0), between the image constructed above (where R,G,B is [1,1,1] and alpha is the alpha from the source image) and the output from feBlend (where the R,G,B = the values we want for the output and alpha = 1)

Effectively, the source alpha is multiplied by 1 (as the image produced from the feBlend operation has the alpha set to 1 for all pixels) and the R,G,B values from the feBlend output are multiplied by 1 (as the constructed image, in the first step above, sets R,G,B to 1 for every pixel).

<!-- Get and use alpha from source image -->
<feColorMatrix in="SourceGraphic" type="matrix" result="whiteAlpha"
    values="0 0 0 0 1
            0 0 0 0 1
            0 0 0 0 1
            0 0 0 1 0" />     

<feComposite in="whiteAlpha" in2="outputFullAlpha" operator="arithmetic" k1="1" k2="0" k3="0" k4="0" />

Pulling everything together

We now have all the pieces for the filter and here’s what the code looks like:

<svg style="width:0; height:0; margin:0; padding:0; border:none;">
    <filter color-interpolation-filters="sRGB" id="colorTransformFilter">

        <feColorMatrix in="SourceGraphic" type="matrix" result="imageA"
            values="1 0 0 0 0
                    0 1 0 0 0.68235294117
                    0 0 1 0 0.93725490196
                    0 0 0 0 1" /> 

        <feColorMatrix in="SourceGraphic" type="matrix" result="imageB"
            values="-1 0 0 0 1.0784313725
                    0 -1 0 0 1.7058823529
                    0 0 -1 0 1.2431372549
                    0 0 0 0 1" />              
                    
        <feBlend 
            color-interpolation-filters="linearRGB"
            in="imageA" 
            in2="imageB" 
            mode="multiply" 
            result="outputFullAlpha" />        

        <!-- Get and use alpha from source image -->
        <feColorMatrix in="SourceGraphic" type="matrix" result="whiteAlpha"
            values="0 0 0 0 1
                    0 0 0 0 1
                    0 0 0 0 1
                    0 0 0 1 0" />     

        <feComposite in="whiteAlpha" in2="outputFullAlpha" operator="arithmetic" k1="1" k2="0" k3="0" k4="0" />

    </filter>
</svg>

The demo below uses the filter, along with a bit of Javascript to cycle and update the input colors (i.e. dynamically updating the values plugged into the first 2 feColorMatrix elements):

Limitations

We have the same limitations I mentioned in part 1:

  • For icons applied as background-images, the CSS filter property isn’t ideal. CSS filter will effect not only the element it’s applied to, but all child elements as well
  • As is the case with mixing the CSS filter property and the SVG filter element, effects governed by the CSS transition property won’t work

In addition, because of how the alpha channel in treated in regards to feBlend (setting all pixels to have alpha=1), you more than likely won’t get good results if the icon has different-colored adjoining or overlapping elements, as you won’t get a smooth transition at the edges/boundaries.

Using feColorMatrix to dynamically recolor icons (part 1, single-color icons)

I’ve been experimenting with using feColorMatrix as an elegant way to dynamically color/re-color SVG icons. Here I’ll look at working with single-color icons.

Working directly with the SVG markup

Changing the stroke and/or fill colors of the SVG elements directly can be a good solution in many cases, but it requires:

  • Placing the SVG markup into the document to query and modify the appropriate elements when a color update is needed (note that this option isn’t viable if you need to place the icon in an <img> tag or it needs to be placed as a background-image on an element, as you can’t reference the SVG element in such cases)
  • Treating the SVG markup as a templated, Javascript, string to make a data-URI, and re-making it when a color update is needed

By using the color transformation matrix provided by feColorMatrix these restrictions go away and we also get back the flexibility of using external files.

Icon color

Keep in mind, we’re only dealing with single-color icons. What color is used doesn’t technically matter, but black is a nice basis and in an actual project, black is beneficial, as you’re able to open your the icon files in an editor or browser and actually see it.

black colored icon

A black pixel within the icon can then be represented by the following vector:

black pixel as column vector

Note that the alpha component may vary due to antialiasing (to smooth out edges) or some translucency within the icon.

The color transformation matrix

feColorMatrix allows you to define a 5×4 transformation matrix. There’s a lot you can do with that, but note that the last column of the matrix is essentially an additive component for each channel (see matrix-vector multiplication), so in that column we enter the desired R, G, B values from top to bottom, which will be added to the zeros in the input vector. Next, we want to preserve the alpha component from the input vector, so the fourth column of the matrix becomes [0, 0, 0, 1]T and the fourth row of the last column is zero, as we don’t want to add anything to the alpha component.

color transformation matrix, [[0, 0, 0, 0, R], [0, 0, 0, 0, G], [0, 0, 0, 0, B], [0, 0, 0, 1, 0]][(0), (0), (0), (A_(src))] = [(R), (G), (B), (A_(src))]

The matrix-vector multiplication gives a new vector (that defines the output pixel) with the entered R, G, B values and the alpha value from the source pixel.

Representing the matrix within an feColorMatrix element is straightforward…

<feColorMatrix in="SourceGraphic" type="matrix"
values="0 0 0 0 R
0 0 0 0 G
0 0 0 0 B
0 0 0 1 0"
/>

… just plug in values for R, G, B.

Applying the color transformation

The color transformation matrix can be applied to an element by wrapping it in an SVG filter element and referencing the filter via the CSS filter property.

With Javascript, the values attribute of the feColorMatrix element can be updated dynamically. The color change will, in turn, be reflected in any elements referencing the SVG filter.

<!DOCTYPE html>
<html>
<body>
<!--
The values of the color matrix defines the color transformation what will be applied.
Here we just setup the elements and define an identity matrix. We'll modify the matrix via Javascript code
to define an actual color transformation.
-->
<svg style="width:0; height:0; margin:0; padding:0; border:none;">
<filter
color-interpolation-filters="sRGB" id="colorTransformFilter">
<feColorMatrix
in="SourceGraphic" type="matrix"
values="1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0"
/>

</filter>
</svg>

<!--
Element with an SVG icon that we want to colorize
Note: that the color transformation is applied to everything not only to the background, but everything
within the element as well.

Typical solution to to isolate background stuff to it's own div and use another div for contents

-->
<div id="logo-colored"
style="
width:300px;
height:300px;
background-color: transparent;
background-image: url(logo.svg);
background-position: center center;
background-repeat: no-repeat;
background-size: cover;
filter:url(#colorTransformFilter);"
>

<p
style="color:#fff">Testing 1 2 3...</p>
</div>

<script
type="text/javascript">
/**
* A little helper function to update the color transformation matrix
*
* @param {Number} _r
* @param {Number} _g
* @param {Number} _b
*/
const setPrimaryColor = function(_r, _g, _b) {

const rScaled = _r / 255.0