Pushing computation to the front: thumbnail generation
Jul 4 2020 · Web Technologies
Frontend possibilities
As the APIs brought forward by HTML5 about a decade ago have matured and the devices running web browsers have continued to improve in computational power, looking at what’s possible on the frontend and the ability to bring backend computations to the frontend has been increasingly interesting to me. Such architectures would see each user’s browsers as a worker for certain tasks and could simply backend systems, as those tasks are pushed forward to the client. Using Canvas for image processing tasks is one area that interesting and that I’ve had success with.
For Mural, I did the following Medium-esque image preload effect, the basis of which is generating a tiny (16×16) thumbnail which is loaded with the page. That thumbnail is blurred via CSS filter, and transitions to the full-resolution image once it’s loaded. The thumbnail itself is generated entirely on the frontend when a card is created and saved alongside the card data.
In this post, I’ll run though generating and handling that 16×16 thumbnail. This is fairly straightforward use of the Canvas API, but it does highlight how frontend clients can be utilized for operations typically relegated to server-side systems.
The image processing code presented is encapsulated in the canvas-image-transformer library.
<img> → <canvas>
A precursor for any sort of image processing is getting the image data into a <canvas>. The <img> element and corresponding HTMLImageElement interface don’t provide any sort of pixel-level read/write functionality, whereas the <canvas> element and corresponding HTMLCanvasElement interface does. This transformation is pretty straightforward:
- Use the Image constructor to load an image or reference an
<img>element from the DOM - Create a canvas with the same width and height as the image
- Get the canvas context and use CanvasRenderingContext2D.drawImage() to draw the image onto the canvas
The code is as follows (an interesting thing to note here is that this can all be done without injecting anything into the DOM or rendering anything onto the screen):
const img = new Image();
img.onload = function() {
const canvas = document.createElement('canvas');
canvas.width = img.width;
canvas.height = img.height;
const canvasCtx = canvas.getContext('2d');
canvasCtx.drawImage(img, 0, 0, img.width, img.width);
// the image has now been rendered onto canvas
}
img.src = "https://some-image-url";
Resizing an image
Resizing is trivial, as it can be handled directly via arguments to CanvasRenderingContext2D.drawImage(). Adding in a bit of math to do proportional scaling (i.e. preserve aspect ratio), we can wrap the transformation logic into the following method:
/**
*
* @param {HTMLImageElement} img
* @param {Number} newWidth
* @param {Number} newHeight
* @param {Boolean} proportionalScale
* @returns {Canvas}
*/
imageToCanvas: function(img, newWidth, newHeight, proportionalScale) {
if(proportionalScale) {
if(img.width > img.height) {
newHeight = newHeight * (img.height / img.width);
}
else if(img.height > img.width) {
newWidth = newWidth * (img.width / img.height);
}
else
{}
}
var canvas = document.createElement('canvas');
canvas.width = newWidth;
canvas.height = newHeight;
var canvasCtx = canvas.getContext('2d');
canvasCtx.drawImage(img, 0, 0, newWidth, newHeight);
return canvas;
}
Getting the transformed image from the canvas
My goto method for getting the data off a canvas and into a more interoperable form is to use the HTMLCanvasElement.toDataURL() method, which allows easily getting the image as a PNG or JPEG. I do have mixed feeling about data-URIs; they’re great for the web, because so much of the web is textually based, but they’re also horribly bloated and inefficient. In any case, I think interoperability and ease-of-use usually wins out (esp. here where we’re dealing with a 16×16 thumbnail and the data-uri is relatively lightweight) and getting a data-uri is generally the best solution.
Using CanvasRenderingContext2D.getImageData() to get the raw pixel from a canvas is also an option but, for a lot of use-cases, you’d likely need to compress and/or package the data in some way to make use of it.
Save the transformed image
With a data-uri, saving the image is pretty straightforward. Send it to the server via some HTTP method (POST, PUT, etc.) and save it. For a 16×16 PNG the data-uri textual representation is small enough that we can put it directly in a relational database and not worry about a conversion to binary.
Alternatives & limitations
The status quo alternative is having this sort of image manipulation logic encapsulated within some backend component (method, microservice, etc.) and, to be fair, such systems work well. There’s also some very concrete benefits:
- You are aware of and have control over the environment in which the image processing is done, so you’re isolated from browser quirks or issues stemming from a user’s computing environment.
- You have an easier path for any sort of backfill (e.g. how do you generate thumbnails for images previously uploaded?) or migration needs (e.g. how can you move to a different sized thumbnail?); you can’t just run though rows in a database and make a call to get what you need.
However, something worth looking at is that backend systems and server-side environments are typically not optimized for any sort of graphics workload, as processing is centered around CPU cores. In contrast, the majority of frontend environments have access to a GPU, even fairly cheap phone have some sort of GPU that is better suited for “embarassing parallel”-esque graphics operations, the performance benefits of which you get for free with the Canvas API in all modern browsers.
In Chrome, see the output of chrome://gpu:
Scale, complexity and cost also come into play. Thinking of frontend clients as computational nodes can change the architecture of systems. The need for server-side resources (hardware, VMs, containers, etc.) is eliminated. Scaling concerns are also, to a large extent, eliminated or radically changed as operations are pushed forward to the client.
Future work
What’s presented here is just scratching the surface of what’s possible with Canvas. WebGL also presents as a ton of possibilities and abstraction layers like gpu.js are really interesting. Overall, it’s exciting to see the web frontend evolve beyond a mechanism for user input and into a layer in which substantive computation can be done.
