Posts Tagged ‘pixel shader’

Post-process shaders in glfx

Pushed an update to glfx to allow for post-process shading. When a post-process shader is defined, the scene is rendered to a screen-space quad (the size of the viewport), and that quad is then rendered to the viewport with the post-process shader applied.

The shader is loaded (asynchronously) like any other:

glfx.shaders.load('screenspace.fs', "frag-shader-screenspace", glfx.gl.FRAGMENT_SHADER);

Once loaded, we create the shader program, and get locations for whatever variables are used. The vertex shader isn’t anything special, it just transforms a vertex by the model-view and projection matrices, and passes along the texture coordinates.

glfx.whenAssetsLoaded(function() {

var postProcessShaderProgram = glfx.shaders.createProgram([glfx.shaders.buffer['vert-shader-basic'], glfx.shaders.buffer['frag-shader-screenspace']],
function(_shprog) {

// Setup variables for shader program
_shprog.vertexPositionAttribute = glfx.gl.getAttribLocation(_shprog, "aVertexPosition");
_shprog.pMatrixUniform = glfx.gl.getUniformLocation(_shprog,
"uPMatrix");
_shprog.mvMatrixUniform = glfx.gl.getUniformLocation(_shprog,
"uMVMatrix");
_shprog.textureCoordAttribute = glfx.gl.getAttribLocation(_shprog,
"aTextureCoord");

_shprog.uPeriod = glfx.gl.getUniformLocation(_shprog,
"uPeriod");
_shprog.uSceneWidth = glfx.gl.getUniformLocation(_shprog,
"uSceneWidth");
_shprog.uSceneHeight = glfx.gl.getUniformLocation(_shprog,
"uSceneHeight");

glfx.gl.enableVertexAttribArray(_shprog.vertexPositionAttribute);
glfx.gl.enableVertexAttribArray(_shprog.textureCoordAttribute);

});

...

We then tell glfx to apply our post-process shader program:

glfx.scene.setPostProcessShaderProgram(postProcessShaderProgram);

This call will result in different rendering path, which renders the scene to a texture, applies that texture to a screen-space quad, and renders the quad with the post-process shader.

Here is the shader for screenspace.fs, used in the demo shown above:

precision mediump float;

uniform float uPeriod;
uniform float uSceneWidth;
uniform float uSceneHeight;
uniform sampler2D uSampler;        
varying vec2 vTextureCoord;

void main(void) {

vec4 sum = vec4( 0. );
float blurSampleOffsetScale = 2.8;
float px = (1.0 / uSceneWidth) * blurSampleOffsetScale;
float py = (1.0 / uSceneHeight) * blurSampleOffsetScale;

vec4 src = texture2D( uSampler, ( vTextureCoord ) );

sum += texture2D( uSampler, ( vTextureCoord + vec2(-px, 0) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(-px, -py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(0, -py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(px, -py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(px, 0) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(px, py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(0, py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(-px, py) ) );
sum += src;

sum = sum / 9.0;

gl_FragColor = src + (sum * 2.5 * uPeriod);

}

Note that it requires a few uniforms to be supplied to it, we use the glfx.scene.onPostProcessPreDraw() callback to setup the variables (before the post-processed scene is drawn):

var timeAcc = 0;
glfx.scene.onPostProcessPreDraw =
function(tdelta) {

timeAcc += tdelta;
var timeScaled = timeAcc * 0.00107;

if(timeScaled > 2.0*Math.PI) {
timeScaled = 0;
timeAcc = 0;
}

var period = Math.cos(timeScaled);
glfx.gl.uniform1f(postProcessShaderProgram.uPeriod, period + 1.0);

glfx.gl.uniform1f(postProcessShaderProgram.uSceneWidth, glfx.gl.viewportWidth);
glfx.gl.uniform1f(postProcessShaderProgram.uSceneHeight, glfx.gl.viewportHeight);
};

What we’re doing is using the scene rendering time deltas to generate a periodic/sinusoidal wave. This results in the pulsing brightness/fading effect of the scene. The brightness effect itself is done by adding the source pixel to a blurred + brightened version of itself. The blurring allows for the soft fade in and fade out.

Circular stipple patterns on an HTML5 canvas

In my previous post on stipple patterns, I presented code to draw a few simple stipple patterns based on drawing single pixels at fixed locations. In this post, I’ll present something just a bit more complex: drawing circles to create a circular stipple patten, again writing a shader that makes use of the GraphicsCore and FXController classes.

Shader.circleStippleShader = function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{
var alpha = 1.0;
var r1 = r / 255.0;
var rF = Math.floor((alpha * r1 + (1.0 - alpha)) * 255.0);

var circleMaxDiam = 12; // circle at every 12th pixel, also defines max diameter of circle,
var circleMaxRadius = circleMaxDiam / 2; // maximum radius of circle

// figure out the x, y indices of the circle we're within
// x,y need to be shifted by the circle radius b/c circleMaxDiam defined the offset of the circle center
// ... e.g. going along the x-axis, we are within the next circle not at x/circleMaxDiam, but at (x+6)/circleMaxDiam
var iX = Math.floor((x + circleMaxRadius) / circleMaxDiam);
var iY = Math.floor((y + circleMaxRadius) / circleMaxDiam);

// multiply the circle indices by the diameter to get the actual coordinates of the circle's center
var targetX = iX * circleMaxDiam;
var targetY = iY * circleMaxDiam;

// calculate squared distance to the circle we are within
var dist = (targetX - x) * (targetX - x) + (targetY - y) * (targetY - y);

if (dist < 25) {
GraphicsCore.setPixel(bufWrite, index, rF, 0, 0, 255);
}
else {
GraphicsCore.setPixel(bufWrite, index, r, g, b, 255);
}

}
Shader.circleStippleShader.numPassesRequired = 1;    

Circle stipple

Conceptually, we define the center of a circle at every 12th pixel (both along the x and y axis). At every pixel (x,y) we figure out which circle we are within, and calculate the distance to the center. If the distance is less than our threshold (25), we change the color of the pixel (use the red channel only).

Stipple patterns on an HTML5 canvas

I wanted to play around a bit with stipple patterns after seeing stippling done with photos on the LinkedIn news feed. However, what I’m going to present is not what LinkedIn does. LinkedIn applies the stipple pattern as a background-image on a DOM element above a <img> element with a (fairly low resolution) JPEG – the stippling may help to alleviate the negative visual impact of the low-resolution image. What I’m going to show is how to do stippling on an HTML5 canvas, which allows for a much greater degree of freedom in terms of what’s possible, but is also slower and requires a modern browser.

I’m going to make use of the GraphicsCore and FXController classes in a previous post, Gaussian blur on an HTML5 canvas. In that post I presented the concept of writing shaders as plug-in to the FXController class to apply different per-pixel effects. What I’m going to present are shaders for a few simple stipple patterns. Applying the shader is simply a matter of passing it into the constructor for the FXController class, e.g.

var theShader = Shader.crossStippleShader;
var fxCtrlr = new FXController(ctxSource, ctxDest, theShader, width, height, 100, 1);
fxCtrlr.init();

Checkerboard Stipple

This shader has the effect of creating a checkerboard pattern.
The source pixel is preserved if (x+y) % 2 == 0, otherwise the pixel’s alpha is reduced to 66.

Shader.checkerboardStippleShader = function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{                
    
if( (x+y)%2 == 0) {
        GraphicsCore.setPixel(bufWrite, index, r, g, b, 255);
    }
    
else {
        GraphicsCore.setPixel(bufWrite, index, r, g, b, 66);
    }
}
Shader.checkerboardStippleShader.numPassesRequired = 1;    

Checkerboard Stipple

Dot Stipple

This shader blends a white pixel into the source image where x%2 == 0 && y%2 == 0, in effect creating a dotted grid pattern.

The alpha blending code is a straightforward implementation of alpha compositing, but since we’re blending with white (where r=1.0, g=1.0, and b=1.0) the equation is simplified and there is no second color value; we’re just biasing the source color by the alpha value. Also note that this is different than simply changing the alpha of the source pixel (as was done in the Checkerboard Stipple shader), here we’re always blending with white, in the previous shader we’re blending with whatever is the background of the DOM element.

Shader.dotStippleShader = function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{    
    
var alpha = 0.8;
        
    
var r1 = r / 255.0;
    
var rF = Math.floor((alpha*r1 + (1.0-alpha)) * 255.0);
            
    
var g1 = g / 255.0;
    
var gF = Math.floor((alpha*g1 + (1.0-alpha)) * 255.0);

    
var b1 = b / 255.0;
    
var bF = Math.floor((alpha*b1 + (1.0-alpha)) * 255.0);            
            
    
if( x%2 == 0 && y%2 == 0) {
        GraphicsCore.setPixel(bufWrite, index, rF, gF, bF, 255);
    }
else {
                
        GraphicsCore.setPixel(bufWrite, index, r, g, b, 255);
    }            

}
Shader.dotStippleShader.numPassesRequired = 1;

Dot Stipple

Quincunx Stipple

With this shader we blend in a white pixel at every 4 pixels (x%4 == 0 && y%4 == 0, the target pixel) and also at the 4 orthogonally adjacent pixels around the target, creating a quincunx pattern.

Shader.quincunxStippleShader = function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{    
    
var alpha = 0.78;
        
    
var r1 = r / 255.0;
    
var rF = Math.floor((alpha*r1 + (1.0-alpha)) * 255.0);
            
    
var g1 = g / 255.0;
    
var gF = Math.floor((alpha*g1 + (1.0-alpha)) * 255.0);

    
var b1 = b / 255.0;
    
var bF = Math.floor((alpha*b1 + (1.0-alpha)) * 255.0);            
            
    
if( (x%4 == 0 && y%4 == 0) ||
        ((x+1)%4 == 0 && y%4 == 0) ||
        ((x-1)%4 == 0 && y%4 == 0) ||
        (x%4 == 0 && (y+1)%4 == 0) ||
        (x%4 == 0 && (y-1)%4 == 0) )
    {
        GraphicsCore.setPixel(bufWrite, index, rF, gF, bF, 255);
    }
    
else
    
{
        GraphicsCore.setPixel(bufWrite, index, r, g, b, 255);
    }            

}
Shader.quincunxStippleShader.numPassesRequired = 1;

Quincunx Stipple

Cross Stipple

Similar to the quincunx stipple, but we blend in a white pixel at every 6 pixels (x%6 == 0 && y%6 == 0, the target pixel), the 4 orthogonally adjacent pixels around the target, and 4 additional pixels extending beyond the orthogonals, creating a cross (“+”) pattern.

Shader.crossStippleShader = function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{    
    
var alpha = 0.78;
        
    
var r1 = r / 255.0;
    
var rF = Math.floor((alpha*r1 + (1.0-alpha)) * 255.0);
            
    
var g1 = g / 255.0;
    
var gF = Math.floor((alpha*g1 + (1.0-alpha)) * 255.0);

    
var b1 = b / 255.0;
    
var bF = Math.floor((alpha*b1 + (1.0-alpha)) * 255.0);            
            
    
if( (x%6 == 0 && y%6 == 0) ||
        ((x+1)%6 == 0 && y%6 == 0) ||
        ((x-1)%6 == 0 && y%6 == 0) ||
        ((x+2)%6 == 0 && y%6 == 0) ||
        ((x-2)%6 == 0 && y%6 == 0) ||                
        (x%6 == 0 && (y+1)%6 == 0) ||
        (x%6 == 0 && (y-1)%6 == 0) ||
        (x%6 == 0 && (y+2)%6 == 0) ||
        (x%6 == 0 && (y-2)%6 == 0)                 
        )
                
    {
        GraphicsCore.setPixel(bufWrite, index, rF, gF, bF, 255);
    }
    
else
    
{
        GraphicsCore.setPixel(bufWrite, index, r, g, b, 255);
    }            

}
Shader.crossStippleShader.numPassesRequired = 1;    

Cross Stipple

That’s all for now. There’s tons of variations possible with only minor code changes to alter blending, color, and the shape of the stipple patten.

Gaussian blur on an HTML5 canvas

I’m actually going to present more than the actual gaussian blur implementation, also showing how to setup a simple animation controller and lightweight pixel shader system, allowing for defining the final color of a pixel on a per-pixel basis and allowing other effects to easily be plugged into the system. Be warned, this stuff is slow. This is all CPU processing (atop JavaScript no-less), there’s, sadly, no GPU hardware acceleration here. If your thinking about doing this on high-resolution images or writing effects that require a ton of passes over the image, you’re going bring the browser to a crawl, even on a fairly high-end system.

gaussian blur on HTML5 canvas element

I can’t embed JavaScript within this post, so you’ll have to go here to view the result (obviously, you’ll need an HTML 5 capable browser). For those who are curious, the very cool test image used is of a bird of paradise flower by the Agricultural Research Service.

So, first things first, the HTML, which is very simple. There’s 2 canvas element, one will hold the source image and the other will be the destination for the post-processed image. The width and height attributes on the canvas elements are set to the width and height of the image.

<!DOCTYPE html>
<html>
<head>
<title>HTML5 Blur FX</title>

<
script type="text/javascript">
// JS code will go here!
</
script>

</
head>
<body>
<canvas id="cvs-source" width="72" height="50">
</canvas>

<canvas id="cvs-dest" width="72" height="50">
</canvas>
</body>
</html>

Next, load the test image onto the source canvas (cvs-source) and setup the destination canvas (cvs-dest) as a blank image, which will occur when the page is loaded. Ignore the reference to the FXController object for now.

window.onload = function ()
{
var img = new Image();
img.onload =
function ()
{
// setup source
var ctxSource = document.getElementById('cvs-source').getContext('2d');
ctxSource.drawImage(img, 0, 0);

// setup destination
var cvsElement = document.getElementById('cvs-dest');
var ctxDest = cvsElement.getContext('2d');

var width = parseInt(cvsElement.getAttribute("width"));
var height = parseInt(cvsElement.getAttribute("height"));

ctxDest.createImageData(width, height);

var theShader = Shader.gaussBlur;
var fxCtrlr = new FXController(ctxSource, ctxDest, theShader, width, height, 10, 10);
fxCtrlr.init();
}

img.src =
'test3.png';
}

Stepping away from the actual code for a minute, it’s important to note how to actually modify the pixels on a canvas element:

  • Get the 2d context of the element by calling getContext(‘2d’) on the DOM element.
  • Call CanvasRenderingContext2D.getImageData(…) to get a buffer with the pixels in RGBA format.
  • To commit changes to the pixels onto a canvas, call CanvasRenderingContext2D.putImageData(…) with the buffer of modified pixels.
var ctxSource = document.getElementById('cvs-source').getContext('2d');
var imageData = ctxSource.getImageData(0, 0, width, height);


ctxDest.putImageData(bufWrite, 0, 0);

Back to the actual code. One of the very simple and primitive operations needed is to set a pixel to a color:

// GraphicsCore object
var GraphicsCore = {};
GraphicsCore.setPixel =
function (imageData, index, r, g, b, a)
{
imageData.data[index + 0] = r;
imageData.data[index + 1] = g;
imageData.data[index + 2] = b;
imageData.data[index + 3] = a;
}

I didn’t implement a corresponding getPixel() function because, as you’ll see, it’s very clean and easy to get the a pixel directly from the buffer and wasn’t worth invoking a function call.

The FXController object is (for the most part) the animation controller.

// FXController object
function FXController(_ctxSource, _ctxDest, _theShader, _width, _height, _fps, _maxFrames)
{
this.ctxSource = _ctxSource;
this.ctxDest = _ctxDest;
this.theShader = _theShader;
this.width = _width;
this.height = _height;
this.fps = _fps;
this.curFrame = 1; // [1, ...]
this.maxFrames = _maxFrames;
this.numPassesPerFrame = _theShader.numPassesRequired;
this.invervalPtr = null;

this.shaderFunc = function (fxCtrlr, passNum, frameNum, maxFrames)
{
Shader.run(fxCtrlr.ctxSource, fxCtrlr.ctxDest, fxCtrlr.width, fxCtrlr.height, fxCtrlr.theShader, passNum, frameNum, maxFrames);
}

this.init = function ()
{
var fxCtrlr = this;
var runFunc = function () { fxCtrlr.run(fxCtrlr); }

this.invervalPtr = setInterval(runFunc, 1000.0 / this.fps);
}

this.unInit = function()
{
clearInterval(
this.invervalPtr);
this.invervalPtr = null;
}

this.run = function (sender /*FXController*/)
{
for (var pn = 1; pn <= sender.numPassesPerFrame; pn++) {
sender.shaderFunc(sender, pn, sender.curFrame, sender.maxFrames);
}

sender.curFrame++;
if (sender.curFrame > sender.maxFrames) {
sender.unInit();
}
}
}

Most of what going on here is simply holding values which are passed to Shader.run(…). However, a few important things are being setup:

  • FXController.run(…) will be called at a certain number of frames per seconds (this.fps), until this.maxFrames is hit.
  • For each frame, Shader.run(…) will be called for each pass necessary (this.numPassesPerFrame). Certain effects will require more passes than other, for example, the gaussian blur implementation will require 2 passes.

The Shader object, the core of which is within Shader.run(…),

// Shader object
// Note: Shader.<shader_name>.numPassesRequired must be defined
var Shader = {};
Shader.run =
function (ctxSource, ctxDest, width, height, shaderFunc, passNum, frameNum, maxFrames)
{
//
// netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead"); // REMOVE ME BEFORE DEPLOYMENT
//

var bufWrite = ctxDest.getImageData(0, 0, width, height);

var imageData = null;
if (passNum == 1 && frameNum == 1) {
imageData = ctxSource.getImageData(0, 0, width, height);
}
else {
imageData = ctxDest.getImageData(0, 0, width, height);
}

for (var y = 0; y < height; y++) {

for (var x = 0; x < width; x++) {

var index = (x + y * imageData.width) << 2;
shaderFunc(imageData, bufWrite, index, x, y, imageData.data[index + 0], imageData.data[index + 1], imageData.data[index + 2], imageData.data[index + 3], passNum, frameNum, maxFrames);
}
}

ctxDest.putImageData(bufWrite, 0, 0);
}

(The UniversalBrowserRead privilege is necessary to run this locally in Firefox)

Shader.run(…) will setup the source and destination buffers, iterate over every pixel, compute the pixel index, get the pixel color, call shaderFunc(…) with all the necessary params, and finally commit any changes to the destination buffer (bufWrite). Note, the source canvas is only used for the first frame, first pass; in all other cases whatever is rendered on the destination canvas is used. This allows for certain effects (such as the gaussian blur) in which the effect can be progressively applied again and again, in a feedback loop, to produce updated iterations of the effect (in the case of a gaussian blur, the image is blurred more and more).

The code presented so far lays the framework to allow for writing a shader, plugging it into the system, and watching the result. Before, getting to the more complex gaussian blur filter, here’s a much simpler one: a single-pass, per-pixel image fade-in.

// fade in shader
Shader.fadeInShader =
function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{
var dt = frameNum/maxFrames; // [0, 1]
GraphicsCore.setPixel(bufWrite, index, r, g, b, dt * 255);
}
Shader.fadeInShader.numPassesRequired = 1;

(Shader..numPassesRequired is required for every shader, as FXController will query the value to determine how many times to call Shader.run(…) per frame.)

The shader function allows us to define what the final color of a pixel will be, given a set of input parameters, on a pixel-by-pixel basis, the core of what a pixel shader system is and allowing for a amazing degree of flexibility.

Finally, the gaussian blur shader. I won’t go into too many details here. If you’re interested in how a guassian blur is actually done, this article on gamedev.net is probably the best out there (esp. for transitioning from theory to practice), and the code here is almost a direct translation of what’s up there. Also note that bitshifts are used to do the power-of-2 divisions and multiplications.

// gaussian blur filter
Shader.gaussFact = Array(1, 6, 15, 20, 15, 6, 1);
Shader.gaussSum = 64;
// not used, >> 6 bitshift used in Shader.gaussBlur()
Shader.gaussWidth = 7;

Shader.gaussBlur =
function (imageData, bufWrite, index, x, y, r, g, b, a, passNum, frameNum, maxFrames)
{
if (passNum == 1 && (x <= 0 || x >= imageData.width - 1)) {
GraphicsCore.setPixel(bufWrite, index, r, g, b, a);
return;
}

if (passNum == 2 && (y <= 0 || y >= imageData.height - 1)) {
GraphicsCore.setPixel(bufWrite, index, r, g, b, a);
return;
}

var readBuf = imageData;
var writeBuf = bufWrite;

var sumR = 0;
var sumG = 0;
var sumB = 0;
var sumA = 0;

for (var k = 0; k < Shader.gaussWidth; k++) {

var nx = x;
var ny = y;

if (passNum == 1) { nx = (x - ((Shader.gaussWidth - 1) >> 1) + k); }
else if (passNum == 2) { ny = (y - ((Shader.gaussWidth - 1) >> 1) + k); }
else { }

// wrap around if we're trying to read pixels beyond the edge
if (nx < 0) { nx = readBuf.width + nx; }
if (ny < 0) { ny = readBuf.height + ny; }
if (nx >= readBuf.width) { nx = nx - readBuf.width; }
if (ny >= readBuf.height) { ny = ny - readBuf.height; }

var pxi = (nx + ny * readBuf.width) << 2;
var pxR = readBuf.data[pxi];
var pxG = readBuf.data[pxi + 1];
var pxB = readBuf.data[pxi + 2];
var pxA = readBuf.data[pxi + 3];

// little hack to make alpha=0 pixels look a bit better
// Note, the proper way to handle the alpha channel is to premultiply, blur, "unpremultiply"
if (pxA == 0) {
pxR = 255;
pxG = 255;
pxB = 255;
pxA = 255;
}

sumR += pxR * Shader.gaussFact[k];
sumG += pxG * Shader.gaussFact[k];
sumB += pxB * Shader.gaussFact[k];
sumA += pxA * Shader.gaussFact[k];
}

GraphicsCore.setPixel(writeBuf, index, sumR >> 6, sumG >> 6, sumB >> 6, sumA >> 6);
}
Shader.gaussBlur.numPassesRequired = 2;

The blur is done with a 3×3 convolution filter, over 2 passes. In the first pass, neighboring pixels are sampled and blurred along the x-axis. In the second pass, the same is done along the y-axis.

A few simple conditionals allow for wrapping around and sampling from the other side of the bitmap, if there’s an attempt to sample beyond the edges.

Note the little hack for transparent/translucent pixels; this is not the proper way to do this (and simply makes the error more grey-ish instead of black-ish), but I didn’t want to deal with premultiplying the alpha, so I’ve left it out.

The demo + all code is up @ http://aautar.digital-radiation.com/HTML5-BlurFX/