Post-process shaders in glfx

Pushed an update to glfx to allow for post-process shading. When a post-process shader is defined, the scene is rendered to a screen-space quad (the size of the viewport), and that quad is then rendered to the viewport with the post-process shader applied.

The shader is loaded (asynchronously) like any other:

glfx.shaders.load('screenspace.fs', "frag-shader-screenspace", glfx.gl.FRAGMENT_SHADER);

Once loaded, we create the shader program, and get locations for whatever variables are used. The vertex shader isn’t anything special, it just transforms a vertex by the model-view and projection matrices, and passes along the texture coordinates.

glfx.whenAssetsLoaded(function() {

var postProcessShaderProgram = glfx.shaders.createProgram([glfx.shaders.buffer['vert-shader-basic'], glfx.shaders.buffer['frag-shader-screenspace']],
function(_shprog) {

// Setup variables for shader program
_shprog.vertexPositionAttribute = glfx.gl.getAttribLocation(_shprog, "aVertexPosition");
_shprog.pMatrixUniform = glfx.gl.getUniformLocation(_shprog,
"uPMatrix");
_shprog.mvMatrixUniform = glfx.gl.getUniformLocation(_shprog,
"uMVMatrix");
_shprog.textureCoordAttribute = glfx.gl.getAttribLocation(_shprog,
"aTextureCoord");

_shprog.uPeriod = glfx.gl.getUniformLocation(_shprog,
"uPeriod");
_shprog.uSceneWidth = glfx.gl.getUniformLocation(_shprog,
"uSceneWidth");
_shprog.uSceneHeight = glfx.gl.getUniformLocation(_shprog,
"uSceneHeight");

glfx.gl.enableVertexAttribArray(_shprog.vertexPositionAttribute);
glfx.gl.enableVertexAttribArray(_shprog.textureCoordAttribute);

});

...

We then tell glfx to apply our post-process shader program:

glfx.scene.setPostProcessShaderProgram(postProcessShaderProgram);

This call will result in different rendering path, which renders the scene to a texture, applies that texture to a screen-space quad, and renders the quad with the post-process shader.

Here is the shader for screenspace.fs, used in the demo shown above:

precision mediump float;

uniform float uPeriod;
uniform float uSceneWidth;
uniform float uSceneHeight;
uniform sampler2D uSampler;        
varying vec2 vTextureCoord;

void main(void) {

vec4 sum = vec4( 0. );
float blurSampleOffsetScale = 2.8;
float px = (1.0 / uSceneWidth) * blurSampleOffsetScale;
float py = (1.0 / uSceneHeight) * blurSampleOffsetScale;

vec4 src = texture2D( uSampler, ( vTextureCoord ) );

sum += texture2D( uSampler, ( vTextureCoord + vec2(-px, 0) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(-px, -py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(0, -py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(px, -py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(px, 0) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(px, py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(0, py) ) );
sum += texture2D( uSampler, ( vTextureCoord + vec2(-px, py) ) );
sum += src;

sum = sum / 9.0;

gl_FragColor = src + (sum * 2.5 * uPeriod);

}

Note that it requires a few uniforms to be supplied to it, we use the glfx.scene.onPostProcessPreDraw() callback to setup the variables (before the post-processed scene is drawn):

var timeAcc = 0;
glfx.scene.onPostProcessPreDraw =
function(tdelta) {

timeAcc += tdelta;
var timeScaled = timeAcc * 0.00107;

if(timeScaled > 2.0*Math.PI) {
timeScaled = 0;
timeAcc = 0;
}

var period = Math.cos(timeScaled);
glfx.gl.uniform1f(postProcessShaderProgram.uPeriod, period + 1.0);

glfx.gl.uniform1f(postProcessShaderProgram.uSceneWidth, glfx.gl.viewportWidth);
glfx.gl.uniform1f(postProcessShaderProgram.uSceneHeight, glfx.gl.viewportHeight);
};

What we’re doing is using the scene rendering time deltas to generate a periodic/sinusoidal wave. This results in the pulsing brightness/fading effect of the scene. The brightness effect itself is done by adding the source pixel to a blurred + brightened version of itself. The blurring allows for the soft fade in and fade out.

Lexiio

Another little experiment of mine: Lexiio, a web-based CLI dictionary.

Lexiio

A few takeaways:

  • Part of the reason for building this was that I wanted to actually make use of the Wiktionary data set snapshot in a real project. The data set is pretty comprehensive, and easy to parse and work with.
  • This was also a learning exercise for Golang. There’s nothing complex here but, so far, working with Go has been enjoyable. I like that I’m building a native application, types are enforced, and the HTTP server included as part of the standard library is incredibly easy to setup and work with.
  • I wanted to experiment a bit with what a web-based CLI would look and feel like. For something like a dictionary, where user interaction revolves around textual input/output, a command-line interface seems to work really well.

The end of XULRunner

I’ve written a bit on XULRunner (and the related technologies, XUL and XPCOM) for application development, so I felt it was apt to mention that Mozilla no longer provides new builds of XULRunner. The recommended way to run XULRunner-based application is via the Firefox executable (Firefox ships with a private private XULRunner package):

firefox -app application.ini

You can take a copy of a Firefox installation and or compile Firefox from source, which is surprisingly easy.

One nice side effect worth mentioning: while I could never get proper high-DPI support with XULRunner, it works perfectly when Firefox is used as the runner.

GLSL variable qualifiers

I’ve been playing around with WebGL shader code recently and found this bit on variable prefixes helpful, particularly in the explanation of the variable qualifiers:

  • Attribute: data provided by buffers
  • Uniform: inputs to the shaders
  • Varying: values passed from a vertex shader to a fragment shader and interpolated (or varied) between the vertices for each pixel drawn

Something important to keep in mind is that this relates to the OpenGL ES Shading Language, Version 1.00, which is (unfortunately) what’s currently supported by WebGL.

A WebGL implementation must only accept shaders which conform to The OpenGL ES Shading Language, Version 1.00 [GLES20GLSL], and which do not exceed the minimum functionality mandated in Sections 4 and 5 of Appendix A.

Attribute and Varying were part of early versions of, OpenGL-supported, GLSL, but are deprecated as of OpenGL 3.0 / GLSL 1.30.10, and replaced with more generic constructs:

  • in is for input from the previous pipeline stage, i.e. per vertex (or per fragment) values at most, per primitive if using glAttribDivisor and hardware instanciation
  • out is for output to the next stage

Vertical centering with flexbox

I’ve only begun delving into doing layouts with flexbox (CSS Flexible Box Layout), but one immediate use case is vertical centering, which has been a pain point in web design since the transition away from table-based layouts.

HTML

<body>    
<
div class="flex-container">    
    <
p class="centered-element-within-body">Center Me</p>            
</
div>
</
body>

CSS

* { margin:0; padding:0; border:none; box-sizing:border-box; }
html, body { width:100%; height:100%; overflow:hidden; }

.flex-container { width:100%; /* width of body (fluid) */
height:100%; /* height of body (fluid) */
display:flex; /* flex display */
flex-direction: row; /* flow of elements in container */
justify-content: center; /* alignment along the main axis (x-axis, since the flex-direction = row) */
                    
align-items: center; /* alignment along the secondary axis (y-axis) */            
                    
border: 5px solid #2C81F5;                    
                }
                
.centered-element-within-body { background:#2C81F5; color:#fff; }

Real-time image processing on the web

A while ago I began playing around with grabbing a video stream from a webcam and seeing what I could do with the captured data. Capturing the video stream using the navigator.getUserMedia() method was straightforward, but directly reading and writing the image data of the video stream isn’t possible. That said, the stream data can be put onto a canvas using CanvasRenderingContext2D.drawImage(), giving you to ability to read the pixel data. When it comes to writing visual data, a few options are available.

var videoElem = document.querySelector('video');

// Request video stream
navigator.getUserMedia({video: true, audio: false},

function(_localMediaStream) {
videoStream = _localMediaStream;
videoElem.src = window.URL.createObjectURL(_localMediaStream);
},

function(err) {
console.log(
'navigator.getUserMedia error' + err);
}

);
var videoElem = document.querySelector('video');
var canvas = document.querySelector('canvas');
var ctx = canvas.getContext('2d');

...

// put snapshot from video stream into canvas
ctx.drawImage(videoElem, 0, 0);

You can read and write to the <canvas> element, so hiding the <video> element with the source data and just showing the <canvas> element is an option, but the CanvasRenderingContext2D.drawImage() call is expensive; looking at the copied stream on the <canvas> element there is, very noticeable, visual lag. Another reason to avoid this option is that the frequency at which you render (e.g. 30 FPS), isn’t necessarily the frequency at which you’d want to grab and process image data (e.g. 10 FPS). The disassociation allow you to keep the video playback smooth, for a better user experience, but more effectively utilize CPU cycles for the image processing. At least in my experiences so far, a small delay in the visual feedback from the image processing is acceptable and looks perfectly fine intermixed with the higher-frequency video stream.

Throwing aside reading and writing to just the <canvas> element, alternative options all involve showing the <video> element with the webcam stream and placing visual feedback on top of the video pixels. A few ideas:

  • Write pixel data to another canvas and render it on top of the <video> element
  • Render SVG elements on top of the <video> element
  • Render DOM elements (absolutely positioned) on top of the <video> element

The third option is an ugly solution, but it’s fast to code and thus allows for quick prototyping. The demo and code below shows a quick demo I slapped together using <div> elements as markers for hot spots, in this case bright spots, within the video.

<!DOCTYPE html>
<
html>
<
head>
<
title>Webcam Cap</title>
<
meta charset="UTF-8">
<
meta name="viewport" content="width=device-width, initial-scale=1.0">

<
style type="text/css">
* { margin:0; padding:0; border:none; }
</style>

</
head>

<
body>
<
div>
<
video style="width:640px; height:480px;" width="640" height="480" autoplay></video>
<
canvas style="display:none; width:640px; height:480px;" width="640" height="480"></canvas>
</
div>

<
div class="ia-markers"></div>

<
script type="text/javascript">

navigator.getUserMedia = (navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia);

if ( typeof navigator.getUserMedia !== 'undefined' ) {

var videoElem = document.querySelector('video');
var canvas = document.querySelector('canvas');
var ctx = canvas.getContext('2d');
var videoStream = null;
var snapshotIntv = null;

var width = 640;
var height = 480;

// Request video stream
navigator.getUserMedia({video: true, audio: false},

function(_localMediaStream) {
videoStream = _localMediaStream;
videoElem.src = window.URL.createObjectURL(_localMediaStream);

// Take a snapshot of the video stream 10ms
snapshotIntv = setInterval(function() {
processSnapshot(videoStream);
}, 100);

},

function(err) {
console.log(
'navigator.getUserMedia error' + err);
}

);


// Take a snapshot from the video stream
function processSnapshot() {

// put snapshot from video stream into canvas
ctx.drawImage(videoElem, 0, 0);

// Clear old snapshot markers
var markerSetParent = (document.getElementsByClassName('ia-markers'))[0];
markerSetParent.innerHTML =
'';

// Array to store hotzone points
var hotzones = [];

// Process pixels
var imageData = ctx.getImageData(0, 0, width, height);
for (var y = 0; y < height; y+=16) {
for (var x = 0; x < width; x+=16) {
var index = (x + y * imageData.width) << 2;

var r = imageData.data[index + 0];
var g = imageData.data[index + 1];
var b = imageData.data[index + 2];

if(r > 200 && g > 200 && b > 200) {
hotzones.push([x,y]);
}
}
}

// Add new hotzone elements to DOM
for(var i=0; i<hotzones.length; i++) {
var x = hotzones[i][0];
var y = hotzones[i][1];

var markerDivElem = document.createElement("div");
markerDivElem.setAttribute(
'style', 'position:absolute; width:16px; height:16px; border-radius:8px; background:#0f0; opacity:0.25; left:' + x + 'px; top:' + y + 'px');
markerDivElem.className =
'ia-hotzone-marker';

markerSetParent.appendChild(markerDivElem);
}
}

}
else {
console.log(
'getUserMedia() is not supported in your browser');
}

</script>

</
body>
</
html>

Function argument tricks

I came across this “trick” on coderwall for avoiding an if statement when you have a a function argument that is allowed to be either an array or a scalar value (something that seems oddly common in loosely typed languages).

function example(ids) {
;[].concat(ids).forEach(
function (id) {
// ...
})
}

The trick in question is just concatenating the argument with an empty array so that, within the function, you’re always dealing with an array and elements of the array.

Taking a step back, the bigger question is why do this?
In my experience, it’s almost always better to keep the code within the function straightforward and force the caller to adapt and give the function what it needs. In this case, that means simply forcing the caller to always pass an array.

The road never built

On reliability, Robert Glass notes the following in Frequently Forgotten Fundamental Facts about Software Engineering:

Roughly 35 percent of software defects emerge from missing logic paths, and another 40 percent are from the execution of a unique combination of logic paths. They will not be caught by 100-percent coverage (100-percent coverage can, therefore, potentially detect only about 25 percent of the errors!).

John Cook dubs these “sins of omission”:

If these figures are correct, three out of four software bugs are sins of omission, errors due to things left undone. These are bugs due to contingencies the developers did not think to handle.

I can’t validate the statistics, but they do ring true to my experiences as well; simply forgetting to handle an edge case, or even a not-so-edge case, is something I’ve done more often than I’d like. I’ve introduced my fair share of “true bugs”, code paths that fails to do what’s intended, but with far less frequency.

The statistics also hint at the limitations of unit testing, as you can can’t test a logic path that doesn’t exist. While you can push for (and maybe even attain) 100% code coverage, it’s a fuzzy metric and by no means guarantees error-free functionality.

Code Coverage

A look at Adobe Edge Animate

I recently played around a bit with Adobe Edge Animate, as I was searching for an SVG animation tool, and jotted down a few notes on my impression of the editor.

Adobe Edge Animate CC 2014
  • The timeline and visual preview are excellent, it’s refreshing to have something like this available for web animations. I did notice a tendency for the preview to tear and flicker when animating but, overall, it’s a minor annoyance.
  • The output is not cross-platform, the generated code is heavily webkit-based and depends on vendor-prefixed styles (-webkit-filter) for certain effects. It’s cool to see the effect and have them in the editor, but it really doesn’t achieve a “write once, run anywhere” development process.
  • This is not an SVG animation tool. I expected transitions and transformation on SVG elements (groups, paths, etc.), instead what’s generated is CSS3 transitions/transformations on a <div> element (the SVG document is simply put as the background-image of the <div>). This severely limits what can be done. Much richer and expressive animations could be achieved by allowing manipulating the points and control points on individual SVG elements.

It’s disappointing that despite being an incredibly visual medium, most web design tasks are still done by writing linear blocks of code. Animation is a task well suited for a WYSIWYG editor and Edge Animate is ultimately a step in the right direction, but it’s basis on CSS3 transition and transform puts some hard limits on what can be achieved and I can’t see myself using it for anything substantial.

WebGL on a high-DPI display

Dealing with WebGL on a high-DPI display isn’t too difficult, but it does require an understanding of device pixels vs CSS pixels. Elements on a page automatically upscale on a high-DPI display, as dimensions are typically defined with CSS, and therefore defined in units of CSS pixels. The <canvas> element is no exception. However, upscaling a DOM element doesn’t mean that the content within the element will be upscaled or rendered nicely – this is why non-vector content can appear blurry on higher resolutions. With WebGL content, not only will it appear blurry, but the viewport will likely be clipped as well, due to the viewport being incorrectly calculated using CSS pixel dimensions.

With WebGL everything is assumed to be in units of device pixels and there is no automatic conversion from CSS pixels to device pixels. To specify the device pixel dimensions of the <canvas>, we need to set the width and height attributes of the element:

CSS Pixels vs Device Pixels on canvas

I like to compute and set the attributes automatically using window.devicePixelRatio.
In glfx I do the following (passing in _canvasWidthCSSPx, _canvasHeightCSSPx):

// Get devicePixelRatio
glfx.devicePixelRatio = window.devicePixelRatio || 1;

// Set the width,height attributes of the canvas element (in device pixels)
var _canvasWidthDevicePx = _canvasWidthCSSPx * glfx.devicePixelRatio;
var _canvasHeightDevicePx = _canvasHeightCSSPx * glfx.devicePixelRatio;    
_canvas.setAttribute(
"width", _canvasWidthDevicePx);
_canvas.setAttribute(
"height", _canvasHeightDevicePx);

// Set viewport width,height based on dimensions of canvas element        
glfx.gl.viewportWidth = _canvasWidthDevicePx;
glfx.gl.viewportHeight = _canvasHeightDevicePx;        

Reference: HandlingHighDPI