This becomes a problem when you have a lot of textures enabled at once, or when they're very large, or when they're high-bit depth, because if they won't fit in VRAM, the current drawing operation will fail and you'll end up with garbage drawn instead. There isn't an error condition or log message output when this happens, so it can be tricky to track down.

To avoid this, you need to work out if your textures will fit in the currently available VRAM before you draw. You can calculate the VRAM usage of an individual texture with

width * height * 4 * bytes per channel

So, a 2048x2048 eight bit per channel texture takes up 2048*2048*4*1 = 16,77,216 bytes, or 16 MB of VRAM.

To work out if you'll run out of VRAM, take all the active texture's memory, add them together, and do the same for the area (pbuffer/window size) that you're drawing to, since that also needs to be in the graphics card while you're drawing.

For a 64 MB graphics card, you can fit four 16 MB textures in at once, so that means you can use three 2048*2048 input textures, if you're drawing into a 2048*2048 pbuffer.

To figure out how much VRAM is available on the card, you need to call

long getAvailableVRAM(void)
{
CGOpenGLDisplayMask displayMask = CGDisplayIDToOpenGLDisplayMask(CGMainDisplayID());
CGLRendererInfoObj info;
CGLQueryRendererInfo(displayMask, &info, &_numRenderers);
long availableVRAM;
CGLDescribeRenderer(_info, 0, kCGLRPVideoMemory, &availableVRAM);
return availableVRAM;
}

and then divide by the number of displays connected to the card, since at least ATI evenly partitions the VRAM between each display.

First, the fundamental rule of debugging OpenGL: call glGetError() early and often. This returns a non-zero value if any of the previous GL calls ran into a problem. I try to call this at the end of every big chunk of GL processing, for example at the end of a model drawing routine, to keep down the clutter and avoid the performance impact of having it in an inner loop.

Now, you only get a number out of the function, so you need to call gluErrorString() to get an understandable description of what went wrong. I usually wrap the checking up into a macro, so I can easily add it, and automatically remove it in release builds. Here's what I use:

#ifdef ENABLE_GL_ERRORS
void Effect_ShowGLErrors(void)
{
GLenum error;
const GLubyte* errStr;
if ((error = glGetError()) != GL_NO_ERROR)
{
errStr = gluErrorString(error);
fprintf(stderr, "OpenGL Error: %s\n", errStr);
}
}
#else // ENABLE_GL_ERRORS
#define EFFECT_SHOW_GL_ERRORS() (void)(0)
#endif // ENABLE_GL_ERRORS

Sometimes it's not obvious which call caused the error, and then you either need to scatter more checks through your GL code to narrow it down, or use a great feature of Apple's OpenGL Profiler, Break on Error.

Start up the profiler, in /Developer/Applications/Graphics Tools/, and either launch your program from it or attach to a copy that's already running. Choose "Views->Breakpoints" from the main menu, and click "Break on Error" on the bottom left of the window. Now, when an error occurs, the program will display a callstack, arguments, and the exact function that caused it. If you've attached to a program you ran from the debugger, you can hit pause in there and examine everything as normal.

Black Screen Blues

One of the hardest things to debug is a black screen in GL. Is the window or pbuffer setup messed up? Is one of the textures you're using bogus? Is your camera pointing at the floor? Is a fragment program having a problem? Is the geometry you're drawing broken, or translated off to infinity? Is your lighting setup busted?

My first test is the programming equivalent of checking pupil dilation with a flashlight; glClear(). If calling a colored clear at the end of rendering doesn't work, then the problem's outside of the drawing code, and I know I have to fix something in my pbuffer or window setup instead.

Clearing the screen is the only way of getting something to appear on screen that it's impossible to screw up by having dodgy gl state. It doesn't matter what your viewport or matrices are, what textures or programs are bound, if GL is working, calling glClear() will have an effect. It's especially useful if you have a chain of rendering running through a series of pbuffers, it can be tough to figure out where a problem's occuring if the end result is blank, using clear lets you work backwards through the chain.

glClearColor(1,1,0,1);
glClear(GL_COLOR_BUFFER_BIT);
glClearColor(0,0,0,0);

As a bonus, you can change the color (I've got yellow above) if you want to indicate different calls.

If this does give you a yellow screen when you call it after all your rendering, then move it to the start of your rendering code, so anything else will be drawn over it. If you then see a black screen, or a silhouette of your model, you know your geometry is getting rendered, but the texturing, lighting or pixel shaders you're using don't work. If you still see a clear yellow screen, then it's the geometry that's the problem.

Always specify if you're only using some components in an instruction, the compilers aren't smart enough generally to figure out if you only use the .x component of the result later on, and both vendors' hardware has clever tricks they can play executing vector and scalar instructions in parallel.

Try and calculate everything you can using arithmetic instructions rather than doing table lookups from textures. Memory access almost always seems to be the limiting factor on the speed of our fragment programs, you've got a lot of free instruction slots that can be filled performing extra calculations while the hardware's waiting on memory.

ATI cards support simple swizzling, either where you're masking out some components in the result register (ADD foo.xy, bob, jim;) or where you're duplicating a single component across the whole register (ADD foo, bob, jim.x;)
Anything more complicated will be emulated using multiple instructions (ADD foo, bob.zyzy, jim; or ADD foo, bob.xxxy, jim;)

On ATI, using GL_TEXTURE_RECTANGLE_EXT textures in TEX instructions (RECT as the target) will generate hidden instructions to convert the coordinates to the 0 to 1 range, from the input range of 0 to that's used for the extension. This is especially tricky because it adds another hidden level of texture indirection.

I've included the code inline below, or you can get a zip of the code from PBufferUtils.zip here

Use PBuffer_Begin() before you start drawing into it, and PBuffer_End() once you're done. When you're ready to use it as a texture, call PBuffer_Use() in the same place you'd normally do a
glEnable(GL_TEXTURE_RECTANGLE_EXT);
glBindTexture(GL_TEXTURE_RECTANGLE_EXT, someTextureID);
and do a
glDisable(GL_TEXTURE_RECTANGLE_EXT);
when you're done with it.

PBuffer_Begin() clears the pbuffer, sets up an orthographic view and resets some common gl state, but you need to be careful to reset gl state manually, a lot of our bugs have been caused by state left hanging around.

You'll get an error if you try to create a pbuffer with different pixel attributes than the shared context, even if the difference doesn't seem relevant. For example, you might need to add kCGLPFANoRecovery to the list of attributes if that's what the shared context has been created with.

Creating and destroying pbuffers is expensive, so we tend to carry them over from frame to frame unless they need to change in size. If size changes are needed, we try to over-allocate and create bigger pbuffers than we need, to avoid frequent reallocations for growing objects.

Destroying a pbuffer immediately after using it as a texture also seems to cause rendering problems, as if the surface is destroyed before the renderer uses it. Waiting a little while, for example until the next time you need to render, seems to prevent this.

Here's the code to go into PBufferUtils.h:
------------
#ifndef INCLUDE_PBUFFER_UTILS_H
#define INCLUDE_PBUFFER_UTILS_H

#include
#include

enum EPBufferFlags
{
ePBufferFlag_ZBuffer=(1<<0),
};

typedef struct PetePBuffer_tag {
CGLPBufferObj pbuffer;
CGLContextObj pbufferContext;
CGLContextObj previousContext;
int width;
int height;
GLuint textureID;
bool needsClearing;
bool needsFlush;
int createdWidth;
int createdHeight;
} PetePBuffer;

PetePBuffer* PBuffer_Create(CGLContextObj sharedContext,int nWidth,int nHeight,int colorDepth, int flags);
void PBuffer_Destroy(PetePBuffer* pbuffer);

// Surround rendering with these to draw into the pbuffer, they handle pushing and popping the old
// context for you
void PBuffer_Begin(PetePBuffer* pbuffer);
void PBuffer_End(PetePBuffer* pbuffer);

// Binds the pbuffer as a texture
void PBuffer_Use(PetePBuffer* pbuffer);

#endif // INCLUDE_PBUFFER_UTILS_H
------
and here's the PBufferUtils.cpp code:
------
#include "PBufferUtils.h"

#include
#include

static void checkCGLErrorImplementation(CGLError error, char* sourceFile, int sourceLine)
{
if (error) {
const char* errStr;
errStr = CGLErrorString(error);
fprintf(stderr, "CGL Error: %s at %s:%d\n", errStr, sourceFile, sourceLine);
assert(false);
}
}

#define checkCGLError(error) checkCGLErrorImplementation(error,__FILE__,__LINE__)

PetePBuffer* PBuffer_Create(CGLContextObj sharedContext,int width,int height,int colorDepth, int flags)
{
PetePBuffer* pbuffer=new PetePBuffer;
pbuffer->pbuffer=NULL;
pbuffer->pbufferContext=NULL;
pbuffer->previousContext=NULL;
pbuffer->width=width;
pbuffer->height=height;
pbuffer->textureID=0;
pbuffer->needsClearing=true;
pbuffer->needsFlush=false;

const int bitsPerPixel = (colorDepth*4);
const bool hasZBuffer = (flags&ePBufferFlag_ZBuffer);

int i = 0;
CGLPixelFormatAttribute pixelFormatAttributes[32];

pixelFormatAttributes[i++] = kCGLPFAAccelerated;
pixelFormatAttributes[i++] = kCGLPFAWindow;
pixelFormatAttributes[i++] = kCGLPFAColorSize;
pixelFormatAttributes[i++] = (CGLPixelFormatAttribute)(bitsPerPixel);

if (colorDepth>8)
pixelFormatAttributes[i++] = kCGLPFAColorFloat;

if (hasZBuffer)
{
pixelFormatAttributes[i++] = kCGLPFADepthSize;
pixelFormatAttributes[i++] = (CGLPixelFormatAttribute)(16);
}

pixelFormatAttributes[i++] = (CGLPixelFormatAttribute)0;

long numPixelFormats = 0;
CGLPixelFormatObj pixelFormat = NULL;
CGLError error = CGLChoosePixelFormat(pixelFormatAttributes, &pixelFormat, &numPixelFormats);
checkCGLError(error);

error = CGLCreateContext(pixelFormat, sharedContext, &pbuffer->pbufferContext);
checkCGLError(error);

CGLDestroyPixelFormat(pixelFormat);

error = CGLCreatePBuffer(pbuffer->width,pbuffer->height,GL_TEXTURE_RECTANGLE_EXT,GL_RGBA,0,&pbuffer->pbuffer);
checkCGLError(error);

return pbuffer;

}

void PBuffer_Destroy(PetePBuffer* pbuffer) {

if (pbuffer!=NULL)
{
CGLDestroyPBuffer(pbuffer->pbuffer);
CGLDestroyContext(pbuffer->pbufferContext);
}

delete pbuffer;

}

void PBuffer_Begin(PetePBuffer* pbuffer) {

// Pete- check to ensure begin() hasn't already been called for this pbuffer
assert(pbuffer->previousContext==NULL);

pbuffer->previousContext=CGLGetCurrentContext();

long screen;
CGLError error=CGLGetVirtualScreen(pbuffer->previousContext,&screen);
assert(!error);

error=CGLSetCurrentContext(pbuffer->pbufferContext);
checkCGLError(error);

error=CGLSetPBuffer(pbuffer->pbufferContext,pbuffer->pbuffer,0,0,screen);
checkCGLError(error);

glClearColor(0.0f,0.0f,0.0f,0.0f);
if (pbuffer->needsClearing)
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT);
glDisable(GL_DEPTH_TEST);
glDisable(GL_FRAGMENT_PROGRAM_ARB);

glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();

glColorMask(GL_TRUE,GL_TRUE,GL_TRUE,GL_TRUE);

glColor4f(1.0f,1.0f,1.0f,1.0f);

glOrtho(0, pbuffer->width, 0, pbuffer->height, -1.0, 1.0);

glActiveTexture(GL_TEXTURE0);

}

void PBuffer_End(PetePBuffer* pbuffer) {

glFlush();

pbuffer->needsFlush=true;

assert(pbuffer->previousContext!=NULL);

CGLError error=CGLSetCurrentContext(pbuffer->previousContext);
checkCGLError(error);

pbuffer->previousContext=NULL;

}

void PBuffer_Use(PetePBuffer* pbuffer) {

if (pbuffer->needsFlush)
{
CGLContextObj currentContext=CGLGetCurrentContext();

CGLError error=CGLSetCurrentContext(pbuffer->pbufferContext);
checkCGLError(error);

glFlush();

error=CGLSetCurrentContext(currentContext);
checkCGLError(error);

pbuffer->needsFlush=false;
}

if (pbuffer->textureID==0)
{
CGLContextObj currentContext=CGLGetCurrentContext();

glGenTextures(1,&pbuffer->textureID);
glBindTexture(GL_TEXTURE_RECTANGLE_EXT,pbuffer->textureID);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT,GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT,GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

CGLError error=CGLTexImagePBuffer(currentContext,pbuffer->pbuffer,GL_FRONT_LEFT);
checkCGLError(error);

}
else
{
glBindTexture(GL_TEXTURE_RECTANGLE_EXT,pbuffer->textureID);
}

glEnable(GL_TEXTURE_RECTANGLE_EXT);

}

ATTRIB inputCoords = fragment.texcoord[0];

TEMP sourceCoords;
TEMP sourceCoordsTopLeft, sourceCoordsTopRight, sourceCoordsBottomLeft, sourceCoordsBottomRight, fraction;
TEMP sourceTopLeft,sourceTopRight,sourceBottomLeft,sourceBottomRight;

SUB sourceCoords, inputCoords, {0.5, 0.5, 0.0, 0.0};
FRC fraction.x, sourceCoords.x;
FRC fraction.y, sourceCoords.y;
SUB sourceCoords, sourceCoords, fraction;
ADD sourceCoords, sourceCoords, {0.5, 0.5, 0.0, 0.0};

ADD sourceCoordsTopLeft, sourceCoords, {1,1,0,0};
ADD sourceCoordsTopRight, sourceCoords, {0,1,0,0};
ADD sourceCoordsBottomLeft, sourceCoords, {1,0,0,0};
ADD sourceCoordsBottomRight, sourceCoords, {0,0,0,0};

TEX sourceTopLeft, sourceCoordsTopLeft, texture[0], RECT;
TEX sourceTopRight, sourceCoordsTopRight, texture[0], RECT;
TEX sourceBottomLeft, sourceCoordsBottomLeft, texture[0], RECT;
TEX sourceBottomRight, sourceCoordsBottomRight, texture[0], RECT;

LRP sourceTopLeft, fraction.x, sourceTopLeft, sourceTopRight;
LRP sourceBottomLeft, fraction.x, sourceBottomLeft, sourceBottomRight;
LRP result.color, fraction.y, sourceTopLeft, sourceBottomLeft;

END

The results you want are easy to visualize, when we see a moving image, we can tell that there are moving objects there, and roughly how fast and in what direction they're moving. The useful way to represent that is as a vector field, an image where each pixel has a velocity instead of a color, showing what direction the pixel in the underlying image is moving.

Determining Optical Flow by Horn & Shunck lays out the classic method for figuring this out. I took it as a starting point for my implementation, since it seemed pretty simple and robust compared to more recent algorithms.

It took me some time to understand the basic ideas behind their algorithm, but once I did, they're remarkably elegant and straightforward. I'll try an informal explanation here to help anyone else whose eyes glaze over when confronted with a journal paper.

Start with the 1D case. You have a 1D wave that looks like this at time 0:

Here's the same wave at frame 1:

Visually, you can see that the line has moved to the left. If you subtract frame 0 from frame 1, you get this line:

This is just the difference between the first and second frames. The crucial step is to multiply this against the slope of the original wave, to get an estimate of the movement. This gives you this wave:

Why does this work? One way that I found to visualize it is to imagine you've got an infinite straight line, that's constantly sloping and sliding to the right over time. For any point that's on the line at the start of time, the line directly below it drops away at a constant speed:

The difference between two frames is like the distance from the point that was on the line, to where the line is now below it. That distance is determined by three variables; the slope of the line, the time that the line's been sliding, and how fast it's been sliding. Since we know the slope, and how long there's been between frames, we can divide those out and be left with the line's speed of movement.

This simple example should also highlight a couple of limitations of optical flow analysis:
- You need a slope or gradient in the image for the analysis to get a grip on. Flat areas of color in the center of objects can't reveal any movement.
- Movements need to be small, compared to the size of the object. The difference is applied to the gradient at each pixel, if that gradient is from a different object than in the first frame, it will give nonsense results. You need a slope that is roughly the same over the distance that the object moves between frames to get accurate results.

To extend the algorithm to two dimensions, do a difference on the two frames, and then multiply dy and dx gradient vectors for the slope of each pixel by the difference to get the optical flow estimate.

Here's some pseudo-code for the basic process:

(1) CurrentDifference = (Frame1[x][y]-Frame0[x][y]);

(2) GradientX = ((Frame1[x+1][y]-Frame1[x-1][y]) + (Frame0[x+1][y]-Frame0[x-1][y])) / 2.0;
(3) GradientY = ((Frame1[x][y+1]-Frame1[x][y-1]) + (Frame0[x][y+1]-Frame0[x][y-1])) / 2.0;

(4) GradientMagnitude = sqrt((GradientX*GradientX)+(GradientY*GradientY));

(5) VelocityX = CurrentDifference*(GradientX/GradientMagnitude);
(6) VelocityY = CurrentDifference*(GradientY/GradientMagnitude);

This is the heart of the technique. To improve the accuracy of the results, Horn and Schunck do this repeatedly, in order to eliminate spurious errors introduced by noise and to propagate movement across areas of flat color.

They use a couple of ways to do this. The algorithm likes long smooth slopes, and gets confused by short steep ones, and short steep ones are more likely to be the result of high-frequency noise than genuine objects in the sequence. To damp down the effect, a term usually called lambda is added to line 4:

(4) GradientMagnitude = sqrt((GradientX*GradientX)+(GradientY*GradientY)+Lambda);

Since the gradient is divided by this magnitude result, large values of lambda mean that steep gradients give lower velocities than gentle slopes. In practice, it's a noise filter, low values give you a very sensitive but noisy result, high values filter out the noise, but miss small objects.

The other method tries to smooth out the result, removing isolated small areas that are moving in a different direction to their neighbors, and also filling in areas with no gradients with movement data from nearby. This is where the iteration comes in; the vector field result is run through a blur kernel, and then the optical flow calculation is re-run, feeding the previous flow vector back in. This is how it looks in the code:

...
(3.1) PreviousVector = PreviousResultBlurred[x][y];
(3.2) PreviousDotGradient = (PreviousVector.x*GradientX)+(PreviousVector.y*GradientY);
...
(5) VelocityX = PreviousVector.x + (CurrentDifference+PreviousDotGradient)*(GradientX/GradientMagnitude);
(6) VelocityY = PreviousVector.y + (CurrentDifference+PreviousDotGradient)*(GradientY/GradientMagnitude);

Visually, the dot product is high when the vector is going in the same direction as the previous result (which, being blurred, represents the average direction in the local neighborhood) and is negative when the vector is going against the local flow. Iterating over this rewards vectors that are in step with their neighbors, and minimises the vectors that are marching to a different drummer:

To do the blurring, I just run a small gaussian kernel over the vector field between each iteration.

Once I understood the algorithm, it was not too hard to convert it over to a multi-pass sequence of fragment programs. In my implementation I take the two frames I'm interested in, and encode the luminance of each pixel into a 16 bit fixed point value, and store this value in two 8 bit channels in a 32 bit pbuffer, packing both images into a single surface.

Inside my iteration loop, I then run a fragment program that contains the gradient calculation and flow estimation algorithm explained in lines (1) to (6), with the core of it looking like this:

# left, right, bottom, top and centre have the luminance of the neighboring pixels of the first frame
# in the x component, and the luminance from the second in z
SUB left, right, left;
DP4 differential.x, left, {0.5, 0.0, 0.5, 0.0};
SUB bottom, top, bottom;
DP4 differential.y, bottom, {0.5, 0.0, 0.5, 0.0};

SUB differential.z, center.z, center.x;

# velocity.x and .y are the previous velocity for the pixel, both 0 first pass through
MOV velocity.z, 1.0;
DP3 flowError.x, differential, velocity;

MOV differential.z, lambda.x;
DP3 differentialMagnitude.x, differential, differential;

RCP differentialMagnitude.x, differentialMagnitude.x;
MUL differentialScale.x, flowError.x, differentialMagnitude.x;
MUL differentialScale.x, differentialScale.x, -1.0;

MAD velocity, differential, differentialScale.x, velocity;

I run two passes of a seperable gaussian blur convolution over the resulting vector field, one horizontal and one vertical, and then iterate over again. The lambda value I have between 0 and 1, depending on the image, and I normally run between 2 and 10 iterations, again depending on my needs and the image quality.

# x and y are either 0 if out of range, or 1 if in. Multiply them
# together so that the result is 0 if either are our of range, or
# 1 if they're both inside. Nudge the result down so we can
# compare easily
MAD compareResult.x, compareResult.x, compareResult.y, -0.1
CMP color, compareResult, borderColor, inputColor;

Convert a straight alpha color to premultiplied with black:

MUL premultColor, straightColor, straightColor.a;
MOV premultColor.a, straightColor.a;

Convert from premultiplied with black to straight alpha:

ADD premultColor.a, premultColor.a, 0.0000001;
RCP recipAlpha.x, premultColor.a;
MUL straightColor, premultColor, recipAlpha.x;
MOV straightColor.a, premultColor.a;

When we hit the limit on indirections, we'd see a black output. We'd either try to rewrite to avoid the indirections, or break the shader into multiple passes.

RCP on zero gives different results.
ATI cards return a very large number when you divide by zero. NVidia cards give a plus or minus infinity NaN result, which infects subsequent calculations.

We preferred ATI's behavior, it wasn't mathematically correct, but was usually what we wanted. To emulate that on the NV34 we had two approaches:

- Detect the zero denominator before the operation, and then replace the result with something sensible:

ABS value.x, denominator.x;
SUB value.x, 0, value.x;
RCP reciprocal.x, denominator.x;
CMP reciprocal.x, value.x, reciprocal.x, someValueYouWant.x;

- Don't try to detect the zero, just nudge all numbers up a bit. This only works for inputs you know will be positive (eg texture samples), and loses a little precision, but often this is not significant.

ADD denominator.x, denominator.x, 0.0000001;

Inexact texture coordinates.
It's been hard to get exactly one-to-one texel to pixel mapping on ATI cards, we want to use bilinear filtering in general but the sampling errors can introduce slight blurring to the image. NVidia cards show much smaller errors. Where possible, we switched to nearest neighbor sampling, we also considered griding up each large quad into smaller polys to minimise the errors.

The NV34 doesn't support a floating point pixel format.
For a high precision accumulation buffer, you can get 16bit fixed point precision by using two 8 bit buffers, and storing half of each 16 bit channel in one of the buffer channels.

Since you can only write to one pbuffer at a time, you need to run two passes. Here's a fragment program to implement the red/green pass:

!!ARBfp1.0
# Emulate a 16 bit accumulation buffer for red and green channels
# author - Pete Warden

ATTRIB tex0 = fragment.texcoord[0];
ATTRIB tex1 = fragment.texcoord[1];

PARAM unpackScale = { 255.0, 1.0, 255.0, 1.0 };
PARAM packScale = { 0.0039216, 1.0, 0.0039216, 1.0 };

TEMP s0, s1, s2;

# Fetch the texel to be added
TEX s0, tex0, texture[0], RECT;
# Fetch the current total from the first 8 bit buffer
TEX s1, tex0, texture[1], RECT;

# Multiply the upper half of each channel by 255
MUL s1, s1, unpackScale;

# Add on the lower half to the upper
ADD s1.r, s1.r, s1.g;
ADD s1.g, s1.b, s1.a;

# Do the accumulation of the current input onto the total
ADD s1,s0,s1;

# Put the lower half of each channels into the result
FRC s2.g, s1.r;
FRC s2.a, s1.g;

# Calculate the rounded upper half of both channels
SUB s2.r, s1.r, s2.g;
SUB s2.b, s1.g, s2.a;

# Compress the channels into a 0-1 range to store in the 8 bit buffer
MUL result.color, s2, packScale;

END

The NV34 only passes 4 texture coordinates through to the fragment program.
The solution is to write a vertex program that emulates the fixed-function pipeline, but passes through all 8 texture coordinates, as on ATI:

!!ARBvp1.0
# Passthru vertex program
ATTRIB vertexPosition = vertex.position;
OUTPUT outputPosition = result.position;

DP4 outputPosition.x, state.matrix.mvp.row[0], vertexPosition;
DP4 outputPosition.y, state.matrix.mvp.row[1], vertexPosition;
DP4 outputPosition.z, state.matrix.mvp.row[2], vertexPosition;
DP4 outputPosition.w, state.matrix.mvp.row[3], vertexPosition;

MOV result.color, vertex.color;
MOV result.texcoord, vertex.texcoord;
MOV result.texcoord[1], vertex.texcoord[1];
MOV result.texcoord[2], vertex.texcoord[2];
MOV result.texcoord[3], vertex.texcoord[3];
MOV result.texcoord[4], vertex.texcoord[4];
MOV result.texcoord[5], vertex.texcoord[5];
MOV result.texcoord[6], vertex.texcoord[6];
MOV result.texcoord[7], vertex.texcoord[7];

END

Normally, texture IDs are local to each context:

if you've created a texture in one context, you can't use it in another. To share textures, you need to use a shared context.

Shared contexts are places to store texture IDs, and other IDs for things like draw lists and programs. Creating a context with a shared context means using that shared context to store those IDs, and information about the objects they point to.

A shared context doesn't even need to be pointing to a surface, since its only purpose is to store all those IDs. The easiest way to use a shared context is to create a single global one when your app starts, don't attach it to any window or pbuffer, and pass it into all create context calls.

Once you've done that, all of the texture IDs you create in any of your contexts will be usable in all of them, and you never need to think about the shared context again.

You can use OpenGL on the Mac in a multithreaded app, but you must never call GL functions on the same context from different threads at the same time.

Think of a GL context like a C++ object, if you call a member function on an object whilst another thread is halfway through another member function on the same object, you may end up with a corrupted object. Corrupting a context by a threading mistake will often cause a kernel panic.

If you're writing a multi-threaded GL app, make sure you periodically run the OpenGL Profiler, and enable 'break on thread errors'. This will stop the app and give a stack trace whenever two threads call into the same context. To find out what the other threads are doing, run the command line gdb with 'gdb -p ' and the process number of your app, and do the command 'thread apply all bt' for a complete thread listing.

CGLMacro.h exposes some of the implementation details of Mac OpenGL calls. Normally, every gl*() call actually ends up doing something like a CGLGetCurrentContext() call internally, and then passing that on to the function that does the actual work.

CGLGetCurrentContext() has to do some magic to figure out which thread it's in, and return the correct context for that thread, so it's a good optimisation to cache the current context if you know it's not changing, and you're making a lot of GL calls. CGLMacros let you do this optimisation but change very little of your code, by letting you specify a variable to use for the CGL context. Then, including the header into a source file will automatically switch all GL calls to use that instead of the standard context system.

if you use bilinear filtering to get color and alpha values for a point halfway between the white pixel and a black one, this is what you end up with:

If you render this texture scaled up using the standard blending equation for straight alpha, glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA) then you'll see black fringing around the white pixel.

In games, this is not a problem, because you just make sure any zero alpha pixels in your artwork have the correct straight color values.

Where this does become an issue is when the image is generated as an intermediate stage of an image processing pipeline. For example, if you read in images that are pre-multiplied with black alpha, and convert them to straight alpha for the pipeline, then you have no way of telling what the correct color values for fully transparent pixels are.

Another example is drawing a polygon into a pbuffer or render texture. Nothing's drawn into the fully transparent areas, and there's no definitive way of picking a color to put in those pixels.

We looked at several solutions to this:

- Do a processing pass that averages the rgb values of the non-transparent neighbors of each zero alpha pixel, and replaces the existing color with that, or some other way of 'growing' good rgb values into the zero alpha areas.

Though this would probably do a decent job of removing the fringing in real world images, I could still construct cases where this would fail, such as a zero alpha pixel between black and white opaque pixels:

This would mean an extra pass over all of our intermediate textures, which would hit performance.

- Ignore any pixels with zero alpha when doing the bilinear filtering

Before each lerp in the bilinear process, do a check to see if one of the pixels to be lerped between has a zero alpha, and replace it with the other non-zero alpha pixel rather than lerping between them if so.

In pseudo-code, here's the original bilinear process

topColor = lerp(topLeftNeighbor,topRightNeighbor, fractionalX);
bottomColor = lerp(bottomLeftNeighbour,bottomRightNeighbor, fractionalX);
result = lerp(topColor, bottomColor, fractionalY);

and here's the version that checks for zero alpha

if (topLeftNeighbor.alpha==0)
topColor = topRightNeighbor;
else if (topRightNeighbor.alpha==0)
topColor = topLeftNeighbor;
else
topColor = lerp(topLeftNeighbor,topRightNeighbor, fractionalX);

if (bottomLeftNeighbor.alpha==0)
bottomColor = bottomRightNeighbor;
else if (bottomRightNeighbor.alpha==0)
bottomColor = bottomLeftNeighbor;
else
bottomColor = lerp(bottomLeftNeighbour,bottomRightNeighbor, fractionalX);

if (topColor.alpha==0)
result = bottomColor;
else if (bottomColor.alpha==0)
result = topColor;
else
result = lerp(topColor, bottomColor, fractionalY);

This would fix the problem, but requires 4 texture access's if implemented using OpenGL fragment programs, as well as being rather inelegant.

- Change the pipeline to use premultiplied with black

Using glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA) lets you render premultiplied with black alpha textures correctly, and bilinear filtering does the right thing. In the original case, the neighborhood of the white pixel evaluates to a full white (premultiplied by the alpha which drops away), removing the dark fringing.

This is what we did, even though it seems to go against the grain of normally using straight alpha with graphics cards, unlike the other solutions it didn't hurt performance.

A naive implementation of this would search through all of the scattered points to find the current closest distance, which gets slow very quickly. Steve's implementation cuts down the search space by dividing space into grid cells and placing a single point randomly within each one, like this:

This allows each evaluation to only check against the point in the cell it's currently in, and its neighbours.

This works great for static images, but I really wanted the points to move to get an animated texture. One way to do this would be go back to randomly scattered points, and then sort them into a spatial sorting structure such as a quad-tree each frame to keep the number of neighbours you need to check to a minimum.

I was aiming at a hardware solution, and wanted to draw a simple 2d image of the pattern, rather than procedurally evaluating it at arbitrary points, so I decided to try an idea I first came across in Stylized Video Cubes; using a z-buffer to construct the voronoi diagram.

I draw a quad centered around each point, enable depth testing and use a small pixel shader that outputs the distance from the central point as the z depth, as well as setting the color from a palette indexed on the depth:

The actual size of the quad that's drawn for each point was chosen pretty arbitrarily, to give a nice look for a typical distribution of points.

I then animate each point by giving it a random velocity vector, and wrapping it's position when it heads off the edge of the image (you can either add a slop border the size of a single quad around the edge to wrap to, or if you want a tileable result, render each point multiple times if it's touching a border).

This works pretty nicely, the main issue was what happened if a part of the image isn't covered by any quads, which leaves a gap in the result. I worked around this by clearing the screen to the color indexed by the maximum distance in the palette.

Having a tileable image means that it's possible to do a poor man's fractal noise, by reusing the constructed image for all the sublayers of the fractal pattern. The repetition can be noticeable, especially when its animated, but it's a very low cost operation once you have the basic image.

I experimented doing multiple passes to implement deeper levels of the algorithm, such as using the distance from the second or third nearest neighbor, but this involved rendering a pass that wrote ID values for each point to compare against in the next pass to figure out if the current nearest point was acceptable, and much larger quads, and so I was unable to get decent performance for L2 and higher textures.

I think the best approach for these on hardware would involve a combination of space-sorting strategies to break areas down into a small enough set of possible neighbor points, and then doing the fine-level sorting within a pixel shader. Space sorting would also be a good way to handle the moving points problem with a software implementation, since you don't have the performance advantage you get from an optimized z-buffer in hardware.