CS424 Notes, 18 April 2012

• Framebuffers
  • In WebGL, a framebuffer refers to the collection of buffers (regions in memory on the GPU) that are used for rendering the image. A framebuffer can have several "attached" buffers in memory. One of these is a color buffer, in which the pixel colors themselves are stored. Another is a depth buffer. The third is a stencil buffer, which we won't discuss further here.
  • There is a default framebuffer, which is the only one that we have used so far. It contains the image that is copied into the canvas on the web page. By default -- unless you specify otherwise in the second parameter to gl.createContext() -- it comes with a depth buffer but no stencil buffer. The depth buffer is only used if the depth test is enabled.
  • It is possible to create framebuffer objects, which represent alternative framebuffers that can be used for drawing. The content of these extra framebuffers does not automatically appear on the screen. However, you can use the pixel data from them in various ways.
  • A frame buffer object is created using gl.createFramebuffer(), which returns a reference to the new framebuffer. Like texture objects, framebuffer objects reside (ideally) on the GPU.
  • When a framebuffer object is initially created, it does not initially have any any actual memory buffers attached to it. One way to get such memory is create a renderbuffer object -- yet another type of object that is managed similarly to framebuffer objects and texture objects. You need to create a renderbuffer object for each type of attachment that you want (color, depth, stencil). All are optional, but you need to have at least one.
  • But another, very interesting, option is to attach a texture object as a memory buffer for use by the framebuffer. In that case, drawing takes place directly to texture memory -- and you can then use the drawing as a texture in further work.
  • Once you have a framebuffer object, osc, set up, you can draw to it simply by saying

        gl.bindFramebuffer(gl.FRAMEBUFFER, osc);

    After that, all drawing operations apply to osc instead of to the default framebuffer. To return to drawing to the default location, say

        gl.bindFramebuffer(gl.FRAMEBUFFER, null);

    Note that you still use the same context, gl to do the drawing, and you will be using the same shader program as well, unless you change to a different program.
• Reading pixels
  • It is possible to read pixel color values from the current framebuffer (usually just the default framebuffer; otherwise, whatever one you have set with gl.bindFramebuffer).
  • The pixel values must be read into a typed array that you have already created. In WebGL, the data for the pixels will be in RGBA format, and the type of the array must be Uint8Array. You will need four bytes per pixel that you want to read. That is, if you want to read a rectangle of pixels whose width is width and whose height is height, you will need to create a variable such as

       var pixelstore = new Uint8Array( width*height*4 );

    Then, to read the pixels, call

       gl.readPixels(x,y,width,height,gl.RGBA,gl.UNSIGNED_BYTE,pixelStore);

    where (x,y) are the coordinates of the lower left corner of the rectangle that you want to read. (Remember that y-values increase upward!) The next two parameters must be gl.RGBA and gl.UNSIGNED_BYTE (full OpenGL allows other values).
  • The pixel data is in raw form: four bytes for each pixel, representing the RGBA components of the pixel color. The data for the bottom row of pixels comes first, followed by the next row above that, and so on.
• Pixels to Texture
  • Now, what can you do with all this raw pixel data? One possibility is to use the pixels in a texture. You do this using an alternative form of the gl.texImage2D function. So far, we have used gl.texImage2D only to load HTML image objects. It is actually more general. To load raw pixel data from a pixelstore variable like the one above, use

       gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, width, height, 
                           0, gl.RGBA, gl.UNSIGNED_BYTE, pixelStore);

    The first zero in this function is the mipmap level, which will ordinarily be 0. Except for width, height, and pixelstore, the other parameters also usually have the values shown here. You should provide enough data to completely fill the texture. That is, the size of pixelstore should be at least width*height*4. (However, one special note: If the last parameter is null, then the texture will be filled with zeros.)
  • One way to use this would be to create pixelstore programmatically, by some sort of computation. You might make a simple checkerboard texture in this way (or maybe compute part of the Mandelbrot set to use as a texture!).
  • Another way would be to draw an image into the framebuffer using WebGL, grab the pixels of the image using gl.readPixels, and then use the pixel data to create a texture image with gl.texImage2D.
  • This idea is used in AnimatedTexture2.html, a sample program that can show an "animated texture" on various 3D models. When this program is animating the texture, it first draws a 2D scene. It captures the data from that scene to make a texture, then draws a 3D scene that uses the texture. Two different shader programs are used, one for the 2D scene and one for the 3D scene. It looks something like this:

       gl.disable(gl.DEPTH_TEST);  // for 2D drawing
       gl.useProgram(world2D.prog);  // the shader program for the 2D scene
       world2D.drawFrame();  // draws the 2D scene
       gl.readPixels(0,0,512,512,gl.RGBA,gl.UNSIGNED_BYTE,pixelStore); // grab pixels
       gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, false);
       gl.bindTexture(gl.TEXTURE_2D, textureID);
       gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 512, 512, 0, gl.RGBA, 
              gl.UNSIGNED_BYTE, pixelStore);  // Make a texture image using the pixels
       gl.texParameteri(gl.TEXTURE_2D, 
              gl.TEXTURE_MIN_FILTER, gl.LINEAR);  // We're not making mipmaps!
       gl.useProgram(prog);  // the shader program for the 3D model
       gl.enable(gl.DEPTH_TEST);
          .
          .  // 3D drawing
          .
       

• Draw Direct to Texture
  • The problem with the above approach is that the pixel data for the texture is first copied from the GPU to the CPU (by gl.readPixels), then copied back from the CPU to the GPU (by gl.texImage2D). This is horribly inefficient. Communication between the GPU and CPU can be a real bottleneck in a program. That's why we want to do as much as possible in the GPU.
  • The solution is to use an alternative framebuffer. As mentioned above, you can attach a texture to a framebuffer object to be used as the color buffer. In our example, the texture is drawn as a 2D scene, which doesn't require a depth buffer. For 3D drawing, we would need a depth buffer. In that case, we might create a renderbuffer object and attach that to the framebuffer for use as the depth buffer.
  • Texture memory must already exist before it can be attached to a framebuffer. You can create the texture object, and then create its memory by calling

       gl.bindTexture(gl.TEXTURE_2D, textureID);
       gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, width, height, 0, gl.RGBA, gl.UNSIGNED_BYTE, null);

    where textureID was previously created with textureID = gl.createTexture. (I told you above that we would need gl.texImage2D with final parameter equal to null!)
  • Assume osc has been create with osc = gl.createFramebuffer(). Then drawing the animated texture example might look like:

       gl.bindFramebuffer(gl.FRAMEBUFFER, osc);  // draw to the framebuffer osc
       gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, 
                gl.TEXTURE_2D, textureID, 0);  // attach texture to framebuffer as color buffer
       gl.disable(gl.DEPTH_TEST);
       gl.useProgram(world2D.prog);
       world2D.drawFrame();
       gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, 
                gl.TEXTURE_2D, null, 0);  // sever the color buffer attachment
       gl.bindFramebuffer(gl.FRAMEBUFFER, null);  // go back to drawing to default framebuffer
       gl.bindTexture(gl.TEXTURE_2D, textureID);  // now, use the texture in the 3D drawing
       gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
       gl.useProgram(prog);  // the shader program for the 3D model
       gl.enable(gl.DEPTH_TEST);
          .
          .  // 3D drawing
          .

  • This technique is used in AnimatedTexture1.html.