Sunday, May 1, 2011

Using the stencil buffer, rendering a stencil mirror

As we've seen the stencil buffer helps us to stop some pixels from ever being drawn to the screen, also the rasterizer is configurable in such a way that we can tell it how to draw geometry. In this short tutorial you will see how to combine these 2 to get a mirror effect.

The starting source of the stencil mirror sample is here. It contains a lot of useful code already written, for starters all necessary objects are already drawn. You will notice that the world matrix of the crate is multiplied by the Reflect matrix, this matrix create a reflection across the YZ plane. To reflect about any arbitrary plane you can use a rotation matrix. Create a rotation matrix that will rotate you plane over one of the easy planes to reflect, perform the reflection then multiply by the inverted rotation matrix.

Before starting to render we have to turn on the stencil buffer, we do this by changing the PreferredDepthStencilFormat of the graphics object. This member is of type DepthFormat enumeration so you just select one of the values there. I would also like to set a different resolution and turn on multi-sampling:

    graphics.PreferredDepthStencilFormat = DepthFormat.Depth24Stencil8;
    graphics.PreferredBackBufferWidth = 1280;
    graphics.PreferredBackBufferHeight = 720;
    graphics.PreferMultiSampling = true;

So now we have a DepthBuffer that stores 24 bit data and a stencil buffer with 8 bits of data for each pixel. Our resolution is 1280*1050 and multi-sampling is on.

Before starting to change the rendring code I will explain how it will work. First we will clear all buffers (back buffer, depth buffer and stencil buffer) then we will draw the normal scene to the back/depth buffers. We will then set the stencil pass operation to increment, stencil compare function to always and then draw the mirror. This will increment all the stencil values corresponding to the mirror pixels. Now we will set the stencil test to equal and the reference to 1. This will make sure only the pixels in the mirror get drawn. But if we do only this nothing will get drawn because the depth values will actually be behind the mirror, so we will need to clear the stencil buffer first, also because we reflected across the YZ plane the winding order of our polygons has changed so we have to change it in the rasterizer too.

To be able to set our desired DepthStencilState we must first create them. The first one (used to draw our mirror) will have to enable the stencil, set the stencil compare function to always and stencil pass operation to increment:

    DepthStencilState addIfMirror = new DepthStencilState() 
        StencilEnable = true,
        StencilFunction = CompareFunction.Always,
        StencilPass = StencilOperation.Increment

The next one (used to render the objects IN the mirror) will have to enable stencil test, have the stencil compare function set to equal and the stencil reference set to 1. Also because we changed the stencil pass operation we should set it back to default (keep):

    DepthStencilState checkMirror = new DepthStencilState() 
        StencilEnable = true,
        StencilFunction = CompareFunction.Equal,
        ReferenceStencil = 1,
        StencilPass = StencilOperation.Keep

In the draw function the first step is to reset the GraphicsDevice using this line of code:

    GraphicsDevice.Clear(ClearOptions.Target | ClearOptions.DepthBuffer | ClearOptions.Stencil,Color.CornflowerBlue,1,0);

The first parameter, the ClearOptions represent what buffer we want to clear, we bitwise OR them to signal that we want to clear all of them. The next one is the value that the color buffer (back buffer/target) will be cleared to. The third parameter represents the value the depth buffer will be cleared to, it's a float value between 0 and 1 where 0 represents the near plane and 1 represents the far plane. The last parameter is the value the stencil buffer will be cleared to. Earlier I specified that we will change the DepthStenilState and the Rasterizer state, so before performing any drawing it's best to reset them:

    GraphicsDevice.DepthStencilState = DepthStencilState.Default;
    GraphicsDevice.RasterizerState = RasterizerState.CullCounterClockwise;

Now we will draw the crate and the ground, no troubles there (code already written) and just before we draw the mirror we have to set our first DepthStencilState:

GraphicsDevice.DepthStencilState = addIfMirror;

Now we will draw our mirror as normal, remembering that it also writes to the stencil buffer. Aster we draw the mirror we have to set the other DepthStencilState, but remember we already have the depth of the mirror on the depth buffer, so we can't draw anything in the mirror, to solve this we have to clear the depth buffer only. Also remember that the winding order is inverted (because of the reflection) and we need to set it to CullClockwise:

    GraphicsDevice.DepthStencilState = checkMirror;
    GraphicsDevice.Clear(ClearOptions.DepthBuffer, Color.CornflowerBlue, 1, 0);
    GraphicsDevice.RasterizerState = RasterizerState.CullClockwise;

Now just draw the crate and ground in the mirror. Don't worry about rendering over the cube (because we reset the depth buffer) as the mirror was only drawn where the cube isn't. This is it, you should now have an effect similar to the video bellow, for the complete source of the stencil mirror tutorial click here.


  1. Excellent tutorial! Very very helpful! One question: How would you go about changing the background color in the mirror? See how it's white? That really ruins the effect, especially for what I'm working on. I just can't seem to figure out how to change it! Any help?

    1. If you want to change the background color (I left it as white so the mirror would be easily visible) you can just change the Color parameter from the simpleColorEffect.

      If you want your mirror to have a tint (like an imperfect mirror) you should draw only to stencil the first time, draw all items in the mirror and after that render the mirror color/texture using blending. I don't currently have a tutorial on blending but everything you need should be on MSDN: