Stencil testing

The depth buffer we just looked at has a companion. In the format every lesson creates, DXGI_FORMAT_D24_UNORM_S8_UINT, each pixel holds 24 bits of depth and, beside them, 8 more bits: the stencil buffer. Those 8 bits give every pixel a small integer, 0 to 255, which shaders never read or write. Instead, draws can be told to write values into it, and other draws can be told to keep or drop their pixels depending on what is there. That second part is the stencil test, and it runs in the output merger together with the depth test; as we will see, what a draw writes into the stencil can depend on the outcome of both.

Here is the idea in LearnOpenGL's picture. The stencil buffer is cleared to 0, a frame of 1s is written into it, and then a scene is drawn that only keeps the pixels where the stencil holds 1:

Whatever the effect, the recipe usually has the same shape:

1. Draw some objects with stencil writes on, so their shape is recorded in the stencil buffer.
2. Turn stencil writes off.
3. Draw other objects with a stencil test that keeps or drops their pixels according to what step 1 left behind.

The stencil buffer is cleared with the depth buffer. Every lesson's clear already includes it, with 0 as the value:

ID3D11DeviceContext_ClearDepthStencilView(context, depth_dsv, D3D11_CLEAR_DEPTH | D3D11_CLEAR_STENCIL, 1.0, 0);

The 1.0 is the depth to clear to, and the 0 the stencil value.

Stencil functions

OpenGL configures stencil testing with three calls: glEnable(GL_STENCIL_TEST), glStencilFunc for the test and glStencilOp for the writes, plus glStencilMask for which bits may change. In Direct3D all of it, together with the depth settings from the previous chapter, is one immutable object: the same depth-stencil state. The lesson builds its states with one helper:

// Pass the stencil test when (reference `func` stored); on passing, write the
// reference if `write`.
create_stencil_state: proc(device: *ID3D11Device, func: D3D11_COMPARISON_FUNC, write: b32, depth: b32) -> *ID3D11DepthStencilState = {
    desc: D3D11_DEPTH_STENCIL_DESC = {
        .DepthEnable = depth,
        .DepthWriteMask = D3D11_DEPTH_WRITE_MASK.D3D11_DEPTH_WRITE_MASK_ALL,
        .DepthFunc = D3D11_COMPARISON_FUNC.D3D11_COMPARISON_LESS,
        .StencilEnable = 1,
        .StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK,
        .StencilWriteMask = write ? D3D11_DEFAULT_STENCIL_WRITE_MASK : 0,
        .FrontFace = {
            .StencilFunc = func,
            .StencilPassOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_REPLACE,
            .StencilFailOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_KEEP,
            .StencilDepthFailOp = D3D11_STENCIL_OP.D3D11_STENCIL_OP_KEEP,
        },
    };
    desc.BackFace = desc.FrontFace;
    state: *ID3D11DepthStencilState = null;
    ID3D11Device_CreateDepthStencilState(device, desc&, state&);
    return state;
}

Piece by piece:

• StencilEnable is GL_STENCIL_TEST. With it off, as in the default state, no stencil test happens and nothing is written.
• StencilFunc is the comparison, with the same eight choices as the depth test (NEVER, LESS, EQUAL, ..., ALWAYS). It compares a reference value against the stored one, "reference function stored". So LESS passes when the reference is less than what is in the buffer.
• StencilReadMask is ANDed with both the reference and the stored value before they are compared. It is the third argument of glStencilFunc. D3D11_DEFAULT_STENCIL_READ_MASK is 0xff, all bits.
• StencilWriteMask chooses which bits a write may change, like glStencilMask. 0xff writes all of them; 0 writes none, so the stencil is tested but left alone.
• The three operations say what to write in each outcome, as the three arguments of glStencilOp do: StencilFailOp when the stencil test fails, StencilDepthFailOp when it passes but the depth test fails, StencilPassOp when both pass.

Two things are different from OpenGL. The reference value is not in the state at all. It is given each time the state is bound, as the last argument of OMSetDepthStencilState, so one state object serves for any reference. And Direct3D has a separate set of operations for front-facing and back-facing triangles, FrontFace and BackFace (OpenGL has the same in glStencilOpSeparate). We draw with culling off, so both kinds of triangle reach the test, and the helper simply copies one into the other.

The operations a stencil write can perform are these:

So with a state from create_stencil_state, a pixel that passes both tests writes the reference value into the stencil (if writes are on), and a pixel that fails either test leaves the stencil as it was.

Object outlining

Descriptions of stencil state only really make sense once you see them used, so let's build something with it: an outline around objects, the kind an editor draws around a selected object or a strategy game around selected units.

The trick is to draw each object twice. The first time, normally, and while doing so we write 1 into the stencil wherever the object lands. The second time, a little larger and in one flat colour, but only where the stencil is not 1. Everywhere the bigger copy covers the original, it is rejected; what is left is the rim that sticks out past the original's silhouette. In steps:

1. Draw the floor, which gets no outline, without touching the stencil.
2. Draw the cubes with a stencil test that always passes and a write of 1.
3. Scale the cubes up slightly and draw them with a flat-colour pixel shader, a stencil test that passes only where the stencil is not 1, no stencil writes and no depth test.

The flat colour comes from a second shader file, single_color.hlsl, whose vertex shader is the usual one and whose pixel shader is as short as they come:

float4 ps_main(vs_out p) : SV_Target
{
    return float4(0.04, 0.28, 0.26, 1.0);
}

s: shader = shader_create(device, filesystem_path("src/4.advanced_d3d11/2.stencil_testing/stencil_testing.hlsl"), vertex_layout);
single_color: shader = shader_create(device, filesystem_path("src/4.advanced_d3d11/2.stencil_testing/single_color.hlsl"), vertex_layout);

Each step's configuration is one depth-stencil state, created once at startup:

plain: *ID3D11DepthStencilState = create_stencil_state(device, D3D11_COMPARISON_FUNC.D3D11_COMPARISON_ALWAYS, 0, 1);
mark: *ID3D11DepthStencilState = create_stencil_state(device, D3D11_COMPARISON_FUNC.D3D11_COMPARISON_ALWAYS, 1, 1);
outside: *ID3D11DepthStencilState = create_stencil_state(device, D3D11_COMPARISON_FUNC.D3D11_COMPARISON_NOT_EQUAL, 0, 0);

• plain: the stencil test always passes and nothing is written. The depth test is on as usual. This is OpenGL's glStencilMask(0x00) for the floor.
• mark: the test always passes, and since the pass operation is REPLACE with writes on, every cube pixel that also passes the depth test writes the reference into the stencil.
• outside: the test passes where the reference is not equal to the stored value, nothing is written, and the depth test is off.

The render loop then reads almost like the list of steps:

// the floor: no stencil writes
shader_use(context, s&);
ID3D11DeviceContext_OMSetDepthStencilState(context, plain, 0);
bind_texture(context, 0, floor_texture);
draw_vertices(context, transforms_buffer, t&, plane_vb, 6, m4_identity());

// 1st pass: the cubes, writing 1 to the stencil where they land
ID3D11DeviceContext_OMSetDepthStencilState(context, mark, 1);
bind_texture(context, 0, cube_texture);
for (i: u32 = 0; i < 2; i += 1) { draw_vertices(context, transforms_buffer, t&, cube_vb, 36, m4_translate(positions[i])); }

// 2nd pass: slightly larger cubes, only where the stencil is not 1
ID3D11DeviceContext_OMSetDepthStencilState(context, outside, 1);
shader_use(context, single_color&);
for (i: u32 = 0; i < 2; i += 1) { draw_vertices(context, transforms_buffer, t&, cube_vb, 36, m4_mul(m4_translate(positions[i]), m4_scale(v3(1.1, 1.1, 1.1)))); }
ID3D11DeviceContext_OMSetDepthStencilState(context, null, 0);

The reference value 1 appears twice, once when binding mark, where it is the value written, and once when binding outside, where it is the value compared against. The outline cubes are scaled by 1.1 about their own centres: m4_translate(position) * m4_scale(1.1) scales first and then moves, as in Transformations. At the end the state is reset to null, the default, so that nothing drawn later is affected by accident.

Why switch the depth test off for the outline? With it on, the outline would lose its bottom edge, which lies below the floor (the scaled cubes reach down to y = -0.55), and it would vanish wherever anything nearer covers it. With it off, the outline shows through everything, which is the look an editor wants for a selection: you can find the object even when it is behind a wall. (A disabled depth test also writes no depth, so the outline does not hide anything drawn after it.)

The outline is not the same width all round: scaling about the centre pushes each face out by a fixed distance in the world, and perspective makes that look wider where the cube is nearer the camera. A scale is not a true constant-width outline, but for convex shapes like a cube it is convincing.

Because both cubes write into the same stencil before either outline is drawn, the outlines merge where the cubes overlap on screen: one border around the pair, not a border on top of each cube. For a group of selected units that is usually what you want. For separate borders you would clear the stencil, or use a different reference value, per object. The full program is here, and its shaders here and here.

Outlines are only one use. A stencil can limit drawing to the inside of a mirror or a portal, so a second view of the scene appears exactly there. Counting with INCR and DECR as shadow geometry is drawn is the basis of stencil shadow volumes. And the stencil is a cheap way to mark which pixels need a costly effect, so a later full-screen pass can skip the others.

Additional resources

• D3D11_DEPTH_STENCIL_DESC and D3D11_STENCIL_OP: the reference for every field and operation.
• Configuring depth-stencil functionality: Microsoft's guide, including the order of the tests.
• LearnOpenGL's Stencil testing: the OpenGL version, with glStencilFunc and glStencilOp.

Exercises

1. Turn the depth test back on for the outline pass (the last argument of create_stencil_state for outside) and fly around the cubes. Where does the outline disappear?
2. Give each cube its own border: write reference 1 for the first cube and 2 for the second, and draw each outline with the matching reference.
3. Draw the floor so that it only shows through a window: before drawing it, write 1 into the stencil with a small square (without drawing its colour, using a blend state whose RenderTargetWriteMask is 0), then draw the floor with EQUAL and reference 1.