Face culling

Picture a cube and ask how many of its faces you can see at once. However you turn it, the answer is at most three; from straight ahead it is one. The other faces point away from you and are hidden behind the ones that face you. Yet so far we have sent all six faces through the pipeline every frame, and for every pixel they cover the pixel shader has run, only for the depth test to throw the result away (or, if the back faces happen to be drawn first, to be painted over). For a cube that is at least half of the pixel shading wasted.

Face culling skips that work. The rasterizer decides, for each triangle, whether it faces the viewer or faces away, and simply drops the ones facing away before a single pixel of them is shaded. The question is how it can tell which way a triangle faces. It cannot look at the normals; those are just data we pass to our shaders. It looks at the order of the corners.

Winding order

The three corners of a triangle, taken in the order we listed them, go around it either clockwise or counter-clockwise. That direction is the triangle's winding order:

The useful part is when the winding is judged: after the vertex shader, on the projected triangle as it lands on screen. A triangle that runs counter-clockwise when you look at its front runs clockwise when you look at it from behind, just as a clock face seen from the back of a glass clock appears to run backwards. So if every triangle of a closed mesh is listed counter-clockwise as seen from outside, then after projection the faces turned towards the camera come out counter-clockwise and the faces turned away come out clockwise:

Both triangles in LearnOpenGL's picture were listed in the same order, counter-clockwise from outside the cube. From where the viewer stands, only the near one still reads counter-clockwise. The rasterizer calls that one a front face and the other a back face, and can cull either kind.

Deciding the winding costs the rasterizer almost nothing. Once the corners have been divided by w, it has their 2D positions \((x_0, y_0)\), \((x_1, y_1)\), \((x_2, y_2)\) on the screen, and the sign of

says which way they go round. This is the signed area of the triangle: with y pointing up, as in normalized device coordinates, it is positive for counter-clockwise corners and negative for clockwise ones. (In pixel coordinates, where y points down, the signs swap; the APIs define clockwise as you see it on screen, so you never need to care.) A triangle seen exactly edge-on has an area of zero and covers no pixels anyway. Since this is computed per triangle, before any pixel is touched, a culled triangle costs only its three vertex shader runs.

Which winding counts as "front" is a convention, and the two APIs disagree. OpenGL treats counter-clockwise as front by default; Direct3D treats clockwise as front. LearnOpenGL's vertex data, and ours, is counter-clockwise from outside. Here is the first face of the cube that the lessons in this chapter share, from cube.asm:

cube_vertices label byte
              ; right (+x)
              real4   0.5,  -0.5,   0.5,     1.0,   0.0,   0.0,   0.0, 1.0
              real4   0.5,  -0.5,  -0.5,     1.0,   0.0,   0.0,   1.0, 1.0
              real4   0.5,   0.5,  -0.5,     1.0,   0.0,   0.0,   1.0, 0.0
              real4   0.5,  -0.5,   0.5,     1.0,   0.0,   0.0,   0.0, 1.0
              real4   0.5,   0.5,  -0.5,     1.0,   0.0,   0.0,   1.0, 0.0
              real4   0.5,   0.5,   0.5,     1.0,   0.0,   0.0,   0.0, 0.0
; ...

Stand on the +x side and look at it: z = +0.5 is on your left and z = -0.5 on your right. The first triangle goes bottom-left, bottom-right, top-right, which is counter-clockwise; the second goes bottom-left, top-right, top-left, counter-clockwise too. It is worth doing this exercise for a face or two of your own vertex data, because a single triangle listed the wrong way round is invisible without culling and a hole with it.

Rather than reorder all of it for Direct3D, we tell the rasterizer which winding we use. window_create in window.asm does that for every lesson, in the rasterizer state it makes at startup, as Hello Triangle did by hand:

; GL's conventions rather than D3D's: counter-clockwise triangles face the viewer, and nothing
; is culled.
invoke mem_zero, ADDR raster_desc, sizeof D3D11_RASTERIZER_DESC
mov raster_desc.FillMode, D3D11_FILL_SOLID
mov raster_desc.CullMode, D3D11_CULL_NONE
mov raster_desc.FrontCounterClockwise, 1
mov raster_desc.DepthClipEnable, 1
vcall device, ID3D11Device_CreateRasterizerState, ADDR raster_desc, ADDR raster_state
vcall context, ID3D11DeviceContext_RSSetState, raster_state

FrontCounterClockwise is OpenGL's glFrontFace(GL_CCW). The other half of the conventions is CullMode. OpenGL culls nothing until you call glEnable(GL_CULL_FACE), while Direct3D's default rasterizer state culls back faces. The lessons have kept OpenGL's behaviour, culling off, so that nothing went missing while the winding was still a mystery. It is time to turn it on.

Face culling

In Direct3D, culling is the CullMode of the rasterizer state, one of three values:

• D3D11_CULL_NONE: draw every triangle, whichever way it faces.
• D3D11_CULL_BACK: drop back faces. This is the one you want for closed, solid objects.
• D3D11_CULL_FRONT: drop front faces, so only the insides remain.

OpenGL's glCullFace also accepts GL_FRONT_AND_BACK, which culls every triangle (and so draws only points and lines); Direct3D has no such mode.

Since rasterizer states are immutable, the lesson makes one per mode at startup, from window.asm's description with a different CullMode each time. The modes are a small table, and the states go into an array of three pointers beside it:

cull_modes dword D3D11_CULL_NONE, D3D11_CULL_BACK, D3D11_CULL_FRONT

cull_states qword 3 dup(0)      ; ID3D11RasterizerState for each of cull_modes

; window.asm's rasterizer state, with each of the three cull modes
invoke mem_zero, ADDR raster_desc, sizeof D3D11_RASTERIZER_DESC
mov raster_desc.FillMode, D3D11_FILL_SOLID
mov raster_desc.FrontCounterClockwise, 1
mov raster_desc.DepthClipEnable, 1
lea rsi, cull_modes
lea rdi, cull_states
xor ebx, ebx
states:
    mov eax, [rsi + rbx * 4]
    mov raster_desc.CullMode, eax
    lea rax, [rdi + rbx * 8]
    vcall device, ID3D11Device_CreateRasterizerState, ADDR raster_desc, rax
    inc ebx
    cmp ebx, 3
    jb states

The loop reads the ebx-th mode (four bytes each) and passes the address of the ebx-th pointer (eight bytes each) for the state to be written to. Both tables are globals, so their addresses go into rsi and rdi with lea before they are indexed: an index register cannot be combined with the instruction-relative address a global normally has.

Tab steps through none, back and front, through window_mode. The program starts in mode 2, culling front faces, because that is the mode where something visibly happens:

invoke window_mode_default, 2

invoke window_mode, 3
mov ebx, eax

In the render loop, the chosen state is bound around the cubes only:

invoke bind_texture, 0, floor_texture
invoke m4_identity, ADDR model
invoke draw_vertices, transforms_buffer, ADDR t, plane_vb, 6, ADDR model

; the cubes with this mode's culling
lea rax, cull_states
mov rax, [rax + rbx * 8]
vcall context, ID3D11DeviceContext_RSSetState, rax
invoke bind_texture, 0, cube_texture
invoke m4_translate, ADDR model, ADDR cube_a
invoke draw_vertices, transforms_buffer, ADDR t, cube_vb, 36, ADDR model
invoke m4_translate, ADDR model, ADDR cube_b
invoke draw_vertices, transforms_buffer, ADDR t, cube_vb, 36, ADDR model
vcall context, ID3D11DeviceContext_RSSetState, raster_state

The floor keeps the default state without culling. It is a single flat quad, not a closed shape, so there is no "inside" for it to hide; with back faces culled it would simply vanish when seen from below. The same goes for the grass and windows of the previous chapter, which should be visible from both sides. Culling is for closed meshes.

With back faces culled, the picture does not change at all, which is exactly the point: the faces we skip are the ones we could never see. On the left, back-face culling; on the right, front-face culling:

Culling the front faces removes the walls facing us and leaves the far walls, seen from inside: the cubes become open boxes. That is the same image LearnOpenGL gets by flipping the front-face convention to clockwise, and you can get it the same way here by setting FrontCounterClockwise to 0 and culling back faces (try it in the exercises). Notice also the stripes on the boxes' floors: that is the z-fighting between each cube's bottom face and the metal floor, described in Depth testing. Fly inside a cube with back faces culled and its walls disappear around you, since from inside every wall faces away.

How much does culling save? For a convex shape such as our cube, the back faces are always hidden behind front ones, so the saving is all the pixel shading those faces would have cost, which can be half or more. If the front faces happened to be drawn first, early depth testing would already have rejected most of the back faces' pixels, and the saving is smaller; but culling also skips rasterizing those triangles at all, and it does not depend on drawing order. Meshes exported from modelling tools come with consistent winding, almost always counter-clockwise, so turning culling on for them is nearly free. The one thing to keep track of is which objects are open or two-sided, and must be drawn without it.

Front or back in the shader

The rasterizer's decision is also available to the pixel shader, which helps with the objects that must not be culled. A pixel shader input marked SV_IsFrontFace, a bool, says whether the triangle being shaded was a front face; it is GLSL's gl_FrontFacing. A two-sided leaf could use it to flip its normal on the back, so that both sides are lit correctly, or a cutaway view could paint the inside of a mesh in a different colour. With FrontCounterClockwise set, "front" means the same here as for culling. Advanced HLSL uses it to draw the insides of the cubes as a checkerboard.

The full program is here, and its shader here.

Additional resources

• D3D11_RASTERIZER_DESC: the rasterizer state, with the defaults of each field.
• Getting started with the rasterizer stage: Microsoft's overview of the stage that does the culling.
• LearnOpenGL's Face culling: the OpenGL version, with glCullFace and glFrontFace.

Exercises

1. Set FrontCounterClockwise to 0 in the lesson's raster_desc and step through the three modes again. Which mode now shows the insides, and why?
2. List the cube's triangles clockwise from outside instead (swap the second and third vertex of every triangle), and find the rasterizer settings that make back-face culling work with the new data.
3. Draw the floor with back-face culling too, and fly below it. Then work out which way the floor's triangles wind as seen from above.