Normal Mapping

Every surface we draw is made of flat triangles. Textures disguise that well from a distance: paint bricks onto a flat quad and it reads as a brick wall. Light it, though, and the disguise slips. A real brick wall is anything but flat. The mortar between the bricks is sunken, the bricks' edges are rounded off, their faces are pitted and cracked, and every one of those details catches the light differently. A flat quad lit by a point light is lit as a flat quad, evenly, whatever its texture shows.

Why is it lit as flat? Because, to the lighting, a surface's shape is nothing but its normal. Diffuse and specular light both come from dot products with the normal, and a flat quad has one normal everywhere (interpolated from identical normals at its corners). Every pixel of the wall answers the light the same way.

So what if each pixel had a normal of its own? Tilt the normals near a brick's top edge upwards, those near its bottom edge downwards, and those in the mortar's grooves at angles, and the lighting will treat the surface as if it had all that shape, though the geometry is still two triangles.

This is normal mapping, sometimes called bump mapping. The per-pixel normals come from a texture, the lighting equations do not change at all, and the improvement for its cost is enormous. Here is the lesson's wall with and without it:

On the left the highlight slides across the wall like a reflection on a printed photograph. On the right the light picks out the edge of every brick.

Normal mapping

To give every pixel its own normal, we store normals in a texture, as we stored diffuse and specular strength in Lighting maps. A normal is a vector of three components, x, y and z, and a texel has three colour channels, so the components go into red, green and blue. Components run from -1 to 1 and colour channels from 0 to 1, so they are squeezed on the way in:

rgb = normal * 0.5 + 0.5;          // [-1, 1] to [0, 1], when the map is made
normal = rgb * 2.0 - 1.0;          // [0, 1] to [-1, 1], when the shader reads it

Normal maps nearly always look pale blue. The normals mostly point straight out of the surface, and if "out of the surface" is +z, such a normal is (0, 0, 1), stored as (0.5, 0.5, 1.0): a light blue. The tilted normals at the bricks' edges show up as shifts towards red, green, or their opposites.

A normal map holds data, not a colour, so it must not be decoded as sRGB when it is read: the shader should see the stored values exactly. As the Gamma Correction chapter said, colour textures get the sRGB flag and data textures do not. The lesson loads its two maps accordingly:

maps: [2]*ID3D11ShaderResourceView = {
    load_texture(device, context, filesystem_path("resources/textures/brickwall.jpg"), 1),
    load_texture(device, context, filesystem_path("resources/textures/brickwall_normal.jpg"), 0),
};

and binds both at once, to registers t0 and t1, with a single call:

ID3D11DeviceContext_PSSetShaderResources(context, 0, 2, maps[0]&);

In the pixel shader, the simplest possible use reads the normal, unpacks it, and lights with it:

float3 n = normalize(normal_map.Sample(linear_wrap, p.uv).rgb * 2.0 - 1.0);
// ... light with n as usual

For a wall facing +z this works, because the map's normals point roughly along +z, and so does the wall. Now lay the same wall down as a floor, facing +y. The map's normals still point along +z, which is now sideways across the floor, and the floor is lit as if it were standing upright. The normals would need turning to match the surface, and turning differently for every surface the map is put on.

We could make one normal map per orientation, six for a cube, but not for a curved model with a different orientation at every vertex. What we need instead is to store normals relative to the surface, and turn them into place in the shader.

Tangent space

That is what the normal map really stores: normals in tangent space, a coordinate system that is local to the surface at each point. In tangent space, z points straight out of the surface, and x and y lie in the surface, along the texture's directions. A normal of (0, 0, 1) means "straight out", whatever way the surface faces in the world. To light with it, we turn it from tangent space into world space with a matrix built from three vectors at each point of the surface: the tangent T, the bitangent B and the normal N. The TBN matrix.

We already have N, the vertex normal. T and B lie in the surface, along the directions the texture runs: T is the direction in which u grows, and B is the map's y direction, which LearnOpenGL's diagram draws running up the image. We will come back to which way B really runs below, because it is where Direct3D and OpenGL part ways.

The tangent has to be computed from the mesh. Take a triangle with corners \(P_1\), \(P_2\), \(P_3\), and texture coordinates \((U_1, V_1)\) and so on. Each edge is a step across the surface, and also a step across the texture. The edge \(E_1 = P_2 - P_1\) moves \(\Delta U_1\) in u and \(\Delta V_1\) in v. If T is the world-space direction (and length) of one unit of u, and \(B_v\) that of one unit of v, then each edge is made of so many of each:

Every \(E\) and \(\Delta\) here is known from the triangle, and T and \(B_v\) are unknown: two vector equations, two unknown vectors. In matrix form:

and multiplying both sides by the inverse of the 2x2 matrix of texture steps solves it. The inverse of a 2x2 matrix is one over its determinant times its adjugate, which gives:

The top row is the tangent: \(T = f(\Delta V_2 E_1 - \Delta V_1 E_2)\), with \(f\) the one over the determinant. Don't worry if the algebra is not all clear; what matters is that the tangent follows from a triangle's positions and texture coordinates, because the texture coordinates say how the texture lies on the surface.

Manual calculation of tangents

The lesson's wall is a 2 by 2 square facing +z, made from four corners and two triangles. create_wall builds it, solving for the tangent exactly as above:

create_wall: proc(device: *ID3D11Device) -> *ID3D11Buffer = {
    p: [4]vec3 = { v3(-1.0, -1.0, 0.0), v3(1.0, -1.0, 0.0), v3(1.0, 1.0, 0.0), v3(-1.0, 1.0, 0.0) };
    uv: [4]vec2 = { v2(0.0, 1.0), v2(1.0, 1.0), v2(1.0, 0.0), v2(0.0, 0.0) };
    // The tangent of a triangle: solve for the direction in which u grows,
    // from how position and uv change along two of its edges. Both triangles
    // of a flat square share it.
    e1: vec3 = v3_sub(p[1], p[0]);
    e2: vec3 = v3_sub(p[2], p[0]);
    du1: f32 = uv[1].x - uv[0].x;
    dv1: f32 = uv[1].y - uv[0].y;
    du2: f32 = uv[2].x - uv[0].x;
    dv2: f32 = uv[2].y - uv[0].y;
    f: f32 = 1.0 / (du1 * dv2 - du2 * dv1);
    tangent: vec3 = v3_normalize(v3_scale(v3_sub(v3_scale(e1, dv2), v3_scale(e2, dv1)), f));

Note the texture coordinates: the bottom left corner (-1, -1) has uv (0, 1), and the top left corner has (0, 0). In Direct3D the first row of an image is at v = 0, so this puts the image upright on the wall without flipping anything. For this square the tangent comes out as (1, 0, 0), along +x, as you would expect: u grows to the right. Because the square is flat, both triangles share one tangent, so it is computed once.

The rest of the function writes the six vertices with a fourth attribute, the tangent, after position, normal and uv:

    order: [6]u32 = { 0, 1, 2, 0, 2, 3 };
    vertices: [66]f32 = {};
    for (i: u32 = 0; i < 6; i += 1) {
        k: u32 = order[i];
        values: [11]f32 = { p[k].x, p[k].y, p[k].z, 0.0, 0.0, 1.0, uv[k].x, uv[k].y, tangent.x, tangent.y, tangent.z };
        for (j: u32 = 0; j < 11; j += 1) { vertices[i * 11 + j] = values[j]; }
    }
    return create_buffer(device, D3D11_BIND_VERTEX_BUFFER, vertices[0]&, sizeof(vertices));
}

and the input layout gets a matching fourth element, with a semantic of its own:

// A 2x2 wall in the xy plane facing +z, with a tangent per vertex: the
// direction texture u runs in across the surface. Position, normal, uv,
// tangent; 44 bytes a vertex.
tangent_layout: [4]D3D11_INPUT_ELEMENT_DESC = {
    { .SemanticName = "POSITION", .Format = DXGI_FORMAT.DXGI_FORMAT_R32G32B32_FLOAT, .AlignedByteOffset = 0 },
    { .SemanticName = "NORMAL", .Format = DXGI_FORMAT.DXGI_FORMAT_R32G32B32_FLOAT, .AlignedByteOffset = 12 },
    { .SemanticName = "TEXCOORD", .Format = DXGI_FORMAT.DXGI_FORMAT_R32G32_FLOAT, .AlignedByteOffset = 24 },
    { .SemanticName = "TANGENT", .Format = DXGI_FORMAT.DXGI_FORMAT_R32G32B32_FLOAT, .AlignedByteOffset = 32 },
};

The wall is drawn with a stride of 44 bytes, rather than through the lessons' draw_vertices helper, which assumes 32-byte vertices.

Which way is the bitangent?

The bottom row of the solved matrix gives \(B_v\), the direction in which v grows, and LearnOpenGL computes its bitangent that way and passes it in as a fifth attribute. We do not, for two reasons.

The first is that B does not need to be stored at all. T, B and N are perpendicular, so B is the cross product of the other two, and the vertex shader can compute it. That saves twelve bytes per vertex.

The second is the sign. What B has to be is the direction the normal map's y component means, and that is a convention of the map, not a property of the triangle. In Direct3D, v grows down the image, and \(B_v\) for our wall comes out as (0, -1, 0), down the wall. A map whose green channel means "down the image" fits that directly; one whose green means "up the image" would be read upside down.

Our brick map, as it sits in its file, is of the first kind, so the lesson's bitangent points down the image, along v. For a surface facing us with T to the right, that is \(T \times N\):

// The map's y points down the image, the way v increases: the files are
// made for GL, which shows images upside down, so seen the right way up
// their +y is down. For a surface facing the viewer that is
// cross(tangent, normal).
o.b = cross(o.t, o.n);

Normal maps in the wild come in both conventions. Up-the-image green is often called the OpenGL convention (or "Y+"), and down-the-image green the DirectX convention ("Y-"), after the APIs whose texture coordinates each suits. Neither is wrong; what matters is that the shader's B matches the map it reads. You cannot tell which you have from the API, only from the map, by checking which edges of a raised feature are green.

Tangent space normal mapping

The vertex shader takes the tangent along with everything else and builds the three vectors in world space:

tangent_out vs_main(tangent_in v)
{
    tangent_out o;
    float4 world = mul(model, float4(v.position, 1.0));
    o.world = world.xyz;
    o.position = mul(projection, mul(view, world));
    o.uv = v.uv;
    o.n = normalize(mul((float3x3)normal_matrix, v.normal));
    // Gram-Schmidt: make the tangent exactly perpendicular to the normal
    float3 t = normalize(mul((float3x3)model, v.tangent));
    o.t = normalize(t - dot(t, o.n) * o.n);
    // The map's y points down the image, the way v increases: the files are
    // made for GL, which shows images upside down, so seen the right way up
    // their +y is down. For a surface facing the viewer that is
    // cross(tangent, normal).
    o.b = cross(o.t, o.n);
    return o;
}

The normal goes through the normal matrix as always. The tangent lies in the surface, so it transforms like a position difference, through the model matrix's upper 3x3. Then it is made exactly perpendicular to the normal, which we will come back to at the end of the chapter, and the bitangent is crossed from the two. All three are passed to the pixel shader as separate interpolated vectors: HLSL could pass a float3x3, but three named vectors are clearer and cost the same.

With T, B and N in world space there are two ways to use them, and LearnOpenGL shows both.

The first, which this lesson uses, turns the sampled normal from tangent space into world space, and lights as usual. In the pixel shader:

float4 ps_main(tangent_out p) : SV_Target
{
    float3 color = albedo.Sample(linear_wrap, p.uv).rgb;
    float3 n = normalize(p.n);
    if (use_map)
    {
        float3 tangent_normal = normal_map.Sample(linear_wrap, p.uv).rgb * 2.0 - 1.0;
        float3x3 tbn = float3x3(normalize(p.t), normalize(p.b), n);
        // tbn's rows are the axes, so this is t * x + b * y + n * z
        n = normalize(mul(tangent_normal, tbn));
    }
    float3 to_light = normalize(light_position - p.world);
    float3 to_eye = normalize(view_position - p.world);
    float3 halfway = normalize(to_light + to_eye);
    float diffuse = max(dot(n, to_light), 0.0);
    float spec = pow(max(dot(n, halfway), 0.0), 32.0);
    return float4((0.1 + diffuse) * color + 0.2 * spec, 1.0);
}

The interpolated vectors are renormalized first, since interpolation shortens them. The matrix multiplication needs a moment's care, because HLSL and GLSL build matrices differently. The float3x3 constructor takes its arguments as rows, so tbn has T, B and N as its rows; GLSL's mat3(T, B, N) takes them as columns. (The row_major pragma at the top of our shaders only changes how matrices are read from constant buffers; it does not affect constructors.) Multiplying a row vector on the left, mul(v, tbn), gives \(x T + y B + z N\): the tangent-space normal's components, used as amounts of each axis. That is the tangent-to-world transform, the same thing GLSL's TBN * normal does with its column matrix.

use_map comes from the light's constant buffer, and Tab flips it, which gives the comparison at the top of this chapter. The light wanders in a loop in front of the wall, and the wall itself turns slowly back and forth around y, so you can see that the lighting stays right however the surface is turned:

light: light_constants = {
    .position = v3(0.8 * cosf(current_frame), 0.8 * sinf(current_frame * 1.3), 0.6),
    .mode = use_map,
};

t.model = m4_rotate(radians(-25.0) * sinf(current_frame * 0.4), v3(0.0, 1.0, 0.0));
t.normal_matrix = m4_normal_matrix(t.model);

The second way goes the other direction: transform the light and eye vectors into tangent space and light there, where the normal from the map can be used as it is. The inverse of the TBN matrix does that, and since its rows are perpendicular unit vectors, its inverse is simply its transpose. In HLSL, with T, B and N as rows, that is mul(tbn, v): each component of the result is the dot product of v with one axis. The advantage is that the light and eye positions, and the vertex's own position, can be transformed into tangent space in the vertex shader and interpolated, which saves the matrix work per pixel. The next chapter uses this direction, because parallax mapping needs the eye direction in tangent space anyway.

The full program is here, and its shader here.

Complex objects

A single square is easy to do by hand. A model has thousands of triangles, and in practice tangents come from the model loader. LearnOpenGL asks Assimp to compute them with aiProcess_CalcTangentSpace. Our own loader, from the Model chapter, does not compute tangents, but the method is the same as create_wall's, done for every triangle:

1. For each triangle, solve for its tangent from its edges and texture coordinates.
2. Add that tangent to each of the triangle's three vertices.
3. When all triangles are done, make each vertex's sum perpendicular to the vertex normal and normalize it.

Summing across triangles gives smooth tangents over a curved surface, the way averaged normals give smooth shading. One complication: artists often mirror a texture, using the same half of an image for the left and right sides of a face, say. On the mirrored half, u runs backwards and the bitangent must be flipped. Loaders handle that by storing a fourth component in the tangent, +1 or -1, that the shader multiplies the crossed bitangent by.

Normal mapping is also a performance tool. A high-detail model of millions of triangles can be baked into a normal map for a low-detail model of a few thousand, and the two look almost the same under lighting, at a fraction of the vertices.

One last thing

When tangents are averaged over shared vertices, as above, they generally end up not quite perpendicular to the normal, and the TBN matrix is then not quite a rotation. The error is small, but easy to remove with the Gram-Schmidt process: subtract from T its component along N, which leaves only the part perpendicular to N, and normalize. That is the line we passed over in the vertex shader:

o.t = normalize(t - dot(t, o.n) * o.n);

Crossing N with the corrected T then gives a B perpendicular to both, and the three make a proper orthonormal frame. For our flat wall the tangent was perpendicular already, so it changes nothing here, but it costs very little and it makes the shader correct for any mesh.

Additional resources

• Tutorial 26: Normal mapping: ogldev's version.
• Normal mapping mathematics: TheBennyBox on the derivation, including Gram-Schmidt.
• Tutorial 13: Normal mapping: opengl-tutorial.org, with notes on mirrored texture coordinates.
• MikkTSpace: the tangent space convention most modelling tools bake normal maps in, and code to compute matching tangents.
• LearnOpenGL's Normal Mapping: the OpenGL version of this chapter.

Exercises

1. Find out which convention the brick map uses. Hold the light still above the wall, at v3(0.0, 1.5, 0.6), and look at the bricks' top and bottom edges; then flip the bitangent to the other convention, o.b = cross(o.n, o.t);, and look again. Which version lights the top edges when the light is above?
2. Draw the wall as a floor: rotate it by -90 degrees about x and put the light above it. With the normal map on, does the lighting still look right? Now replace the TBN transform with the naive n = tangent_normal; and see why tangent space is needed.
3. Compute the bitangent \(B_v\) in create_wall from the bottom row of the solved matrix, print it, and check that it points down the wall.