Geometry Shaders
The geometry shader is unique in contrast to the other shader types in that it processes a whole primitive (triangle, line, or point) at once and can actually change the amount of data in the OpenGL pipeline programmatically. A vertex shader processes one vertex at a time; it cannot access any other vertex’s information and is strictly onein, oneout. That is, it cannot generate new vertices, and it cannot stop the vertex from being processed further by OpenGL. The tessellation shaders operate on patches and can set tessellation factors, but have little further control over how patches are tessellated, and cannot produce disjoint primitives. Likewise, the fragment shader processes a single fragment at a time, cannot access any data owned by another fragment, cannot create new fragments, and can only destroy fragments by discarding them. On the other hand, a geometry shader has access to all of the vertices in a primitive (up to six with the primitive modes GL_TRIANGLES_ADJACENCY and GL_TRIANGLE_STRIP_ADJACENCY), can change the type of a primitive, and can even create and destroy primitives.
Geometry shaders are an optional part of the OpenGL pipeline. When no geometry shader is present, the outputs from the vertex or tessellation evaluation shader are interpolated across the primitive being rendered and are fed directly to the fragment shader. When a geometry shader is present, however, the outputs of the vertex or tessellation evaluation shader become the inputs to the geometry shader, and the outputs of the geometry shader are what are interpolated and fed to the fragment shader. The geometry shader can further process the output of the vertex or tessellation evaluation shader, and if it is generating new primitives (this is called amplification), it can apply different transformations to each primitive as it creates them.
The PassThrough Geometry Shader
As explained back in Chapter 3, “Following the Pipeline,” the simplest geometry shader that allows you to render anything is the passthrough shader, which is shown in Listing 8.16.
Listing 8.16. Source code for a simple geometry shader
#version
430 core layout (triangles) in; layout (triangle_strip) out; layout (max_vertices = 3) out;void
main(void
) {int
i;for
(i = 0; i < gl_in.length(); i++) { gl_Position = gl_in[i].gl_Position; EmitVertex(); } EndPrimitive(); }
This is a very simple passthrough geometry shader, which sends its input to its output without modifying it. It looks similar to a vertex shader, but there are a few extra differences to cover. Going over the shader a few lines at a time makes everything clear. The first few lines simply set up the version number (430) of the shader just like in any other shader. The next couple of lines are the first geometry shaderspecific parts. They are shown again in Listing 8.17.
Listing 8.17. Geometry shader layout qualifiers
#version
430 core
layout (triangles) in;
layout (triangle_strip) out;
layout (max_vertices = 3) out;
These set the input and output primitive modes using a layout qualifier. In this particular shader we’re using triangles
for the input and triangle_strip
for the output. Other primitive types, along with the layout qualifier, are covered later. For the geometry shader’s output, not only do we specify the primitive type, but the maximum number of vertices expected to be generated by the shader (through the max_vertices
qualifier). This shader produces individual triangles (generated as very short triangle strips), so we specified 3 here.
Next is our main() function, which is again similar to what might be seen in a vertex or fragment shader. The shader contains a loop, and the loop runs a number of times determined by the length of the builtin array, gl_in. This is another geometry shaderspecific variable. Because the geometry shader has access to all of the vertices of the input primitive, the input has to be declared as an array. All of the builtin variables that are written by the vertex shader (such as gl_Position) are placed into a structure, and an array of these structures is presented to the geometry shader in a variable called gl_in.
The length of the gl_in[] array is determined by the input primitive mode, and because in this particular shader, triangles are the input primitive mode, the size of gl_in[] is three. The inner loop is given again in Listing 8.18.
Listing 8.18. Iterating over the elements of gl_in[]
for
(i = 0; i < gl_in.length(); i++)
{
gl_Position = gl_in[i].gl_Position;
EmitVertex();
}
Inside our loop, we’re generating vertices by simply copying the elements of gl_in[] to the geometry shader’s output. A geometry shader’s outputs are similar to the vertex shader’s outputs. Here, we’re writing to gl_Position, just as we would in a vertex shader. When we’re done setting up all of the new vertex’s attributes, we call EmitVertex(). This is a builtin function, specific to geometry shaders that tells the shader that we’re done with our work for this vertex and that it should store all that information away and prepare to start setting up the next vertex.
Finally, after the loop has finished executing, there’s a call to another special, geometry shaderonly function, EndPrimitive(). EndPrimitive() tells the shader that we’re done producing vertices for the current primitive and to move on to the next one. We specified triangle_strip
as the output for our shader, and so if we continue to call EmitVertex() more than three times, OpenGL continues adding triangles to the triangle strip. If we need our geometry shader to generate separate, individual triangles or multiple, unconnected triangle strips (remember, geometry shaders can create new or amplify geometry), we could call EndPrimitive() between each one to mark their boundaries. If you don’t call EndPrimitive() somewhere in your shader, the primitive is automatically ended when the shader ends.
Using Geometry Shaders in an Application
Geometry shaders, like the other shader types, are created by calling the glCreateShader() function and using GL_GEOMETRY_SHADER as the shader type, as follows:
glCreateShader(GL_GEOMETRY_SHADER);
Once the shader has been created, it is used like any other shader object. You give OpenGL your shader source code by calling glShaderSource(), compile the shader using the glCompileShader() function, and attach it to a program object by calling the glAttachShader() function. Then the program is linked as normal using the glLinkProgram() function. Now that you have a program object with a geometry shader linked into it, when you draw geometry using a function like glDrawArrays(), the vertex shader will run once per vertex, the geometry shader will run once per primitive (point, line, or triangle), and the fragment will run once per fragment. The primitives received by a geometry shader must match what it is expecting based in its own input primitive mode. When tessellation is not active, the primitive mode you use in your drawing commands must match the input primitive mode of the geometry shader. For example, if the geometry shader’s input primitive mode is points, then you may only use GL_POINTS when you call glDrawArrays(). If the geometry shader’s input primitive mode is triangles, then you may use GL_TRIANGLES, GL_TRIANGLE_STRIP, or GL_TRIANGLE_FAN in your glDrawArrays() call. A complete list of the geometry shader input primitive modes and the allowed geometry types is given in Table 8.1.
Table 8.1. Allowed Draw Modes for Geometry Shader Input Modes
Geometry Shader Input Mode 
Allowed Draw Modes 

GL_POINTS 

GL_LINES, GL_LINE_LOOP, GL_LINE_STRIP 

GL_TRIANGLES, GL_TRIANGLE_FAN, GL_TRIANGLE_STRIP 

GL_LINES_ADJACENCY 

GL_TRIANGLES_ADJACENCY 
When tessellation is active, the mode you use in your drawing commands should always be GL_PATCHES, and OpenGL will convert the patches into points, lines, or triangles during the tessellation process. In this case, the input primitive mode of the geometry shader should match the tessellation primitive mode. The input primitive type is specified in the body of the geometry shader using a layout qualifier. The general form of the input layout qualifier is
layout (primitive_type) in;
This specifies that primitive_type is the input primitive type that the geometry shader is expected to handle, and primitive_type must be one of the supported primitive modes: points
, lines
, triangles
, lines_adjacency
, or triangles_adjacency
. The geometry shader runs once per primitive. This means that it’ll run once per point for GL_POINTS; once per line for GL_LINES, GL_LINE_STRIP, and GL_LINE_LOOP; and once per triangle for GL_TRIANGLES, GL_TRIANGLE_STRIP, and GL_TRIANGLE_FAN. The inputs to the geometry shader are presented in arrays containing all of the vertices making up the input primitive. The predefined inputs are stored in a builtin array called gl_in[], which is an array of structures defined in Listing 8.19.
Listing 8.19. The definition of gl_in[]
in
gl_PerVertex {vec4
gl_Position;float
gl_PointSize;float
gl_ClipDistance[]; } gl_in[];
The members of this structure are the builtin variables that are written in the vertex shader: gl_Position, gl_PointSize, and gl_ClipDistance[]. You should recognize this structure from its declaration as an output block in the vertex shader described earlier in this chapter. These variables appear as global variables in the vertex shader because the block doesn’t have an instance name there, but their values end up in the gl_in[] array of block instances when they appear in the geometry shader. Other variables written by the vertex shader also become arrays in the geometry shader. In the case of individual varyings, outputs in the vertex shader are declared as normal, and the inputs to the geometry shader have a similar declaration, except that they are arrays. Consider a vertex shader that defines outputs as
out vec4 color; out vec3 normal;
The corresponding input to the geometry shader would be
in vec4 color[]; in vec3 normal[];
Notice that both the color and normal varyings have become arrays in the geometry shader. If you have a large amount of data to pass from the vertex to the geometry shader, it can be convenient to wrap pervertex information passed from the vertex shader to the geometry shader into an interface block. In this case, your vertex shader will have a definition like this:
out VertexData { vec4 color; vec3 normal; } vertex;
And the corresponding input to the geometry shader would look like this:
in VertexData { vec4 color; vec3 normal; // More pervertex attributes can be inserted here } vertex[];
With this declaration, you’ll be able to access the pervertex data in the geometry shader using vertex[n].color and so on. The length of the input arrays in the geometry shader depends on the type of primitives that it will process. For example, points are formed from a single vertex, and so the arrays will only contain a single element, whereas triangles are formed from three vertices, and so the arrays will be three elements long. If you’re writing a geometry shader that’s designed specifically to process a particular primitive type, you can explicitly size your input arrays, which provides a small amount of additional compiletime error checking. Otherwise, you can let your arrays be automatically sized by the input primitive type layout qualifier. A complete mapping of the input primitive modes and the resulting size of the input arrays is shown in Table 8.2.
Table 8.2. Sizes of Input Arrays to Geometry Shaders
Input Primitive Type 
Size of Input Arrays 

1 

2 

3 

4 

6 
You also need to specify the primitive type that will be generated by the geometry shader. Again, this is determined using a layout qualifier, like so:
layout (primitive_type) out;
This is similar to the input primitive type layout qualifier, the only difference being that you are declaring the output of the shader using the out keyword. The allowable output primitive types from the geometry shader are points
, line_strip
, and triangle_strip
. Notice that geometry shaders only support outputting the strip primitive types (not counting points—obviously, there is no such thing as a point strip).
There is one final layout qualifier that must be used to configure the geometry shader. Because a geometry shader is capable of producing a variable amount of data per vertex, OpenGL must be told how much space to allocate for all that data by specifying the maximum number of vertices that the geometry shader is expected to produce. To do this, use the following layout qualifier:
layout (max_vertices = n) out;
This sets the maximum number of vertices that the geometry shader may produce to n. Because OpenGL may allocate buffer space to store intermediate results for each vertex, this should be the smallest number possible that still allows your application to run correctly. For example, if you are planning to take points and produce one line at a time, then you can safely set this to two. This gives the shader hardware the best opportunity to run fast. If you are going to heavily tessellate the incoming geometry, you might want to set this to a much higher number, although this may cost you some performance. The upper limit on the number of vertices that a geometry shader can produce depends on your OpenGL implementation. It is guaranteed to be at least 256, but the absolute maximum can be found by calling glGetIntegerv() with the GL_MAX_GEOMETRY_OUTPUT_VERTICES parameter.
You can also declare more than one layout qualifier with a single statement by separating them with a comma, like so:
layout (triangle_strip, max_vertices = n) out;
With these layout qualifiers, a boilerplate #version
declaration, and an empty main() function, you should be able to produce a geometry shader that compiles and links but does absolutely nothing. In fact, it will discard any geometry you send it, and nothing will be drawn by your application. We need to introduce two important functions: EmitVertex() and EndPrimitive(). If you don’t call these, nothing will be drawn.
EmitVertex() tells the geometry shader that you’ve finished filling in all of the information for this vertex. Setting up the vertex works much like the vertex shader. You need to write into the builtin variable gl_Position. This sets the clipspace coordinates of the vertex that is produced by the geometry shader, just like in a vertex shader. Any other attributes that you want to pass from the geometry shader to the fragment shader can be declared in an interface block or as global variables in the geometry shader. Whenever you call EmitVertex, the geometry shader stores the values currently in all of its output variables and uses them to generate a new vertex. You can call EmitVertex() as many times as you like in a geometry shader, until you reach the limit you specified in your max_vertices
layout qualifier. Each time, you put new values into your output variables to generate a new vertex.
An important thing to note about EmitVertex() is that it makes the values of any of your output variables (such as gl_Position) undefined. So, for example, if you want to emit a triangle with a single color, you need to write that color with every one of your vertices; otherwise, you will end up with undefined results.
EmitPrimitive() indicates that you have finished appending vertices to the end of the primitive. Don’t forget, geometry shaders only support the strip primitive types (line_strip
and triangle_strip
). If your output primitive type is triangle_strip
and you call EmitVertex() more than three times, the geometry shader will produce multiple triangles in a strip. Likewise, if your output primitive type is line_strip
and you call EmitVertex() more than twice, you’ll get multiple lines. In the geometry shader, EndPrimitive() refers to the strip. This means that if you want to draw individual lines or triangles, you have to call EndPrimitive() after every two or three vertices. You can also draw multiple strips by calling EmitVertex() many times between multiple calls to EndPrimitive().
One final thing to note about calling EmitVertex() and EndPrimitive() in the geometry shader is that if you haven’t produced enough vertices to produce a single primitive (e.g., you’re generating triangle_strip
outputs and you call EndPrimitive() after two vertices), nothing is produced for that primitive, and the vertices you’ve already produced are simply thrown away.
Discarding Geometry in the Geometry Shader
The geometry shader in your program runs once per primitive. What you do with that primitive is entirely up to you. The two functions EmitVertex() and EndPrimitive() allow you to programmatically append new vertices to your triangle or line strip and to start new strips. You can call them as many times as you want (until you reach the maximum defined by your implementation). You’re also allowed to not call them at all. This allows you to clip geometry away and discard primitives. If your geometry shader runs and you never call EmitVertex() for that particular primitive, nothing will be drawn. To illustrate this, we can implement a custom backface culling routine that culls geometry as if it were viewed from an arbitrary point in space. This is implemented in the gsculling example.
First, we set up our shader version and declare our geometry shader to accept triangles and to produce triangle strips. Backface culling doesn’t really make a lot of sense for lines or points. We also define a uniform that will hold our custom viewpoint in world space. This is shown in Listing 8.20.
Listing 8.20. Configuring the custom culling geometry shader
#version
330 // Input is triangles, output is triangle strip. Because we're going // to do a 1 in 1 out shader producing a single triangle output for // each one input, max_vertices can be 3 here.layout
(triangles
)in
;layout
(triangle_strip
,max_vertices
=3)out
; // Uniform variables that will hold our custom viewpoint and // modelview matrixuniform vec3
viewpoint;uniform
mav4 mv_matrix;
Now inside our main() function, we need to find the face normal for the triangle. This is simply the cross products of any two vectors in the plane of the triangle—we can use the triangle edges for this. Listing 8.21 shows how this is done.
Listing 8.21. Finding a face normal in a geometry shader
// Calculate two vectors in the plane of the input trianglevec3
ab = gl_in[1].gl_Position.xyz  gl_in[0].gl_Position.xyz;vec3
ac = gl_in[2].gl_Position.xyz  gl_in[0].gl_Position.xyz;vec3
normal = normalize(cross(ab, ac));
Now that we have the normal, we can determine whether it faces toward or away from our userdefined viewpoint. To do this, we need to transform the normal into the same coordinate space as the viewpoint, which is world space. Assuming we have the modelview matrix in a uniform, simply multiply the normal by this matrix. To be more accurate, we should multiply the vector by the inverse of the transpose of the upperleft 3 × 3 submatrix of the modelview matrix. This is known as the normal matrix, and you’re free to implement this and put it in its own uniform if you like. However, if your modelview matrix only contains translation, uniform scale (no shear), and rotation, you can use it directly. Don’t forget, the normal is a threeelement vector, and the modelview matrix is a 4 × 4 matrix. We need to extend the normal to a fourelement vector before we can multiply the two. We can then take the dot product of the resulting vector with the vector from the viewpoint to any point on the triangle.
If the sign of the dot product is negative, that means that the normal is facing away from the viewer and the triangle should be culled. If it is positive, the triangle’s normal is pointing toward the viewer, and we should pass the triangle on. The code to transform the face normal, perform the dot product, and test the sign of the result is shown in Listing 8.22.
Listing 8.22. Conditionally emitting geometry in a geometry shader
// Calculate the transformed face normal and the view direction vectorvec3
transformed_normal = (vec4
(normal, 0.0) * mv_matrix).xyz;vec3
vt = normalize(gl_in[0].gl_Position.xyz  viewpoint); // Take the dot product of the normal with the view directionfloat
d = dot(vt, normal); // Emit a primitive only if the sign of the dot product is positiveif
(d > 0.0) {for
(int
i = 0; i < 3; i++) { gl_Position = gl_in[i].gl_Position; EmitVertex(); } EndPrimitive(); }
In Listing 8.22, if the dot product is positive, we copy the input vertices to the output of the geometry shader and call EmitVertex() for each one. If the dot product is negative, we simply don’t do anything at all. This results in the incoming triangle being discarded altogether and nothing being drawn.
In this particular example, we are generating at most one triangle output for each triangle input to the geometry shader. Although the output of the geometry shader is a triangle strip, our strips only contain a single triangle. Therefore, there doesn’t strictly need to be a call to EndPrimitive(). We just leave it there for completeness.
Figure 8.16 shows a the result of this shader.
Figure 8.16. Geometry culled from different viewpoints
In Figure 8.16, the virtual viewer has been moved to different positions. As you can see, different parts of the model have been culled away by the geometry shader. It’s not expected that this example is particularly useful, but it does demonstrate the ability for a geometry shader to perform geometry culling based on applicationdefined criteria.
Modifying Geometry in the Geometry Shader
The previous example either discarded geometry or passed it through unmodified. It is also possible to modify vertices as they pass through the geometry shader to create new, derived shapes. Even though your geometry shader is passing vertices on onetoone (i.e., no amplification or culling is taking place), this still allows you to do things that would otherwise not be possible with a vertex shader alone. If the input geometry is in the form of triangle strips or fans, for example, the resulting geometry will have shared vertices and shared edges. Using the vertex shader to move shared vertices will move all of the triangles that share that vertex. It is not possible, then, to separate two triangles that share an edge in the original geometry using the vertex shader alone. However, this is trivial using the geometry shader.
Consider a geometry shader that accepts triangles and produces triangle_strip
as output. The input to a geometry shader that accepts triangles is individual triangles, regardless of whether they originated from a glDrawArrays() or a glDrawElements() function call, or whether the primitive type was GL_TRIANGLES, GL_TRIANGLE_STRIP, or GL_TRIANGLE_FAN. Unless the geometry shader outputs more than three vertices, the result is independent, unconnected triangles.
In this next example, we “explode” a model by pushing all of the triangles out along their face normals. It doesn’t matter whether the original model is drawn with individual triangles or with triangle strips or fans. As with the previous example, the input is triangles, the output is triangle_strip
, and the maximum number of vertices produced by the geometry shader is three because we’re not amplifying or decimating geometry. The setup code for this is shown in Listing 8.23.
Listing 8.23. Setting up the “explode” geometry shader
#version
330 // Input is triangles, output is triangle strip. Because we're going to do a // 1 in 1 out shader producing a single triangle output for each one input, // max_vertices can be 3 here.layout
(triangles
)in
;layout
(triangle_strip
,max_vertices
=3)out
;
To project the triangle outward, we need to calculate the face normal of each triangle. Again, to do this we can take the cross product of two vectors in the plane of the triangle—two edges of the triangle. For this, we can reuse the code from Listing 8.21. Now that we have the triangle’s face normal, we can project vertices along that normal by an applicationcontrolled amount. That amount can be stored in a uniform (we call it explode_factor) and updated by the application. This simple code is shown in Listing 8.24.
Listing 8.24. Pushing a face out along its normal
for
(int
i = 0; i < 3; i++) { gl_Position = gl_in[i].gl_Position +vec4
(explode_factor * normal, 0.0); }
The result of running this geometry shader on a model is shown in Figure 8.17. The model has been deconstructed, and the individual triangles have become visible.
Figure 8.17. Exploding a model using the geometry shader
Generating Geometry in the Geometry Shader
Just as you are not required to call EmitVertex() or EndPrimitive() at all if you don’t want to produce any output from the geometry shader, it is also possible to call EmitVertex() and EndPrimitive() as many times as you need to produce new geometry. That is, until you reach the maximum number of output vertices that you declared at the start of your geometry shader. This functionality can be used for things like making multiple copies of the input or breaking the input into smaller pieces. This is the subject of the next example, which is the gstessellate sample in the book’s accompanying source code. The input to our shader is a tetrahedron centered around the origin. Each face of the tetrahedron is made from a single triangle. We tessellate incoming triangles by producing new vertices halfway along each edge and then moving all of the resulting vertices so that they are variable distances from the origin. This transforms our tetrahedron into a spiked shape.
Because the geometry shader operates in object space (remember, the tetrahedron’s vertices are centered around the origin), we need to do no coordinate transforms in the vertex shader and, instead, do the transforms in the geometry shader after we’ve generated the new vertices. To do this, we need a simple, passthrough vertex shader. Listing 8.25 shows a simple passthrough vertex shader.
Listing 8.25. Passthrough vertex shader
#version
330in vec4
position;void
main(void
) { gl_Position = position; }
This shader only passes the vertex position to the geometry shader. If you have other attributes associated with the vertices such as texture coordinates or normals, you need to pass them through the vertex shader to the geometry shader as well.
As in the previous example, we accept triangles as input to the geometry shader and produce a triangle strip. We break the strip after every triangle so that we can produce separate, independent triangles. In this example, we produce four output triangles for every input triangle. We need to declare our maximum output vertex count as 12—four triangles times three vertices. We also need to declare a uniform matrix to store the modelview transformation matrix in the geometry shader because we do that transform after generating vertices. Listing 8.26 shows this code.
Listing 8.26. Setting up the “tessellator” geometry shader
#version
430 corelayout
(triangles
)in
;layout
(triangle_strip
,max_vertices
= 12)out
; // A uniform to store the modelviewprojection matrixuniform mat4
mvp;
First, let’s copy the incoming vertex coordinates into a local variable. Then, given the original, incoming vertices, we find the midpoint of each edge by taking their average. In this case, however, rather than simply dividing by two, we multiply by a scale factor, which will allow us to alter the spikiness of the resulting object. Code to do this is shown in Listing 8.27.
Listing 8.27. Generating new vertices in a geometry shader
// Copy the incoming vertex positions into some local variablesvec3
a = gl_in[0].gl_Position.xyz;vec3
b = gl_in[1].gl_Position.xyz;vec3
c = gl_in[2].gl_Position.xyz; // Find a scaled version of their midpointsvec3
d = (a + b) * stretch;vec3
e = (b + c) * stretch;vec3
f = (c + a) * stretch; // Now, scale the original vertices by an inverse of the midpoint // scale a *= (2.0  stretch); b *= (2.0  stretch); c *= (2.0  stretch);
Because we are going to generate several triangles using almost identical code, we can put that code into a function (shown in Listing 8.28) and call it from our main tessellation function.
Listing 8.28. Emitting a single triangle from a geometry shader
void
make_face(vec3
a,vec3
b,vec3
c) {vec3
face_normal = normalize(cross(c  a, c  b));vec4
face_color =vec4
(1.0, 0.2, 0.4, 1.0) * (mat3
(mvMatrix) * face_normal gl_Position = mvpMatrix *vec4
(a, 1.0); color = face_color; EmitVertex(); gl_Position = mvpMatrix *vec4
(b, 1.0); color = face_color; EmitVertex(); gl_Position = mvpMatrix *vec4
(c, 1.0); color = face_color; EmitVertex(); EndPrimitive(); }
Notice that the make_face function calculates a face color based on the face’s normal in addition to emitting the positions of its vertices. Now, we simply call make_face four times from our main function, which is shown in Listing 8.29.
Listing 8.29. Using a function to produce faces in a geometry shader
make_face(a, d, f); make_face(d, b, e); make_face(e, c, f); make_face(d, e, f);
Figure 8.18 shows the result of our simple geometry shaderbased tessellation program.
Figure 8.18. Basic tessellation using the geometry shader
Note that using the geometry shader for heavy tessellation may not produce the most optimal performance. If something more complex than that shown in this example is desired, it’s best to use the hardware tessellation functions of OpenGL. However, if simple amplification of between two and four output primitives for each input primitive is desired, the geometry shader is probably the way to go.
Changing the Primitive Type in the Geometry Shader
So far, all of the geometry shader examples we’ve gone through have taken triangles as input and produced triangle strips as output. This doesn’t change the geometry type. However, geometry shaders can input and output different types of geometry. For example, you can transform points into triangles or triangles into points. In the normalviewer example, which we’ll describe next, we’re going to change the geometry type from triangles to lines. For each vertex input to the shader, we take the vertex normal and represent it as a line. We also take the face normal and represent that as another line. This allows us to visualize the model’s normals—both at each vertex and for each face. Note, though, that if you want to draw the normals on top of the original model, you need to draw everything twice—once with the geometry shader to visualize the normals and once without the geometry shader to show the model. You can’t output a mix of two different primitives from a single geometry shader.
For our geometry shader, in addition to the members of the gl_in structure, we need the pervertex normal, and that will have to be passed through the vertex shader. An updated version of the passthrough vertex shader from Listing 8.25 is given in Listing 8.30.
Listing 8.30. A passthrough vertex shader that includes normals
#version
330in vec4
position;in vec3
normal;out
Vertex {vec3
normal; } vertex;void
main(void
) { gl_Position = position; vertex.normal = normal; }
This passes the position attribute straight through to the gl_Position builtin variable and places the normal into an output block.
The setup code for the geometry shader is shown in Listing 8.31. In this example, we accept triangles and produce line strips, each of a single line. Because we output a separate line for each normal we visualize, we produce two vertices for each vertex consumed, plus two more for the face normal. Therefore, the maximum number of vertices that we output per input triangle is eight. To match the Vertex output block that we declared in the vertex shader, we also need to declare a corresponding input interface block in the geometry shader. As we’re going to do the objectspacetoworldspace transformation in the geometry shader, we declare a mat4 uniform called mvp to represent the modelviewprojection matrix. This is necessary so that we can keep the vertex’s position in the same coordinate system as its normal until we produce the new vertices representing the line.
Listing 8.31. Setting up the “normal visualizer” geometry shader
#version
330layout
(triangles
)in
;layout
(line_strip
)out
;layout
(max_vertices = 8
)out
;in
Vertex {vec3
normal; } vertex[]; // Uniform to hold the modelviewprojection matrixuniform mat4
mvp; // Uniform to store the length of the visualized normalsuniform float
normal_length;
Each input vertex is transformed into its final position and emitted from the geometry shader, and then a second vertex is produced by displacing the input vertex along its normal and transforming that into its final position as well. This makes the length of all of our normals one but allows any scaling encoded in our modelviewprojection matrix to be applied to them along with the model. We multiply the normals by the applicationsupplied uniform normal_length, allowing them to be scaled to match the model. Our inner loop is shown in Listing 8.32.
Listing 8.32. Producing lines from normals in the geometry shader
gl_Position = mvp * gl_in[0].gl_Position;
gs_out.normal = gs_in[0].normal;
gs_out.color = gs_in[0].color;
EmitVertex();
gl_Position = mvp * (gl_in[0].gl_Position +
vec4
(gs_in[0].normal * normal_length, 0.0));
gs_out.normal = gs_in[0].normal;
gs_out.color = gs_in[0].color;
EmitVertex();
EndPrimitive();
This generates a short line segment at each vertex pointing in the direction of the normal. Now, we need to produce the face normal. To do this, we need to pick a suitable place from which to draw the normal, and we need to calculate the face normal itself in the geometry shader along which to draw the line.
As in the earlier example given in Listing 8.33, we use a cross product of two of the triangle’s edges to find the face normal. To pick a starting point for the line, we choose the centroid of the triangle, which is simply the average of the coordinates of the input vertices. Listing 8.33 shows the shader code.
Listing 8.33. Drawing a face normal in the geometry shader
vec3
ab = gl_in[1].gl_Position.xyz  gl_in[0].gl_Position.xyz;vec3
ac = gl_in[2].gl_Position.xyz  gl_in[0].gl_Position.xyz;vec3
face_normal = normalize(cross(ab, ac));vec4
tri_centroid = (gl_in[0].gl_Position + gl_in[1].gl_Position + gl_in[2].gl_Position) / 3.0; gl_Position = mvp * tri_centroid; gs_out.normal = gs_in[0].normal; gs_out.color = gs_in[0].color; EmitVertex(); gl_Position = mvp * (tri_centroid +vec4
(face_normal * normal_length, 0.0)); gs_out.normal = gs_in[0].normal; gs_out.color = gs_in[0].color; EmitVertex(); EndPrimitive();
Now when we render a model, we get the image shown in Figure 8.19.
Figure 8.19. Displaying the normals of a model using a geometry shader
Multiple Streams of Storage
When only a vertex shader is present, there is a simple onein, oneout relationship between the vertices coming into the shader and the vertices stored in the transform feedback buffer. When a geometry shader is present, each shader invocation may store zero, one, or more vertices into the bound transform feedback buffers. Not only this, but it’s actually possible to configure up to four output streams and use the geometry shader to send its output to whichever one it chooses. This can be used, for example, to sort geometry or to render some primitives while storing other geometry in transform feedback buffers. There are a couple of pretty major limitations when multiple output streams are used in a geometry shader; first, the output primitive mode from the geometry shader for all streams must be set to points
. Second, although it’s possible to simultaneously render geometry and to store data into transform feedback buffers, only the first stream may be rendered — the others are for storage only. If your application fits with these constraints, then this can be a very powerful feature.
To set up multiple output streams from your geometry shader, use the stream layout qualifier to select one of four streams. As with most other output layout qualifiers, the stream qualifier may be applied directly to a single output or to an output block. It can also be applied directly to the out
keyword without declaring an output variable, in which case it will affect all further output declarations until another stream layout qualifier is encountered. For example, consider the following output declarations in a geometry shader:
out vec4
foo; // "foo" is in stream 0 (the default).layout
(stream=2)out vec4
bar; // "bar" is part of stream 2.out vec4
baz; // "baz" is back in stream 0.layout
(stream=1)out
; // Everything from here on is in stream 1.out int
apple; // "apple" and "orange" are partout int
orange; // of stream 1.layout
(stream=3)out
MY_BLOCK // Everything in "MY_BLOCK" is in stream 3. {vec3
purple;vec3
green; };
In the geometry shader, when you call EmitVertex(), the vertex will be recorded into the first output stream (stream 0). Likewise, when you call EndPrimitive(), it will end the primitive being recorded to stream 0. However, you can call EmitStreamVertex() and EndStreamPrimitive(), both of which take an integer argument specifying the stream to send the output to:
void
EmitStreamVertex(int stream
);void
EndStreamPrimitive(int stream
);
The stream argument must be a compile time constant. If rasterization is enabled, then any primitives sent to stream 0 will be rasterized.
New Primitive Types Introduced by the Geometry Shader
Four new primitive types were introduced with geometry shaders: GL_LINES_ADJACENCY, GL_LINE_STRIP_ADJACENCY, GL_TRIANGLES_ADJACENCY, and GL_TRIANGLE_STRIP_ADJACENCY. These primitive types are really only useful when rendering with a geometry shader active. When the new adjacency primitive types are used, for each line or triangle passed into the geometry shader, it not only has access to the vertices defining that primitive, but it also has access to the vertices of the primitive that is next to the one it’s processing.
When you render using GL_LINES_ADJACENCY, each line segment consumes four vertices from the enabled attribute arrays. The two center vertices make up the line; the first and last vertices are considered the adjacent vertices. The inputs to the geometry shader are therefore fourelement arrays. In fact, because the input and output types of the geometry shader do not have to be related, GL_LINES_ADJACENCY can be seen as a way of sending generalized fourvertex primitives to the geometry shader. The geometry shader is free to transform them into whatever it pleases. For example, your geometry shader could convert each set of four vertices into a triangle strip made up of two triangles. This allows you to render quads using the GL_LINES_ADJACENCY primitive. It should be noted, though, that if you draw using GL_LINES_ADJACENCY when no geometry shader is active, regular lines will be drawn using the two innermost vertices of each set of four vertices. The two outermost vertices will be discarded, and the vertex shader will not run on them at all.
Using GL_LINE_STRIP_ADJACENCY produces a similar effect. The difference is that the entire strip is considered to be a primitive, with one additional vertex on each end. If you send eight vertices to OpenGL using GL_LINES_ADJACENCY, the geometry shader will run twice, whereas if you send the same vertices using GL_LINE_STRIP_ADJACENCY, the geometry shader will run five times. Figure 8.20 should make things clear. The eight vertices in the top row are sent to OpenGL with the GL_LINES_ADJACENCY primitive mode. The geometry shader runs twice on four vertices each time—ABCD and EFGH. In the second row, the same eight vertices are sent to OpenGL using the GL_LINE_STRIP_ADJACENCY primitive mode. This time, the geometry shader runs five times—ABCD, BCDE, and so on until EFGH. In each case, the solid arrows are the lines that would be rendered if no geometry shader were present.
Figure 8.20. Lines produced using lines with adjacency primitives
The GL_TRIANGLES_ADJACENCY primitive mode works similarly to the GL_LINES_ADJACENCY mode. A triangle is sent to the geometry shader for each set of six vertices in the enabled attribute arrays. The first, third, and fifth vertices are considered to make up the real triangle, and the second, fourth, and sixth vertices are considered to be in between the triangle’s vertices. This means that the inputs to the geometry shader are sixelement arrays. As before, you can do anything you want to the vertices using the geometry shader; GL_TRIANGLES_ADJACENCY is a good way to get arbitrary sixvertex primitives into the geometry shader. Figure 8.21 shows this.
Figure 8.21. Triangles produced using GL_TRIANGLES_ADJACENCY
The final, and perhaps most complex (or alternatively the most difficult to understand), of these primitive types is GL_TRIANGLE_STRIP_ADJACENCY. This primitive represents a triangle strip with every other vertex (the first, third, fifth, seventh, ninth, and so on) forming the strip. The vertices in between are the adjacent vertices. Figure 8.22 demonstrates the principle. In the figure, the vertices A through P represent 16 vertices sent to OpenGL. A triangle strip is generated from every other vertex (A, C, E, G, I, and so on), and the vertices that come between them (B, D, F, H, J, and so on) are the adjacent vertices.
Figure 8.22. Triangles produced using GL_TRIANGLE_STRIP_ADJACENCY
There are special cases for the triangles that come at the start and end of the strip, but once the strip is started, the vertices fall into a regular pattern that is more clearly seen in Figure 8.23.
Figure 8.23. Ordering of vertices for GL_TRIANGLE_STRIP_ADJACENCY
The rules for the ordering of GL_TRIANGLE_STRIP_ADJACENCY are spelled out clearly in the OpenGL Specification—in particular, the special cases are noted there. You are encouraged to read that section of the specification if you want to work with this primitive type.
Rendering Quads Using a Geometry Shader
In computer graphics, the word quad is used to describe a quadrilateral – a shape with four sides. Modern graphics APIs do not support rendering quads directly, primarily because modern graphics hardware does not support quads. When a modeling program produces an object made from quads, it will often include the option to export the geometry data by converting each quad into a pair of triangles. These are then rendered by the graphics hardware directly. In some graphics hardware, quads are supported, but internally the hardware will do this conversion from quads to pairs of triangles for you.
In many cases, breaking a quad into a pair of triangles works out just fine and the visual image isn’t much different than what would have been rendered had native support for quads been present. However, there are a large class of cases where breaking a quad into a pair of triangles doesn’t produce the correct result. Take a look at Figure 8.24.
Figure 8.24. Rendering a quad using a pair of triangles
In Figure 8.24, we have rendered a quad as a pair of triangles. In both images, the vertices are wound in the same order. There are three black vertices and one white vertex. In the left image, the split between the triangles runs vertically through the quad. The topmost and two side vertices are black and the bottommost vertex is white. The seam between the two triangles is clearly visible as a bright line. In the right image, the quad has been split horizontally. This has produced the topmost triangle, which contains only black vertices and is therefore entirely black, and the bottommost triangle, which contains one white vertex and two black ones, therefore displaying a black to white gradient.
The reason for this is that during rasterization and interpolation of the pervertex colors presented to the fragment shader, we’re only rendering a triangle. There are only three vertices’ worth of information available to us at any given time, and therefore, we can’t take into consideration the “other” vertex in the quad.
Clearly, neither image is correct, but neither is obviously better than the other. Also, the two images are radically different. If we rely on our export tools, or worse a runtime library, to split quads for us, we do not have any control over which of these two images we’ll get. What can we do about that? Well, the geometry shader is able to accept primitives with the GL_LINES_ADJACENCY type, and each of these has four vertices — exactly enough to represent a quad. This means that by using lines with adjacency, we can get four vertices’ worth of information at least as far as the geometry shader.
Next, we need to deal with the rasterizer. Recall, the output of the geometry shader can only be points, lines, or triangles, and so the best we can do is to break each quad (represented by a lines_adjacency
primitive) into a pair of triangles. You might think this leaves us in the same spot as we were before. However, we now have the advantage that we can pass whatever information we like on to the fragment shader.
To correctly render a quad, we must consider the parameterization of the domain over which we want to interpolate our colors (or any other attribute). For triangles, we use barycentric coordinates, which are threedimensional coordinates used to weight the three corners of the triangle. However, for a quad, we can use a twodimensional parameterization. Consider the quad shown in Figure 8.25.
Figure 8.25. Parameterization of a quad
Domain parameterization of a quad is twodimensional and can be represented as a twodimensional vector. This can be smoothly interpolated over the quad to find the value of the vector at any point within it. For each of the quad’s four vertices A, B, C, and D, the values of the vector will be (0, 0), (0, 1), (1, 0), and (1, 1), respectively. We can generate these values per vertex in our geometry shader and pass them to the fragment shader.
To use this vector to retrieve the interpolated values of our other perfragment attributes, we make the following observation: The value of any interpolant will move smoothly between vertex A and B and between C and D with the x component of the vector. Likewise, a value along the edge AB will move smoothly to the corresponding value on edge CD. Thus, given the values of the attributes at the vertices A through D, we can use the domain parameter to interpolate a value of each attribute at any point inside the quad.
Thus, our geometry shader simply passes all four of the pervertex attributes, unmodified, as flat
outputs to the fragment shader, along with a smoothly varying domain parameter per vertex. The fragment shader then uses the domain parameter and all four pervertex attributes to perform the interpolation directly.
The geometry shader is shown in Listing 8.34, and the fragment shader is shown in Listing 8.35 — both are taken from the gsquads example. Finally, the result of rendering the same geometry as shown in Figure 8.24 is shown in Figure 8.26.
Listing 8.34. Geometry shader for rendering quads
#version
430 corelayout
(lines_adjacency
)in
;layout
(triangle_strip
,max_vertices
= 6)out
;in
VS_OUT {vec4
color; } gs_in[4];out
GS_OUT {flat vec4
color[4];vec2
uv; } gs_out;void
main(void
) { gl_Position = gl_in[0].gl_Position; gs_out.uv =vec2
(0.0, 0.0); EmitVertex(); gl_Position = gl_in[1].gl_Position; gs_out.uv =vec2
(1.0, 0.0); EmitVertex(); gl_Position = gl_in[2].gl_Position; gs_out.uv =vec2
(1.0, 1.0); // We're only writing the output color for the last // vertex here because they're flat attributes, // and the last vertex is the provoking vertex by default gs_out.color[0] = gs_in[1].color; gs_out.color[1] = gs_in[0].color; gs_out.color[2] = gs_in[2].color; gs_out.color[3] = gs_in[3].color; EmitVertex(); EndPrimitive(); gl_Position = gl_in[0].gl_Position; gs_out.uv =vec2
(0.0, 0.0); EmitVertex(); gl_Position = gl_in[2].gl_Position; gs_out.uv =vec2
(1.0, 1.0); EmitVertex(); gl_Position = gl_in[3].gl_Position; gs_out.uv =vec2
(0.0, 1.0); // Again, only write the output color for the last vertex gs_out.color[0] = gs_in[1].color; gs_out.color[1] = gs_in[0].color; gs_out.color[2] = gs_in[2].color; gs_out.color[3] = gs_in[3].color; EmitVertex(); EndPrimitive(); }
Listing 8.35. Fragment shader for rendering quads
#version
430 corein
GS_OUT {flat vec4
color[4];vec2
uv; } fs_in;out vec4
color;void
main(void
) {vec4
c1 = mix(fs_in.color[0], fs_in.color[1], fs_in.uv.x);vec4
c2 = mix(fs_in.color[2], fs_in.color[3], fs_in.uv.x); color = mix(c1, c2, fs_in.uv.y); }
Figure 8.26. Quad rendered using a geometry shader
Multiple Viewport Transformations
You learned in “Viewport Transformation” back in Chapter 3 about the viewport transformation and how you can specify the rectangle of the window you’re rendering into by calling glViewport() and glDepthRange(). Normally, you would set the viewport dimensions to cover the entire window or screen, depending on whether your application is running on a desktop or is taking over the whole display. However, it’s possible to move the viewport around and draw into multiple virtual windows within a single larger framebuffer. Furthermore, OpenGL also allows you to use multiple viewports at the same time. This feature is known as viewport arrays.
To use a viewport array, we first need to tell OpenGL what the bounds of the viewports we want to use are. To do this, call glViewportIndexedf() or glViewportIndexedfv(), whose prototypes are
void
glViewportIndexedf(GLuint index, GLfloat x, GLfloat y, GLfloat w, GLfloat h);void
glViewportIndexedfv(GLuint index,const
GLfloat * v);
For both glViewportIndexedf() and glViewportIndexedfv(), index is the index of the viewport you wish to modify. Also notice that the viewport parameters to the indexed viewport commands are floatingpoint values rather than the integers used for glViewport(). OpenGL supports a minimum^{6} of 16 viewports, and so index can range from 0 to 15.
Likewise, each viewport also has its own depth range, which can be specified by calling glDepthRangeIndexed(), whose prototype is
void
glDepthRangeIndexed(GLuint index,
GLdouble n,
GLdouble f);
Again, index may be between 0 and 15. In fact, glViewport() really sets the extent of all of the viewports to the same range, and glDepthRange() sets the depth range of all viewports to the same range. If you want to set more than one or two of the viewports at a time, you might consider using glViewportArrayv() and glDepthRangeArrayv(), whose prototypes are
void
glViewportArrayv(GLuint first, GLsizei count,const
GLfloat * v);void
glDepthRangeArrayv(GLuint first, GLsizei count,const
GLdouble * v);
These functions set either the viewport extents or depth range for count viewports starting with the viewport indexed by first to the parameters specified in the array v. For glViewportArrayv(), the array contains a sequence of x, y, width, height values, in that order. For glDepthRangeArrayv(), the array contains a sequence of n, f pairs, in that order.
Once you have specified your viewports, you need to direct geometry into them. This is done by using a geometry shader. Writing to the builtin variable gl_ViewportIndex selects the viewport to render into. Listing 8.36 shows what such a geometry shader might look like.
Listing 8.36. Rendering to multiple viewports in a geometry shader
#version
430 corelayout
(triangles
,invocations
= 4)in
;layout
(triangle_strip
,max_vertices
= 3)out
;layout
(std140
,binding
= 0)uniform
transform_block {mat4
mvp_matrix[4]; };in
VS_OUT {vec4
color; } gs_in[];out
GS_OUT {vec4
color; } gs_out;void
main(void
) {for
(int
i = 0; i < gl_in.length(); i++) { gs_out.color = gs_in[i].color; gl_Position = mvp_matrix[gl_InvocationID] * gl_in[i].gl_Position; gl_ViewportIndex = gl_InvocationID; EmitVertex(); } EndPrimitive(); }
When the shader of Listing 8.36 executes, it produces four invocations of the shader. On each invocation, it sets the value of gl_ViewportIndex to the value of gl_InvocationID, directing the result of each of the geometry shader instances to a separate viewport. Also, for each invocation, it uses a separate modelviewprojection matrix, which it retrieves from the uniform block, transform_block. Of course, a more complex shader could be constructed, but this is sufficient to demonstrate direction of transformed geometry into a number of different viewports. We have implemented this code in the multipleviewport sample, and the result of running this shader on our simple spinning cube is shown in Figure 8.27.
Figure 8.27. Result of rendering to multiple viewports
You can clearly see the four copies of the cube rendered by Listing 8.36 in Figure 8.27. Because each was rendered into its own viewport, it is clipped separately, and so where the cubes extend past the edges of their respective viewports, their corners are cut off by OpenGL’s clipping stage.