I made the discovery this week that the vector math library I had written for my current project wasn’t right: I needed to transpose any matrix I got from an OpenGL glGet… call before I could use it properly, otherwise my matrix-vector and matrix-matrix operations would just return junk values. A quick Google search revealed a pretty big point of confusion around the OpenGL spec: column-major or row-major matrix representation? Which as it turns out is two points: how you store your matrices and how you represent your vectors.
There are two schools of thought on how to treat vectors when doing matrix-multiplication: traditional math treats them as columns in a 1-dimensional matrix, like in Figure 1.
Computer science, on the other hand, has tended to write them as a row, like in figure 2. I assume the difference comes about because the first method is an absolute bitch to try and type, but I’m only guessing. Anyway, the confusion comes about because the OpenGL spec is written using vectors-as-columns notation, in an attempt to bring us unruly computer scientists in line with the rest of mathematics. The way in which the code for a matrix-vector operation is written will be different depending on whether a vector is treated as a row or as a column.
The second point is how to store a 4 by 4 matrix in a contiguous bit of memory: to map it columns-by-column, like in figure 3; or store each row in order, like in figure 4. Obviously the position of data in memory will also affect how the actual code to perform vector-matrix operations is written.
So, as it turns out there was an issue with backwards compatibility when writing the OpenGL spec: they wanted to use vectors-as-columns, but to do so would break compatibility with the previous GL, which treated vectors as rows. So they did a sneaky on us. They wrote into the spec that OpenGL was column-major. The thing with column-major and row-major matrices is that you can switch between them by transposing i.e. a column-major translation matrix can become the same matrix in a row-major system by flipping its values along the diagonal like in figure 5.
And the thing with treating vectors as columns, is that you can switch between them by transposing i.e. in a vectors-as-rows system, you could treat vectors as columns by using the transpose of the matrix you’re multiplying against. See figure 6 (at which time I realised that Open Office comes with a perfectly good math editor).
So by taking the existing vectors-as-rows system, with row-major storage, and making it a vectors-as-columns system with column-major storage, OpenGL leaves us with the transpose of the transpose: the exact same matrix we started with!
Moral of the story: if implementing a vectors-as-rows math library is your thing – as it is mine for previously metnioned typing issues – and storing matrices row-first comes natural: keep doing both! If vectors-as-columns is your secret perversion, make sure you implement it with a healthy dose of column-major matrix storage or people will look at you weird. But no matter what you do, don’t waste almost a week writing and rewriting your matrix code to try to get it to play nice with OpenGL – read this article again instead