Visualizing Matrix Multiplication

In linear algebra, the ability to shift between different perspectives on the same concept is very useful. A classic example is the long list of equivalent statements about matrix invertibility.¹

Just as invertibility can be understood in multiple ways, so can matrix multiplication. I was therefore happy to come across Eli Bendersky’s excellent article on visualizing matrix multiplication as linear combinations of columnns and rows. Inspired by his post, I wanted to highlight yet another way of looking at it:

Figure 1. Matrix multiplication can be viewed as a sum of rank-one matrices, where each term is the outer product of a column from the first matrix and a row from the second matrix.

The images illustrate how a matrix product can be viewed as a sum of rank-one matrices– a perspective that appears in various contexts, such as matrix decomposition and attention mechanisms in Transformers.

In general, thinking of familiar concepts in new ways helps with intuition. I’ve found it helpful to create flashcards based on Bendersky’s article and the decomposition above, and I would highly recommend doing the same.

1. See e.g. https://en.wikipedia.org/wiki/Invertible_matrix. ↩