11
$\begingroup$

The wiki article on eigenvectors offers the following geometrical interpretation:

Each application of the matrix to an arbitrary vector yields a result which will have rotated towards the eigenvector with the largest eigenvalue.

Qn 1: If there is any other geometrical interpretation particularly in the context of a covariance matrix?

The wiki also discusses the difference between left and right eigenvectors.

Qn 2: Do the above geometrical interpretations hold irrespective of whether they are left or right eigenvectors?

  • 2
    The left eigenvectors of M are the right eigenvectors of M^T, so if you understand the transpose then the geometric interpretations are essentially the same.2010-07-29
  • 2
    [Principal Component Analysis](http://en.wikipedia.org/wiki/Principal_component_analysis) transforms the data to a basis of eigenvectors with the largest eigenvalues in order to reduce dimension while minimizing information loss.2010-07-29
  • 0
    @Qiaochu: That makes a lot of sense.2010-07-29
  • 0
    @Kaestur: So my understanding of your comment and based on the wiki article is this: The eigenvector associated with the highest eigenvalue is parallel to the major axis of the ellipsoid associated with the covariance matrix. Would that be correct?2010-07-29
  • 0
    @Srikant: Mostly correct. Clustering and such can make it a little complicated. I realized wikipedia does not have a good picture, perhaps [this](http://www.med.govt.nz/templates/MultipageDocumentPage____1065.aspx#P1096_144597) may help.2010-07-29
  • 0
    @Kaestur Hakarl - that link is broken :(2012-07-03

4 Answers 4

12

Here is a partial answer in the case where M is a real symmetric matrix. This is to ensure, by the real spectral theorem, that M has real eigenvectors with real eigenvalues, so there is a chance for a genuine geometric interpretation which stays in $R^n$.

M acts on the unit sphere in $R^n$ in the following way: it sends the unit sphere $v^T v = 1$ to $v^T (M^T M) v = 1$ . This modified shape is not generally a sphere, but is generally an ellipsoid. The axes of this ellipsoid are the eigenvectors of M, and the sizes of each axis are given by the squares of the corresponding eigenvalues.

  • 0
    +1 @Qiaochu: I find your method of thinking about "how M modifies the unit sphere" to be very interesting and useful. By the way, why do you say it is a partial answer? Your comment on my second qn + the above response answers my question (at least from my perspective).2010-07-29
  • 1
    I haven't addressed the case where M is allowed to be symmetric or complex.2010-07-29
  • 0
    Fair enough but I am ok with your answer.2010-07-29
6

If you are interested in covariance matrices, then the eigenvectors of the covariance matrix tell you how to change variables to make your random variables uncorrelated.

Specifically, let $M$ be a covariance matrix of the random variables $X_1,...,X_n$. For simplicity, lets assume that all of these random variables are zero mean. Lets also define $X$ to be the random vector whose $i$"th component is the random variable $X_i$. Let the eigenvectors of $M$ be $v_1, v_2, ..., v_n$. We can assume that these are orthogonal since $M$ is symmetric. Consider the random variables $Y_i$ obtained by taking the dot product of $v_i$ and $X$.

Then the random variables $Y_1, ..., Y_n$ are uncorrelated!

Indeed:

$$ E[ Y_i Y_j] = E[ v_i^T X X^T v_j] = v_i^T M v_j = \lambda_j v_i^T v_j = 0 $$

  • 1
    that's what i was looking for. good show!2010-07-29
6

Instead of giving an answer, let me point out to you this chapter in Cleve Moler's book "Numerical Computing with MATLAB", there is a nice geometric demonstration in MATLAB on how eigenvalues/eigenvectors (as well as singular values/vectors) of an order-2 square matrix are involved in how a circle is transformed into an ellipse after a linear transformation represented by the matrix.

  • 0
    Current projections indicate tomorrow. :-)2012-07-31
  • 0
    Lo and behold, [the screenshot](http://i.imgur.com/dwZgd.png).2012-08-01
2

Of course! Consider a coordinate transformation of rotation and/or scaling (but not translation):

v = Au

where v and u are vectors, and A is a transformation matrix. Then the eigenvectors, if they have real components, are the axes which are left unrotated (scaling only) by the transformation. (see wikipedia)

A covariance matrix is a symmetric, positive definite matrix, so it has orthonormal eigenvectors, and these form a tuple of axes; I am fairly sure the eigenvectors form a new basis of linear combinations of the input variables where the basis variables are uncorrelated, but I can't remember how to show this.

For example, if w1 = [x;y] is a pair of independent unit-variance zero-mean Gaussian random variables, consider w2 = [u;v] = [1 1; 2 1][x;y] = (x+y,2x+y), so that w1 = [-1 1;2 -1][u;v] = [v-u;2u-v]. Then cov(w2) = [2 3; 3 5]. This has eigenvectors which have sqrt(5) in them, hmmmm...

As for question 2, I'm not sure.

  • 1
    But, this seems trivial as it follows from the definition of the eigenvector or is there something deeper I am missing?2010-07-29
  • 0
    @Srikant, could you explain in more detail what sort of answer you are looking for?2010-07-29
  • 0
    I thought the question was clear. I am looking to see if there is any deeper insight along the lines of the one suggested by the wiki. The geometrical interpretation that repeated applications will rotate a vector towards the eigenvector with the highest eigenvalue is not at all obvious at first sight. I also want to know if there is a difference between left and right eigenvectors with respect to the geometry. Does that help clarify?2010-07-29
  • 0
    I added the section on covariance matrices. forgot how to show what I wanted to show, though.2010-07-29