Let $T:X\to Y$ be a bounded linear operator between Banach spaces $X,Y$. A vector $x_0\in {\mathsf{S}}_X$ in the unit sphere ${\mathsf{S}}_X$ of $X$ is called a supporting vector of $T$ provided that $\Vert T(x_0)\Vert =sup\{\Vert T(x)\Vert :\Vert x\Vert =1\}=\Vert T\Vert$. Since matrices induce linear operators between finite-dimensional Hilbert spaces, we can consider their supporting vectors. In this manuscript, we unveil the relationship between the principal components of a matrix and its supporting vectors. Applications of our results to real-life problems are provided.