In pattern recognition, the task of image set classification has often been performed by representing data using symmetric positive definite (SPD) matrices, in conjunction with the metric of the resulting Riemannian manifold. In this paper, we propose a new data representation framework for image sets which we call component symmetric positive definite representation (CSPD). Firstly, we obtain sub-image sets by dividing the images in the set into square blocks of the same size, and use a traditional SPD model to describe them. Then, we use the Riemannian kernel to determine similarities of corresponding sub-image sets. Finally, the CSPD matrix appears in the form of the kernel matrix for all the sub-image sets; its $i,j$-th entry measures the similarity between the $i$-th and $j$-th sub-image sets. The Riemannian kernel is shown to satisfy Mercer’s theorem, so the CSPD matrix is symmetric and positive definite, and also lies on a Riemannian manifold. Test on three benchmark datasets shows that CSPD is both lower-dimensional and more discriminative data descriptor than standard SPD for the task of image set classification.