Let $A$ be a finite-dimensional algebra with identity over the field $\mathbb{F}$, $U(A)$ be the group of units of $A$ and $L(A)$ be the set of left ideals of $A$. It is well known that there is an equivalence relation $\sim$ on $L(A)$ by defining $L_1\sim L_2\in L(A)$ if and only if there exists some $u\in U(A)$ such that $L_1=L_{2}u$. $C(A)=\{[L]|L\in L(A)\}$ is the set of equivalence classes determined by the relation $\sim$, which is a semigroup with respect to the natural operation $[L_1][L_2]=[L_1{L}_2]$ for any $L_1,L_2\in L(A)$. The purpose of this paper is to describe the structures of semigroup of conjugacy classes of left ideals for the Sweedler's four-dimensional Hopf algebra $H_4$.