This paper addresses a highly challenging yet critical task: unsupervised person re-identification (Re-ID) under clothing changes. Unfortunately, existing unsupervised person Re-ID methods are mainly designed for short-term scenarios that rely heavily on RGB cues, thus failing to exploit effective features invariant to clothing changes. To tackle this limitation, we propose a novel and efficient approach, called silhouette-driven contrastive learning with hierarchical neighborhood structure (SiCL-HN). Our framework integrates RGB and silhouette information within a contrastive learning paradigm, where silhouettes provide clothing-invariant shape cues that complement RGB features, and constructs a hierarchical neighborhood structure among instances to learn features robust to clothing variations. We conduct extensive experiments to evaluate our proposed approach against state-of-the-art unsupervised person Re-ID methods on six representative long-term datasets (LTCC, PRCC, VC-Clothes, Celeb-ReID, Celeb-ReID-Light, and DeepChange), and observe consistent improvements on mAP and Rank-1; for example, on DeepChange our method surpasses SpCL by +3.5 mAP and +8.1 Rank-1. Experimental results demonstrate that our approach significantly outperforms existing unsupervised Re-ID methods. Our code is available at https://github.com/MingkunLishigure/SiCL.