Stimuli-responsive polymers are promising to achieve targeted delivery, improved stability during circulation, and controlled release of therapeutic and diagnostic agents. Among them, pH-responsive polymeric nanocarriers have attracted significant attention as pH varies in different body fluids (e.g., stomach, intestine, and colon) and intracellular organelles (e.g., endosome, lysosome, and mitochondria) to maintain homeostasis, while distinctive pH changes are also found in certain pathological states. For example, the extracellular environment of the tumor is acidic, which can be employed to drive selective delivery. During the internalization process, since most nanocarriers enter cells upon endocytosis where a drop of pH from 6.5 to 5.0 can occur from endosome to lysosome, pH-sensitive groups have been developed for enhanced cargo release. In this review, both non-covalent and covalent interactions responsive to pH changes are introduced, with a focus on the structure–property relationship and their applications in cancer targeting and endosomal escape.