Existing human pose estimation methods seldom consider the impact or constraint of different types of contact. In this paper, we elaborate on the impact of both body-scene contact and self-contact on pose estimation and refer to them as general contact. First, we extend existing datasets by calculating additional contact labels for general contact inference. Moreover, based on the extended dataset, we present the first network to predict dense general contact from a single RGB image. Finally, we develop a novel optimization method that successfully utilizes the inferred general contact information for accurate 3D pose estimation. Our results show that knowledge of contact can provide strong constraints and resolve pose ambiguity, thus significantly improving human pose estimation accuracy, especially for challenging poses that cannot be well handled by existing methods. Experimental results and comparisons further demonstrate the effectiveness of the proposed method. Our results are even more reasonable than certain pseudo-ground truth determined from multi-view images.