Image restoration is the inverse problem of recovering high-quality images from knowledge of degraded images; it includes image super-resolution, image denoising, image deblurring, etc. The objective of image restoration methods is to minimize the error defined by the loss function between the network output image and the corresponding ground truth. Distortion-oriented loss functions are fundamental and commonly used in image restoration. However, these functions treat all pixels as equally important without differentiating between sharp and blurred edge areas, which does not match human visual perception. As a result, these methods can produce accurate but blurred images. To address this issue and achieve both accurate and perceptually satisfactory results, we propose a novel perception-enhanced distortion-oriented loss (PE loss) for image restoration, inspired by the Mach band effect. This effect demonstrates that sharp edges are perceived as having better quality than blurred edges by the human visual system. Our approach includes designing a blur factor map that detects blurred pixels and penalizes them by amplifying their error. The PE loss is a simple yet effective plug-and-play method, and we apply it to state-of-the-art networks. Extensive quantitative and qualitative experiments show that our method can restore images with sharp edges and high perceptual quality.