Rice grain yield is determined by three major “visible” morphological traits: grain weight, grain number per panicle, and effective tiller number, which are affected by a series of “invisible” physiological factors including nutrient use efficiency and photosynthetic efficiency. In the past few decades, substantial progress has been made on elucidating the molecular mechanisms underlying grain yield formation, laying a solid foundation for improving rice yield by molecular breeding. This review outlines our current understanding of the three morphological yield-determining components and summarizes major progress in decoding physiological traits such as nutrient use efficiency and photosynthetic efficiency. It also discusses the integration of current knowledge about yield formation and crop improvement strategies including genome editing with conventional and molecular breeding.