Nexus: On-Device Lightweight Real-Time Super-Resolution System for 360^◦ Videos

Delivering high-quality 360^◦ video on resource-constrained head-mounted displays (HMDs) is challenging due to limited bandwidth and compute. We present Nexus, a lightweight, on-device real-time super-resolution system. Nexus transmits only low-quality frames and performs high-quality reconstruction on the HMDs for predicted viewports. We also design a viewport filling scheme to enhance the system’s robustness. Our viewport prediction algorithm achieves a mean angular error of 0.58, and our lightweight SR algorithm reduces inference time by 61% compared to state-of-the-art lightweight SR methods. Under a frame drop rate below 0.1%, compared to full-frame super-resolution on VR devices, Nexus supports resolutions up to 720p (up from 480p), fixed at 30 fps, and increases the maximum tractable bitrate from 10 Mbps to 25 Mbps.