Diffusion models have revolutionized text-driven video editing. However, applying these methods to real-world editing encounters two significant challenges: (ⅰ) the rapid increase in GPU memory demand as the number of frames grows, and (ⅱ) inconsistency between frames in edited videos. To address these issues, we propose NVEdit, a novel text-driven video editing framework designed to mitigate memory overhead and improve editing consistency for long real-world videos. Specifically, we construct a neural video field, based on tri-plane and sparse grid representations, to enable the encoding of videos with hundreds of frames in a memory-efficient manner. Next, we update the video field through off-the-shelf text-to-image (T2I) models to impart text-driven editing effects. A progressive optimization strategy is used to preserve original temporal priors. Importantly, both the neural video field and T2I model are adaptable and replaceable, thus providing a basis for future research. Experiments demonstrate the ability of our approach to edit hundreds of frames with strong inter-frame consistency. Our project is available at https://nvedit.github.io/.
- Article type
- Year
- Co-author
Open Access
Research Article
Issue
Open Access
Research
Issue
Single image-to-3D generation is pivotal for crafting controllable 3D assets. Although recent inference-only methods have achieved impressive effects, their generation quality still lags behind that of image generation models. We attempt to leverage 3D geometric priors from the novel view diffusion model and 2D appearance priors from an image generation model to combine the geometric messages of the former and appearance priors of the latter. We note that there is a disparity between the generation priors of these two diffusion models, leading to different appearance outputs. Specifically, image generation models tend to deliver more detailed visuals, whereas novel view models produce consistent yet over-smooth results across different views. Directly combining them leads to suboptimal effects due to their appearance conflicts. Hence, we propose a 2D-3D hybrid Fourier score distillation objective function, called hy-FSD. It optimizes 3D Gaussians using 3D priors in the spatial domain to ensure geometric consistency, while exploiting 2D priors in the frequency domain through the Fourier transform for better visual quality. The proposed hy-FSD can be integrated into existing 3D generation methods and produce significant performance gains. With this technique, we have developed an image-to-3D generation pipeline to create high-quality 3D objects within one minute, named Fourier123. Extensive experiments demonstrate that Fourier123 excels at efficiently generating results with rapid convergence speed and a visually appealing output.
京公网安备11010802044758号