Researchers at Cornell University and Google Research have developed an algorithm, known as DynIBar, that offers exciting possibilities for filmmakers. With this software, it may soon be possible to stabilize shaky video, alter viewpoints, and create captivating effects like freeze-frame, zoom, and slow-motion—all without the need to shoot additional footage.
Unlike previous attempts, DynIBar surpasses expectations by generating new views through the clever utilization of pixel data from the original video. This innovative algorithm can handle challenging scenarios involving moving objects and unstable camerawork. In contrast to earlier methods, which often resulted in mere seconds of usable footage with blurry or glitchy artifacts, DynIBar presents a significant advancement.
Furthermore, the researchers have made the code for this groundbreaking research freely available. However, it's important to note that the project is still in its early stages and has not yet been integrated into commercial video editing tools.
Abstract
We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. State-of-the-art methods based on temporally varying Neural Radiance Fields (aka dynamic NeRFs) have shown impressive results on this task. However, for long videos with complex object motions and uncontrolled camera trajectories, these methods can produce blurry or inaccurate renderings, hampering their use in real-world applications. Instead of encoding the entire dynamic scene within the weights of MLPs, we present a new approach that addresses these limitations by adopting a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views in a scene-motion-aware manner. Our system retains the advantages of prior methods in its ability to model complex scenes and view-dependent effects, but also enables synthesizing photo-realistic novel views from long videos featuring complex scene dynamics with challenging camera and object motion, where prior methods fail to produce high-quality renderings.
Discussion (1)
looks promising