Video structure analysis is a basic requirement for most content-based video editing and processing systems. This paper presents a fast video structure analysis method based on image segmentation in each frame, with region matching between frames. The structure analysis decomposes the video into several moving objects, including information about their colors, positions, shapes, movements, and lifetimes. The method also supports user interactions to improve the results. The result shows that this method is fast and stable and can complete video analyzing interactively.