Back

 Industry News Details

 
Google’s new AI can intelligently crop videos for any screen size Posted on : Feb 14 - 2020

How many times have you seen a video being badly cropped when you watch it on a mobile device? It’s quite frustrating and annoying, and most of the time, there’s not much you can do about it.

To address this problem, Google’s AI team has developed an open-source solution, Autoflip, that reframes the video that suits the target device or dimension (landscape, square, portrait, etc.).

Autoflip works in three stages: Shot (scene) detection, video content analysis, and reframing. The first part is scene detection, in which the machine learning model needs to detect the point before a cut or a jump from one scene to another. So it compares one frame with the previous one before to detect the change of colors and elements.

Once the model determines a shot, it moves on to the video content analysis to determine important objects in a scene. It uses a deep learning neural network to determine not just people or animals, but motion and moving balls in sports, and logos in commercials.

For the final stage, the AI model determines if it will use stationary mode for scenes that take place in a single space, or tracking mode for when objects of interest are constantly moving. Based on that, and the target dimensions in which the video needs to be displayed, Autoflip will crop frames while reducing jitter and retaining the content of interest.

Google researchers said that Autoflip can be used to convert videos to many formats and screens without much effort. For the next stage, the team wants to improve object tracking in interviews and animation films. It wants to use text detection and image inpainting techniques to better place foreground and background objects in one frame. Source