Big Buck Bunny but it's seam-carved to a 4:3 aspect ratio

This video is the result of an old project of mine to develop an algorithm that intelligently resizes videos to a different aspect ratio. Loosely speaking, it works by finding vertical lines of "least interesting" pixels in each frame and deleting them until the frame is in the desired new aspect ratio, in this case, from 16:9 to 4:3.

The algorithm is based on the algorithm presented by Avidan et al. in their 2007 paper "Seam Carving for Content-Aware Image Resizing". The energy matrix for each frame is calculated from the magnitude of the Sobel of the original frame, with the following modifications:
1. Keypoints are found on the original frame using SURF and are given extra weight in the energy matrix to discourage seams from going through the most important details.
2. The energy matrix of each frame is averaged with the energy matrix of the previous frame in an 85%/15% ratio to keep it from changing too quickly frame-to-frame.

As you can see, the final results are rather mixed. Some frames are carved very cleanly, while others are horribly distorted, and poorly-carved frames can be right next to well-carved ones. The low hysteresis between each frame means that which pixels the algorithm decides to delete can vary drastically between frames, causing features to "vibrate" side-to-side in a distracting manner. A working adaptation of the seam-carving algorithm to video will need to take into account the movement of features between frames, or it will need to carve 3D "sheets" of pixels along the X, Y, and time axes.

I decided to upload my results to YouTube anyway because I found them to be hilariously bad.

"Big Buck Bunny" is an animated short produced by the Blender Foundation and released under a Creative Commons Attribution 3.0 license.

Видео Big Buck Bunny but it's seam-carved to a 4:3 aspect ratio канала squawkBirdies
opencv, python, computer vision, seam carving, big buck bunny, funny