Category: Experiments in Computer Vision

These posts document my experiments in the aesthetics of computer vision applied to cinema.

Optical Flow, Part 1

tyler

This is another sparse post that I’ll fill back in with more details shortly.  In the meantime, check out my GitHub for the project associated with this post, optFlowTest.  It tests measuring optical flow in a short film clip.

Here’s a sample of the program running:

Mini-explanation

This project uses ofxCv’s Flow function to analyze Farneback flow in a short shot from Hitchcock’s Rear Window (1954).

Farneback flow is an algorithm in OpenCV that measures the frame-by-frame motion for each pixel in an image sequence.  In the clip above, you can see that the program visualizes the motion as a flow field (the white lines over the video).  Each line’s direction and magnitude corresponds to the amount of motion measured at that pixel, compared to the previous frame of the clip.

I’ve also added some visualizations of the average motion.  The red line over the video shows the magnitude and direction of the average motion of the current frame, and the red line on the left show the average motion of the whole clip so far. The light blue dot travels along a path generated by the average flow, to give a sense of how the motion changes over of the course of the shot.

Purpose

There are many reasons for measuring motion in film.  For one, it’s used commercially in video-stabilization algorithms, and it can even be used to generate a 3D model of a scene from 2D images.

For my own purposes, I want to build out a database of film movement.  This may involve separating the motion of the camera from the motion of the characters and objects within the frame. You can see that the clip I chose from Rear Window contains both camera movement and character motion – so the average motion is a combination of the two.

Often, films are edited with motion in mind: when an shot is cut on the motion of a character or an object, and the motion continues in the next shot (but from a different camera angle), this is called a match-on-action edit, and it creates a seamless impression of movement and space over time.  Using optical flow could be one way of measuring, and even generating this type of edit.

Big props:
Today’s project soundtrack is brought to you by xzayoso
a.k.a. Udit Mahajan

[soundcloud url=”https://api.soundcloud.com/tracks/255054873″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=true” width=”100%” height=”300″ iframe=”true” /]

Next

Next I plan to start applying feature matching and tracking, as well as optical flow detection to building out a database.  We’ll see how that goes!

SIFT Implementation in OpenFrameworks, Part 2

tyler

This post is a continuation of the last post which introduces the SIFT algorithm as well as some info on OpenCV in openFrameworks through the ofxCv addon. This post is going to be a change of pace, less tutorial-esque and more experimental.  I’ll still try and distill what I’m doing into a guide-like format – but I’m literally writing this while I’m coding, so errors and failures (and hopefully, happy accidents) will be part of the process.

Continue reading “SIFT Implementation in OpenFrameworks, Part 2”