ENLIGHTEN3D

By Joshua Koopferstock and Christian Laforte

With 50 startups launching at the TechCrunch50 conference last week, the blogosphere has been abuzz with high tech news. In all that media frenzy, one company stood out and impressed us more than the rest. Videosurf, a video search engine, combines traditional text metadata search with computer vision to provide higher quality search results. After seeing a video of their presentation at TC50, I wanted to learn more, so I arranged an interview with Videosurf CTO and computer vision expert Dr. Eitan Sharon. The following article is based on our discussion.

How Videosurf Works

Videosurf starts by looking through textual tags and dividing video into shots. Unlike other video search engines though, Videosurf goes much further in its analysis. A facial recognition system goes through the films frame-by-frame, collects faces that look alike in different shots, and tries to match them against the appearance of known actors using the cast information. In the beginning, an operator had to assist the system, but now, most of the common actors are easily recognized automatically.

Here’s an example. I put in a search for “Star Trek”. At the top of the page, I am presented with a thumbnail of people that are associated with my search. In this case the list includes William Shatner, Jeri Ryan, Leonard Nimoy, and Patrick Stewart, all actors who have played major characters in a Star Trek TV series. By clicking on one of the thumbnails (Patrick Stewart), I can find all of the Star Trek-related clips in which Patrick Stewart appears.

From the screenshots, you may have noticed that next to each clip there appears to be a storyboard. Appearing simple at first glance, these “video summaries” actually employ a complex computer vision approach in trying to automatically determine the most “important” frames in a clip. According to Dr. Sharon, creating these visual summaries involves a combination of techniques such as determining the uniqueness or similarity of objects across frames, the depth of field, motion in the scenes, etc. If these automatic storyboards work effectively, and from my own testing they seem to do a good job, this should save users time so that they don’t have to wait for a video to finish downloading just to realize that it is not what they were looking for.

Other Features

Perhaps the feature with the most immediate profit potential is their adult content filter. Though we did not get too far into the details, they have employed a type of “safe search” filter that adds computer vision techniques to traditional text search to determine whether content should be filtered. Like Google’s Safe Search, this filter can be toggled on or off. If I was Yahoo’s chief marketing officer, I would immediately license this technology exclusively for a few years. Then I would run an ad campaign about how our search engine is the only safe place for children on the internet. But that’s just me.

One other neat feature lets you share a specific segment from a video, increasing the viral potential of online video beyond where it already is today.

Will it take off?

Saying “Will I use this?” is not really the most unbiased kind of market analysis, but I can say that I will use this (free!) service now. I already have, actually; having “Videosurfed” 3 times in the past 24 hours since receiving my beta invite, and I am confident that I will keep coming back if it keeps giving me quality results.

From a profitability perspective, Videosurf may have an ace in the hole that they haven’t talked about yet: copyright detection. After Google (owner of YouTube) was sued for one billion dollars by Viacom last, it is safe to bet that they and others are interested in being able to better detect whether the material on their sites is copyrighted. Computer vision has finally come to online video search, and we can safely say that it is here to stay.

You can find a list of Dr. Sharon’s computer vision publications on his personal web site: http://www.dam.brown.edu/people/eitans/