ENLIGHTEN3D

By Joshua Koopferstock and Christian Laforte

With 50 startups launching at the TechCrunch50 conference last week, the blogosphere has been abuzz with high tech news. In all that media frenzy, one company stood out and impressed us more than the rest. Videosurf, a video search engine, combines traditional text metadata search with computer vision to provide higher quality search results. After seeing a video of their presentation at TC50, I wanted to learn more, so I arranged an interview with Videosurf CTO and computer vision expert Dr. Eitan Sharon. The following article is based on our discussion.

How Videosurf Works

Videosurf starts by looking through textual tags and dividing video into shots. Unlike other video search engines though, Videosurf goes much further in its analysis. A facial recognition system goes through the films frame-by-frame, collects faces that look alike in different shots, and tries to match them against the appearance of known actors using the cast information. In the beginning, an operator had to assist the system, but now, most of the common actors are easily recognized automatically.

Here’s an example. I put in a search for “Star Trek”. At the top of the page, I am presented with a thumbnail of people that are associated with my search. In this case the list includes William Shatner, Jeri Ryan, Leonard Nimoy, and Patrick Stewart, all actors who have played major characters in a Star Trek TV series. By clicking on one of the thumbnails (Patrick Stewart), I can find all of the Star Trek-related clips in which Patrick Stewart appears.

From the screenshots, you may have noticed that next to each clip there appears to be a storyboard. Appearing simple at first glance, these “video summaries” actually employ a complex computer vision approach in trying to automatically determine the most “important” frames in a clip. According to Dr. Sharon, creating these visual summaries involves a combination of techniques such as determining the uniqueness or similarity of objects across frames, the depth of field, motion in the scenes, etc. If these automatic storyboards work effectively, and from my own testing they seem to do a good job, this should save users time so that they don’t have to wait for a video to finish downloading just to realize that it is not what they were looking for.

Other Features

Perhaps the feature with the most immediate profit potential is their adult content filter. Though we did not get too far into the details, they have employed a type of “safe search” filter that adds computer vision techniques to traditional text search to determine whether content should be filtered. Like Google’s Safe Search, this filter can be toggled on or off. If I was Yahoo’s chief marketing officer, I would immediately license this technology exclusively for a few years. Then I would run an ad campaign about how our search engine is the only safe place for children on the internet. But that’s just me.

One other neat feature lets you share a specific segment from a video, increasing the viral potential of online video beyond where it already is today.

Will it take off?

Saying “Will I use this?” is not really the most unbiased kind of market analysis, but I can say that I will use this (free!) service now. I already have, actually; having “Videosurfed” 3 times in the past 24 hours since receiving my beta invite, and I am confident that I will keep coming back if it keeps giving me quality results.

From a profitability perspective, Videosurf may have an ace in the hole that they haven’t talked about yet: copyright detection. After Google (owner of YouTube) was sued for one billion dollars by Viacom last, it is safe to bet that they and others are interested in being able to better detect whether the material on their sites is copyrighted. Computer vision has finally come to online video search, and we can safely say that it is here to stay.

You can find a list of Dr. Sharon’s computer vision publications on his personal web site: http://www.dam.brown.edu/people/eitans/

By Joshua Koopferstock

If creating 3D models was as easy as taking photos, it is safe to say that the use of 3D would be far more widespread than it is today.  From e-commerce to virtual tourism to casual games, reducing the cost and complexity of creating 3D models would have a widespread effect on multiple industries.

Feeling Software is making that possible.  Over the last 2 years, we have worked to develop a technology that allows anyone to create 3D models with little effort and no training.  Our goal: simplicity.  You take a bunch of photos with a regular camera from any angle you please, and we automatically create a 3D model.  The demo video below discusses our project in detail.

Feeling Software Demo from joshk on Vimeo.

We have thought of a variety ways that this technology can be applied to solve problems for consumers.  For our readers, imagine that you could take photos of an object or scene, press a button, and instantly have a high-quality 3D model of that object or scene.  If this technology were available today, how would you use it?

By Joshua Koopferstock

After months of development, Presto3D left closed beta last week and has opened to the public.  If you missed the post a few weeks ago, Presto3D is a 3D model marketplace that automatically creates 3D previews of the content that is submitted.

The reception has been positive, and we have gotten press coverage on several major animation and game development sites.  If you haven’t taken a look at the site yet, come check out what we’ve been up to for the last few months.  If you already visited the site during the closed beta, it’s worth going back just to see the fullscreen 3D previews that we’ve added in the latest release.

All feedback is appreciated!

By Joshua Koopferstock

You tag this photo:

Source: www.bt.dk

Polar Rose finds this photo:

Source: www.newprophecy.net

With facial recognition in Picasa Web Albums launched yesterday, an exciting computer vision application once again bumps heads against privacy concerns.  On the one hand, automatic tagging of photos through facial recognition can be a useful time saver; if I have an album of 100 pictures of my family vacation with the same 5 people, I would much rather have my computer tag them for me than having to sift through them one-by-one adding tags.  On the other hand, I might not necessarily want all photos of me to be so easily found by anyone.

Picasa Web Albums seems to be OK in how it deals with this issue, at least for now.  Users only tag their own albums, and, as far as I can tell, the information gathered is not used to search Google Imagesand automatically tag images of the people you tag in your web albums.

This is not the case with every player stepping into this industry.  Polar Rose, announced late-2006, uses facial recognition on user-tagged photos to search for more photos of an individual in any publicly available images.  The service is designed as a browser plug-in and an embeddable widget for photo-sharing sites, and rumour has it that a partnership with one of the major sites is imminent.  Users tag photos of people that they find anywhere online, and Polar Rose uses that information to find that same face across other images.

The example at the beginning of this post should illustrate why this may be a concern.  For Paris Hilton, perhaps her image benefits when scandalous pictures surface, but this is not the case for most of us.  Should photos of people really be that searchable?

In reality, though, the point is moot.  Using Google Image search, I probably could have found the same 2 pictures shown above; with facial recognition, this just becomes more efficient.  If you have been using Facebook for the past few years, you have probably already come to terms with the fact that people can quickly find many pictures of you, including ones that others took without your consent (though in fairness to Facebook, you can untag and render unsearchable pictures you don’t like).

Conclusion: from a computer vision standpoint, it is neat to see these technologies reach the mass market.  From a privacy outlook, more (visual) information about us is going to become accessible without our direct consent, but only information that was publicly available in the first place, and this development is probably inevitable.

On a final note, there is one feature proposed by Polar Rose which I can’t help but find far more creepy than useful: personal photo RSS feeds.  Basically, get instantly updated by RSS each time a new photo of a targeted person is found.  Stalkers rejoice!